What gets measured: Bibliometrics and education

The world of research has been changing radically in recent decades with the ascendence of bibliometrics. All journals are now quantified and ranked by various measures, such as Source Normalised Impact per Paper (SNIP), the Impact Factor (IF) or the SciMago Journal Ranking (SJR). These are all variations on a theme of looking at how journals cite each other, on the basis that heavily cited articles and journals are of higher quality.

Criticism of such quantification is well documented and widely available, but the interest in this article is in this friction between what we might call the need to quantify and those things that are not quantifiable.

In this age of management we often hear the catchcry that “what gets measured gets managed“. That’s the thinking behind the use of KPIs in organisations. It’s also the thinking behind standardised tests in schools and this is where a lot can be learnt from making an analogy between bibliometrics and education.

What are the assumptions?

The assumption in bibliometrics is that citations are a good indicator of impact. This has been refined (i.e. in the SJR) so that citations in a well-respected journal count for more towards a weighted score (using the same algorithm that Google PageRank uses) and has been augmented by other innovations that make for better predictions (e.g. this technique) but the overall assumption is still the same – that more citations (in the past or the predicted future) is what you’re after.

Similarly, the assumption in education is that standardized tests (e.g. NAPLAN in literacy and numeracy) are measuring the thing that we want to be producing. In other words, it has the assumption that our education system is about producing adults that can perform well on these kinds of tests.

Clearly nobody (well, hopefully nobody) actually thinks this way, but the problem is if you don’t take this as the measure of successful education, what can you use?

Another way to measure?

The problem with measuring things (like test scores) that people do is that they will invariably game the measurements to maximise results. In this case, teach to the test at the expense of other things (e.g. producing whole and complete human beings capable of being a positive force of change in the world).

The bibliometric measure is particularly interesting (as is the Google PageRank) in that it is far more difficult to game, being fundamentally a socially driven measure (although it can still be gamed, obviously, but much less directly).

Networked Learning for example is another way of thinking about education and suggests a different set of measurements. This is one way we could go.

Or not measuring at all?

Really though, humans work best when they have intrinsic motivation. Extrinsic motivation has been a fantastic discovery that we have been using for many centuries now to make people do things that they don’t really want to (e.g. work or you don’t get fed), but basing everything in society upon this mechanism of extrinsic reward and punishment, whilst in vogue, is unlikely to create the kind of world that we want to live in.

The alternative (or at the very least augmentation) to the ubiquitous measure-and-manage approach is to create environments within which people want to do the things you want them to do – for example, in education, emphasising the reasons for learning, or using the student experience as a starting point to create something that they see has value.

This may all sound very much like hand-waving (and it is), but it has a strong basis in educational philosophy.

My memory of people trying to motivate me during school was a long chain of extrinsic motivations: Why do I need to learn algebra? (To pass the test). Why do I need to pass the test (To get into University)… (To get a good job)… (To earn money) (…).

Why do only some teachers tap into the intrinsic motivation for learning? This is topic too big for this post that merits much further discussion.

It is fundamentally a different focus to look towards building a culture rather than building a compliance. This applies as much to academia as it does to education.