Categories
Research

What gets measured: Bibliometrics and education

The world of research has been changing radically in recent decades with the ascendence of bibliometrics. All journals are now quantified and ranked by various measures, such as Source Normalised Impact per Paper (SNIP), the Impact Factor (IF) or the SciMago Journal Ranking (SJR). These are all variations on a theme of looking at how journals cite each other, on the basis that heavily cited articles and journals are of higher quality.

Criticism of such quantification is well documented and widely available, but the interest in this article is in this friction between what we might call the need to quantify and those things that are not quantifiable.

In this age of management we often hear the catchcry that “what gets measured gets managed“. That’s the thinking behind the use of KPIs in organisations. It’s also the thinking behind standardised tests in schools and this is where a lot can be learnt from making an analogy between bibliometrics and education.

What are the assumptions?

The assumption in bibliometrics is that citations are a good indicator of impact. This has been refined (i.e. in the SJR) so that citations in a well-respected journal count for more towards a weighted score (using the same algorithm that Google PageRank uses) and has been augmented by other innovations that make for better predictions (e.g. this technique) but the overall assumption is still the same – that more citations (in the past or the predicted future) is what you’re after.

Similarly, the assumption in education is that standardized tests (e.g. NAPLAN in literacy and numeracy) are measuring the thing that we want to be producing. In other words, it has the assumption that our education system is about producing adults that can perform well on these kinds of tests.

Clearly nobody (well, hopefully nobody) actually thinks this way, but the problem is if you don’t take this as the measure of successful education, what can you use?

Another way to measure?

The problem with measuring things (like test scores) that people do is that they will invariably game the measurements to maximise results. In this case, teach to the test at the expense of other things (e.g. producing whole and complete human beings capable of being a positive force of change in the world).

The bibliometric measure is particularly interesting (as is the Google PageRank) in that it is far more difficult to game, being fundamentally a socially driven measure (although it can still be gamed, obviously, but much less directly).

Networked Learning for example is another way of thinking about education and suggests a different set of measurements. This is one way we could go.

Or not measuring at all?

Really though, humans work best when they have intrinsic motivation. Extrinsic motivation has been a fantastic discovery that we have been using for many centuries now to make people do things that they don’t really want to (e.g. work or you don’t get fed), but basing everything in society upon this mechanism of extrinsic reward and punishment, whilst in vogue, is unlikely to create the kind of world that we want to live in.

The alternative (or at the very least augmentation) to the ubiquitous measure-and-manage approach is to create environments within which people want to do the things you want them to do – for example, in education, emphasising the reasons for learning, or using the student experience as a starting point to create something that they see has value.

This may all sound very much like hand-waving (and it is), but it has a strong basis in educational philosophy.

My memory of people trying to motivate me during school was a long chain of extrinsic motivations: Why do I need to learn algebra? (To pass the test). Why do I need to pass the test (To get into University)… (To get a good job)… (To earn money) (…).

Why do only some teachers tap into the intrinsic motivation for learning? This is topic too big for this post that merits much further discussion.

It is fundamentally a different focus to look towards building a culture rather than building a compliance. This applies as much to academia as it does to education.

Categories
Education Research

Independent public schools – Why competition in public education is a bad thing

THE SLIPPERY SLOPE OF INDEPENDENT PUBLIC SCHOOLS

The current government is committed to creating “independent” public schools, a change to bring more autonomy to around 25% of Australian public schools. A $70 million fund has been set up to make this a reality.

According to Education Minister Christopher Pyne, independent public schools will give principals the autonomy to hire and fire their staff and control more of the school budget.

What is not being talked about is that a brief tour of recent history in other Western countries shows that dividing the public education system in two can be a very slippery slope. It has a tendency lead to negative social consequences and there is a lack of evidence of better student outcomes. So why then is this happening?

BIG IDEAS: TO SUBSIDISE THE SCHOOL OR THE STUDENT?

Influential Chicago economist Milton Friedman wrote more than 60 years ago about how to formulate government policy in education. Friedman’s big idea was that just because education is provided by a government, it does not mean that the forces of competition cannot be useful here too.

Friedman’s suggestion was that governments have a choice, between giving money to schools (to educate our children), and giving money to the parents of children (with which to choose a school). The advantage of the latter, says Friedman, is that schools will have a reason to compete with each other – with a result of greater quality for less money.

For this to work how Friedman intends, public (state-run) schools need to be in competition with other schools – and private schools, with their large annual fees, aren’t competing for the same students.

The way competition is achieved by policymakers is to give money to schools based on how many students they have, whether they are run by the state or whether they are independent – what is sometimes called a “voucher” system, in which parents theoretically have more choice.

In the US they created the competition for public schools by giving out “charters” for non-profit schools to be formed that are independent of the state in many ways, but are still subsidized by the state. In the UK they created “academies” under a similar model with government funding but school independence.

Here in Australia the suggestion is to follow a similar program and make roughly a quarter of the nation’s schools “autonomous”. The motivation in all three cases is the same, a desire to introduce choice and competition into public schooling – and this is where the problem lies.

THE DANGERS OF COMPETITION IN EDUCATION

Whilst superficially appealing, evidence shows that introducing competition to public education can have a raft of undesirable social consequences that are very difficult to undo.

Some of the most harmful effects of introducing competition to public schooling have been identified by researchers as:

1) More pronounced social stratification. Dividing the public school system into autonomous and state-run schools is likely to widen inequality. The case of education in Chile, where a similar system (the first in the world back in 1981) provides an extreme example of the way that competition in public schools can lead to a cycle of widening social inequality.

In Chile, autonomous schools have the ability to select or reject students. In the drive for better results and the development of reputation these schools tend to favour students from wealthier backgrounds that are already achieving better results. Consequently, the 38.5% of students that remain in the fully public system are much more likely to be from disadvantaged backgrounds and perform more poorly in national tests.

Further, the idea that parents now have more “choice” in where to send a child with the advent of competition is not as straightforward as it might seem. Will an autonomous school, which is being measured on academic results, spend its advertising budget in a wealthy or a disadvantaged neighbourhood? Does a parent that has the time and money to transport a child long distances have the same amount of “choice” as a parent that does not?

2) Impoverishment of the educational mission. Consider two schools in the US. One is a state-run school, the other a non-profit autonomous charter school. The state school spends their allocated $500 on advertising. The other school spends a full $325,000 on advertising. The autonomous school can do this because they have a different mission to the public school. Their “success” is entirely determined by the number of enrolments they can get and their score on a very limited range of indicators (often standardized tests). The two schools are playing by different rules but are evaluated by the same measures. It serves neither society nor the aims of public education to have schools spending their budgets on advertising or in targeting standardized tests, yet a competitive system encourages such spending.

3) Teacher remuneration and satisfaction. The charter school movement in the US was used as an opportunity to attack unions, with by far the majority of charter schools being non-union. This led to serious fissures within the profession and in many cases led to reduced teacher wages. Charter schools in the US have been shown to have a much greater turnover of staff leading to teachers in these schools with less experience.

4) Exploitation by for-profit entities. In the United States, charter schools are able to enter into agreements with for-profit organisations. This can lead to effective ‘subcontracting’ of education where a school pays a for-profit company to manage the school. Autonomy in schools can also open the door to corporate sponsorship, introducing commercialisation into the early years of schooling – a polluted mental environment in the very place where students are meant to be learning the skills for life.

5) Contentious educational outcomes. For all of this, the educational benefits are from most evidence non-existent. Chrisopher Lubienski, a Professor of Educational Policy with a focus on public and private interests, summarises the situation:

“Advocates argued that autonomy from bureaucratic regulation allows these schools to react to the needs of individual learners and be more effective at their core academic mission as measured by standardised tests… [Yet] Large-scale empirical analyses consistently find that charter schools are no better – and often somewhat worse – than public schools at boosting student achievement, even after controlling for demographic differences in the populations served at different types of schools”

Lubienski is talking about the situation in US charter schools, but the first sentence sounds eerily familiar to anybody listening to the rhetoric around the move towards independent public schools.

IT HASN’T HAPPENED YET, BUT THE RISK IS REAL

Here in Australia, we have a well-respected public system facing a serious change with the move towards independent public schools.

The assurance we are being given by government is that the independent schools policy is simply a way to give more power to principals regards staffing and operations. Whilst a reduction in bureaucracy within schools is welcomed by many there are other ways to achieve this outcome without creating this split between those that are autonomous and those that are not.

The risk we run by creating a division within our public school system is serious. Once a nation heads down the slippery slope of introducing more “choice” and “competition” into public education, the door is left wide open for any well-meaning ideologue to follow the ideas of Friedman.  All that is needed is to keep giving this subset of our public schools ever more autonomy and it will be to the detriment of the system as a whole.

This article is written as a response to Pyne’s claim that “the more autonomous a school, the better the outcomes for students”.

Categories
Research

Publications

Some recent work on teacher education (free and downloadable):

http://ro.ecu.edu.au/ajte/vol39/iss4/4/

And on creativity in design (paywalled, free version here):

http://link.springer.com/journal/163

Categories
Research

Distributing determination of quality #2

In the previous article, as in the book chapter below, we discuss the idea of ‘distributing the determination of quality’. I think a better term for what we are trying to say can be devised, one that is less of a mouthful (any ideas welcome).

This is a short post to describe some parts of the animal, even if we’re not sure what beast it is yet.

– Peer assisted learning is a fantastic example of distributing the determination of quality in higher education. For example, having peers that took a course mark that course in the following year as a part of their second year studies. If implemented badly this is exploitation. If implemented well then all can benefit

– I attended an session with the Queensland Department of Education, Training and Employment (DETE) on the theme of doing research in education in Queensland. One idea that came up was the idea that instead of each school striving to “maximise their NAPLAN results” and being judged on this (as is the current practice), imagine if each school was judged on how well the four closest schools performed? The idea is that instead of encouraging competition and individualism, it encourages altruism and reaching out to help those schools closest. It was not a serious suggestion, but rather a nice sketch of how the world could be if KPIs were something distributed to provide external motivation for altruism, rather than concentrated to provide external motivation for self-interest.

More examples to follow (any suggestions welcome too).

Categories
Education Research

Innovation and distributing determination of quality

I’ve recently published an ADFI Blog post here. This post is a cut and paste of the content from this article:

An image drawn from Buddhist philosophy is of a spider’s web covered with dew-drops. The significance of the image is that its beauty comes from the way in which each dew drop reflects all other drops. The droplets are interconnected in this complex way, and their beauty comes from these connections.

Dewy spider web
Image Source: http://en.wikipedia.org/wiki/File:Dewy_spider_web.jpg
(Licensed as Creative Commons: Attribution-Sharealike, image by http://en.wikipedia.org/wiki/User:Fir0002)

The image is a powerful one to introduce the theme of how we determine quality (of resources, people, information) in a society that increasingly relies upon the internet. The internet has changed and will continue to change many sectors of society by making it cheap (trivially so) and accessible (we all know how to do it) to duplicate and communicate data anywhere in the world. As a society we are still exploring the effects and potential of this.

This blog post is about one idea within this context; that much of the disruptive innovation driven by the internet has come about not simply through increased inclusivity but, rather, through innovation in ways to distribute the determination of quality.

Whilst it is often simple to ‘scale things up’ with the internet and get more people involved in something, the hard part is finding a way to similarly scale up the way that quality is determined. (For a subjective definition of quality we adopt the notion of ‘fit for purpose’)

Some examples are helpful for introducing the idea and distilling a common narrative (Kelly, Sie, & Suwer, In press):

  • In the 90′s anybody could create a web page with HTML, and many people did. However, judging the usefulness of a given site was a difficult problem. Google changed all that with their use of eigenvector centrality (within the ‘PageRank’ algorithm) to quantify quality based upon the interconnected network of web page links (Page, Brin, Motwani, & Winograd, 1999). A site’s value is judged based upon the links to it from other sites, and these links are weighted based upon their respective value. By mining the graph of connections the value of each site can be determined. (A fantastic intuitive understanding of the complex nature of this calculation can be gained through this NetLogo simulation of PageRank).
  • In research administration there is a desire to ‘manage’ the quality of research and hence to measure it. Many academics publish many articles, but which ones can be judged as high quality? The field of bibliometrics attempts to respond to this problem through content and citation analysis. Some of most popular metrics (e.g. the SCIMago Journal Rank or SJR as it is commonly known) utilise applications of the same eigenvector centrality measure used in Google’s PageRank.
  • The company AirBnB has provided a platform such that anybody can open up their house for guests, thus allowing individuals to compete with hotels to provide accommodation. The difficulty in setting up this platform was not so much the ability to list their house (increasing inclusivity), but rather the challenge of ensuring quality in the listings – people wanting to stay in accommodation register by uploading ID documents, people listing houses need to provide accurate photos, and ratings and reviews play a part in providing constant community feedback.
  • A new company Uber allows for anybody to use their car to provide ‘taxi’ services to others using the platform that requires this service. The online platform removes barriers to entry (inclusivity) allowing anyone with a car to compete in providing taxi services and it maintains quality in a way that is similar to AirBnB.
  • eBay similarly allowed for increased inclusivity in online trading, allowing anybody to turn their home into a warehouse for selling goods. The challenge of ensuring quality involves legal protections, reviews, ratings and metrics such as number of goods sold.

These few cherry-picked examples have a common narrative: the internet makes it possible to implement processes on a grand scale. Wherever this occurs, a need is created to distinguish quality. The above examples can all be seen as disruptive innovations within their sector. It can be argued that the significant innovation in each case is the way in which quality is determined within the large-scale community.

The current fashion for MOOCs (the acronym for which is increasingly irrelevant) in higher education provides a useful case study. The model of students being taught by a lecturer and tutors on campus has been around for centuries and is difficult to scale. Putting digital course content online and making it open has been around in various forms for a long time but offered a different kind of education. Downes and Siemens introduced the term MOOC to emphasise the way that online courses designed in a specific way could be massive, open and, in this way, fulfil the aims of connectivist pedagogy (what are now known as cMOOCs). However, determining the quality of students within MOOCS (both formative to aid the students and summative for the ‘gatekeeping’ role of university education) is difficult within this expanded context. Innovations such as automated marking and peer-assessment have been used within the MOOC context but, as the two links show, also strongly criticised.

Recent work has begun developing a collection of ways in which quality is assured online and organising them into a taxonomy:

  • Implicit measures: Users of a service to do what they would normally do. Quality is measured using implicit metrics, such as Eigenvalue analysis and content analysis (e.g. PageRank) or indicators of behaviours (e.g. seller ratings of eBay based upon volume sold, and measures of contributions on StackOverflow)
  • Explicit measures: Users of a service or paid professionals take specific actions to ensure quality. Examples include paid staff rating contributions to the OER commons, ebay members rating their experiences, or Amazon buyers reviewing books.

A theory for quality in connected systems?

It is useful to identify and describe these measures of quality, but a more profound question is: Can we identify a technique that could aid this kind of innovation for determining quality?

An inspiring example is Shannon’s work in developing information theory (Shannon, 2001). Shannon saw that engineers were coming up with ways of sending and receiving signals with greater or less bandwidths and in the presence of noise (interference). Rather than contribute to ad-hoc, domain specific solutions, Shannon was able to develop a mathematical representation for the problem and consequent implications of this representation that make up the basis for much of the cryptography and communications that we use today.

What could such a representation, abstracted away from eigenvector centrality or specific measures look like? Or perhaps, at the very least, a general formulation of an approach that could be used in determining quality regardless of the medium or context.

This blog post is much more about questions than answers. The spiders’ web remains a beautiful symbol for our ever more connected world. Within recent centuries we have moved from a philosophy of the whole-in-the-parts to mathematical and applied representations of it. We propose that this way of thinking is particularly useful for distributing the determination of quality – there remains much more to be uncovered in applying this type of thinking.

If you would like to be involved in this research as a collaborator or higher degree research student, contact Dr Nick Kelly.

Image credit: Dewy Spider Web by User:Fir0002 Used under Creative Commons Attribution-Share Alike 3.0 Unported licence.

References

Kelly, N., Sie, R., & Suwer, R. (In press). Innovating processes to determine quality alongside increased inclusivity in higher education. In M. Keppell, S. Reushle & A. Antonio (Eds.), Open Learning and Formal Credentialing in Higher Education: Curriculum Models and Institutional Policies: IGI Global.

Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: bringing order to the web.

Shannon, C. E. (2001). A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1), 3-55.

If you would like to be involved in this research as a collaborator or higher degree research student, contact Dr Nick Kelly.

References
Kelly, N., Sie, R., & Suwer, R. (In press). Innovating processes to determine quality alongside increased inclusivity in higher education. In M. Keppell, S. Reushle & A. Antonio (Eds.), Open Learning and Formal Credentialing in Higher Education: Curriculum Models and Institutional Policies: IGI Global.
Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: bringing order to the web.
Shannon, C. E. (2001). A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1), 3-55.

Categories
Education Research

Good teachers are leaving the profession

Nobody really knows how exactly many teachers drop out of the profession. A couple of recent articles are keeping this issue in the spotlight, which is a good thing:

This from the conversation

And this a reference to the Gallant and Riley study (Monash).

50% in the first five years is a figure thrown around, 30% in the first three years is another. And unfortunately there is reason to believe that many of the leavers are good quality teachers.

How to keep them in the profession?

Some suggestions that have been raised: increase salaries of experienced teachers and finding a way for society to recognise the professionalism and technical skills required for the profession, giving them the social status they ought to have.

Many ideas about what it means to be a quality teacher are very well articulated in this article by Raewyn Connell.

Categories
Research

ASCILITE Presentation

Presenting about support for beginning teachers at ASCILITE. Most useful thing about the conference was finding out about what the Keeping Cool people are doing in this area.

Categories
Research

Academic WordPress Todo List

This is a short post about the joys of being an academic in the 21st century. There are more digital communities available than there are predatory open access journals these days and so what is useful online and what is not?

Clearly actually doing research and publishing is the important thing, but my thinking is that the following things are also useful:

  • Slideshare is actually helpful. I’ve noticed some professors that put their slideshare on the last slide, and if they’ve been interesting then it makes it easy for a followup. I’ve also stumbled across collaborators by first coming across their slideshare in Google search
  • Find one place to store all your papers. There are lots of options, like your institutional ePrints or Mendeley or Academia.edu or your personal website. I’m finding that currently I’m trying to keep three of these updated and it doesn’t work so well

I’m still figuring out if blogging is useful (the fact that my colleague’s blog was cited in George Siemens’ book is a good indicator that it probably is if done well) but the point of this website is to try it out and see. I’m sure there’s a clever way of doing this online presence thing that involves half an hour a week, so I’ll keep this updated as I figure it out.

I feel like the ideal kind of a pipeline would be:

  1. Write an article and publish it
  2. Deposit in the institutional ePrints
  3. Write a short blog post about why the world is a different place with this incredibly important finding and linking to the paper
  4. This blog post would then be automatically echoed in social media in some useful way. (What does this look like? Facebook? Twitter?)

I’m figuring all of this out (including the purpose of any of it, given that quality research is the hard part, but then I’m still figuring out the purpose of that is too, so bear with me for some time).

Categories
Research

Research overview

I am a Research Fellow based at the Australian Digital Futures Institute (ADFI). This web site is about bringing together all of my research to communicate the ideas beyond publications and to listen to any comments.

A good place to start is with an overview of all the research that I’m actively involved in at the moment, with some links to the projects and the organisations responsible:

    • Virtual Communities of Practice for Transition from Higher Education to Profession (ADFI)

Universities build strong communities amongst pre-service teachers. Often much of the community support is lost in the transition into practice. How could we design and foster a virtual community of practice within the university that could then support these teachers as well as possible in their first year of practice?

    • How students ground their knowledge when learning from models (ADFI, CoCo)

A collaboration led by Lina Markauskaite of the CoCo research centre using results from a study of students as they learn about the complexity of climate change with a model. The work examines epistemic fluency in model based learning.

    • Learning Analytics: Natural Language Processing (ADFI, CoCo)

More info to follow, but for now view papers here where you can see the published work that has come out recently.

A thread that runs through the work is creating communities for supporting learning – where learning is something takes place both inside and outside institutions.