March 2016 - Dr Nick Kelly

This post is about designing learning analytics tools and some theory around their innovation.

Having said that, this site is at risk of becoming a litany of posts about papers that have come out, such as this one on teacher support and this one on developing online communities. I’m curious, does anybody have any good examples of academic websites that illuminate? One that I’ve found is the blog by David Jones (USQ) which is extremely generous without being outlandish – a place where new ideas happen as well as discussing established ideas. Something to aspire to perhaps.

This post is about the design of learning analytics tools, and sharing a paper that I wrote with Kate Thompson and Pippa Yeoman. The paper is interesting in that it discusses the design of learning analytics tools in the big picture, before using the innovation of a specific tool to provide an example.

The abstract for the paper:

This paper describes theory-led design as a way of developing novel tools for learning analytics (LA). It focuses upon the domain of automated discourse analysis (ADA) of group learning activities to help an instructor to orchestrate online groups in real-time. The paper outlines the literature on the development of LA tools within the domain of ADA, and poses an argument for conducting tool development based upon first-principles. It describes first principles as being drawn from theory and that these principles can subsequently inform the structure and behaviour of tools. It presents a framework for this process of theory-led design. The framework is substantiated through the example of developing a new tool for assisting instructors with the orchestration of online groups. A description of the tool is given and examples of results from use with real-world data are presented. The paper concludes with a call for intent on the part of designers to connect the design process explicitly to theory on the basis that this has the potential to yield innovation when developing tools as well as the prospect of outcomes from tools connecting back to theory.

It was reviewed by Chris Teplovs who asks tricky questions around how learning analytics innovators can work towards tools that scale along with the innovations occurring in education. He goes on to discuss the work occuring at the Michigan Innovation Greenhouse, which focusses upon this kind of innovation. To quote Chris:

One question that emerges has to do with the sustainability of the development of theory- and principle-based tools and techniques. Are there better models for getting to scale with educational innovations? Those of us at the University of Michigan’s new Digital Innovation Greenhouse think there are. Researchers are good at innovating but their creations can seldom be scaled up for widespread adoption. They are focused on creating and testing innovations but typically have little experience with developing software that can be supported as infrastructure. On the other end of the spectrum, Information Technology Services (ITS) organizations are very good at staging and supporting mature software systems. Their skills are ill-matched to the loose, rapid, duct-tape development methods of researchers. It is not possible to take code from a research group and hand it off to ITS for staging at scale. We have tried for several years; the mismatch is too large. This chasm between innovation and infrastructure is present in all kinds of technology transfer. Our team posited that higher education needs a greenhouse for propagation; an interim space that understands both why innovations arrive so fragile and how to make them stronger before they’re taken “outdoors.” In the world of entrepreneurial business, these spaces are often called “incubators.” We have adopted an “incubator” model that seeks to take educational technology and learning analytics innovations and grow them to scale. Time will tell how successful we are. Innovators such as Kelly, Thompson, and Yeoman can help usher in a new era of learning analytics tools and techniques that incorporates not only powerful design frameworks but also concern themselves with the design, development, and deployment of robust and scalable tools and techniques.

Both the paper on designing learning analytics tools and the commentary are freely available the Journal of Learning Analytics website. The references for the two papers are:

Kelly, N., Thompson, K., & Yeoman, P. (2015). Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning. Journal of Learning Analytics, 2(2), 14-43.

https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4265/5088

Teplovs, C. (2015). Commentary On “Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning”. Journal of Learning Analytics, 2(2), 44-46.

https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4606/5091

Interested in the ways that teachers support one another online in teacher social networks? A recently published paper by myself and Amy Antonio looks at open Facebook groups of teachers to examine the ways in which they support one another.

Some elements of this paper are discussed in another post about the limits and potential of online communities for teachers.

The paper has been published in the journal Teaching and Teacher Education and is titled teacher peer support in social networks (free link).

The crux of the paper is that six roles can be identified that experienced teachers take on to support other teachers online:

Advocates the practical. Teachers help one another with day-to-day pragmatic aspects of the profession, such as finding resources for a lesson or navigating the bureaucracy.
Conveners of relations. Teachers instigate relationships with other teachers, and can make introductions to other useful contacts.
Agents of socialisation. Teaching as a profession has cultural norms. Experienced teachers induct other teachers into these norms, such as in the way that they share stories and the ‘memes’ that they promulgate.
Modelers of practice. Teachers give a rich description of what they are doing in the classroom, providing a model of teaching practice.
Supporters of reflection. Collaborative reflection is often considered the most important kind of knowledge for beginning teachers, to make sense of confusing situations and learn from their experiences.
Providers of feedback. Teachers provide a constructive source of feedback, such as pedagogical and curriculum advice or in reconstructing an event that has occurred.

In a review of existing online communities, it appears that certain conditions are needed for teachers to be willing to engage in the most important of these roles: modelling practice, supporting reflection and providing feedback. Such a connection appears to have preconditions of a trusted environment with stable relationships and a sense of privacy

The logic for reaching these six categories comes from two places. The first is from Clarke et al. who identify eleven roles for co-operating teachers of which only six apply to the online context. Secondly, there is much in the literature on forms of social support onto which these six roles can be mapped:

Emotional support in the form of esteem, affect, trust, concern and listening
Appraisal support in the form of affirmation, feedback and social comparison
Informational support in the form of advice, suggestion, directives and information
Instrumental support in the form of aid in kind, money, labour and time.

The main contribution of the paper is to define these six roles for online teachers. The paper then uses these roles to analyse teachers interacting in social network groups (on Facebook).

The results show that teachers support each other in open groups of teacher social networks in very pragmatic ways – there is very little in the way of reflection upon practice or modelling of teaching occurring in these groups.

The citation for the paper is:

Kelly, N., & Antonio, A. (2016). Teacher peer support in social network sites. Teaching and Teacher Education, 56, 138-149. doi:http://dx.doi.org/10.1016/j.tate.2016.02.007