This post is about designing learning analytics tools and some theory around their innovation.
Having said that, this site is at risk of becoming a litany of posts about papers that have come out, such as this one on teacher support and this one on developing online communities. I’m curious, does anybody have any good examples of academic websites that illuminate? One that I’ve found is the blog by David Jones (USQ) which is extremely generous without being outlandish – a place where new ideas happen as well as discussing established ideas. Something to aspire to perhaps.
This post is about the design of learning analytics tools, and sharing a paper that I wrote with Kate Thompson and Pippa Yeoman. The paper is interesting in that it discusses the design of learning analytics tools in the big picture, before using the innovation of a specific tool to provide an example.
The abstract for the paper:
This paper describes theory-led design as a way of developing novel tools for learning analytics (LA). It focuses upon the domain of automated discourse analysis (ADA) of group learning activities to help an instructor to orchestrate online groups in real-time. The paper outlines the literature on the development of LA tools within the domain of ADA, and poses an argument for conducting tool development based upon first-principles. It describes first principles as being drawn from theory and that these principles can subsequently inform the structure and behaviour of tools. It presents a framework for this process of theory-led design. The framework is substantiated through the example of developing a new tool for assisting instructors with the orchestration of online groups. A description of the tool is given and examples of results from use with real-world data are presented. The paper concludes with a call for intent on the part of designers to connect the design process explicitly to theory on the basis that this has the potential to yield innovation when developing tools as well as the prospect of outcomes from tools connecting back to theory.
It was reviewed by Chris Teplovs who asks tricky questions around how learning analytics innovators can work towards tools that scale along with the innovations occurring in education. He goes on to discuss the work occuring at the Michigan Innovation Greenhouse, which focusses upon this kind of innovation. To quote Chris:
One question that emerges has to do with the sustainability of the development of theory- and principle-based tools and techniques. Are there better models for getting to scale with educational innovations? Those of us at the University of Michigan’s new Digital Innovation Greenhouse think there are. Researchers are good at innovating but their creations can seldom be scaled up for widespread adoption. They are focused on creating and testing innovations but typically have little experience with developing software that can be supported as infrastructure. On the other end of the spectrum, Information Technology Services (ITS) organizations are very good at staging and supporting mature software systems. Their skills are ill-matched to the loose, rapid, duct-tape development methods of researchers. It is not possible to take code from a research group and hand it off to ITS for staging at scale. We have tried for several years; the mismatch is too large. This chasm between innovation and infrastructure is present in all kinds of technology transfer. Our team posited that higher education needs a greenhouse for propagation; an interim space that understands both why innovations arrive so fragile and how to make them stronger before they’re taken “outdoors.” In the world of entrepreneurial business, these spaces are often called “incubators.” We have adopted an “incubator” model that seeks to take educational technology and learning analytics innovations and grow them to scale. Time will tell how successful we are. Innovators such as Kelly, Thompson, and Yeoman can help usher in a new era of learning analytics tools and techniques that incorporates not only powerful design frameworks but also concern themselves with the design, development, and deployment of robust and scalable tools and techniques.
Both the paper on designing learning analytics tools and the commentary are freely available the Journal of Learning Analytics website. The references for the two papers are:
Kelly, N., Thompson, K., & Yeoman, P. (2015). Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning. Journal of Learning Analytics, 2(2), 14-43.
https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4265/5088
Teplovs, C. (2015). Commentary On “Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning”. Journal of Learning Analytics, 2(2), 44-46.
https://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/4606/5091