Quality Assuring Online Learning: ACODE’s Work on Benchmarking

This presentation, of 26 minutes, unpacks the concepts behind the 'TEL hierarchy of needs', as exemplified through current practices of the Higher Education in the Australasian sector. And importantly promoting the work being done by the Australasian Council on Open Distance end eLearning (ACODE). It was presented at the Quality Assurance of Online Learning Forum run by TEQSA in Melbourne, on 27 November 2018.

To set the scene:

• We know students are seeking consistency within their courses/units in the online learning environment 
• TEQSA, and we (the sector), also want a level of consistency for the learning outcomes between f2f and online courses
• This means institutions need to have quality processes in place to ensure both course quality and process quality
• Over the last few year TEQSA have been focusing on process quality
• They are now turning their attention to course/unit quality
• But It takes a village to raise a child

In relation to this village, of particular note is the emphasis TEQSA has placed on Benchmarking (external referencing) amongst members of our Higher Education village. This is seen in their Guidance Note released in July last year (2018). In this document is says:

"Monitoring, review and improvement processes can and should 
encompass review against comparators, both internal to the provider 
and external.

A number of approaches and techniques can be used for external 
referencing, such as benchmarking, peer review and moderation. 
Benchmarking is perhaps the most elaborate form of external 
referencing and typically consists of focused improvement through 
relationships with a benchmarking partner or partners, but can also 
include comparing course design against publicly-available information and market intelligence."

Of course there is no one activity that can mediate the level of quality in TEL within an institution, there are many factors that need to be considered; measures to be applied and activities to enact these measures. And many of these need to be applied at different levels within the organisation to encompass a wide variety of stakeholder groups.

A few ways to do this at the institution level

At the institutional level, this presentation is obviously promoting the ACODE Benchmarks for TEL. but to be fair there are a number of other measures that could be applied. Including this list taken from the 2015 ICDE Report titled, Quality models in online and open education around the globe: State of the art and recommendations:

These tools, and those like them, Benchmark activities at the institutional (meso) level, but generally do not go down to the Course/Unit/Subject level, which is now becoming increasingly important.

Enter then the notion of Course level standards (the micro level), and not surprisingly there are a range of tools that can do this also, some of which are free and some that come at a cost to the institution:

• OLC quality scorecard and toolkit
• Quality Matters (QM)
• ACODE Threshold Standards for Online Learning Environments
• eLearning Guidelines (New Zealand)
• JISC - eLearning Quality Standards
• European set associated with eExcellence
• E-learning Quality Model (ELQ) out of Sweden
• ASCILITE TELAS
• CoL (Commonwealth of Learning)
• There are a number of others (too many to list here)

But what provides the overall framework (the meta level) for an institution to apply these measures. In some cases it is contained in institutional policy, but more often than not, there are only vague references to this within the overarching Learning and Teaching Policies within our institutions.

Presumably this could be handled by an institution implementing some form of technology enhanced learning framework, that explicitly aligned all these different elements, making it easier for an institution to ensure their practices are being handled in a consistent and reportable way (for when the quality agency comes knocking at the door). We are getting better at constructive alignment at the course level (aligning our assessments with our course learning outcomes), as the quality agency is very interested in this, to ensure students are getting equivalent experiences across institutions (think TEQSA Higher Education Standards Framework). Well, funnily enough, how are institutions going to evidence that their online learning is equivalent to their face to face offerings, and that this is being done in a consistent way, particularly compared to the other institution down the Street?

Enter the notion of Technology Enhanced Learning Framework, one that aligns to benchmarks and course level standards. One that is flexible enough to be applied to suit an individual institutions context. This has been the work recently conducted by a team of 14 institutions (the ACODE representatives) from across Australasia and is currently being piloted by 8 institutions.

This framework document was released today for broader consultation and trial and we (ACODE) look forward to receiving your feedback.

However, at the end of the day we mustn't loose sight of the fact that:

…a truly practical standard is one that will be used because it is 
simple enough to follow and flexible enough to allow for creativity … a tool that allows you to do more, rather than a grim necessity to 
which you must adhere. (Welsch 2002) 

References

• Ossiannilsson, E., Williams, K., Camilleri, A.F., & Brown, M. (2015). Quality models in online and open education around the globe. State of the art and recommendations. Oslo: International Council for Open and Distance Education. Retrieved from https://icde.typepad.com/quality_models/
• Welsch, E. (2002). SCORM: Clarity or calamity? Online Learning Magazine. [verified 31 Oct 2004] https://www.onlinelearningmag.com/onlinelearning/magazine/article_display.jsp?vnu_content_id=1526769