How can you effectively evaluate the effectiveness of e-learning simulations using learning analytics?

As far as using LA tools and models on E-Learning simulations is concerned, it is essential to ensure that LA are deployed with the same level of diligence that currently-run courses are to be treated to. The first core principle of making simulations representative of the ‘’real world’’ is ensuring that they are carried out via valid and reputable analytical frameworks that the training provider uses on for course that are already available to the learners. Secondly, for simulations for courses that differ from the ones currently on offer, comprehensive assessment metrics should be tailored specifically to the unique requirements. Last but not least, simulated environments should incorporate a number of scenarios that may occur.

Starting with the end in mind and mapping skills and learning stages toward that end goal has always been how a curriculum or instructional designer tackled this work. Now, newer tools and visualization options make progress toward learning even more accessible for more significant swaths of demographic groups (such as learners who speak languages other than English, those with diverse needs, or our youngest learners). I want to understand how these diverse groups interact with and learn from the simulation and content and how we can make the outcomes equitable across groups.

Everything is too generalized. Could we please first start by at least mentioning a specific subject, background of the learners, background of the trainers, type of e-learning, casual or professional? And so on. We could start by asking each learner to provide his / her qualifications, reason for joining the training, and their ratings of the e-learning exercise. We must first have a reliable rating or benchmarking of the training, the subject matter, sequence of topics, depth of explanation, etc. The responses of learners gives us the data. For analysis we would need data from a large number of people suitably grouped. If the resources are available, we should get some useful results.

Identify the gaps and set goals that close the gaps Contextualize custom learning content models to help learners understand real-time scenarios Break them down to digestible nuggets of meaningful knowledge Introduce concepts that help establish understanding of application leading to performance driven knowledge

In any learning environment a set of meaningful learning outcomes need to be set such that the learner can see a definition of what they should expect to learn during the learning experience, and be tested against them. Learning outcomes therefore become definitions for analytics which should evaluate whether learning outcomes have been met adequately (e.g. exam, project, student satisfaction). Any outcomes not met would become a source of information to define and refine further development. For example, research into the problem may find that activities need to be defined in different ways to meet different learner types, or indeed, it may highlight that the learning outcome cannot be met adequately due to the environment design overall.

Organizational size matters when you begin to talk about analysis and tools. If you are a smaller organization moving into e-learning and simulations for the first time, you'll start smaller with the analytics, but recognize that regardless of xAPI, LMS or headless approaches, you will probably need an analytics-specific tool, not just canned reports.

Implement a learning analytics system within your e-learning platform to collect relevant data. Common data sources include user interactions within the simulations, quiz and assessment results, and user feedback.

Your metrics should be technical in addition to learning-based. Why? While not inherently indicative of learning, if your learning model being incorporated is efficient, but your system is not, then that can affect the data from your learning metrics, too. The overall E-Learning experience does not segregate learning and usability. They are correlative and need assessing regularly.

Assuming the XAPI can harvest sensory data that quantifies the state of the brain and body, for example, BCI (brain-computer interface) devices that measure EEG and describe emotional states, eye tracker camera systems leveraging the device camera, perhaps users using wearable devices that measure, heart rate, blood pressure, and glucose levels, then data can be unified to not only determine the "realness" of the simulation but also matched to the user response. A XAPI that utilises wearables can measure fields with an accuracy that tells us if the experience matches a roller coaster ride where riders are scared witless even when they know they are safe. The ride is a perfect simulation for adrenaline junkies, even when they know it.

While most LMS available in the market have built-in analytics functionality that churns in reports for you, it’s also important to analyze historicals to see where your blindspots are. With that, you may need use the pre-built reports and feed them to a separate analytics app (sometimes Excel formulas are enough) in order to stratify and re-arrange data based on need. Some LMS (like Adobe Learning Manager) allow you to build Dashboards via PowerBI for example by giving you the capability to gather the data points directly from the database.

Understanding and publishing results is important, but don't forget that you have at least 4 potential audiences that need information: (1) the learner, (2) the learner's organization/manager, (3) your team of ID/LX'ers, and (4) your leadership. Crafting the data so that each of these groups can understand what's happening, and being able to make suggestions to move forward are key.

Communication of the results with all stakeholders should be a priority, especially with the learners: this communication should be mediated by the trainers, so that they themselves receive feedback from the training experience and can identify areas for improvement.

Adding short quizzes and or chances to virtually perform tasks throughout the E-Learning will help to engage as well as reinforce what has been presented.

Reviewing user profile for his progress on regular basis will always help in deciding about content of simulations for e-learning. Learners interest in various parts of the content is to be analyzed for improving the whole course as per its objective.

The tricky part of reporting these data points is understanding that as the L&D professional, it is your duty to translate* these data points into something that stakeholders understand—which also includes highlighting data points that may be important to other business arms but not for you.

I’ll add to this in that acting upon the feedback we get from users/learners is paramount as well. I’ve had learners make valuable suggestions I would have never thought about.

Data-driven insights pave the way for enhancements. Identifying strengths and weaknesses allows for targeted improvements. Whether it’s adjusting difficulty levels, enhancing interactivity, or creating new challenges, using analytics to guide these decisions ensures e-learning simulations evolve to meet learners’ needs effectively. Incorporating learning analytics in the evaluation process transforms raw data into strategic actions. By understanding, defining, collecting, interpreting, and utilizing this data, educational simulations can continuously evolve, offering richer, more effective learning experiences.

It should be a regular feature of e-courses, modules, course components to seek three good things and three things from the users about things to be improved in the content.

Adapting, adjusting and correcting our e-learning simulator training proposal is part of a necessary process if we want the impact on learners to be relevant and lasting. And, of course, the more we can incorporate feedback from users' first-hand experiences, as @DerekSchwartz points out, and I would add, by establishing a fruitful dialogue with them, the more we can refine and improve the training for effectiveness and quality.

There are instances that rapid prototyping is more efficient, especially if feedback comes from intermittent timings of different streams. All feedback should be processed, and all systems should be adaptive as feedback comes.

I was involved in rock climbing wall training, where the instructor was giving a lecture to some primary children, they were training for mountain climbing, so this was a simulation for the real thing, with harnesses and ropes. After 30 minutes, I asked the instructor to let the children loose on the wall. Let them figure it out by experimenting and watching each other. Not surprisingly, the children quickly learned everything they needed to do that the instructor had been eagerly waiting to lecture about. Needles to say, she left the session after watching them for an hour. The children eagerly carried on their self-directed training simulation and covered everything they needed to The best learning simulations must mimic this.

It is a continuous process where learner and teacher have to regularly interact with each other, content, peers and other resources. These considerations to be prioritized in all modules.

In my research, I have stumbled on several ideas to invigorate teaching using simulations, without having proper learning objectives defined at the start. If you do not know what the students need to learn/understand/prove, there is no use in discussing the tool or the metrics surrounding it.

It's important to remove bias from your evaluations as much as possible. It can be easy to find the data that supports what you want or expect to find, but being open to what the data will be is an opportunity for growth in ways you may not have envisioned.

When creating a simulation, it's important to think about the learners' needs. Consider their current knowledge, skills, and motivation level. Feedback is crucial to help them improve and achieve their goals. Make sure the feedback is timely and constructive.