Assessing the effectiveness of Business Simulation Games: A short guide

The common perception is that business simulation games (BSG) improve learner engagement and learning outcomes and are therefore effective learning tools. This view is backed up by numerous studies that provide evidence for the effectiveness of simulations (although it is possible that this preponderance of positive results is a consequence of selection bias and that negative experiences are not reported).

In practice, instructors must ask themselves if any particular teaching intervention is reaching its objectives. If learners appear engaged with the simulation, actively communicate with each other and show their enthusiasm in class, then these are all positive signs. However, although such personal impressions are valuable, they are insufficient, as the loudest people in the class disproportionally drive impressions gained by the instructor. Therefore, a more systematic review assessment is a useful complement to the instructor’s own feelings about the experience.

?

Measuring effectiveness

The academic literature contains many assessments of the effectiveness of BSG. These studies include a wide variety of definitions of effectiveness, such as the level of learner engagement, exam results, other learning outcomes (like teamwork or critical thinking skills) or even professional success in the workplace.

Generally speaking, the more meaningful the measure of effectiveness, the more difficult it is to generate evidence that the BSG intervention had an impact. For example, it may be possible to show that learners are more engaged with BSG than in lectures, but engagement is not the ultimate objective of education. On the other hand, showing that superior learning outcomes or future professional success result from using BSG would be meaningful, but is difficult to prove.

This difficulty to provide clear evidence on the effectiveness of BSG stems from the challenge of causality. If a course that uses BSG appears successful, what is the evidence that the success is due to the BSG and that it is superior to other ways of teaching? Researchers often involve the use of control groups (i.e. a non-BSG teaching intervention), but defining appropriate control groups in this context is not easy. Control groups will always have some differences from the experimental groups in terms of student profile, making it difficult to make valid comparisons.

Although it is useful to see how researchers have approached the question of assessing effectiveness, instructors need to find a balance between relying only on their own feelings and carrying out an extensive study with credible control groups. In practice, a few simple steps can yield important insights.

Engagement

Instructors can begin to assess learner engagement informally by just asking learners how they felt about the experience, for example, during the debrief session. Subsequently, learner engagement can be measured more systematically by administering a simple online survey. This is easy to do and will allow all learners to make their opinions heard. A sample engagement questionnaire is available upon request from Sim Institute.

Although survey results can provide valuable information, they lack the richness provided by anecdotes. Therefore, a short (but structured) focus group discussion with around five to ten learners can yield useful additional insights. Focus groups can discuss a variety of questions, including feedback on the quality, length and complexity of the BSG, the instructions and support available, the fit with the course, and the way it was implemented.

The instructor functionality of the BSG itself may also provide data that reflect learner engagement, including the number of games played, the amount of time spent online and when the BSG has been played (only during class time or also outside class). When learners spend much time playing the BSG outside class, this is usually a sign of high engagement levels or strong competitive spirits!

Any of these methods can help to gain valuable insights into the level of engagement of students with the simulation.

Learning outcomes

The measurement of learning outcomes can start by considering the game scores obtained by learners. If the BSG has been well designed, better scores usually indicate a higher level of skills and knowledge. For games that are played several times, it is instructive to see if learner scores are improving. However, game scores by themselves do not provide real evidence of learning, as it is not certain whether the skills and knowledge that are reflected in game scores have been developed during the BSG exercise or prior to it. When BSG are played in teams then game scores are especially difficult to relate to individual learning.

A more robust way to assess learning is through learning assessments related to the BSG, such as exams, quizzes, written learning reflections or presentations. If the learning objective of the BSG is very closely aligned with that of the course, then the existing assessments and Assurance of Learning practices for the course may already be sufficient.

However, just like with game scores, good assessment results are not always a reflection of learning that has taken place as a result of the BSG. Administering a quiz before and after the BSG can identify improvement that has taken place, but the questions in the two quizzes would need to be different but of similar difficulty.

?

Conclusion

In conclusion, assessing the effectiveness of teaching with BSG can be complex if the purpose is to come to scientifically robust conclusions. But if the main objective is to assess effectiveness of BSG and to get the most benefit from using BSG in future, then simple tools such as a short online survey or a 30-minute focus group discussion provide a wealth of insight. If instructors use the same evaluation tools within one institution and over time, they build up data against which each new class can be benchmarked.

Subscribe to Sim Institute's Newsletter and stay up to date with developments in simulation games in Sustainability and Management Education.