What are the risks of using evaluation data to make decisions about workshops and training programs?

In my experience, you must ALWAYS include the comments in context with a star or number rating system. There will always be a learner who rates a 1, or the least positive result, while simultaneously leaving a comment that it was the best course they've ever taken. If the course or conference session is in a physical location, there will always be 1-star reviews because the room was too hot or cold. The absolute best advice I've ever received is: 1. Take out the best/highest rating. 2. Take out the lowest/worse rating. 3. Use everything in between as your real, actual evaluations.

Data, however it is gathered, does not understand many aspects, such as the human element for one. Data is gathered by biased and unbiased methods. The relevance of the data may be in question. Often times data does not represent the whole, but rather the person who gathered it.

When I used a quiz to assess the learning outcomes of a training session. I made sure that the quiz questions were- 1. Aligned with the learning objectives, 2. That the difficulty level was appropriate 3. That the scoring system was fair and transparent. This way, I could ensure that the quiz data reflected the true performance of the participants. Reliability and validity may not be the most important for evaluation data, other factors may matter more. For example, if I need to make a quick decision based on feedback from a focus group, I may not have the time or resources to check the reliability and validity of the data. Instead, I may focus on the key insights and recommendations that emerge from the discussion.

Data, however it is gathered, does not understand many aspects, such as the human element for one. Data is gathered by biased and unbiased methods. The relevance of the data may be in question. Often times data does not represent the whole, but rather the person who gathered it.

The primary risks lie in reliability and validity. Especially with small sample sizes, it's challenging to accurately capture the majority's experience or needs. Moreover, your audience and the broader climate may evolve, necessitating adjustments to remain relevant and effective in addressing the current needs and interests of your audience.

Certainly, another critical perspective is acknowledging the potential influence of bias and subjectivity in evaluation data. Bias, whether intentional or unintentional, can skew results by favoring certain groups or outcomes. Subjectivity, driven by personal feelings or opinions, may introduce variability in participants' self-reported data. For instance, participants might adjust their responses based on motivation, expectations, or social pressure. Similarly, observers should be mindful of their own biases, assumptions, and preferences when using observational methods. Recognizing and addressing these factors are essential steps in ensuring the objectivity and reliability of the evaluation process.

when we used a rating scale to measure the satisfaction of a workshop. We realized that the scale was skewed towards positive responses, and that the participants may have given higher ratings than they actually felt, due to the presence of the facilitator or their peers. This may have resulted in inflated satisfaction scores that did not reflect the true quality of the workshop or the training. To identify bias and subjectivity is to consider the different sources and types of bias and subjectivity that may affect evaluation data, such as selection bias, confirmation bias, social desirability bias, halo effect, or anchoring effect.

A different viewpoint underscores the risk that evaluation data may fall short in capturing the intricate context and complexity of workshops and their outcomes. Context, encompassing environmental, cultural, resource, and stakeholder factors, plays a significant role. Likewise, the complexity involves the dynamic interactions among workshop components, such as content, methods, facilitators, and participants. Consider a scenario where a test is used for evaluation; the test score might not fully represent participants' learning or performance across various situations. Similarly, feedback forms may struggle to encompass the myriad dynamic factors influencing the workshop experience and results.

Par expérience il ne faut pas prendre de décision immédiate Un retour terrain pour jauger l'efficacité de la formation selon des critères prédéfinis est indispensable Le suivi des stagiaires post formation durant un trimestre permet d'évaluer sérieusement si le message a été assimilé

Contrary to the mentioned perspective, some argue that using evaluation data introduces ethical and confidentiality concerns. Ethical considerations involve the principles guiding both evaluators and those being evaluated. For instance, in interviews used for evaluation, ensuring informed consent, avoiding harm or coercion, and reporting with honesty and fairness are paramount. Additionally, if a database is employed for storing and analyzing evaluation data, the need to secure and safeguard against unauthorized access, use, or disclosure becomes crucial. This perspective emphasizes the importance of maintaining ethical standards and confidentiality throughout the evaluation process.

Effectivement très souvent l'évaluation de la formation est remplie en fin de session et remise en main propre au formateur ! Difficile dans ce cas de parler de confidentialité Toutefois il est quasiment impossible d'obtenir les formulaires à posteriori, les stagiaires étant éparpillés et n'ont jamais le temps... Une bo?te pour glisser les feuilles d'évaluation avant de sortir de la salle est un bon compromis

Correct data ensures right decisions whereas poor or defective data leads to bad decisions. In my opinion Geospatial intelligence provides the kind of data that guarantees right policy decisions.

The concept of usefulness, focusing on relevance and value for intended purposes and users, implies that data should align with workshop objectives, outcomes, and indicators. In the same vein, actionability, emphasizing feasibility and applicability, suggests that clear and concise presentation of results is essential. For instance, if a questionnaire is used, ensuring alignment with workshop goals is crucial. However, it's worth noting that some argue that even with careful alignment, evaluation data might still lack practicality.

data might be misleading due to under reporting or over reporting and thus is related to safety culture within the organisation