Mapping the Unmappable. The Evaluation Methods, the Human Factors and the Qualitative Unease.

Do you know this (immediate) answer after your evaluation presentation of interventions or similar actions in the field of human factors? "Do you have anything tangible with you?"

Invisible effects, behavioural changes, thought tornadoes, situational anomalies, puzzling processes, illogic, personal values and norms, incomprehensibility, joy, trust, enthusiasm, touching moments, amazement, breathlessness, kindness of heart and magnetizing intensity of collaboration up to collective intelligence. How can all this be measured, evaluated and hammered into traditional statistics? The "previous"/traditional KPIs, performance indicators or rather quantitative reporting show little understanding of these "cosmic effects".

Together with Maria Kromlidou , I have been working for many years on numbers, qualitative methods and content analysis of free text and comments (recently with AI Power) to map these "star systems" of unmappable achievements.

Our reports guide viewers and readers through a polyculture-like landscape of numbers and letters and, above all, sophisticated infographics. We focus on a multidimensional moment of encounter instead of a static and formalistic wave of numbers.

The infinite space of human nature needs reverse representation methods and courage for alternative result images in the visualization of the myriad of invisible aspects. Does such an understanding match the measured values in your corporate area? Many decision makers are ambivalent about such thoughts and even reporting. And the anxious question of whether it is possible to leap over one's own evaluation habits?

In fact, there is an intensive framework for conducting evaluations of this invisible world. Often these are overlapping methods of different theories from psychology, education, social sciences and human-computer interaction.

A possible overview:

1. Behavior Change Theories

• Transtheoretical Model (Stages of Change)?| Stages that individuals go through to change their behavior, from pre-consideration to maintenance/preservation.
• Theory of Planned Behavior?| Intention, attitude, subjective norms and perceived behavioral control influence behavioral change.
• Self-Determination Theory?| Focuses on intrinsic and extrinsic motivation and the role of autonomy, competence, and relatedness in fostering behavior change.

2. Trust Building Theories

• Trust and Technology Acceptance Model (TTAM)?| Combines elements from the Technology Acceptance Model (TAM) and theories of trust - the focus is on how trust influences technology acceptance and use.
• Trust Theory?| Examines how trust is built, maintained and preserved in interpersonal relationships and in organizations.

3. Empowerment Theories

• Empowerment Theory?| Often used in community psychology - emphasizes processes that enable individuals to gain control over their lives and influence the decisions that affect them.
• Psychological Empowerment?| Empowerment through the dimensions of meaning, competence, self-determination and impact.

4. Curiosity and Learning Theories

• Self-Regulated Learning (SRL)?| Focus on how individuals manage their own learning process by setting goals, monitoring progress and reflecting on results.
• Constructivist Learning Theory?| Learners build their knowledge through experience and reflection - their curiosity is often encouraged through active engagement and exploration.

5. Collective Intelligence Theories

• Crowdsourcing and Collective Intelligence?| How can large groups of people work together to solve problems, often facilitated by technology.
• Wisdom of Crowds?| Large groups of people make better decisions than individuals due to diversity of opinion, independence, decentralization and aggregation.

Evaluation Approaches

1. Mixed-Methods Evaluation?| Combines qualitative and quantitative research methods to provide a comprehensive understanding of the interventions and their outcomes.
2. Logic Models and Theories of Change?| Visual representations that outline the expected causal pathways from interventions to desired outcomes and are used to plan and evaluate programs.
3. Participatory Evaluation?| Involves stakeholders in the assessment process to ensure that the assessment is relevant and useful to those affected.
4. Formative and Summative Evaluation?| Focuses on improving programs during their development and implementation - with summative evaluation assessing their overall effectiveness after completion.

Integrative Frameworks

• The COM-B Model?| Integrates the components of ability, opportunity and motivation to understand behavior and develop interventions.
• Behavior Change Wheel?| Identify intervention functions and policy categories to support behavior change.

Practical Application

• Human-Centered Design (HCD)?| Incorporate user feedback and iterative design to create solutions that meet the needs of the people they are intended to serve.

Participatory Action Research (PAR)?| Researchers and participants work together to identify a problem, develop a solution and evaluate the results.

I occasionally use the non-scientific method Anecdotal Evidence and Stories for inspiration:

- What are the building blocks? Collect and analyze, for example, stories about kindness, behaviors and habits, (breathtaking) collaborations, etc. told by employees. The consolidated narratives with observations according to the most frequently used writing patterns provide complementary evidence of the impact of the chosen topic in the work environment.

- If you find yourself shaking your head or frowning as you read this, it's a good sign. Such evaluations of results are generally regarded as "weak evidence". In the qualitative field, supplementary evaluations may be "weak", but they always show more authentic evidence than interviews or multiple choice works. About the good sign - irritations are a source of reflections.

- Have scientists ever made comments on anecdotal reports? Francis Bacon, British author, philosopher and statesman from the 16th century / René Descartes, French philosopher, scientist and mathematician from the 17th century / Sir Karl Raimund Popper, Austrian-British philosopher, emphasize the importance of systematic observation, experiments and falsifiability (proof of the invalidity of a statement, etc.).

Application tips

1. Collection

- Interviews | Conduct in-depth interviews with participants of interventions/actions/prompts to gather personal stories and experiences.

- Focus groups | Organize groups where participants can share their stories and experiences in a supportive environment.

- Diaries and Journals | Participants record their daily experiences and progress in a diary.

- Observations | Documenting observations and interactions during interventions.

2. Analysis

- Thematic analysis | Identify recurring themes, motifs and patterns in the stories.

- Narrative analysis | Analyze the structure of stories to gain insights into individual experiences and meanings as well as thematic hotspots.

- Categorization | Sorting the stories into different categories such as challenges, successes, changes in behavior, etc.

3. Presentation of Results

a) As a presentation

- Case studies | Present detailed case studies of individual participants to gain deeper insights.

- Quotes | Use meaningful quotes from the stories to reinforce the messages, perspectives and focus points.

- Storyboards | Visual representations to illustrate the progression (timelines, growth or shrinkage changes) and impact of interventions.

- Theme-based reports | Structuring the reports according to the identified themes and relevant stories.

b) In terms of media and formats

- Written reports | Write comprehensive reports to summarize stories and their analysis.

- Presentations | (Animated) slides, digital whiteboards, digital clustering such as mind mapping or similar tools to effectively showcase the results.

- Video documentaries | Maximum 2.5 minute videos capturing the stories by the participants themselves.

- Infographics | Infographics in TV news format or a similar number of fascinating explanatory graphics in balance sheets to illustrate the most important findings in the viewing and reading fascination.

4. Recapitulation of results management via anecdotes and stories

a) Advantages

- Human connection | Stories create a deeper emotional connection and understanding of the people and situations involved.

- Context and nuances | Anecdotes provide context and nuance that is often lost in quantitative data.

- Motivation and engagement | Stories can motivate and increase engagement among stakeholders and participants.

b) Challenges

- Subjectivity | Anecdotal evidence is subjective and can be influenced by individual perceptions and biases.

- Generalizability | Results are not necessarily transferable to a larger population*.

- Validity | Ensuring that the stories are authentic and representative can be difficult.

*At the same time, a positive approach when local collections reveal specific multicultural signals for effective interventions from a global perspective.

Conclusion

By combining anecdotal evidence and storytelling with other qualitative and quantitative methods, researchers and practitioners can gain a broader and deeper understanding of human-centered empowerment, collaboration trust and behavior interventions, and more. | Or shrunk down to an unscientific basis: "When new complementary horizons of knowledge open up through active listening instead of immediate objections in the plenum. And thus the holistic thinking and acting that has already been promised internally and externally suddenly becomes noticeable as a spirit that flows through the cells."

Result technique: Don't do without one, add the other - with a different evaluation mix, I have observed the first signs of collective intelligence forming among team members in global activities; for example, when individual members devise and implement locally effective interventions in self-directed learning and application mode. | These are the moments that make me happiest as a learning transfer expert.

On the one hand, unmappable transformations, on the other hand, mappable moments of happiness for all involved when the activists tell their anecdotes and stories to the others. | Almost like the land of smiles in results meetings.

With the most measurable greetings