How To Identify and Document Good Quality Lessons Learned

Hello and welcome to the 2nd edition of Learn Adapt Manage. Thank you to the 650 subscribers ?? walking this journey of using #evidence to inform decisions and actions in #internationaldevelopment. Sign up by clicking the Subscribe button in the upper right corner to get future editions directly to your feed.?

A couple of weeks ago, I co-facilitated a workshop to identify lessons learned from stakeholders based on their involvement as key actors in delivering development programs. By the end of the workshop, we quickly realized that what we had were not lessons learned but a mix of personal opinions, findings, observations, recommendations, and a list of community needs to be addressed in the future. So what really are lessons learned, and how do we identify and document them?

In this edition, we will learn more about lessons learned and how to define, identify, and document them. In future episodes, we will focus on utilizing lessons learned for adaptive management in programs.

What are good quality program lessons learned?

There are many definitions, but here are my two go-to definitions of lessons learned to start the conversation.

Lessons Learned are generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact. (OECD-DAC)

A lesson learned is knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure…A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result (Secchi, 1999 in Weber 2001).

What does a good-quality lesson-learned statement look like?

Based on these definitions, here are a couple of questions you can use to assess the quality of your lessons-learned statements;

1. Does the statement specify the context from which the lesson is derived? A lesson learned statement can answer these questions: what happened, why it happened, why it is important, and what conditions or assumptions the lesson was learned.
2. Does it establish its relevance beyond that context? A lesson-learned statement includes details on where the lesson can be applied and by whom.
3. Is the statement concise enough to suggest or guide some type of action? The statement has a clear call for action, including what can be repeated or avoided in future contexts.
4. Do multiple sources and types of evidence support the statement? Justify the lesson with proof of why it is valid. Consider if the lesson is supported by more than one source of evidence.

What are the evidence sources for supporting lessons learned?

Now that you know what makes good quality lessons learned, let us review some evidence sources that can help you gather supporting evidence for your lessons learned statement. These sources are directly borrowed from M.Q. Patton's work.

Lessons learned with only one type of supporting evidence would be considered a “lessons learned hypothesis.”

1. Practice wisdom and experience of practitioners;
2. experience from program participants/clients/intended beneficiaries;
3. evaluation findings – patterns across programs;
4. basic and applied research;
5. expert opinion;
6. cross-disciplinary connections and patterns;
7. assessment of the importance of the lesson learned; and
8. strength of the connection to outcome attainment.

How do you document a good-quality lesson-learned statement?

Rick Davies (2009) recommends the following structure and walks us through an example of what a good quality lessons-learned statement would look like.

If you do…X….and these …Z…conditions are present, then …Y….will happen.

Example: "Where possible, use NGOs for community-level work. The experience of using Govt in NTT and NGOs in NTB to do Desa Siaga implementation confirms the views the IMET team has previously expressed about the potential usefulness of NGOs in this role. According to GTZ, "NTT works through government structures. This slows down the process as implementation is done using local subsidies and district health offices - DHO still has problems with accountancy. For this reason, the targeted 50 villages in NTT are only partly ready to cover all aspects of the project DS concept and will need further support in 2009."

Feedback: This Lessons Learned statement has evidence about what happens when one approach is taken but only an assumption about what will happen when the other approach is taken. It might have been better to say something like, “Using government agencies to do community mobilization around Desa Siaga leads to significant delays in implementation.”

This quote from G.J. summarises my learning philosophy. For us to learn, we need first to know what we expect to happen when we take action; second, have systems and processes of tracking what we are doing, the context we are working in, and the results; third, be curious to understand if our expectations match the reality; and fourth take appropriate action based on this comparison of expectations versus reality.

In this edition, I 1) defined lessons learned, 2) identified questions to help us decide if a statement is a good quality lesson learned, 3) highlighted evidence sources and types that support the validity of lessons learned, and 4) provided guidance and an example of what constitutes a well-documented lesson learned.

Thank you?? for reading! What are your key takeaways from this edition? Please let me know in the comments section below ?? and share this newsletter with others you think would benefit.

References:

M. J. Spilsbury, C. Perch, S. Norgbey, G. Rauniyar and C.Battaglino (2017) Lessons Learned from Evaluation: A Platform for Sharing Knowledge

M. Q. Patton (2001) Evaluation, Knowledge Management, Best Practices, and High-Quality Lessons Learned American Journal of Evaluation, 22(3), 2001

Patton, M. Q.?and Millett, R. A., (1998). Lessons Learned, The evaluation exchange,?Volume IV, Number 3 & 4.

Davies, R. (2009).?Expectations about identifying and documenting “Lessons Learned.”

Florence Randari?is a Monitoring, Evaluation, and Learning (MEL) professional who seeks to provide evidence-based guidance to international development actors so that they can achieve sustainable development. She is also a Collaboration, Learning, and Adaptation (CLA) practitioner seeking to empower all implementors with the required knowledge and skills to apply CLA principles in their day-to-day work.