Bringing the power of evidence to the City of Toronto youth programs

Strong youth programs create a stronger community. But how could we possibly measure this impact? Blueprint has been given a unique opportunity to find a solution with the City of Toronto . Together, we built the Youth Outcomes Framework— a common vision for tracking the outcomes of youth taking part in their programs. The City’s goals aimed to embed a culture of learning and continuous improvement, better understand program effectiveness and, ultimately, ensure youth across Toronto receive effective support through the City’s programs.??

Three years ago, the City of Toronto asked themselves “What change are our programs individually and collectively making in the lives of youth?” In response, Blueprint worked with City staff from seven different divisions to build the Youth Outcomes Framework, a common set of goals for the City’s 40+ youth programs, with corresponding indicators that would track the progress of these goals.?

We’ve been excited to see the Youth Outcomes Framework project receive both international and local recognition. Last fall, we were selected to present our work at the American Evaluation Association Conference in New Orleans alongside our City partners. And more recently, the project earned the City Manager’s Office Award of Excellence in the Innovation category, the highest form of recognition for outstanding achievement by the Toronto Public Service.??

This achievement speaks to the strong collaborative approach that underpinned the Youth Outcomes Framework project—involving Blueprinters Jessica Schmidt (Senior Associate), Molly Doan (Director of Inclusive Practice), Ulises Pallares (Associate) and Sandra Nkusi (Manager of Evidence Generation).

This project was led by the Social Policy Unit at the City of Toronto’s Social Development, the Finance and Administration (SDFA) division and involving staff from across seven different divisions, agencies and corporations – SDFA, Parks Forestry and Recreation (PFR), Toronto Public Libraries (TPL), Toronto Community Housing (TCH), Toronto Employment and Social Services (TESS), Economic Development and Culture (EDC) and Toronto Public Health (TPH). ?

To our knowledge, this was one of the first initiatives of its kind at the municipal level. Collaboration across so many different divisions and programs is not always easy, but together we were able to develop an approach that represented the diverse range of programs at the City of Toronto—including library programs, job placement programs, mental wellbeing and leadership programs and more. As we worked with program staff to create surveys and analyze findings, we also fostered enthusiasm for evidence generation along the way.?

How it works

By using the Youth Outcomes Framework with the City’s youth programs, we are now able to collect and use data to explore questions such as: Who are City youth programs reaching? How can we ensure that City programs are inclusive and meet the needs of the youth they serve? What are the experiences of young people in City programs? What changes do City programs contribute to across communities???

Not only has this process allowed staff to have the information they need to answer such important questions, but it has also helped build a culture of evidence generation in the City of Toronto. By implementing the Youth Outcomes Framework, staff learned how to embed data collection and use in their day-to-day program delivery. In this way, staff are equipped to gather information about how their programs are experienced by young people, and the change they are making. This knowledge has also allowed staff to identify opportunities to strengthen their programs over time and ensure they are responsive to the needs and interests of youth, particularly as they change and evolve.?

Building and testing the Youth Outcomes Framework??

We began our work by collaborating with our City of Toronto colleagues to narrow down the goals all City youth programs were working towards as a whole. We drew on examples from other jurisdictions and literature on youth development to make sure the framework was based on strong practices. We also drafted indicators that would tell us whether or not the outcome had been achieved.?

Staff were able to choose outcomes and indicators from five broad categories:??

1. Supportive peers and adults?
2. Physical health and mental well-being?
3. Life and social skills??
4. Employment and entrepreneurship?
5. Civic and community engagement?

We also identified some information that all programs would collect consistently, including demographic questions to help understand the range of youth City of Toronto programs are reaching, as well as whether young people experience barriers to participating in the program and whether they have suggestions to improve the program.??

Next, we built a survey bank of questions that corresponded to each indicator, and engaged young people to ensure the phrasing was youth-friendly. We also created user guides and templates to help City of Toronto staff take the framework and apply it to their programs in the form of an end-of-program survey. We tested these materials in a sample of City of Toronto youth programs, working with staff to problem-solve any challenges that arose. For example, we considered how to compensate survey respondents while ensuring their confidentiality, and how to make sure youth can answer the survey if they don’t have a laptop or smartphone.?

Once all program staff analyzed and shared their findings, we collaborated with our City colleagues to explore how we might combine findings from multiple programs to tell the bigger story of the City of Toronto’s youth programs.??

What’s next??

Blueprint’s work with the City of Toronto on the Youth Outcomes Framework is ongoing. After the framework was developed and tested with a subset of youth programs, Blueprint is supporting the expansion of the framework to more City of Toronto youth programs. In particular, Blueprint has been coaching program staff from the City of Toronto’ Youth Development Unit on how to implement the Youth Outcomes Framework in their work. This involves supporting them to develop a logic model, create a participant survey, work with delivery partners to field their survey, analyze their data and ultimately learn what is working well and how each program might better align with the needs and interests of the young people they serve.??

Through this work, we’ve seen a growing enthusiasm for evidence generation, and we’re excited to continue to collaborate with our City of Toronto partners to embed data collection and learning among the City’s youth programs and beyond.?

Learn more about our work with equity-deserving youth here .?

#innovation #community #youth #torontoyouth #outcomes #evidence #evaluation #datacapacity