Measure f-ing everything, and assume f-ing nothing!! - Or how mentoring ruined lives :-(
Satyajeet Salgar
Director of Product and UX, Google AI | previously YouTube and Google Search | Advisor
I've been really enjoying the Freakonomics podcast of late. This episode and the lesson we should take a away from it, was a stark reminder of one of the most important things we should be doing - but often don't - in building products or making any decisions: measuring the impact of absolutely everything we do, including the things that seem obviously good.
I recommend listening to the podcast if you have the time, but here's the summary. Stephen Dubner describes the Cambridge Sommerville Youth Study. The impact of social intervention programs in general is hard to measure and so they seldom are. This was the first attempt at measuring the impact over a long period of time.
It's a great story and there are a few good take-aways, but here's the main one: troubled or at-risk youth that received mentoring (good mentoring!) had worse life outcomes across every dimension than the kids that were left alone. Despite the recipients saying that the mentoring was incredibly valuable and helped them, they were actually worse off across a bunch of measured dimension.
Everyone's assumption: mentoring troubled youth will help them in life was wrong - at least the kind of mentoring that they were doing then. They were able to actually measure and determine this only because computers were just starting to become available and the person leading the research was determined enough to figure out how to use them.
This isn't unusual. A lot of things can seem obviously good especially to people working on them, but most don't really do the work to understand (or measure) the long-term impact of the changes they're making. Sometimes it's because they don't feel the need to - a lesson learned in this Steven Seagal classic. Other times it's because it's really, really hard to measure these things.
This is a great reminder that the impact can be very, very different from what you expect or what seems "obvious". It can even ruin people's lives.
So, find a way to measure everything. Always.
Founder | Data & AI | Product Manager | Gaming | Google & McKinsey Alum | Wharton MBA | IIT CS
7 年Very curious to learn what the correlated and proposed causal relationships could be! Also the kind of problem where deep learning might throw out some thought provoking ideas - I'm theorizing here, but perhaps parameters such as the age of participants, time of mentorship, perhaps even the Day when the sessions were happening could have been correlated to the outcomes, with some interesting implications.
I have not read the original research pointing to the negative effect of mentoring but would be surprised if the causal effect is as clear as you make it sound. While I support measurement, I also think we should be aware of its limitations and not draw general conclusions - in this case, that mentorship made the boys worse off and thus we should not mentor troubled or at-risk youth