Searching for your keys under the street light

There’s an old joke that goes something like this:

I got out of work late one night, and I saw my friend standing under a street lamp, staring at the sidewalk, slowly pacing around.

I ask my friend “Hey buddy, what’s going on?” and he tells me “I dropped my keys, I’ve been looking for them for over an hour!”

I wanted to help him out, so I asked, “Well, where do you think you were standing when you dropped them?”, to which my buddy points halfway down the block and says, “Oh, somewhere over there.”

I was gobsmacked, so I asked him, “If you dropped your keys all the way over there, why are you looking over here?”, to which my friend replies, “Well, the light is just so much better under this street lamp!”

It’s a silly story, but it’s amazing how many times we let ourselves fall into the trap of looking for our keys where the light is good, rather than where we might actually find them.

For example: one of my clients was recently telling me all about the great work their team has done on achieving four 9’s of uptime for their product. For an early stage startup with fewer than a dozen engineers, I was pretty impressed that they had that kind of reliability. Then I started asking some hard questions, like “What would happen if reliability fell to ‘just’ 99%?” and “How is high reliability helping to grow the business?”.

After a few minutes of discussion, I learned that reliability wasn’t a particularly important factor in the growth of the business. Potential customers weren’t asking about reliability, existing customers had no feedback about reliability, and the company had never had a revenue-impacting outcome from an outage.

At the same time, the company had a long backlog of customers with signed contracts who weren’t yet producing revenue; the product was an enterprise offering with an intricate onboarding process, and the company only had the bandwidth to onboard 1-2 customers per month. The onboarding backlog was growing without bound. The team had some ideas about how they might automate parts of the onboarding process, or build some tools to accelerate, but no one had prioritized those efforts.

This engineering team was looking for their keys under the streetlight - they had picked a priority (reliability) which was easy to measure and could be addressed through purely technical solutions. Their keys, halfway down the block, were the opportunity to reduce the customer onboarding lag, grow revenue, and grow the business.

Investing in improvements to the onboarding process, though, was a gnarly problem. What does it mean to measure onboarding time? How could you be sure that any changes you make will actually be an improvement? What happens to your metric if you build a ton of great tools to improve onboarding, but then a customer takes two weeks to reply to an email?

There are countless engineering metrics that we’re all familiar with, because (a) they can be applied to almost any system, and (b) in mature products, they’re excellent leading indicators. Uptime, regression rate, response latency, the list goes on and on. None of these are vanity metrics - they measure meaningful indicators of system health, and improvements reflect real work and real talent.

The great thing about most of these engineering metrics is that there’s a whole toolbox of well-understood approaches to help make improvements. A lot of times, these approaches are also really fun to work on - building in failover mechanisms, working on horizontal scaling, or optimizing a storage layer are all cool projects to tackle.

But the role of the engineering leader is not to prioritize the problems that the team knows how to measure, or how to solve; the role of the engineering leader is to understand the business, measure the team’s progress towards the goals that will do the most to grow the business, and then focus the team on achieving those goals.

It usually requires stepping out from under the street lamp, but, I promise, you’ll have a much better chance at finding your keys!