Case Study: How We Did Project Prioritization at Meta

Working at Meta was a very interesting experience. One of the most unusual aspects of their culture was how much of a bottom-up approach they take to decision making. A lot of the decision power is pushed all the way to the bottom of the hierarchy, only expecting people to report on what they had decided. A consequence of this in practice is assigning the responsibility of allocating people to projects on the shoulders of software engineering managers (EMs) and product managers (PMs) who are directly involved.

Most EMs support a team of 8–16 people, representing a compensation budget between one and five million dollars annually. Thus, people allocation to various projects is likely one of the biggest decisions the EMs gets to make. But it is also one that managers receive surprisingly little support for.

This is a large part of why I’ve started looking into how to make decisions using a more structured approach. Today, I’d like to share the process our team of EMs settled on for roadmapping. Please note that while I’ve modified certain details to protect Meta’s intellectual property, the method described remains the same.

For context, our team did roadmapping twice a year. Each time, we had to come up with a set of projects we would invest in over the upcoming “half”(-year). This was done to coordinate:

• three engineering teams,
• one science team, and
• support from data scientists, product designers, content designers, marketing specialists, etc.

Sometimes, we also had to coordinate those projects with upstream development teams to ensure we’d have the required data and functionality needed to implement our prioritized features. For us, roadmapping had the dual goal of project prioritization and coordination between the various people involved.

First Loop: What Are Our Options?

The first step is generating alternatives: what projects are we considering? We would gather ideas from engineers, sales staff, PMs, etc. Everyone would be involved in this step to avoid being blindsided by an issue we hadn’t considered. At least once a year, we’d also invite people to contribute more pie in the sky ideas that sparked discussions about the future of the product and kept our innovation pipeline full.

For each project, we’d collect:

• a project title,
• a project description that is precise enough for others to understand the gist of the work to be done,
• reasons to invest in that project, and
• rough estimates of how many full-time equivalent (FTE) employees are needed to deliver that project, broken down per role (front-end, back-end, scientists, designers, etc.).

?Full-Time Equivalent (FTE)

Unit representing the work a typical mid-level employee would require to deliver a project assuming they are working alone and solely on that project. One unit corresponds to one employee dedicated to that project for the full duration of the period under consideration (here, a half).

FTE measurements assume that employees take time off, attend required training and meetings, and any other usage of their time typical of an employee in an organization. Specifically, FTE does not presume an employee dedicates 100% of their time to executing on a project.

Consequently, a team of 10 people would have 10?FTE to allocate to projects.

Once done, PMs, EMs and senior engineers go through the list and do a first rough ranking of the projects. They identify:

• projects that are critical (i.e.,, the product would be at risk if we didn’t invest in those projects),
• projects that are high-impact,
• exploratory projects (i.e., projects that are not as mature and would need someone to assess their viability and prepare a high-level plan that could be executed during a subsequent half).

After this exercise, we usually had enough work to keep the teams occupied for roughly two halves.

Second Loop: Project Decomposition

At this stage, we would normally have a shorter list of roughly 20–50 projects we would seriously consider investing in. Each one would be assigned to a lead engineer who is tasked with breaking the project down into milestones, estimate their duration (in engineer-weeks), and estimate the project’s likelihood of success.

Going through the motions of breaking down the project into milestones forced the team to come up with a plan of attack to deliver the project that is more concrete than the high-level description used in the previous loop. This helps in two ways:

• It forces people to think through how the project can be executed, often identifying risks that weren’t obvious from the high-level description.
• It gives a roadmap to follow for the engineer who will execute on the project, which is especially useful guidance for either the more junior people tasked with its execution, or anyone without as much context about the subject matter.

Once broken down into concrete chunks, it becomes easier to estimate the duration of each milestone. It is important at this step to clarify what the estimates include: should time be added for holidays, time off, meetings, training, conferences, etc.? Make sure everyone agrees on the definition of a week of work!

As seen in the description of the First Loop, we trained people to communicate uncertainty by estimating the duration’s 5th and 95th percentile. This means that 9 times out of 10, we’d expect the milestone duration to fall between those two bounds. Then, we’d combine the estimates to find the overall uncertainty distribution for the duration of the whole project. Next week’s article will go into the details of how this is done correctly.

??TIP: Convert durations from its number of weeks to FTEs. This facilitates making decisions according to the team’s capacity.

Finally, the leads were asked to provide an estimate of the probability of success of the solution represented by their specific project decomposition. This may not always be relevant, but in our case, the science team dealt a lot with state-of-the-art problems, for which we had to find innovative solutions. Not all of them were a success. This estimate would be useful for making the final decision to invest.

Review and Commit

The final step is to reconvene the EMs and senior engineers. Using the new estimates, we decided on the roadmap for the next half. In an idealized world, we would simply take the projects with the best return on investment, but a few considerations made that difficult:

• Some projects require expertise that exists only in limited supply (for us, it was the scientists).
• Some projects would require so much time that they’d represent too large a portion of the overall project portfolio. In these cases, we’d often break the project down to be executed over multiple halves.
• Some projects had very uncertain outcomes. Depending on the cost, we’d decide whether we could afford to bet on the project that half. Some of those would be transformed into exploratory projects with the purpose of mitigating the identified risks.
• There’s also the usual internal political shenanigans people in big corporations have to contend with.

In this step, EMs will often experience pushbacks on the estimates, especially when the initial estimate differs significantly from the engineer’s estimate once the project has been broken down. I’ve always regretted making changes to the larger estimate made by the engineers. If nothing else, in my experience, engineers underestimate the true cost of delivering a project unless they have been very well calibrated to give unbiased estimates. We’ll explore calibration tools in a future article.

A Retrospective on This Approach

This was our process when I left Meta in the summer. Here are some learnings and thoughts on how I’d improve the process.

What Worked Well

This is a collaborative approach to roadmapping that fits especially well with Meta’s bottom-up approach to decision making. It is a good tool for EMs to encourage engineers to reflect on the bigger picture and helps them improve their ability to direct new projects. The approach can also be adapted to the engineers’ level by giving them projects that have an appropriate degree of complexity to stretch their abilities without overwhelming them. Overall, I’ve observed increased ownership when engineers were involved very early on in the project’s definition and its cost assessment.

Furthermore, what is true of managers is also true of individual contributors: almost no one learns how to prioritize projects, often relying on what they think feels right to come up with estimates. This usually results in a significant underestimation of the effort required to deliver on a project. Using a more structured approach, we can more easily train new team members in how to plan projects and prioritize them. I find otherwise the planning quality to be very uneven, depending on people’s background. Given the importance of people allocation, it’s the manager’s duty to teach their team how to conduct an effective roadmapping exercise.

What Could Be Improved

You may have noticed that I haven’t mentioned impact estimations. This is because our teams had a much harder time estimating the impact of a project than estimating its cost. When asked to come up with impact estimates, they didn’t know where to start. There are two ways to look at this.

On the one hand, not being able to estimate the impact of a project means engineers are too disconnected from the outcome of their work. This obviously affects the roadmapping process, but I’d argue that it also reflects their lack of understanding of the effect their work has on users. I believe this can lead to employee disengagement, as they can’t clearly see why their work matters. Closing that gap may have benefits beyond making the roadmapping process easier.

On the other hand, it could be argued that impact estimation should belong to PMs, who are closest to user feedback. The problem is that this doesn’t scale to the 20–50 projects under consideration during each half.

The approach we settled on was to rely on PMs to rank each project according to whether they felt its impact was worth the lead engineer’s cost estimate. Even though there was some amount of gut feeling involved, I believe it still gave better results than the old method of only relying on the high-level estimates the teams used to produce during roadmapping.

Something else to keep in mind is that producing and using probabilistic estimates does not come naturally to most people. It requires training (which includes learning about the basics of probability distributions and bias calibration) before one is able to produce reliable uncertainty estimates that are as unbiased as possible. This training has to be refreshed every planning season since there are new employees, or because people simply forget. Additionally, doing estimates of impact would require further training to teach people how to assess the impact of their work, connecting the features being developed with impact metrics used by leadership to measure the health of the product.

Overall, I firmly believe a more structured approach to roadmapping is worth the investment. You have to recognize that people allocation is one of the most important decisions a line manager makes and needs to be done with more care for the process than most organizations employ. The process presented here is a good start, but will likely require adjustments to fit your company’s culture. In any case, having some structure in place to guide new managers and senior individual contributors with less familiarity with project prioritization will be an asset for your teams.

As mentioned, next week I will present a detailed guide to gather and aggregate estimates using a simple spreadsheet. Be sure to subscribe to be notified!

If this was useful to you, also give it a thumbs up. It helps me gather information about what people want out of these articles.