10-Step Product Management Cycle

Introduction

I was asked the other day how we, at Amazon, think about Product Management and the software development cycle. What mechanisms does one need to create a repeatable process? Who owns each step and who should be involved? How do you develop ideas? How do you prioritize? How do you keep the process on track? I realized, after writing out the entire flow, that this was the first time I had written it out on a single piece of paper. I thought this could be helpful for others and so wanted to share it with you here. The target audience for this overview is primarily new Product Managers as these are the first principal concepts. Seasoned PMs will know these steps intimately, but I hope there are a few nuggets for them as well. I also want to be clear that I do not consider this “the” product development flow — there are undoubtedly alternatives and specific modifications required depending on the industry or situation. I, therefore, do not consider this to be an exhaustive review but rather a high-level blueprint. This ‘blueprint’ is for a single program or feature, and therefore there are mechanisms that sit on top of this cycle for longer-term strategic investment purposes. Some organizations will also have different ordering or grouping of activities, which is of course completely fine. The important thing is to build strong mechanisms for each of these activities: create clear ownership, develop objectives and output expectations, and regularly inspect and improve your mechanisms. I summarize the 10 steps below, which will be the framework for this article.

Step 1: Product and Feature Ideas

So, what should we build for customers? There are many potential recurring mechanisms that you can use to develop ideas for either an entirely new product for customers, platform-level products to serve multiple internal or external customers, feature extensions for existing products, re-factoring existing products, or simply fixing bugs. These ideas can then be used as part of your annual/quarterly investment and planning cycles or for pitching new ideas that require additional headcount. We built several mechanisms to facilitate this cycle: annual strategy and investment planning, quarterly prioritization planning, bi-weekly status reviews and sprint planning. We also share these opportunities with stakeholders including customers (especially if in the Business to Business world), internal and external partner teams, vendors/suppliers, and leadership. For new product opportunities, the real question(s) we are trying to answer is: What is the most valuable experience that we can provide customers? And, what is the most valuable experience we can provide our suppliers/vendors (or insert other key stakeholders here) Value is defined by the organization and is likely different for different industries or where the organization is within its growth maturity. Generally, I recommend your best proxy for incremental long-term free cash flow. Long-term free cash flow includes both short-term transactional value (inclusive of advertising) and long-term value in the form of subscriptions, longer-term engagement/conversions, or ‘unlocking’ value elsewhere in the organization (i.e., cross-sell / up-sell opportunities or discovery of new products and services). Long-term free cash flow is the engine of growth, is aligned with investors, and will not sacrifice long-term gain for short-term improvements. One concept that is crucial but very challenging to predict is incrementality. We have run countless experiments where we learned that a new product generates value but may offset value from somewhere else or even fundamentally change the customer’s behaviour. This new experience may therefore not be entirely accretive. As the organization’s products and services become more and more complex, it is imperative to build this into your value equation consistently. And equally, if not more, important to include it in your marketing activities.

Here are a few mechanisms for your consideration when developing new ideas for customers and stakeholders:

Direct customer feedback

I define direct customer feedback as explicit signals intended to provide feedback on a product.

1. Usability studies: typically used early in product development, you facilitate a session with approximately 10 customers to get feedback on specific flow and design choices. You can utilize a product prototype of the entire flow, or you could even utilize mocks if the investment is large to create a prototype. Customers will give countless suggestions and opportunities for further investigation. My suggestion here is to take a 5-ways mindset: customers will rarely give you the answer — or even clearly articulate what they want. Keep digging to find their true motivation.

2. Customer focus groups/diaries: this mechanism is helpful to dive deeper into why a customer does what they do (or don’t do). You may want to know why a customer loves a competing feature, why they don’t use certain features, why they stopped using a product, etc. This step is not to get statistical significance and an ordering of the top issues but rather to build a list of the top issues with a hypothesis of the ordering. Diaries here are also very helpful, where you ask a customer to write down every time they do X, why they did it, what they were trying to accomplish, and what they wish was different. This can be helpful to uncover behaviors that are automatic or subconscious.

3. Customer surveys: Once you have a grasp of the top issues from your focus groups/diaries, a customer survey can be used to get closer to statistical significance to order the top opportunities.

4. Customer reviews / App Store reviews: customers that typically write an app store review or customer review are really, really annoyed. It is typically not easy to provide feedback and so they are willing to jump through the hoops to do so. While I do not necessarily recommend this to create “the” top issues, they are great sources of “diamonds in the rough” for new ideas and of course bugs or breakages in the existing experience.

5. Panel/beta apps: it is helpful to get early feedback on products, especially when building more complex features or services that will take months to complete. To get this feedback, you can create a customer panel (especially if you have large, important customers) or a beta app where customers are passionate about your product and willing to try new features. You obviously need to take this data with a grain of salt given the source is biased, but it will give you valuable early indicators.

6. Explicit customer signals or feedback: these can be built into the product itself to get the customers’ reaction to the experience. This can be implemented in several different ways, from a simple “thumbs up / thumbs down” to enabling the customer to fill out a feedback form. For customer feedback, it is most useful when the customer immediately has an issue because they can best articulate it then, and they are motivated to provide feedback. To accomplish this, you can build a mechanism that enables the customer to provide feedback anywhere in the flow (e.g., enabling one of the Operating System’s macro features such as “shake”).

Indirect customer feedback

I define indirect feedback as implicit signals that indicate the customer’s resonance with a product.

In addition to directly asking the customer for feedback, you can also answer the question, “What are customers resonating with outside your organization?” It is obviously challenging to collect the same data as you would with internal experiences (i.e., retention data, financial information, or even simply click-through rates), but there are ways to collect data to directionally understand whether you should consider these experiences. I would also highly encourage you to refrain from direct benchmarking — in my experience, in the vast majority of cases, you expend a disproportionate amount of time collecting data, comparing it to your own data, and trying to bridge or make sense of differences, and which almost always leads to a dead end and wasted effort. It is tempting (especially for senior leadership or investors) to try to understand “why aren’t we performing at X?” But the devil is always in the details, and those details usually are not available. We have had enough challenges comparing our own internal data!

7. Customer behavioural data: this is one of the most obvious ways to understand the customer’s experience of an existing product. We could dedicate an entire article on how to think about this topic. Some examples include metrics like tap-through rate, retention rates over time, conversion rates, drop-off rates, etc. One of the most powerful ways to understand this data is through cohorts over time to see how and why the customer’s behaviour is changing.

8. Customer service: this is an excellent source to identify bugs and breakages in existing products. Customer service associates are also excellent sources of knowledge to understand what most bothers customers and get nuggets like “Why can’t I just do X?”

9. Data aggregators: We’ve found it useful to look for data aggregators that collect data in a consistent fashion. This is also not ‘pure’ and likely not comparable to your internal data, but it is relative and therefore differences and trends can be insightful. Therefore, the absolute data is not useful, but the deltas and changes over time are. Some examples of data aggregators include data.ai (formerly App Annie), Sensor Tower, Appfigures, and Quantum Metric. These organizations track the customer behaviour of iOS and Android applications and can provide information on what apps customers are most downloaded, the retention of different apps, relative size, and how these changed over time. In addition to app tracking sites, Similarweb and Statista have information on a variety of topics that can be used for market sizing.

Ideas from the team or stakeholders

10. The team: we enable team members to create a one-page ‘pitch doc’ and evaluate these ideas in a bi-annual ‘shark tank’ like the process to see whether any ideas will be included in the roadmap — or — whether to be presented to senior leadership for additional funding. The intention is to enable our team members to participate in the idea-generation process in a relatively lightweight way. The idea can come from data they have collected, something they have experienced in another app or a recommendation from a friend. Our one criterion is that the idea has some basis in data — an article indicating the merit of the idea, All team members are encouraged to participate, including engineers right out of school. If the idea is selected, we will have the author partner with a Product Manager to flesh the idea out in a fully developed PRFAQ (more on this in step 2 below). Three other mechanisms we employ (as do many other organizations) is 1) Hackathon: enabling team members to partner up to create a prototype of any idea they find interesting, and 2) Creator days: twice a year, we dedicate one sprint to enable team members to work on anything they want — from cleaning up code to working on a new idea, 3) Walk the store: once a month, a PM will take the lead on walking the team, step by step, through an existing customer experience, just as a customer would. This should be done live (to the extent possible), with the PM clicking along and enabling team members to ask questions. This will, most of the time, highlight multiple opportunities for improvement of the existing experience, but it will also surface opportunities for new features of experiences.

11. Partner team in-take: if other teams around your organization use your services or are affected by your products, you can solicit their own ideas. We typically create a simple template (i.e., idea, why customers will love it and any data indicating why this will be helpful for customers, high-level long-term free cash flow impact).

Step 2: Working Backwards Document

Once you have decided on your top product or feature ideas, you need to detail the minimum lovable customer experience (short-term and long-term), the business plan, the logging, data, and metrics strategy, and what risks you will encounter and how you will mitigate them. This can all be documented in what we refer to as a working backward document, as in, a document that articulates the full end-to-end experience that we will deliver to customers, why customers will love the product, how they use the product, and where they can find it. The level of depth you go to in this document will vary, but it is roughly correlated with the amount of investment. If the investment is high, e.g., one or more two-pizza teams for over six months (we organize around two-pizza teams, which is a team that has a well-defined and specific objective and is staffed with approximately 6–8 engineers, a product manager, and a software development manager. Together, two pizzas would be enough to feed them), then we would deliver a fully developed Press Release accompanied by Frequently Asked Questions (FAQ). Much has been written on Amazon’s PRFAQ, so I will not go into detail here, but at a high level, the Press Release is typically no more than a page, written to describe what the experience will be at launch, why customers will love it, how they use it, and how customers can find the experience. The FAQs are typically three to five pages with the entire document not exceeding six pages, although one can add an appendix with User Interface mocks (i.e., step-by-step screenshots of the customer experience) or other relevant information. I cannot stress enough how important it is to document, upfront, the customer experience and key business decisions before the software development process begins. Creating this document enables one to communicate the strategy to stakeholders, including your own team members. It enables you to articulate the customer flow and debate, if necessary, key decisions upfront. One high-level Amazon executive once told us, “We always start with a PRFAQ before one line of code is written. This allows us to align on the experience and debate key issues before placing it in front of the customer.”

As mentioned above, you can start the process with ‘pitch docs’ or a version thereof to not require team members to write six pages before you have a conversation. If the idea is a feature of an existing product, you can also skip to step 5 to detail out the use case(s) and translate them to technical specifications.

Step 3: Prioritization

I include prioritization as step three, but, prioritization is taking place across the product development cycle. I include it here because once you have your top product ideas, most of which will have to work backward documents, you will have the information you need to make investment decisions for the year or for the quarter (or month or sprint). Feature or sub-feature decisions may change throughout the process as you collect more information.

To facilitate the prioritization process, it is incredibly useful to have a metric or set of metrics that aligns with teams across the organization. There is a wide variety of prioritization metrics, but typically include at least the following three: 1) Customer or business impact: this can be viewed either as a financial metric (e.g., increase in revenue, profit), viewed as an increase in engagement (e.g., visits or repeat visits, dwell time), or an increase in customer rating/satisfaction with an experience, or a reduction in complaints. You will need to trade off the accuracy of measuring the value to the customer vs. the simplicity of having a single metric that goes across the company. I personally push for the latter, because the team will get better and better at estimating the impact of the single metric over time, 2) Time to implementation: higher value, and shorter times to market are obviously preferred. When considering time to market, ask yourself whether a version of the experience could be put in front of the customer to get a faster estimate of the product value. This will depend on the totality of the cost to create the product; the higher the cost, the more you should look for these opportunities before fully investing in building the product, and 3) Risks: Are there internal or external factors that could block your efforts? How likely are these scenarios? Is there a piece of the experience that the team has less experience in, e.g., Machine Learning?

Step 4: Breakdown of customer stories

Using the working backward document, you can capture all the intended functionality that the customer and/or stakeholders would like to accomplish with this product. This is typically written from the perspective of the recipient of the value of the product. For example, we built a product that provides expert articles on products (e.g., from publishers such as Wirecutter) for customers searching for product review information (beyond the customer reviews). We built one version of this into Amazon’s search results, e.g., if the customer searched for “best TVs” we would surface a widget within the search (or more accurately, we would offer Amazon search the opportunity to surface the article). One customer story was, “As a customer, I would like to see expert articles on products I am interested in when I search for a product. I would like to see the product, the star rating, which publisher has a review on this product, and ‘bite-size’ information from the article to determine whether I want to engage.’ This also enabled us to offer different product interfaces to customers. Another use case was “As a customer, I would like to see the top three articles related to the product I searched for.” You should consider all stakeholders associated with this product. In the case above, publishers are clear stakeholders. One use case was therefore “As a publisher, I want my brand clearly displayed to the customer.” An example from a partner team might be, “As the Search team, we require a ranked set of article widgets sent through our API within X milliseconds’. You can also consider other partner teams such as Legal, Finance, PR, Operations, and other groups as stakeholders from which you can create use cases.

You can then prioritize all use cases to form the Minimum Lovable Product (MLP). Some organizations use the term “Minimum Viable Product (MVP)” as the first iteration of the product. This was debated heavily within Amazon and was determined that the minimum bar is a product that customers will love. We, therefore, debate what features should be part of the MLP, which will then be a collection of P0 features. To facilitate the prioritization of features, you can utilize usability tests to provide data as to what customers find critical vs. nice to have.

Step 5: Business Requirements Document

The objective of the business requirements document is to go deeper than the working backward document and be the key input into the technical design. This artifact will go deeper and outline 1) why we are building this product — why will the customer love this product and what is the business case?, 2) Dive deeper into the customer experience. You should have the full customer flow and mocks for each step in the customer’s journey — which will be based on the P0 features identified in the previous step. This can then be used to articulate what the intended functionality and outcomes are for the customer and also what the product is not intended to do. Typically during this process, you will have several debates within the team and with stakeholders on the design and functionality, 3) Understand how exceptions and errors will be addressed, 4) Detail the experiment strategy, design, and any critical elements that need to be addressed (e.g., how will experiment triggering be handled to ensure an equal split between treatment and control?). This document will be the primary artifact that the technical team will use to develop the technical design.

Step 6: Technical Design and Tech Breakdown

It is important to engage the technical team throughout the entire product development cycle. If you are engaging the technical team at Step 6, there is a very high likelihood that the MLP would be different (and better for customers) with their input. They will have insights, questions, and experience to help determine what is easy vs. hard to build for customers, and what is risky, and ask hard questions on why customers will love the product. While Product Management owns completing steps 1–5 (and subsequently jointly owns steps 8–10), the technical team will take ownership of steps 6 and 7.

At this stage, it is important for the Product Manager to get a working backward timeline for the final technical design. To get there, the team will write drafts of the design, review with principal and senior engineers, and engage with partner teams as necessary. They will also include a plan for testing and building in time to fix any blocking bugs. The team should also have a discussion on Operational/Engineering Excellence and the experiment design and setup. For Operational/Engineering excellence, the technical team will need time for documentation, latency testing, security, privacy, resiliency efforts (i.e., availability levers, load and chaos testing), and logging the operational metrics to understand the customer experience and where it is failing and to what degree. The question for the team is: how much of this does the team complete up-front before experimentation? In our view, there are several table stakes components that cannot be pushed back: security, privacy, availability and operational metrics. Resiliency efforts are particularly important for high-traffic scenarios, but if the product is not going to launch (because it does not pass the experiment), then those investments may be throw-away work. As a result, I suggest differentiating between what is critical for the customer experiment vs. what is critical for high traffic. The primary roles of the Product Manager are to 1) Prepare the experimentation plan. What needs to be tested — and not just the customer-facing metrics such as clicks, revenue, advertising, long-term value, etc. These are of course important, but you should also measure all operational metrics, including the latency between clicks and the painting of experiences and the readiness of additional engagement points, the number of errors that the customer experiences and to what degree. The PM will also be responsible for User Acceptance Testing (UAT) where you walk through click by click to make sure the experience is happening as expected, in every locale and every language. We suggest writing a document outlining the testing approach and recruiting team members from each locale to help with the testing.

Step 7: Technical Implementation and Testing

As mentioned previously, this step is owned by the technical team. The PM should be working closely with the Software Development Manager and the relevant engineers to make sure they are aligned on all data flows, the testing approach, and the experimentation approach. For purposes of this Product Development Cycle framework, this step includes all the technical testing, e.g., ensuring the services are responding with the correct data, the experiences are rendering as expected, and the services are operating at the expected availability and latency.

Step 8: Customer Flow and User Acceptance Testing (UAT)

Parallel to Step 7, the PM will take the lead on the customer flow and UAT. We typically conduct a “bug bash” where team members walk through the entire experience, clicking on everything they can to 1) Make sure the experience happens as expected — not only that the flow is correct, but that the latency and any fallback experiences are correct as well, 2) Try to break the experience and go back and forth throughout the flow as a customer would. Once you identify all issues, you can prioritize based on the frequency and magnitude of each issue. You can also determine what is critical to fix prior to experimentation vs. what can be fixed after the experience but before the production launch.

Step 9: Final Fixes and experimentation setup

By now, the PM should have the full list of prioritized bug fixes and a detailed experimentation plan. This step is intended to dedicate time to complete any final modifications before you begin experimentation. It is important to discuss whether and how you should experiment. Generally speaking, it is helpful to run experiments whenever you can because they 1) Produce output that represents the “will of the customer.” I am a big proponent of letting the customer decide what the product will be and whether it adds value to their experience, 2) Produce incremental attribution, meaning that whatever you display to the customer will be above and beyond what they are experiencing today — measured in a way that assigns the true attribution. Attribution is a complex topic that can be its own dedicated paper. Experiments, however, automatically and accurately assign attribution. They are the gold standard for determining incremental customer behavior. However, I do want to be clear that experimentation comes at a cost and in some cases, a very high cost. In addition, it is non-trivial to receive truly statistically significant — and correct — results. There are many ways that your experiment can produce bias and therefore incorrect results, e.g. experiment triggering issues, customer biases, and experience biases (e.g. if there are latency issues with Treatment or Control). It is important to consult with statisticians to ensure that your results are accurate. In some cases, it may not make sense to experiment. Maybe you’ve already launched a product in 10 of 15 countries. Or maybe you are improving the performance of your product (e.g. latency), in which case you may not need to run an experiment.

Step 10: Experimentation, Customer Feedback, and Analysis

In addition to running an experiment, you will want to consider collecting additional customer feedback. In theory, the results of the experiment should tell you whether the customer prefers this new experience over the control — assuming you have all the correct metrics in place to accurately measure the customer’s preferences. However, it is challenging and expensive to have every single metric logged and analyzed in an experiment. We ran an experiment in the Amazon app to hard-gate the customer experience based on whether they are signed into the app. Signing into the app significantly improves the experience for the customer — we can provide a personalized experience based on their past engagement behaviour, we can show the customer what they’ve purchased and the order tracking history, etc. All the metrics from the experiment were positive in the extreme — every single metric we look at told us to launch this new experience. However, we reached out to customers that were part of the experiment to understand their reactions. Their answer could not have been clearer — they despised the new experience. But they had to do it to shop, so they were willing to jump through the hoops to get what they needed. Despite the experiment results, we decided not to launch the new experience.

I also recommend documenting the results of the experiment and the customer feedback. I always tell my team that the only failure we can achieve is to not learn. We may not launch a new experience based on the experiment results and customer feedback, but I consider the experiment a success if we learn more about what customers want and what we can do differently moving forward. These learnings can then be input back into Step 1 for future product features and new programs.

Below are some additional questions for you and your team to consider as you build your own product development cycle and a few thoughts/suggestions for each:

Who owns each step in the process??To be comprehensive, you could create a RASCI framework for each step in the process. This framework outlines who is Responsible for each step: there is only one individual responsible for owning the step, and they are empowered to assign tasks and deadlines of others to get it done. The individual Accountable for a task is the one who is accountable for the task being completed. In my teams, the R and A are typically the same people. This is somewhat controversial, as the RASCI theory suggests that the R and A should be separate individuals. Use your judgment based on the situation at hand. The Supportive individual does the work of a task. Each step above has multiple tasks and therefore multiple people can be an S. Individuals Consulted are individuals who participate in a task before it is completed. These can be experts (e.g. Principal Engineers, Senior Designers) that can provide feedback before marking a task complete. Individuals that are Informed about a task or decision are stakeholders that need to understand the outputs and make changes or updates to their own processes. Generally speaking, I typically assign a R/A to a given step, and it is up to that individual to determine who should be involved to complete the step.

What recurring mechanisms do you use to keep the process on track??After step 6, we will create a working backward timeline of each key step in the implementation. This timeline includes key milestones to completing the product, e.g. major technical epics (i.e., an aggregation of features/services that meet key requirements), key stakeholder reviews, testing, and experimentation. We then send out a bi-weekly or monthly newsletter to all key stakeholders to share an update on the status of each milestone. Since we have multiple products underway at the same time, we have an additional mechanism that tracks these steps as a portfolio and enables the team to surface risks for any product in any step. We have a bi-weekly roadmap review where we review the status of each product and if there are risks, they are surfaced here. If we deem it necessary, we then set up a follow-on meeting to problem solve on how to eliminate the risk and/or bring the project back on timeline. The technical team will have a parallel process (i.e., bi-weekly sprint planning, retrospectives, daily stand-ups) that facilitates the completion of Step 7.

What potential pitfalls could you encounter and how do you mitigate these risks??There are of course many risks throughout this process. I want to share three of the most common that I have encountered and my thoughts on each.

1) Not treating data as a product. There are three types of data critical for every product a) Customer data: this input data informs how many customers are engaging with a product, how they are engaging, and to what depth. By customers, I mean any key stakeholder. This could include suppliers, publishers, advertisers, etc, b) Output data: this will typically be the financial outputs, e.g. conversions, revenue, long-term events, etc. and c) technical operational data: this data informs how well the services are performing, where the issues are, and how it affects the customer (frequency and significance of the problem). PMs should treat this data as they would the experience itself — by working backwards. Metrics are simply just an aggregation of data. What metrics will inform whether the experience is successful? Whether the experience is rendering as expected? You then work backwards from defining your success criteria by the data you need, how often you need it, and what slices of data you need (by country, by customer segment, by device type, by the operating system, etc.) to understand how different customers are reacting to the product. You then bake in time to log the correct data, build the pipelines to aggregate the data and build the dashboards to display the metrics. All too often the product will go live, an issue arises, and everyone is looking at each other for the data to better understand what is happening to the customer. To avoid this, treat data just as you would the customer experience, and have dedicated workstreams to log and surface the data properly.

2) Underestimating coordination costs. Generally speaking, we are all terrible at estimating how long it will take to complete a task. Add this up across tasks, and the product will be significantly delayed. Daniel Kahneman and Amos Tversky outlined the planning fallacy in their book “Thinking Fast and Slow,” where they detailed their argument and evidence that humans tend to underestimate the amount of time needed to complete a future task, due in part to the reliance on overly optimistic performance scenarios. In my experience, the risk goes up significantly if your product relies on or is an input into another teams product, i.e., you are reliant on input from another team, or your product is an input to another team, it is entirely possible that issues will arise, and resolving these issues could take significant time. We have encountered this countless times with each situation being slightly different. The key across all of these is to first identify where issues could occur, and then have as low a latency as possible between identifying an issue and raising it to leadership (if needed) to get resolution. Escalation strategy and protocol are worthy of their own article, but at a high level, the only thing you need is: do both parties have full information? If not, get what you need as soon as possible, and share it as soon as possible. If you have full information, and you still disagree, escalate. This will minimize non-value-added dead time and align expectations.

3) Cultures are averse to bad news and/or lacking mechanisms to surface risks. I’ve heard people refer to the status of a goal as a ‘watermelon,’ i.e., green on the outside and red in the middle. This is a strong signal that the culture is averse to surfacing risks and issues. While it is certainly a natural human tendency to not want to deliver bad news, not addressing these issues head-on can cause much more pain down the road. It is challenging to build a culture that rewards surfacing bad news and it is so easy to build the opposite. From a team member’s point of view, any hint that surfacing bad news will be received negatively will result in finding ways to hide issues — or — find data that tells a rosier picture than what is actually happening. Solving this problem is similar to solving other cultural challenges: leadership needs to provide the example and reinforce this time and time again. Culture is built brick by brick, with each brick being a seminal moment where a leader takes the time to reinforce and articulate why decisions are made the way they are. In this case, leaders need to reward team members for surfacing challenging situations and remind them why it is important to do so. Leaders need to ‘be in the boat’ with their team members, helping them solve problems. It may come time to have a conversation on why the issue surfaced late or in an incomplete way. But in a similar fashion, it is another problem to be solved. Not a blame-and-shame game.

#productmanagement #softwaredevelopment