Analysis of the Avito Product bootcamp 2024 case

Hi!

Let's delve into an extended version of hypothesis research and prioritization within the scope of the Avito Product Bootcamp 2024, in which I participated. The data for the case were synthesized but closely approximate real-world scenarios. According to the case conditions, my calculations were based on the beginning of 2023. If you wish to explore the data of my calculations in more detail, you can view them at the provided link.

Table of contents

Case Tasks

• General information about the case assignment

Analysis

• General Overview
• ?Electronics? Section
• External Environment
• Analysis of the Provided Data

Checking the Feasibility of Achieving the Set Goal

• Achievability of the growth goal

Formulation and Prioritization of Hypotheses

• Hypothesis Formulation
• Framework Description
• Audience
• Interviews
• Hypothesis Solution Formulation
• Prioritization

Design of the Experiment for the Chosen Hypothesis

• Mechanics of the Chosen Hypothesis
• Hypothesis Operation Scheme
• High-Level Economic Model
• Unit Economics
• Experiment Design Description
• Conclusion of Research and Findings

Case Tasks

General information about the case assignment

Increase Avito delivery's revenue share in the Electronics category by five times over five years

• Analysis of the current situation with Avito delivery
• Market situation analysis
• Product improvement hypotheses. What will contribute to its growth
• Detailed description of the first hypothesis to be tested
• Prioritization of products according to your chosen criteria
• Detailed scheme, sketch, or draft of the product's function

Analysis

General Overview

?Electronics? Section

Sources: *from the terms of the case, analysis of data from the case, new-retail, Avito terms of delivery

External Environment

Source: Vedomosti.ru

Sources: avito.ru, medium.com, exmailpoints.ru

Analysis of the Provided Data

Let's take an overall look at the ratio of shares in the C2C and B2C sector GMV through a diagram, based on a combination of the provided GMV (Gross Merchandise Value) data and the penetration rates of delivery in both sectors.

• We can observe that the technical capability for delivery is at a high level, and from the ?backend? perspective of this product, delivery is feasible in almost all regions of the listings.
• The average ratio of enabled delivery to the total number of listings in GMV is 63% in C2C and 70% in B2C.
• We see that the GMV of goods purchased through delivery is significantly less than what sellers allow. The average conversion to purchase through Avito delivery is 23% for C2C and 14% for B2C.
• We observe a traditional dominance of the ?Phones? product group (telecom market) in the ?Electronics? section.
• We see a high ratio of GMV of goods with delivery to the total volume of listings in certain product categories, such as ?Office equipment and supplies?, ?Photographic equipment?, ?Games, consoles, and programs?. We can assume that buyers in these categories face fewer barriers to its use.

Let's take an overall look at the average monthly purchase funnel metrics with Avito delivery in C2C and B2C for 2022.

Key points: There is a general trend of low conversion to purchase through Avito delivery. It can be assumed that either the customer does not see the value of delivery or secure transaction, or does not feel confident when purchasing complex items like electronics through it. A study should be conducted to identify the problem, and in case it is found, look for additional value for the product.

Let's continue analyzing the data. Let's highlight several categories in which Avito delivery has a high conversion rate to purchase.

Key points: In some categories, we observe a sufficiently high conversion to Avito delivery. It's necessary to conduct a study examining the experience of both buyers and sellers. We should explore the possibility of increasing the commission for selected categories to enhance delivery revenue due to the high conversion rates in these categories.

Checking the Feasibility of Achieving the Set Goal

Achievability of the growth goal

Goal: To achieve a fivefold increase in profit from Avito delivery in the ?Electronics? category over five years, reaching 65%. I interpreted this as a combination of increasing revenue from delivery commissions and gradually changing the proportion of this income towards a larger share relative to other revenue channels in the category. Let's identify the main metrics and actions that influence Avito delivery conversion:

To start, we want to determine the feasibility of achieving the set goal. Let's find the total revenue from commissions in the C2C and B2B sectors in the ?Electronics? section. For this, we will analyze the average monthly GMV (Gross Merchandise Value) data in shares for each product group. The provided data on delivery penetration and the data in the tables with GMV in C2C and B2C will assist us. An example calculation in the ?Phones? product group using the formula below:

We calculate each product group in a similar manner and sum all the delivery order shares into a total value using data from both C2C and B2C sectors. Then, by summing up the revenue from commissions in C2C and B2C, we obtain the total monthly revenue in the ?Electronics? category for 2022.

The total average monthly revenue from the entire ?Electronics? section in 2022 was 97.843 million. Moving forward, this value will help us confirm or refute the achievability of the set goal.

From the case conditions, we know that the revenue share from delivery in the ?Electronics? category in 2022 was 13% of the total revenue in the category. In the analysis block, we found the monthly revenue from commissions in 2022, which is 97.843 million. Let's find the target revenue in 5 years, which is outlined in the case using the formula below:

Goal: 5870.55 million on average per year, revenue from delivery commissions in the ?Electronics? category in 5 years, or 489.21 million per month. We can consider various external factors, such as overall inflation, but at this stage, we want to quickly estimate the number we are aiming for.

We know that Avito has set a goal for itself - 100 million MAU within 3 years. Source: Avito job posting

At the beginning of 2023, Avito had 60.1 million MAU. We can estimate the overall growth that will reflect on our product in three years in case of reaching 100 million MAU, through a proportion:

We obtained a coefficient of 1.66 (growth of 66.38%) so that, by multiplying by it, we can estimate revenue in 3 years, provided the overall MAU metric in Avito reaches 100 million.

Presumably, we can conduct experiments with increased commission rates and observe the churn rate in those categories where we see high conversion to Avito delivery and competitive conditions in the external market. I built a forecast model based on delivery penetration data, average GMV in categories, and commission data. During calculations and testing with changes in the commission percentage in high GMV categories up to 4% in C2C and 5% in B2B, combined with improved delivery conversion to 40-50% in these categories, it was possible to achieve an estimated revenue of about 500 million per month from commissions, which meets the stated goal. The categories affected in the model include ?Phones?, ?Laptops?, ?Desktop PCs?, ?Audio and video?.

Key points: During the calculation of metric growth in Excel with changes in commission percentage in high GMV categories up to 4% in C2C and 5% in B2B, combined with an improvement in delivery conversion in these categories, it was possible to model the estimated revenue of around 500 million per month, which meets the stated goal.

We hypothesize that identifying a pivotal value-add (Aha-moment) in using Avito delivery for buyers will enable us to achieve the desired growth metrics and adhere to our Growth objective. After analyzing the data, we identified focus points around which we deliberated and formulated hypotheses. Let's articulate them and commence their validation.

Formulation and Prioritization of Hypotheses

Hypothesis Formulation

Let's assume that improving the funnel through UX/UI enhancements will aid in boosting conversions for Avito delivery, but it won't suffice to achieve the desired increase to 40-50% in orders through Avito delivery. We speculate that a comprehensive approach involving commission increases and identifying added value for the product is needed to reach our ?5 in 5? Goal. Let's define research questions and gathered hypotheses for validation, conduct the research, and then contemplate a product that could create a pivotal Aha-moment in the use of delivery.

Description of the framework

Let's describe the method of conducting an analysis according to this framework in general. We expand the list of hypotheses for testing. After identifying the hypotheses, we define the scope of the problem through qualitative and quantitative analysis, gradually narrowing down to the key issue that needs to be addressed (JTBD*). Then, we begin to describe the range of solutions, expanding their list. We proceed to investigate the hypotheses for solving the problem/task through qualitative and quantitative analysis, various tests, gradually narrowing down the list of solutions that will allow addressing the focal issue under current conditions with the current resources of the internal and external environment.

Source: *gopractice.io

Audience

To conduct the research, let's first determine with whom we can and should interact to test our hypotheses. The choice of respondents should be determined by the selected hypotheses. Our sample will include:

Experienced users: people who have used Avito delivery in the electronics category in the last six months and have stopped (we look at the Churn rate). Also needed are users who use Avito delivery but not in the ?Electronics? category.

New users: those who have not yet used Avito in general, but use solution interpreters (competitors).

Interviews

To begin, we will conduct a qualitative analysis with in-depth interviews. Following Jakob Nielsen's theory, 5 in-depth interviews with users can uncover 85% of existing problems.

We will take at least 5 in each user segment: 5 users from the ?Electronics? section who have used delivery in the last six months and have stopped using it, 5 users who use delivery but in other sections (not ?Electronics?), and 5 users who use solution interpreters (competitors). In total: a minimum of 15 in-depth interviews, but we plan to conduct interviews with new respondents until we start receiving the same answers to our questions.

The questions themselves must be open-ended and neutral, without personal opinions about the product. We should not try to ?sell? the product during the interview or prove its value. Our task is to gain insights with non-leading questions. Within the organization, we can use internal resources and find the selected target audience. We go to the Analyst and request IDs and contact details of clients that match our sample. We will send an SMS or email campaign inviting users to the in-depth interview, considering personal data usage rules. To find new users, we go for external recruitment. Or we can find respondents among our surroundings, with 31.7 million MAU relative to Russia's population of 143.4 million, indicating that the search will not be long. Practically every fourth person interacts with the service in the ?Electronics? category.

Next, it's important to conduct a quantitative research of the identified insights to test the hypotheses. Determine the frequency and criticality. We survey an identical audience of users, conducting a questionnaire. For this, we will need to determine the size of the survey cohorts to ensure statistical significance.

This is a whole separate topic and a very useful tool for determining the significance of research results. It's beneficial to know the basis of the method, but for convenience of calculation, one can use, for example, Evan’s Miller Calculator. After conducting the quantitative research, we have the opportunity to determine whether our focus problem has been statistically confirmed or not. If the hypothesis is not confirmed, we move into a new iteration of forming hypotheses, their qualitative and quantitative verification based on current conditions and opportunities. During the research through interviews, completely unexpected problems and opportunities often, and do, arise. Suppose all hypotheses except for ?exaggeration of the description? and ?most goods are in poor condition? are statistically confirmed, and we have identified the focus problem JBTD (Job to be done).

Key points: The focus problem JTBD (Job to be done) has been identified - the need for a tool that allows buyers to gain added value in Avito delivery to overcome fear and uncertainty when purchasing through delivery. In conjunction with the identified problem, we have statistically confirmed that sellers in the C2C sector are less concerned about the percentage taken by the platform in exchange for the opportunity to use the platform to sell their goods due to the lack of competitive alternatives when selling used goods. (Trade-off)

Let's move on to formulating ideas to address our focus problem JTBD. Let's assume that we will try to find sufficient growth in purchases of used electronics through delivery, considering the insights gathered. We know that the average monthly GMV for the ?Phones? section is 9300 million, accounting for 34% of the entire ?Electronics? section. From Avito reports, we know that 42% of this section is used electronics, 3906 million in GMV. Considering the conversion rate (10%) in purchase through Avito delivery, we assume there is good potential for revenue growth here. If we ultimately improve the conversion to 40% and, due to this indicator and conditions of the external market, adjust the commission in the ?Phones? section to 4% in C2C and 5% in B2C, we predictably can achieve an average GMV of 138.907 million per month. Not forgetting the planned growth in MAU, we multiply by a coefficient of 1.66, forecasting 230.534 million average GMV per month. This is approximately half of the set goal ?5 times in 5 years?.

Source: new-retail.ru

Key points: by modeling an increase in the conversion to Avito delivery orders in the ?Phones? section to 40% and raising the commission in this section to 4% in C2C and 5% in B2C, we forecasted sufficient growth to support the set goal. We will look for solution hypotheses to find enough added value for the ?Phones? section with the possibility of subsequent scaling to other categories.

Hypothesis Solution Formulation

Source: avito.ru

Prioritization

To prioritize the goal, I want to use the RICE method.

Reach* - the scope of the intended audience in millions. We know that the MAU of the ?Electronics? section is 31.7 million, and 42% of our section is used electronics.

Impact - the impact of the idea on solving our JTBD hypothesis. Subjectively measured based on ratings from 0.1 to 1.

Confidence - Shows how confident we are in our estimates of reach, impact, and effort.

Efforts - the amount of work needed to test the hypothesis.

The calculation is performed using the formula:

In addition to a low RICE score, the AI Assistant appears to lack sufficient added value, and moreover, I have not figured out how to sustain conversion to delivery purchases with this tool. I also suspect that this product may create internal competition through its analysis, which could decrease seller loyalty. Such a hypothesis could be tested using the Wizard of Oz MVP method.

Online inspection through specific templates. In my view, this could improve the customer experience and increase conversion to purchase, but it remains unclear to me how to motivate the buyer to use Avito delivery after such an inspection. Moreover, we have a low RICE score.

A high-level calculation using the RICE method showed the advantage of the hypothesis of checking goods at an Authorized Service Center. Our hypothesis is that this added value for the delivery service has a significant tipping point to start using delivery. We can quickly and inexpensively test this idea using a Fake door MVP and collect research data due to the high DAU in the product. Furthermore, in the long term, this could become one of Avito's standalone products with significant growth potential. Next, we have the final part - the experiment design.

Key points: prioritized the hypothesis using the RICE method. Similarly, we're testing the hypothesis of increasing the commission to 4-5% in C2C. We assume that external conditions (competitors' commissions) will not prevent us from raising the commission in selected product groups.

Design of the Experiment for the Chosen Hypothesis

Mechanics of the Chosen Hypothesis

Mechanics using the category ?Phones? as an example:

• The seller adds a product card and enables the option to sell the phone with a verification service through Avito delivery.
• The buyer, when purchasing the phone, adds the paid service ?check the phone? available only with Avito delivery+safe deal. They complete a transaction that includes the cost of the product+verification+delivery, and the funds are frozen.
• The seller takes the phone to an Avito delivery point. The experience for the seller does not change.
• The purchased phone via Avito delivery is sent to an authorized center, where it undergoes an expert evaluation of its condition.
• The buyer receives a notification and within a limited time must confirm/deny the purchase.
• If the purchase is confirmed, the phone is sent to the buyer through Avito delivery. If not confirmed, the phone is sent back to the seller. The commission for phone verification is returned under the ?safe deal? terms with Avito delivery. The safe deal can be enabled or disabled. If the safe deal is turned off, the verification cost is not refunded.
• If the phone is purchased - the funds are unfrozen and sent to the seller, minus the reduced commission. If the phone is not purchased - the funds return to the buyer, and the phone is sent back to the delivery point for dispatch.

Hypothesis Operation Scheme

In addition to validating this hypothesis for value, it's essential to construct and assess its economic model. This is a complex product that exists at the intersection of online and offline environments, involving work with contractors. Calculating all hypothetical costs and risks is required to understand whether it will be economically feasible.

High-Level Economic Model

Revenue structure:

• Service sale commission: Setting a fixed price for the phone inspection service at the service center. This amount can include both the cost of the inspection itself and an additional commission for Avito as the service organizer.
• Premium options: Offering additional premium services, such as express checking, an extended diagnostic report.
• Possible consideration of adding a warranty upon confirmation of this economic model.

Expenditure structure beyond current:

• Payments to service centers: Negotiating terms and paying service centers for inspections. The cost can be fixed per inspection or vary depending on the complexity and volume of work.
• Logistics and delivery: Organizing the delivery of phones from the buyer/seller to the service center and then to the buyer or seller. The cost may include both the delivery cost itself and insurance for goods in transit. Hypothesis for improving conversion to ?safe deal?.
• Marketing and promotion: Expenses for marketing campaigns to draw attention to the new service, including online advertising, promotional activities, and discounts for early users.

Benefits for participants:

• For sellers: Increased trust and attractiveness of listings through verification of product quality, and the opportunity to sell with a lower sales commission percentage.
• For buyers: Reduced risks when purchasing used phones, confidence in the product's conformity to description and functionality.
• For Avito: Increased conversion of transactional deals with delivery by adding a new service. Improved user loyalty thanks to better quality transactions on the platform.

Scalability:

• The hypothesis suggests rolling out the service to other ?Electronics? section categories by adding specialized service centers for inspection.
• Expanding the geography of the new feature by adding new service centers.

Growth points:

• In the future, we consider growing the product into a full-fledged resale platform with authorized services that will conduct electronics checks in the backend, adding warranties to products to increase conversions. The purchase of phones and cooperation with service centers are already established. Successful international implementations in electronics resale: Blackmarket, Swappie

Unit Economics

Let's evaluate the hypothesis of inspection in an authorized center by calculating a high-level unit economy for the C2C sector:

Assume the inspection service costs the buyer 600?, the cost of conducting one inspection is 300?, and the customer pays for delivery independently. The anticipated future commission from the sale is 4% of the product's price. We can calculate the average price of a product in the phone category from the available GMV and the number of transactions - 13,235?. Assume that 80% of inspected goods are eventually purchased.

• Service price for the buyer: 600?
• Cost of inspection for us: 300?
• Sales commission: 13,235? * 0.04 (4%) = 529.4?
• Profit per unit: (600? + 529.4?) - 300? = 829.4?

CAC (Customer Acquisition Cost) and LTV (Lifetime Value) require a separate calculation, based on marketing expenses and the expected number of purchases from a customer. It's also important to consider risks associated with potential returns and customer dissatisfaction, and to develop strategies to minimize them. In the future, we may switch to contracts with authorized centers with fixed monthly payments.

If we calculate the current average revenue (2% commission) from the average bill in the ?Phones? category in C2C (13,235?), we get 265?. With the new inspection service, we will be receiving an average of 829.4? per bill, which is 3.13 times more. Calculating similar metrics for B2C, we get an average earning of 1,391? with the inspection service compared to 764? under current conditions, which is 1.82 times more. As we have already mentioned, the market volume of the ?Electronics? section for used goods is 42%, accordingly, the new product with the inspection of used phones technically can increase conversion up to 42%. If we assume a conversion growth up to 30% in delivery+inspection orders with an increased percentage, then just the ?Phones? section in both C2C and B2C sectors could bring us an average revenue of 257 million ? per month. Applying the previously found forecast growth coefficient of MAU 1.66, we get 426.62 million ?, which is almost the set goal from just one "phones" section.

Key points: After constructing the high-level unit economics model, we understand the value of the hypothesis and the approximate growth potential that aligns with the stated Growth goal. Now we can move on to designing the experiment, which will require human resources.

Experiment Design Description

Our task is to test the hypothesis quickly and affordably. Let's formulate an experiment design to check it.

Hypothesis: We aim to increase the demand for Avito delivery service by adding the value of the ?item inspection at the Authorized Service Center? service, which will be available only through Avito delivery. For testing, we will select the Fake door MVP type. We will lead the user to the point of purchasing the new service, but at this stage, we will not implement the real product or inspection at the service center.

Product actions: We will conduct an experiment in the ?Phones? product group. The control group of users will not experience any change in product usage. The test group of buyers will have an added step/pop-up/hint at the order stage with the option to choose ?item inspection at the Authorized Service Center?, where we will inform the user about the operation of this service and provide the opportunity to proceed to the payment stage. For the test group of sellers, at the stage of creating a product card, we create the possibility to enable the sale of the item with the inspection service at the Authorized Service Center through Avito delivery under special conditions with reduced transaction commission. For both buyers and sellers, the user path does not change; the item needs to be handed over to Avito delivery.

Which users we include in the experiment: We take registered users who use the ?Electronics? section. For a more quality collection of the test group cohort, the user will enter the experiment at the moment of opening the step/pop-up/hint about the possibility of the new feature.

Metrics for evaluating the experiment: for the buyer - conversion to the ?proceed to payment? click with the added new service. For the seller - saving the product card with the added new service of inspecting this item at the Authorized Service Center through Avito delivery.

Determining the group size for the study: we will use the Evan’s Miller Calculator again to determine the size of the test group. We know that in the control group, the conversion to making an Avito delivery in the ?Phones? product group is 10%. Assuming we want to know with an 80% probability a conversion change of 2 percentage points, then our test group must have 3,623 buyers who clicked on the ?proceed to payment? button with the phone inspection function at the Authorized Center through Avito delivery enabled.

Action plan after the experiment: If the experiment is confirmed, we can move on to designing an MVP product for checking goods at authorized centers. If the experiment does not succeed, we will return to the step of forming ideas.

More about experiments: GoPractice.io

Then, after the quantitative experiment, we need to find the Confidence Interval and check it for statistical significance.

End of research and conclusions

Let's assume we were able to prove the value of the idea with verification at authorized service centers through the experiment. We present the hypothesis to stakeholders/management, and if approved, we proceed to build a real MVP/Lean startup with a soft launch. If the hypothesis is not confirmed or we fail to defend it before management, we move into new iterations of searching for a solution to the JTBD problem.

Key points: We conducted research to achieve the set goal, analyzed the external and internal environments to define hypotheses for the direction of the research. We conducted customer development and formed problem hypotheses, which we then tested both qualitatively and quantitatively. We developed hypotheses for increasing added value, prioritized them, and proposed a new product. We defined its high-level economic model and unit economics, created a description of the MVP's operation, and designed an experiment that will allow us to quickly and with minimal costs test the hypothesis.

Presented by: Vadim Novikov