Product Metrics

Just to be clear guys, I am talking about agile ways of working & Product development in an agile context. Agile ways of working is like Visiting a Barber : You get feedback quickly and frequently. You are involved and collaborate on what is needed (nice haircut :) ).

Now, every work does not fit in an agile context. Some work is not feedback driven and will take as much as it will take. Like giving your car for repair. If mechanic has said it will take between 2 to 3 weeks, Asking "Has it been done? OR how long it will take ?" frequently will not help you. Believe me!!. Now, let us talk about risks and metrics for various endeavors.

E Goldratt, who wrote The Goal—said “Tell me how you measure me, and I will tell you how I will behave.” That’s absolutely correct.?Be careful what you measure.

Metrics are only helpful if it helps in decision making. Metrics does not matter if they are not useful for people who are being measured. Also, be careful in measuring because there are high chances that you will get what you measure.?Example?: If you hire and judge a scrum master to run scrum ceremonies, you are getting only half of their Value.

Now, let's talk about Product Metrics including before, during and after Product development. Just to be clear, Before development = Product discovery, During development = Team metrics and After development = Product delivery Metrics.

Product Discovery Metrics :?This is before you write a single line of code or start development of any kind. Also known as?Problem solution fit stage.?Here we are trying to identify what problems we have to solve, are those problems worth solving AND most importantly?can we solve them in a way that satisfies the customer and works for our business.?So, here we tackle 4 types of risks (and we have to identify metrics for each of them) for our solutions which we are planning to build.

Value Risk?: Is anybody going to buy our product OR people will choose to use it. Answer will be yes ONLY if we have a way better product than any body else.

Usability Risk?: Can people figure out how to use our product?

Feasibility Risk?: Do we have tech stack & all skills necessary to build the product in our team ?

Business Viability Risk?: Do we have sales channel to sell this product ? Can we do cost effective marketing ? is it legal to build this product ? Is this product in line with our business strategy ?

Metrics during product development :

<<?Only one metric : How frequently we are taking feedback from real users in real environment??>>

Team level metrics?: The only metric that matters is whether the team is meeting the stated goals of the team and the expectations of the customer. / stakeholders Any other metric is just about checking boxes for someone and a good indicator they aren’t taking agile seriously.

• Quality standards?: Identify Quality standards for your teams(if you do not have one) and make sure you adhere to them without exception.
• Value delivery per time box?: Customers need answer to only one Q : What Value have you delivered to me lately?
• WIP per iteration. Are we reducing WIP per Iteration or not ? This does not mean complete less rather it means take less and finish more. Taking 10 items and completing 4 (40% Completion) is better OR taking 5 and completing 4(80% Completion) is better.
• Mean time to recover?from any failure. Is it getting shorter or not ?
• Cross skilling every iteration.
• Escaped defects to production.?Ideally it should trend towards Zero.

Product delivery Metrics :?Some consideration after product is shipped to market and we have got a few customers. These are again Product focused metrics , not customer focused metrics. Customer focused are AARRR metrics and HEART Framework which we will talk later.

Scalability?: Can we scale this product from 10 to 1000 users? How quickly can we do it?

Reliability?: Can our customers say that this product is reliable? Can they trust us ?

Performance?: How is our product performing in our market ? with our customers and with our competitors? What is our market share etc ?

Maintenance?: How much does our product cost for regular maintenance?

Now let's talk about beating the competition?: Are we uncovering problems in our own products ? Why because if we don't do , our competitors will do it for us and then they can anchor against us. We really need to prevent that.

What's your market? What's your share? How can you improve that? What's your strengths from the perspective of your market? How can you leverage these? What's the competition doing? What are approaches you can take to differentiate yourself or create greater value than the competition?

Are we doing more job than what is expected from our product ? like when we use google map , they connect us to Uber etc.This is because they are trying to be helpful. When we search for airport , they tell us when to leave for the airport so that we don't miss our flight and so on.......?Obviously we cannot do all jobs available out there and that is where scoping comes into play.

Caution : Just be careful of Vanity Metrics

Vanity metrics are those which gives you rosy picture but does not improve anything.?Example?: Daily Active Users ? Ad Impressions ? Number of Downloads ? Page Views ? Registered Users ??Story?Points Delivered ? Time on Page

Why these are Vanity metrics ? Let us understand by an example :?Registered Users.?If you have 1000 users registered today and 10000 users registered after a month, will you be happy ? I hope not because we do not want users to just register and then leave. We want them to complete the transaction end to end and (hopefully) tell others about how good our product is.

AARRR Metrics :?These are also called Pirate Metrics.

Acquisition?: Where users come from ? What is ur plan to acquire more users? Are we doing cost effective marketing , ad campaigns , SEO's etc ?

Activation?: First happy users experience.

Retention?: How are you planning them to come back ?

Referral?: Are they referring your product ? If not , what are we going to do about it?

Revenue?: How are you planning to make money?

HEART Framework :?This can be used for specific feature or for an entire product.

Happiness?: This is how your users feel about your product or service. Talk to them and observe. What are you going to do about unhappy customers so that other customers don't get the same experience ?

Engagement?: Level of user involvement like number of visits per user per week on your website etc.

Adoption?: Acquiring new users for your product or service

Retention?: The rate at which existing users are returning. Example , how many of active users from a given period are still logging in after sometime later.

Task Success?: This is focused on areas of your product which are task focused. Example : Searching for an item. How quickly can users search for the book etc.

Agile Transformation Metrics

Although transformation is a journey , not a destination, so it is very hard to measure transformation and in my opinion , it is not useful to measure transformation from a Customer Standpoint. It is in itself a vast topic but let us talk about it very briefly. However, if we have to measure it , here are few ideas :

• Start with WHY, the purpose. Then look at WHAT needs to change and the HOW you intend to change the WHAT. WHAT refers to the things which you Value. HOW refers to ways of working. Once you have your deeper WHY you can start to look at WHAT should change. WHAT is the measures, the things you value are the things you measure. They are your WHAT. Once you have discovered the WHAT, with your WHY at the core you will need to collaborate and define HOW.The key to agility is Lead with WHY, Align around WHAT and Trust the teams with HOW.
• When the WHAT changes, the HOW changes. If the team doesn’t connect with the WHAT or doesn’t see how it directly contributes to the WHY then they will find a way to satisfy the WHAT but HOW they do it may not please you.?Alignment of WHY, WHAT & HOW is critical for real agility.
• When the transformation is company wide , so metrics should also be company wide. For Example :?How is the company doing now and how do we want the company to do in next 2 years (or may be 5 years of time).
• You can measure how many silos (in a Value Stream) or handoffs you had when you started and how many you have now(in that same Value Stream) ?
• You can also look at what is your current state and what is the desired future state and then figure out the Gap. Try experiments to fill those gaps.

Now, the bigger Question :?Which metric to choose out of all these.?And answer is very simple. Start with the Goal of your business and your product. Wait, Relax, Think deeply & Go from there.These are my notes on what I have learned in last couple of weeks. I am sure , I am wrong at some places and would love to be pointed out :)