When Quality Falls Below MVQ - Take a Break

The original title of this post was how to tell when quality falls below Minimum Viable Quality (MVQ). I cover that point but I realized that I need to update my original MVQ model to include a new capability, “Take a Break.” 

BOTTOM LINE UP FRONT: If you implement an MVQ strategy you will on occasion ship below your target minimum quality. When you do this, users will get very mad and abandon you. For a proactive program that needs users to opt in, losing a user that is willing to beta test your product is costly. You should implement the “Take a Break” feature I just used within Window.

In 2014 I published some papers on the concept of Minimum Viable Quality (MVQ). The easiest to find and most complete is posted here: The Future of Testing is easy with EaaSy and MVQ.

MVQ relies largely on the rings model. Rings like the ones shown in Figure 1 have some similarity to A/B testing and flights with control and treatment. Rings however are designed more for apps with desktop bits where those other models are designed for cloud services.

Figure 1: Rings Model for build release

With rings, users are divided up into risk pools. Those willing to accept more bugs, more risk, are placed in the inner rings. The builds of code released to those rings are less stable and do have more bugs than the ones released to the larger outer rings.

MVQ also runs the risk of any release actually being under tested. Under-testing is actually hard to quantify before the update is released. This is because good enough quality varies based upon your business objectives at that point in time and the tolerance level of those involved in your risk pools (rings). 

There is one certain way you can tell if your quality is too low. Users will tell you either with inaction or abandonment. If usage of your app dramatically drops then your quality was too low (on occasion it could be a competing product). That is inaction. Abandonment happens when the proactively drop out of your beta program or uninstall your app.   

Figure 2: Minimum Viable Quality (MVQ) Overview

MVQ works when you have users trying your product update and providing enough usage data that you can assess quality. Sometimes those users are beta users and for incentives like a chance to see the new weapons in Halo before the general public or a tShirt they use your app and provide feedback. Some rings the users may not actually know you are pushing new bits to them in a slow rolling release.

You know that your quality is too low for a release if either usage plummets or users quit one of your rings. I finally become one of those individuals that had enough. 

Windows Insider Builds pushed me to the breaking point

With Windows 10 and the Insider program we have largely implemented MVQ as a cornerstone of Windows as a Service. Recently though I had an experience that caused me to go to the Windows Insider Settings on my Surface Book and try to escape. Wow, I work on shipping Windows. I’m always wallowing in the data from our devices and user feedback and here I was so frustrated that I was planning on taking my primary device out of the Insider program all together.

Here is my cautionary tale of what it looks like to fall below MVQ. 

To be fair, I have signed up to be part of one of the most unstable rings in the entire MVQ process for Windows. I am on the Insider Canary ring for Prerelease builds. That means I get updates multiple times a week. Some builds are better than others. Feeling the pain is just part of the job.

Image 1: Windows Insider Program Settings

Each update though does have an impact on productivity. I know what it is and it’s a cost I’ve paid for more than a year now.

My updates seem to always trigger first thing in the morning just as I want to log in and start working. The first message I get is “we need to reboot and update your system.” Click okay and the update starts. 

KEN ASIDE: It really bugs me that updates count up to 100% and then the computer turns off and restarts and then it starts counting to 100% again. I’m thinking, why didn’t it just restart and apply all the updates after the restart.  Why pretend to get to 100% in the first place.

I know that at this point I have at least 15 minutes for the updates to apply, reboot, and finish the process. Finally, the login prompt comes up and I can get back to work, well not quite. 

What I find maddening, what drives me the most bonkers, is the annoying “Hi. We’re setting things up for you.”

Image 2: Windows 10 “Hi. We’re setting things up for you” message after major update

When I see “Hi,” that means I have another two to three minutes to go get coffee and come back and maybe become productive again.

KEN ASIDE: You know how sometimes there is a cool new feature in a program and you start off really liking it but then you stop? Don’t “Hi” me. I know the apps I use and the ones I’ve uninstalled. Update them in the background later. Let me get to work. Stop with the "Hi" thing.

Yes, there can be a lot of tedious overhead being an Insider in the Canary ring. It is however important catch bugs before we unwittingly release them to our customers. So, I pay the tax.

The truth is, I would much rather deal with bugs on my devices at work than release something that impacts friends and family. That goes for all of us here at Microsoft and for everyone that participates in the broader Windows Insider program. We want the world to have a better experience. It’s what we do, we all participate in the Insider process.

Everyone has a tipping point.

It started for me about two weeks ago. Instead of updates 2-3 times a week I seemed to be getting new builds every single night. That meant every single morning I’d go through the update process and see my friend “Hi.” Occasionally I’d notice something new in the build, some tweaked feature, but usually it was just updated bits with some bug fixes and other new bugs introduced. Last week I started getting more than one update a day. I can’t take multiple updates a day. It’s too much. Who’s deciding to push so many updates?

Then it happened, two unforgivable bugs. Bugs that cost me so much time and frustration I was ready to opt my Surface Book out of the Insider program.

The first one was IE crashing all the time and I mean all the time! 

I’m a heavy browser user. I run 3 to 4 different browsers at a time logged into different Microsoft Account (MSA) profiles, LinkedIn profiles, twitter accounts. You name it. Some are open holding some data science papers I’m hoping to find time to read, another my Christmas shopping search history and another has YouTube videos on First Lego League robotics queued up.   

KEN ASIDE: I love that when I get a Windows update and the PC reboots most browsers will re-open all my tabs from the last session. 

IE stopped working. Literally I launched it and it immediately went into app crash and recovery mode over and over again. 

Image 3: Crash and crash again

Unstable browser in a build has happened before but this was particularly bad. I do use Edge a lot but without IE I ended up needing to use Chrome and Firefox more than usual.

The second bug hit me the next morning when I had to work from home. I was waiting on a roof inspector and needed to work from my home. Morning came and I went through the twenty plus minute morning update. 

KEN ASIDE: Sorry, but this next bit gets heavy with Windows jargon.

I then tried to log in to my domain account but it failed. Hmm, did my password expire? No, my cell phone still worked. I could log in to web mail on another PC so my laptop was having issues. I tried at least ten times, fiddled with network settings, nothing. 

One of my friends online let me know there was a bug that you couldn’t log on with domain credentials without being on the corporate network. I was hosed. I needed to use Skype for a meeting in less than ten minutes. I did the only thing I could think of. I logged in with my personal MSA. Fortunately I’d created that profile some time ago for a different test. Great, just 5 minutes until my meeting. Type in the password, press enter…

Image 4: “Hi”

That was it. The last straw.

The next day when I was at work and managed to logon to my corporate profile, the very first thing I did was bring up the “Windows Insider Program” dialogue. When it came up I saw an option to “Stop Insider Preview builds.” I just couldn’t afford the productivity interruptions.

I clicked the link and was surprised by a small tweak to the user experience.     

Image 5: Want to take a break?

Do I want to take a break? NO! I want out! I want out now!

Well, maybe I could do a break instead of completely quit. 

I do have to give credit to the team that builds the features for our insider program. If they didn’t have the option to take a break, I think I would have completely quit. 

I would call this an MVQ guardrail feature and one that everyone should look at implementing. We spend a lot of time getting users to join us on this MVQ journey and losing any of them is a huge loss. You have to realize if you implement an MVQ strategy, you will eventually come up short. In that case you will need an escape valve to help try to retain users. 

The “Take a break,” option let me take a proactive action without completely dropping out. I chose that option and I am currently on a seven-day break. No updates for almost a week now. 

Mostly I don’t mind the bugs. I know all the great engineers pouring over the telemetry and how fast we can find and fix bugs. Bugs happen and it’s better they happen to me than to friends and family.

I’ve written this blog, and vented my frustrations. Time to take a deep breath and apply some updates tomorrow morning. 

MVQ is the right model for a ship fast world

MVQ comes from the whole MVP (Minimum Viable Product) concept outlined in Lean Startup. The entire theory behind MVQ is that you want to ship as fast as possible with the lowest possible quality you can get away with. 

It sounds horrible but the truth is that if you can ship a release of your product that gets enough usage by enough users that you find and fix bugs with maximum efficiency then you were right to release it. When I first published MVQ though I didn’t think about a program that needed insiders to stick around over multiple years. I would amend the MVQ premise to say that you should ship as quickly as possible with the lowest possible quality that does not cause your users to churn out. If users quit being part of your inner rings then you don’t have that safety net and therefore your quality was too low.

I hope this expands your understanding of MVQ.

Research and References:

The Future of Quality is easy with EaaSy and MVQ: (blog 2014 by Ken Johnston): Covers the core concepts and capabilities for MVQ.

The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses (book 2011 by Eric Ries)

The Ultimate Guide to Minimum Viable Products (blog 2013 by Vladimir Blagojevic): Mostly provides an overview of key approaches to MVP from Lean Startup

How To Create A Minimum Viable Product (article 2013 on Tech Crunch by Emre Sokullu): Article is mostly a list of open source technologies and SOA design ideas for launch an online MVP based upon Emre’s experiences launching Grou.ps

In Pursuit of Quality: Shifting the Tester mindset (blog 2014 by Brent Jensen): Wake up testers, “fixing correctness issues on a piece of code that no one is using is a waste of time & resources.”

Stop, if You Want to… (Blog 2014 by Alan Page): Testers should stop writing so much automation. The world of data is changing how we can approach testing and quality.

Testing in Production, Your Key to Engaging Customers (presentation 2011 by Seth Eliot at STP Con): Scenarios and Testing in Production produce High Quality Product Delighted Customers.

Create Your Roadmap to Data Driven Quality (presentation 2014 by Seth Eliot at STP Con): Ditch the HiPPO (Highest Paid Person’s Opinion) and shift to data from real users to gain real insights on quality.

Good Enough Quality: Beyond the Buzzwords (IEEE paper 1997 by James Bach): Paper and article on reducing up front testing of product code.

Journey to Continuous Deployment (presentation 2009 by Nathan Dye): I couldn’t find the white paper that I originally read by Nathan but he covers all the key points in this presentation.

Streamlining the development process with feature flighting and Azure cloud services: Technical Case Study that references MVQ (2015).

Minimum Viable Quality – Why startups should start thinking about it: (blog 2016 by QAPitol): Pitches MVQ as a tool for startups and should be part of assessing the readiness of an MVP.

Minimum Viable Quality: (blog 2014 by Rob DeRosia): Quick summary of MVQ.