Quality Engineering for CEOs: A Series Introduction

TL/DR: This article establishes the reasons behind—and the pathway ahead for— an upcoming multi-post series on software quality. The series, written by the CEO of a global software testing company, will be targeted at CEOs focused on digital products.

As a technology-focused CEO, you have a lot on your plate. Customers. People. Product. Market. Capital. Growth. Competitors. Operations. And more.?

With all of that, you may leave quality assurance and quality engineering to your CTO, VP of Software Development, and/or Director of Quality Engineering. QA may sound old school to you. Software testing may feel like an arduous but necessary step in a release process that you intentionally keep below your radar.?

You likely even yearn for your software to just work. To be bug free. To release and update frequently and fluidly. To be great, fast, nimble, and truly wonderful for your customers to use.?

I get it. I’m a technology CEO myself. And I’ve been working in the software development industry since 1993 (yes, that’s 30+ years if you’re counting). After seeing a dysfunctional record company on the inside as an intern (I’m a musician), at 22 years old I dove into tech. My first job was in NYC, helping banks like Morgan Stanley build teams that created some of the earliest Unix trading systems (often using C++, Encina, and Sybase running on Solaris). I loved it.?

From there, I’ve been fortunate in my journey with software technology companies. Two IPOs (MOTV and DMD). Two acquisitions (iChat and Pluck). 2 board seats. 8 advisory roles (working mostly with other CEOs). 11 angel (seed) investments.

Admittedly, over the first 20 years of my career, I was frequently frustrated by quality assurance: economics, processes, strategy, and a too-often lack of rapid, clear impact.

Don’t get me wrong. I worked with some great people in quality. Greg at Motive. Chris at Pluck. Others. But my frustrations were often based on the amount of spend relative to tangible results. Quality assurance sometimes felt like a necessary task, not something that might give us a competitive advantage. It appeared to be a tactical, not a strategic function.

That changed when, in 2013, I met Kristel Kruustuk and Marko Kruustük , the co-founders of Testlio . They had just won a global hackathon (AngelHack). Jason Seats , the then MD of Techstars Austin, found them and convinced them to join his accelerator. At that point, they had an idea and a prototype.?

It was the very beginning but I could tell they were onto something. Big. Possibly transformative for the entire industry of software quality. They had a new way of thinking about—and looking at—quality assurance. I found their vision compelling and inspiring.

So, I invested in their seed round in 2014. I became their first company advisor. I helped them build their team. And, after 5 years of partnership on the sidelines, and after considering dozens of other opportunities, I joined them full-time as CEO in late 2018. You can read about how Kristel and I managed our transition here.?

Over the last 5.5 years I’ve gone deep, challenged my previous assumptions, and learned how to strategically think about software quality from the shoes of a CEO. I’ve been fortunate to work with scores of other CEOs as clients of Testlio, along with their engineering, product, and quality leaders.?

Through my journey, I’ve built a set of perspectives and experiences. With this article series, I will share many of those with you.?

In the posts ahead, I’ll work hard to help with candid questions I’ve had myself—and often hear from others. Even if I don’t explicitly hear some of the questions I’ll present, I sometimes implicitly sense them, directly from other CEOs or from the CTOs and VPs of Engineering who work for them.?

Questions like:?

• How much should I spend on quality engineering relative to software development?
• How much coverage (devices, languages, locations, payments, etc.) in my software testing journey should I pursue??
• How should I think about capacity management, given that it often seems like we have either too many or too few quality resources???
• Does test automation really make sense economically? If so, how much should we do???
• When should I expect quality processes, and my overall pipeline of software development, to go fast? What can exceptional speed look like from both clock and calendar time standpoints?
• Does software quality really impact my customer experience? If so, how can I measure it? Can great quality help me drive revenue growth??
• When does it make sense to partner for quality engineering? What should I keep in-house for quality? And if I do partner, how can I ensure success???
• What are testing gates? Does my company need them? If so, how do I think about investing in them? Measuring their effectiveness???
• Why should I think about integrating quality and DevOps systems? Is integration worth the security risk and technology hassle???
• What about AI and software testing? Won't AI bots eventually just do all the testing for us? And how should I think about the quality assurance of my AI-powered experiences???
• How should I consider the psychology of my engineering and product team in relation to quality? Can I place an economic value on confidence? Sleeping well at night???
• Are there superstars in the world of quality? If so, what makes them tick? How should I attract and reward them??
• What does a great quality strategy entail? How often should my team update their quality strategy? Does strategy matter in quality?
• And possibly more.

If you’re still with me on this first, introductory article—thank you. I encourage you to leave comments as I welcome ideas, topics, provocations, alternative viewpoints, requests, questions, and/or challenges.

I plan to be back before too long with Post 2. There, I intend to tackle the meaty topic of quality engineering spend relative to software development costs. I’ll leave you with a hint and a teaser: consider 15%.

Wishing you well until next time.

Steve Semelsberger is the CEO of Testlio, the originator of fused software testing.