A Performance Engineer Named Calvin

There once was a young engineer named Calvin who discovered he had a talent for measuring and optimizing the end-user experience of client/server software applications. He decided to become a Performance Engineer, and he got a great job at Big Software, where he became a valued member of their PC/Internet development team in the late 1990’s.

Over the years, he learned new performance testing tools and techniques, and his assignments got more interesting and challenging. Soon Big Software was producing huge B2C applications with millions of simultaneous users on mobile devices in over 200 countries all over the world. But as the company grew, Calvin’s budget got smaller, the schedules got tighter, and his responsibilities got narrower. His company adopted CI/CD/Agile/Scrum methodologies, and soon Calvin was reduced to acting as a policeman watching the nightly build. Anytime Smithers on the developer team checked in a poorly-performing new build of the server software, Calvin’s job was to sound the alarm and report on the problem. And while this arrangement kept the server software from getting slower, the mobile users all over the world noticed that their experience was actually getting worse and worse. They complained to the support line, but nothing ever got done. No-one was measuring the end-user experience anymore, and so no-one knew it needed to be fixed.

After about 5 years of this, Calvin began to question what he was doing with his career. He used to take pride in his company, and in his work, measuring and optimizing end-user experience, and now he was doing limited server policeman duties for a company whose product was getting a bad reputation. He thought about quitting. He wrote a post on LinkedIn asking, “Do you feel like no-one cares about your work as a Performance Engineer anymore?” Dozens of his friends replied in the affirmative. One good friend wrote “Sometimes I feel like a motherless child…”  Several others wrote that the project managers and company executives didn’t understand them and their work, and wouldn’t give them enough budget, schedule or support to do their jobs properly. What happened to this profession that Calvin once loved?

One day, a new person named Hobbes showed up, with a new technology that would allow Calvin and his friends to measure the real end-user experience for all of those users with mobile devices all over the world, so they could actually make real improvements for the end users. But Calvin was a good engineer, and he was skeptical. “This new technology will never work,” he said. “It’s too difficult to solve the end-user measurement problem. It’s too expensive. It’s too different from what I am used to. How would it even work? Our measurement systems are built for testing servers, not whole systems with servers and millions of mobile devices. Our server tests are really well-controlled. If you include all those dirty mobile devices and real networks, you will get a noisier and less repeatable measurement, so we could never use that for daily regression tests. There’s no point in doing any of this if you don’t measure everything on the user’s device, and that would be ridiculous.  The network delay component is not in our control, why would we measure that? The CDN component is out of scope; that makes it even more complicated. Can’t we just model the global and last-mile network delays in software and just add them in as fudge factors? Or just fake up some limited tests with some little tweaks to our current system?”

Hobbes let Calvin finish. 

“Here,” he said.  “Just try it.”

They ran the demo together. It was easy. It worked just like Calvin’s server tests, but it included the global last-mile mobile effects that real end-users experience. They ran a big test together. Because there were so many devices all over the world, the measurement was not noisy or unrepeatable. It just reliably showed how much worse the experience was for all the end-users than for the server alone. It wasn’t more expensive. The new test was actually half the cost of doing it Calvin’s old way.   It included browser rendering effects. It included the network effects and CDN effects, which allowed Calvin to see problem spots with their CDN that were affecting 3 countries in Asia and 2 countries in Europe. There was no need to model anything or add in any fudge factors, or fake anything in the test. They just measured the global last-mile mobile end-user experience. Directly.

Calvin took the report to his boss, Miss Wormwood. She put the new report next to the previous day’s report. “What the heck?” she said. “This says that performance in Asia, Australia, and Europe is WAY worse than the U.S. And that is exactly where all of those complaints have been coming from. And it is a disaster in those 5 countries in Asia and Europe. Wilkins! Our CDN is busted again in Asia and Europe. Fix it, now! And the performance is about 0.75s worse on average everywhere, including the U.S.! Smithers! We have to squeeze another 0.75s out of our mean server response time to meet our Service Level Agreement for AT&T in the U.S. and Orange in France! What the heck? Calvin, great work! But why didn’t you tell me this sooner?!”

Within a few hours, Wilkins had the CDN fixed. 4 days later, Smithers completed his last server SQL and memory optimizations that he had been wanting to do for months, but Miss Wormwood wouldn’t let him because they already met the performance requirements according to the previous server-only tests, and she wanted him to work on something else.  Calvin did another Global Last-Mile Load Test and confirmed that they now met the AT&T and Orange SLA requirements. Two days later, the Support team sent a delighted email to the VP of Software Development, asking, “What did you do last week? Complaints are way down in Asia, Australia and Europe, essentially matching the levels in the U.S.” Two weeks later, the Sales team sent a company-wide email reporting a 14% increase in users switching from their competitor’s app, and a 31% decrease in users switching to their competitor’s app, starting about 2 weeks earlier (and of course, taking credit for it). The trend continued for the next several months; the CEO reported this in their quarterly earnings report and their stock went up 1.9% on the news.

Miss Wormwood came into Calvin’s office three days later, and asked, “How much more expensive is that new service from Hobbes?” Calvin said, “Um, it’s actually about 50% cheaper.” Miss Wormwood said, “OK, great. Keep running your normal daily build tests just like before with the original service. We don’t want to break anything. But add in a weekly Global Last-Mile Load Test from Hobbes’ company. What’s it called again? Oh, yeah. Neocortix.”

And Hobbes took Calvin, Wilkins, and Smithers out to Starbucks to celebrate with a hot chocolate and marshmallows.