The Top Five Myths of Software Engineering Productivity

Myths are public dreams, dreams are private myths. (Joseph Campbell)

Introduction

Throughout history, mythology has played a profound role - both beneficial and detrimental - in the development and evolution of human cultures. Myths give communities a sense of shared identity, collective origin, and common destiny. Whether true or false (and most are, in fact, historically false), myths often bind communities together, for better or for worse.

One realm, however, where myths and fables decidedly do not belong, is that of hard science and logic. Here the only thing that counts is what is empirically and demonstrably factual. Great minds like Galileo and Copernicus experienced firsthand the suppressive power of the prevailing religious mythologies of their respective eras. Today's "great minds" likewise encounter a sea of misinformation that prevents society from maximally benefiting from productive efforts in their respective arenas.

And one of those arenas is the field of software engineering.

Productivity

Everybody in today's workforce, it seems, wants to be "productive" - meaning, typically, that they want to generate maximal value for both their organizations and their clients as quickly as is practical (in regards to long-term sustainability) and at as low a cost (both in terms of time and money) as possible. Few would deny this tenet. This is of course not to minimize the importance of intangibles, such as work-life balance and a concern for both the environment and the greater good of society at large.

But what if the very practices that businesses adhere to work against them not only in terms of the "intangibles" but also in terms of actual, measurable, productive output? Herein dwell the myths of software engineering productivity.

Myth 1: Humans are Capable of Multitasking

Browse any job-seeker site and you will encounter a slew of role descriptors such as "Candidate must be able to multitask in a fast-paced cutting-edge environment." And yet, the idea that such a thing is possible at all has been debunked time and time again by the developed world's most prestigious institutions. The brain simply isn't designed that way. Attempting do so not only ruins your productivity but incurs a significant penalty to both mental and physical health.

What actually happens when we try to multitask is that the brain kicks off a series of context switches. We think we're multitasking, but we aren't. But is context switching any less detrimental to productivity and health? Let's explore that next.

Myth 2: Workers are Good at Rapidly Switching Contexts

Many corporate types, if pressed on the issue, will admit that of course I know that multitasking is impossible. But we are good at context switching, right?

Alas, this is not always the case. While this is true for relatively simple activities like washing dishes or shopping in a supermarket, highly-creative knowledge work of the type performed by software engineers belongs to an entirely separate category. Consider these observations made by the authors of the Reclaim.ai article above:

• 97.5% of people cannot multitask effectively.
• The average person is interrupted 31.6 times/day.
• 20% of cognitive capacity is lost when context switching.
• It takes over 20 minutes to get back on track with a task after being interrupted.?
• At least 45% of people are less productive while context switching.
• The average professional attends 25.6 meetings/week, causing them to switch context 5.1 times/day.

Modern software engineers (and, often more saliently, their managers) worship on the altar of tools such as Slack and Microsoft Teams. Such tools allow real-time interaction and sharing between department members, particularly in the case of a remote and widely-distributed workplace. But the continual slew of popups and "mentions" often serves to interrupt the flow state that is essential to individual-contributor productivity and wellbeing.

Working on a cognitively-demanding task for 15 minutes, being interrupted, and then resuming the task afterwards for another 15 minutes is not the same as working on that task for a straight 30 minutes. In fact, it takes over 20 minutes to regain one's focus after such an interruption! Thus, even if you are one of those hardnosed, bottom-line-centric management executives unconcerned with your employees' happiness, you might want to ask yourself if you are willing to sacrifice 20 minutes worth of monetizable productivity - per employee, every day, multiple times per day, and across your entire engineering workforce.

No matter how oblivious one is towards worker wellbeing, nobody is happy when money is wasted and profits are down (especially not the previously-mentioned "bottom-line-centric management executives"). Ensuring minimal interruption and maximal flow-state time is a simple way to keep both your employees and your stakeholders happy.

For this reason, I encourage software engineers and other knowledge workers to periodically log off their communications apps when the need to focus is paramount - and for their managers to be one hundred percent supportive of this.

Myth 3: More Pressure Equals More Productivity

Returning again to your typical technology job-board listing, you will notice plenty of copy extolling an employer's "Fast-paced, high-pressure" work environment.

This has been the norm for quite some time now, particularly with startups. The underlying assumption is that there is a consistent, linear, and ever-increasing relationship between the amount of pressure an organization puts on their employees and the amount of valuable work product they output.

But this is not what empirical science shows - at least when it comes to cognitively-demanding or highly-creative tasks. Instead, the Yerkes-Dodson law posits a Gaussian ("bell curve") relationship between pressure and productivity. In other words, some degree of pressure (or "arousal" as the linked Wikipedia article calls it) is helpful up to a point - after which work performance begins to rapidly decline.

The site WorldOfWork.io explains this more succinctly in this article, and even includes a helpful graphic:

There are, of course, variations between individuals. Some workers seem to thrive when the heat is cranked up, whereas others crumble when exposed to the same degree of stress. But the overall relationship still applies (nearly everyone needs some urging at times, and nearly everyone has a "breaking point"). An organization should strive to create an environment where the stress level falls within a few standard deviations of the curve's peak, and management should be continually aware of both employee morale and performance as they relate to the environment.

The "pressure equals performance" mindset is an anachronism from the days of repetitive warehouse, factory, assembly-line work, where the activities do not involve a great deal of creativity or mathematical analysis. In such environments, the Yerkes-Dodson relationship does apply so much.

But in the realm of software engineering and similar professions, it is best to heed the words of the World of Work authors:

The relationship between performance and pressure follows a bell curve. It shows that individuals have an optimum level of pressure under which they perform at their highest levels. If pressure is above or below this, their performance falls. At this level they have the highest chance of being in “flow”

Myth 4: Frequent Meetings and Calls Ensure Productivity

Meetings and videoconferences have their place. They are the most helpful when they arise organically in the context of specific topics - especially technical or interpersonal dependencies - among members of the team and/or their business partners.

But in many cases, they are overemphasized and can cause interruptions in flow (as touched upon in the Myth 2: Workers are Good at Rapidly Switching Contexts section above).

Another reason for keeping meetings in check is the the idea of asynchronicity.

The beauty of software engineering (and technical/creative work in general) is that it can be performed in an asynchronous manner. In other words:

1. Different engineers can be working on different things in parallel with one another at the same time without interfering with one another's work. Jane can work on the web application while Jim types up the documentation while June configures the hosting and deplyment environments in the cloud.
2. A worker isn't bound by space and time. They can choose to work on a task at 5:00 AM, or 1:00 PM, or midnight, or whenever (or wherever for that matter: at their desk, in a coffee shop, on a train) - as long as they accomplish their work items in a timely fashion.

Calling a meeting necessarily breaks this asynchronicity. Now, everyone must stop what they're doing and put themselves in the same space (virtual or otherwise) at the same time. They have to become synchronous, and remain in sync for a set period of time.

I am not suggesting an end to meetings, or to the idea of synching up from time to time. It is absolutely necessary at times - not just for resolving urgent dependencies or putting out fires, but for cultivating camaderie and team morale. Rather, I am suggesting that engineering teams be judicious with regards to scheduling meetings, being sensitive to the time constraints of each member. Before setting up a call, ask yourself the following:

1. How urgent is the issue to be discussed?
2. Who absolutely needs to be in this conference?
3. What are the intended outcomes of the meeting?
4. How will this meeting affect the team's morale?
5. Can this be handled simply by an email?

For more reading on this topic, I suggest reading Maker's Schedule, Manager's Schedule by Paul Graham.

Myth 5: Hiring More People will Make us More Productive.

It is necessary at times, of course, to onboard more staff members in order to accomplish the objectives of a growing business. This is particularly the case when it comes to long-term strategic objectives. An understaffed team will likely suffer the pressure/performance dilemma described in the Myth 3 section.

But one must also consider the challenges associated with engineer onboarding (a pain point for engineers and managers alike in many organizations). Even after the formal onboarding process, new engineers often require a lot of time to get up to speed (not because of their lack of aptitude, but because every software-centric company has wildly divergent and often non-canonical ways of doing things).

In the long term, hiring qualified software developers will increase a team's morale and productivity. But one must be prepared for an inevitable and often significant drop in short-term output. More importantly, one cannot expect a linear, proportionate improvement in productivity. In other words, doubling the number of software engineers on a team is unlikely to halve the development time - and might actually lengthen it.

Fred Brook's seminal tome The Mythical Man Month should be required reading for anyone serious about becoming a software engineer or an engineering manager. He discusses this and several other topics with a degree of eloquence and detail that I can only hope to aspire for.

Afterword: Why do Engineering Myths Persist?

In most cases, software engineering myths persist for the same reason that all myths persist.

Studying the writings of Joseph Campbell may prove helpful here. In short, narratives which are passed on from generation to generation tend to inhere in the collective consciousness of the descendants of the cultures from whence they originated. This inherence occurs at an emotional and visceral (rather than a logical or intellectual) level. Even after advances in science and human psychology have invalidated them, it's hard to let go of storylines that civilizations have told themselves fo eons - storylines that wrap life's complexities into a neat little black-and-white box with a ribbon on top.

In the business world, this takes the form of, "That's just the way things are. That's the way we've always done it. That's how all companies do it." Or, perhaps, "It's just the nature of the beast".

But it need not be so. Societies (and companies) benefit when they periodically question and re-examine the usefulness and the factual basis of their long-cherished traditions.

Changing the way we think about anything (software engineering included) - let alone changing our day-to-day behavior - can be a scary proposition. Few want to risk rocking the boat and upsetting the status quo. But in the interest of maturing the oftentimes Wild West-like field of software engineering into a respectable, sustainable, and manageable profession - the effort is worth it.