Laws of nature dictate that measuring developer productivity should be hard

This started as a comment to this LinkedIn post that itself links to Measuring developer productivity? A response to McKinsey article by Gergely&Kent, went over the character limit very fast and turned into a separate post, went over character limit again and become an article.

Full disclosure: I am a theoretical physicist by education and a software developer by profession. My view is obviously biased, and I don't claim to know the full truth. However, I have been thinking about this topic for a long time, and I hope someone could find my idea interesting.

One of the central notions of quantum mechanics is the idea of "quantum experiment". In layman's term it basically means "If we measure a quantum system to know its state, the very act of measurement will irrevocably change the state of the system in an unknown manner, and after performing the measurement we sill not know fully the new state of the system". This idea is roughly a hundred years old, but it still hasn't entered the general use -- I feel that philosophers of XX century were too concentrated on studying history of philosophy and didn't put enough efforts into thinking over and popularizing "modern" philosophical ideas, but this is a different topic. The sad reality is that many people believe all system behave in classic Newtonian way where measurements do not alter the system being measured in any way, we can get the complete understanding of the system given enough efforts, and we can then use the knowledge and measurements to steer the system -- many people believe in that despite very clear evidence of the contrary: Goodhart's law is almost 50 years old by now, it was stated by a economist, but it bears striking resemblance to the idea of the quantum experiment! To quote from Wikipedia:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

It even uses the word "collapse" which is very common in quantum mechanics, "wave function collapses when a measurement is performed"

With that it should be no surprise that attempts on measurements of developer productivity change their performance in an unknown way. To quote from Gergely&Kent article linked at top:

Things went from, “we’d like to know how things are going,” to, “we know even less about how things are going <...>"

is a (somewhat pessimistic) view of the outcome of a quantum experiment I described above. Now, the reasons for this outcome are, of course, completely different fro electrons and humans. Goodhart's law only stated the observed effect, while Gergely&Kent are brutally honest: the <...> from above actually stands for

But they (thing -- MN) are definitely going worse because people are now gaming the system

And indeed people will try to game the system, and smart people -- and normally a company wants their developers to be smart -- will game it smartly.

My main point is not that I know how to measure developers productivity (I do not), but that measurement developer productivity is extremely hard and it is a fundamental law of nature that it is hard, it was one many things that are extremely hard to measure. We are not surprised that no person (to my knowledge) is able to swim across Atlantic without any gear, and we shouldn't be surprised that no company (to my knowledge) is able to measure developer productivity in a good way.

In fact, Gergely&Kent have a couple of points that illustrate that mind-boggling complexity in a indirect way, by assumption. For example, let's take this one:

#2: To compare two investment opportunities. “How can I measure the productivity of teams, and allocate additional headcount to the team that will make the most efficient use of additional headcount?”

To answer this question, you want to compare the return on investment. This is not a question about productivity, it’s about the impact of allocating more headcount to one team or another

So, making a good investment is silently assumed here to be something easier than measuring a developer productivity. But as many people will tell you, investments are anything but easy. If I have an extra $1000 I want to invest, what should I do with it? Buy Bitcoins? Or Tesla stock? Or gold? Ask this question to a person who manages investments, and they'll probably start with "Well, it all really depends on your goals, timelines, risk your are willing to take" and then proceed with a long lecture on investment basics.

In fact, there is anecdotic data that investments are so complicated, that people actually can't do it in a meaningful way, the best strategy is "buy a little bit of everything and see what sticks". See for example https://medium.com/@ellekaplan/why-most-investors-perform-far-worse-than-the-stock-market-1bb19cfea3e0 -- just the first link Google showed me.

Tangentially related is a pet peeve of mine -- higher-level engineers are expected to understand business at least a little bit, but higher-level business are NOT expected to understand engineering. Gergely&Kent silently acknowledge this line of thought by describing how easy it was to understand sales, the underlying assumption is that engineers understand this as well. I could go on, but I'd better stop on this topic, since I am sure my biases color my perception too much here.

The part which really rubs me the wrong way in otherwise extremely smart article Gergely&Kent wrote is how they separate software engineering, for me it has an elitist smell to it, "software engineering is not like other department":

CEOs and CFOs are increasingly frustrated by CTOs throwing up their hands and saying software engineering is too nuanced to measure, when sales teams have individual measurements and quotas to hit, as do recruitment teams in the number of positions to fill.

I don't like that "us and others" juxtaposition. Different department are, or course, different, but it is "sales are different from HR different from software development different from legal different from security" thing, not "software develmpemtn is diffrent from everyone else" thing.

In fact, let's look at the recruitment. So, how do we know a good recruiter?

By the amount of conversations they initiate on LinkedIn? This is no better than measuring line of code for developers, clearly we don't want to reward spammers

By amount of position they filled? By the take it takes them to fill the position? This is better, but then there are much more at play than the recruiter's skill and effort. Lower the compensation twice, and even the best recruiter won't be able to fill a position. Raise compensation ten times, and even a person with no recruiting skill can probably close the vacancy in a day. The metric that claims to be objective is clearly not fair -- and I think everyone agrees that performance metrics should be fare.

This is not all, however! We can easily challenge the objectiveness of the above metric as well: clearly a developer who stayed with a company for ten years and brought a lot a value was a better hire than a person who left after a week. So, if we want to measure a recruiter productivity, we need to measure productivity of the persons the hired, developers included. Ta-dam! What was seen as an easy solved problem -- measuring a recruiter productivity -- actually includes measurement of software developers productivity as one of the steps! Icing on cake is that we only know the full value a hire brought to the company only after they leave, and therefore we can only measure productivity of a recruiter only after people they recruited have left the company. Perhaps recruiter should receive bonuses an long as the persons they hired are with the company, i.e. on a continuous monthly basis, not only once after the recruited someone.

So here is what we get in the end, and this is actually my main point: measuring developer productivity is extremely hard, but measuring productivity of other roles is just as hard, if not harder.

Thank you for listening to my TED talk!