Grab the Nearest Weapon

R&D teams are like gladiators – win the fight quickly and you earn the privilege to fight another day. The probability of injury or death increases as the fight drags on. In this entry I write about winning the day with efficiency, speed, and resourcefulness. 

A popular show from the 1980’s in the United States was “MacGyver.” The protagonist, Angus MacGyver, was a secret agent who repeatedly found himself in impossible situations. His only way out was to quickly combining ordinary items within reach (e.g., a gum wrapper and a battery) to create an instrument of escape. I loved this show because it resonated with my own innovation philosophy – create new things by quickly combining ordinary things in novel ways. The contemporary equivalent could be Matt Damon in The Martian.

For the better part of the last 20 years, I have had the privilege of working with a colleague who is a technology MacGyver as well as a mentor and friend. Repeatedly I watched him run headlong in to seemingly impossible business problems. He quickly pulls together whatever technology is within reach to save the day (most of the time). He doesn’t care what technology he uses, only that it will end the fight as quickly as possible. Most remarkable is his ability to expend minimal energy to solve problems quickly. He knows time is of the essence. This person recently combined Hadoop with -

• 300 lines of Java code
• Some Excel spreadsheets
• A 10-year-old Java technology

He weaponized a technology lost to the sands of time by combining it with contemporary technologies and boring spreadsheets creating a mushroom cloud over a tough problem. I literally laughed until tears streamed down my face when I reviewed it because it was so ingenious and simple.

Another technology MacGyver colleague of mine recently drove me to tears again. Over 6 weeks he created a solution to a problem that had bedeviled his company for 15 years. He did it with a program written in Scala that ingested a JSON specification to generate a wildly complicated ternary operator in Java! The ternary operator is one of the oldest computer language constructs dating back to the early 60’s. In one fell swoop he quickly combined technologies spanning over 50 years to blow everyone’s hair back. I goaded him to generate that ternary operator in C for fun. But, that would have involved JNI increasing the energy required to solve the problem – how gauche.

Show these solutions to other engineers and they may turn their nose up at them because it does not use [technology buzzword salad]. Lost is the most important lesson for any R&D engineer – speed is life for R&D. Investors don’t care about the means; they just care about a return, which will of course degrade over time.

Imagine you are a gladiator in the ring. If your competitor is rushing you, who cares if you neutralize them with a laser or a rock? You have more competitors than you know about. Those competitors are faster than you think. Make no mistake; they want you out of the market completely. They don’t care how they win your market so why should you care how you win theirs? The longer the fight endures the greater the chances for injury and death. R&D is a form of asymmetric warfare so grab the closest weapon, rock or laser, and stop them in their tracks before they reach you.

Something I’ve noticed over the years is a deep-seated phobia of the technology equivalents of using a shiv over a laser. A common way to die in R&D battle is screaming “technical debt” as an excuse for not whipping up a shiv. Technical debt is a metaphor for introducing potentially hard to maintain solutions to gain short-term speed in exchange for long-term drag. Think of it as a loan where the interest payment is the additional time required to maintain a sub-optimal solution.

I recently had the opportunity to watch a horse race between two teams to produce two solutions to the same problem. The team using contemporary technologies was slowed by their fear of “technical debt” and distaste for “old” technologies. Meanwhile another team with a greater desire for survival delivered a solution for a win. The contemporary team may be faster in the long run but they didn't win the market in the moment.

Contemporary development teams move at far higher rates of speed and efficiency than their historical brethren. If a team can rewrite the entire solution in days or weeks (or even months) then deliver now to earn the opportunity to refactor later. Arguing about the right way to do something is a privilege for the living. Not repaying that refactoring loan is gambling your life on that shiv.

I have also seen this phenomenon happen in reverse. I once witnessed an engineer quickly create an arguably over-engineered (but elegant) solution that worked. A heated debate then ensued over the complexity of the solution. One side argued for a reductionist approach while the other side argued for a feature rich approach. In other words, one engineer produced a laser but the other engineer wanted a rock. Meanwhile I am thinking, “The guy that knows how to use the laser is standing there! Just pick up the [colorful explicative] laser and shoot at the [colorful explicative] problem!”

In fairness, “technology MacGyver’s” will often produce solutions designed for a single use to get out of a jam. The vernacular in development communities is a "hack."  Try to use the solution again and you may get a puff of smoke and a sticky end. This is a regular debate with my DevOps mentor. He regularly coaches me on the perils of deploying in to production a solution held together with duct tape and bailing wire. He is right, of course. It is all fun and games until your weapon crumbles in your hands the moment your enemy attacks.

The other extreme is also a dangerous proposition. The more sophisticated a solution the greater the chances for deployment error. The solution that hit 1.0 last week with limited deployment is less proven than the one that hit 1.0 five years ago with broad deployment. My DevOps mentor will regularly argue for more using older, older technologies because they are well understood, reliable, and safe. Why deploy a hydrogen bomb when a firecracker will do? This creates constant tension between engineers that want to use the latest thing from Hacker News and those that have keep a technology deployed and alive.

Balancing this tension requires an appreciation for the context. Are you building a solution to post cat videos or to keep a plane in the air? Greater mission criticality should push the needle towards conservative technologies. Alternatively, lesser mission criticality should push the needle towards progressive technologies. The productive team will recognize and use the context to their advantage to stay simultaneously fresh and grounded.

The unproductive team will suffer from a neurosis that the weapon you have in your hand will be obsolete tomorrow. This “fear of better options,” or FOBO, is a scourge on contemporary R&D. It slows teams down as they waste time thrashing over options. This new reality requires a mastery of API definitions that is difficult to find in the contemporary development community. The resourceful engineer will quickly define an API for protection and then plug in whatever is handy to survive. A mastery of API’s enables you to use the old while waiting for the new. 

So what differentiates someone who wants to win quickly regardless of how from those who would rather die in battle than use the “wrong” weapon? One key is a visceral appreciation for utter defeat. Over the last 30+ years I barely survived the .com crash, disastrous company strategies, epically bad fads, and egomaniacal bosses. The colleagues to whom I referred above survived worse. Losing everything can replace pride with humility in the service of survival. The greatest innovators of our time know this lesson well. Yet, this is the hardest lesson to teach those who have not experienced the agony of defeat first hand. 

Another key is the ability to see the constituent components of a problem. At the very core of a computing problem, no matter how exotic, you will typically find the usual suspects – databases, threading models, clustering, rules, workflow, ETL, time honored algorithms, etc. Sure there are exotic new technologies that pack a huge punch, but grab all the boring stuff first because doing so will introduce fewer risks and free up time to focus on the hard bits. This skill often comes with age but can also come from exposure and always requires humility.

Challenging is watching imminent failure in the face of urgent counsel. I want to scream when I witness some academic argument over which weapon to choose to deflect an impending blow to the head. However, yelling will not help so I am left to watch and wince (“not the face!”). Strong-arm the team to your will and they will not learn and likely repeat the failure. Helping them grow requires nuance and patience –

• Accelerate failures the best you can
• Help them learn to support one another through the process
• Team health is paramount so provide counsel at opportune moments to maintain relationships
• Orchestrate quick failures when necessary to grow the team in a safe environment. Navy Seals do not learn hand-to-hand combat from textbooks and they are not afraid to use a rock or a laser.

As painful as they are, grievous (but survivable) injuries can create MacGyver’s. Appreciating that pain sets you free to grab the nearest weapon the next time you need to move quickly. The worst part is waiting to see if your gamble on a lesson learned pays off.

A word of caution – drawn out systemic failure can cause irreparable team damage. This often happens when a minority in the team is able to hold bad decisions in place for too long and against ardent disagreements with the broader team. General consensus on a path forward is more important than an individual’s passion for a disagreeable approach regardless of their technical purity petard. This kind of organizational engineering is often, and unfortunately, lost on technologists. They often don’t recognize that team strength is so much more valuable than any piece of technology. Damage to relationships can cause morale to tank and kick off a destructive spiral that can be difficult to reverse after which no technology will save the day.

So what about business problems that are inherently intractable and will require a long time to solve? In that case break the problem down in to valuable increments that can deliver faster wins and still drive the larger strategy as I describe in my previous post, “Pave the path to your Big Idea with value.” 

So, go break a leg and do it quickly. Then pick up the nearest weapon and go on the offense.