Looking for a Lighthouse in a Sea of Complexity

I've recently been reflecting on the dynamics between data science, engineering, and product teams and how these dynamics shape the destiny of a project (short-term or long-term) and by extension, the company.?

Some of these reflections have refracted internally and those shards of insights have encouraged me to write this post regarding this symbiotic dance between teams- the dance of vision, skill, and execution.

<<prelude>>

The captain has iteratively optimized the route, played out "what if" scenarios, and installed a special device to get insights on the headwind (we expect a lot of those this quarter; oops, I mean en route to this destination). The ship is sailing, sometimes in uncharted territories- sometimes discovering new opportunities. There is a lot that happens in the quarterdeck but what I wanted to discuss is the engine department.

?

<<start of hypothetical scenario>>

Let's suppose my team's job is to deliver coal to a boiler. Although our main engine is run on diesel, we have this secondary engine that runs on coal. One day we get our hands on this coal substitute that we want to experiment with. This substitute is proven to be environmentally friendly and more efficient than coal.?

We are excited. We throw some in our mockup engine and ignite- the engine purrs and delivers. We alter the substitute a few ways, fiddle with the engine knobs a few ways, and experiment for a few days. There are one or two scenarios where it is not as efficient but overall, the results are promising. We summarize our findings in a report and hand it to our manager.?

We are asked to start replacing coal with this new substitute. We want to do this gradually. Start with a 90-10 ratio and depending on the results, gradually move to 10-90 and then 100%.?

We have a shaft system to deliver coal. We don't want to build another automated system, so we decide to shovel the alternative manually. This creates a manpower resource problem. As a temporary solution, some of the other work gets deprioritized so that the team can take turns shoveling the alternative.?

When the ratio reaches 60-40 (in two months), two issues emerge that we had not anticipated. The amount of shoveling is just too tedious, time-consuming, and uninspiring, and the team is overloaded. More alarmingly, there are issues with the engine itself. The vent system is at capacity most of the time. We need help from the engineering.

The vent system expert engineer gives our deck a visit- the good news: she knows what the problem is and how to solve it; the bad news: she is at capacity and won't be able to work on it for another 3 months.?

We, the coal delivery team, set up a brainstorming session and conclude that we should try to fix the vent system ourselves. One week max, a coworker who has some experience with the vent system, promises.?

After a month of spending knee-deep in the vent system and implementing modifications, we discover that the underlying problem is on the piston system. Nobody understands what the exact problem is but all we know is it has to do with the intake and exhaust parts of the piston.?

Fixing this issue would mean that we would have to stop the coal engine for a few days. This would create issues with the fishing department as they heavily rely on the power generated by the coal engine.?

The coal delivery department is at a crossroads. The vent system has already been modified so rolling back to coal-only system is not trivial. Moving forward with the alternative system is an uphill battle.

<<end of hypothetical scenario>>

The above scenario has become very common in organizations with data science or machine learning teams. It is easy to say: collaboration is the key, don't work in silos, triangulate with product and engineering frequently, etc. But when you are in the engine room, surrounded by shelves filled with ideas that were never materialized, models that were never deployed, and projects that were never productized, the reality feels completely different.?

In the above scenario, "We are asked to start replacing coal with this new substitute" launched this mini-ship in the wrong direction. From the start itself, it should have been a collaborative effort between teams (coal delivery, engineering, and fishing). Goals should have been defined and responsibilities divided.?

?

But in a world where every department has its own goals and KPIs, I wonder who should be responsible for orchestrating this symbiotic dance between teams- the dance of vision, skill, and execution??

Also, what is the responsibility of an individual contributor in harmonizing this dance? Personally, I feel, that when asked to "jump", rather than asking "how high?", I think we should ask "why?".?

Thank you for taking the time to read. I’d love to hear your experiences and thoughts on this subject.