Fostering Creativity and Innovation in Software Products

For quite a while now, I have thought about the role of cultivating creativity (and innovation, its product) as a product manager. I had always felt that how I act as a product manager had an influence on how creative our engineering team could be. And I’ve always believed that great innovation came from certain types of relationships with engineering. This is especially true as the founding idea behind your product has come to fruition and you want to cultivate incremental innovation that keeps your product fresh and ahead of the competition. In such scenarios, you don’t have the luxury of incubating ideas in the CTO office, and you must be much more deliberate in your approach.

Rick Rubin is the recent author of The Creative Act: A Way of Being. As a record producer of an astonishing variety of musical genres, from hip-hop (he co-founded Def Jam) to country to alt-rock he has mastered an approach in which he was not the creator, but had to create the circumstances under which greatness can flourish. I approached this book in hope it would help me sharpen some of my ideas about fostering creativity. The enterprise software world is not the same as the music or other fine arts worlds, so not all ideas from the book are directly relevant, but if you want to foster an innovative culture, the book contains a framework to help structure the process.

Why is Creativity Important?

Let's start out by answering a seemingly obvious question. Why do we care about creativity in the first place? Why can’t we just move from solving one problem to another incrementally, and drive the process via agile methodology?

I think about it this way: your market is paying you to solve a hard problem that it will not solve on its own. Why won’t it solve it? The most interesting reason (as a product manager) is that some problems are too expensive to solve unless you are able to sell the solution to a larger market. Internal solutions are economically infeasible. If the problem is easy to solve, this economic barrier goes away as does the opportunity for a productized solution that commands a premium.

Hard problems in the software world have several characteristics:

• The shape of the solution is not known. Think, for example, of cryptocurrency and its underlying reliance on blockchain. Even if you are a crypto-skeptic, you can appreciate the beauty of the underlying solution.
• The implementation of the solution, even once defined, is not known. In keeping with crypto, this is why there are so many blockchain solutions vying for leadership.
• Investment in a product that has gone to market, and the ongoing innovation required to keep it going, never ends. Think of the amount of R&D that goes into some of the products you have used for decades - from productivity software to databases.

–

“The Object isn’t to make art, it’s to be in that wonderful state which makes art inevitable.” - Robert Henri

Rubin’s quote from Henri reminds us that to be a truly innovative organization, it is not enough to “implement the creative process” for certain projects. Instead, you must cultivate a culture that allows ideas to flourish. A culture in which engineers are constantly context switching, being rushed from one problem to another, is not going to be innovative. Rubin defines one critical aspect of the solution: “we create an open space that allows it. A space so free of the normal overpacked condition of our minds that it functions as a vacuum.”

Some companies like Google pioneered the idea of giving engineers a day a week or so to work on their own projects. I’ve seen that done with some good results, but none that produced groundbreaking ideas. I believe in part that is because one day a week is not enough, with all the itinerant deadlines surrounding that one day, to truly put the mind into that phase where it can be most creative. At the same time, we cannot allow our engineering teams to sit on the beach all day while there are real customer problems to solve.

To me, this all or nothing approach misses the point. Not everyone can be or wants to be an innovator. Many prefer the rush of working the front line of customer problems, or the stability of being told what tasks to work on (perhaps their creativity is not in their work, but in what they do outside of work). It takes all types of people to make a product successful, and so there is no one-size-fits-all solution.? Rubin identifies the Experimenter and the Finisher as two distinct types.

Rubin talks about artists' proclivity to destructive tendencies, self-sabotage, struggles with addiction, or facing obstacles producing and sharing their work. My experience suggests the software engineering world has a different kind of artist who suffers in a unique way. Here are three recent examples of highly creative people I have worked with:

• Software Architect who was responsible for many of the newer innovative ideas on a fairly mature product. He had the proclivity to work feverishly for months on an idea, and then burnout and require time off work.?
• Two creative engineers who burnt out so much they just had to quit despite having a passion for the work.
• Architect who was responsible for one of the major aspects of a recent product I managed. He worked part-time throughout the duration of the project, from brainstorming through ideation and general availability.?

This burnout is, to me, the most unrecognized sign of a truth of the most highly creative and innovative engineers: They are working all the time, whether on the clock or not. This is different from employees who seem to be workaholics: the creative people may seem to shun work, but they are buried in it all the time.

If we want to cultivate innovation and take care of our most innovative people, we must realize that some of their best work, their breakthrough ideas, will occur when doing something else not related to work. While in the shower, taking a walk, driving the car. We must give them room for distraction and seemingly non-productive time. Rubin says “distraction is a strategy in service of the work.” Want to destroy innovation? Start measuring your creative people based on some quantity metric.

Time and Creativity

“Art doesn’t get made on the clock, but it can be completed on the clock.”?

“Time is something that we have no control over. So patience begins with acceptance of natural rhythms. The implied benefit of impatience is to save time by spending up and skipping ahead of those rhythms. Paradoxically, this ends up taking more time and using more energy. It’s wasted effort.

When it comes to the creative process, patience is accepting that the majority of the work we do is out of our control. We can’t force greatness to happen. All we can do is invite it in and await it actively. Not anxiously, as this might scare it off. Simply in a state of continual welcoming.

If there is a rule to creativity that’s less breakable than the others, it’s that the need for patience is ever-present.”

Rubin identifies three phases of the creative process - Seed, Crafting, Completing -? that maps well to how we develop software.?

He suggests the first two-phases not be time bound, but that the last, Completing, having timelines.

The first phase, the Seed phase, is where the fundamental idea germinates. In this phase, from a software perspective, we have an idea of what the problem is we want to solve, and we’ve stated it in clear terms. We may not know how to solve the problem, but we can now know enough about it to determine whether to move it to the Crafting phase. That is, we can now estimate what it would be worth to solve the problem and what direction it would take our product. In other words, we know The WHY.

The second phase, the Crafting phase, is what we would call the prototyping phase. Prototyping takes as long as it takes: until you have an adequate prototype solution, you have no line of sight into what you are building. A prototype doesn’t have to be fully realized in code to count - but it must engender a high-degree of confidence that it will solve the problem and that it can be built. This phase takes as long as it takes. At the end of this phase, we should know WHAT we are building.

Rubin notes the tensions that arise as we seem to end the Crafting phase. Stakeholders may be notified and delivery pressure will mount. But cutting off the Crafting phase too early may leave better options on the table and compromised results. Setting internal deadlines can help move a project along, but be careful sharing those deadlines.

At the end of the Crafting phase, we know the WHAT and the WHY, and we are ready to put a plan in place for the HOW and WHEN. It is this phase, the Completing phase, that so much effort has gone into: from waterfall to agile, the software industry has focused on this phase without thinking too hard about the conditions it needs to get to this phase.?

This is the phase where impatience and timelines are welcome. At this point, we can share project roadmap dates (with their own itinerant uncertainty.Where we scale back and talk about MVPs and releasing early and often.?

Engineers and Motivation

I want to end this article on something near and dear to my heart. I have always defended engineering from executive management who may think that sometimes they “just don’t get it.” I.e. they are not thinking about the business, not thinking about the customers, etc. I always think this cannot be further from the truth,

Engineers must be intrinsically motivated to get their best work. I’ve found it very hard to incentivize engineering through quantifiable metrics. My mantra has been to hire engineers who seem to be passionate about what they do, and provide an environment in which they stay motivated. Rubin calls this continued motivation “energy in the work.”??

As a manager, you have to look out for a flagging of that energy. “Early in a project, excitement is the inner voltmeter to watch to help choose which seed to develop. I made the mistake on a job several years ago of not watching that voltmeter. My own team was telling me they did not understand what we were building, and why, though most of the leadership team thought we had a clear vision for what was being built. This was not a failure of communication. Instead it was a failure to listen to our most creative engineers that what we were doing was not lighting the spark of creativity, and that the effort was not worthwhile.?

–

Rick Rubin’s book is some 400 pages along, and eminently readable. Many of his ideas need adaptation for the software workplace - art has no right and wrong, and can thrive from quirks, whereas software products require elegance that enable mental models and simplicity of operation. His book is intended for the painter, the songwriter, the choreographer, not the record label. But the advice he gives best be understood by the project underwriters.