The Design of Systems of Innovation, Part II: Uncertainty and Constraints

This article is a continuation of "The Design of Systems of Innovation," article published last week discussing the context, key factors, constraints, impediments, and concerns of both culture and leadership requisite for intentionally designing and managing systems of innovation in large enterprises. - WE

“If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible.” – Friedrich August von Hayek

Uncertainty in Complex, Turbulent Environments

Intentionally designing and managing a system of innovation is a leadership capability that requires a different set of perspectives, skills, and behaviors than managing a system of delivery. A key factor of success, as it deals with several impediments raised during the new product development process, is the appropriate governance and decision rights for dealing with high levels of complexity and uncertainty. While management in general involves coping with uncertainty, creating controls and processes to reduce uncertainty, is among the raisons d'être of innovation managers. Through the proper application of context-specific constraints, innovation leaders can manage the increased uncertainty of exploration and experimentation efforts to improve the system and structures themselves to deliver better outcomes.

The management of the innovation processes also requires effective collaboration between different functions, perspectives, and interests. Managing innovation projects often involves a complex series of interactions among different subsystems and sub-cultures as well as the use of various portfolio strategies across different time horizons. When it comes to managing innovation initiatives, three important issues should primarily be taken into account. The first is related to the high amount of uncertainty that inherently exists when exploring the unknowns or third horizon space. Hence, managing uncertainty (lack of perfect knowledge in a domain) is a central feature of new product development as it affects innovation outcomes. Second, maintaining a sustainable competitive advantage not only requires the achievement of higher organizational effectiveness but also the enhancement of organizational learning. As a result, promoting resource allocation efficiency and improving employee’s knowledge, skills and capabilities are two interdependent goals of new innovation efforts. Third, creating new value, and new value propositions for customers should be a key objective of any innovation initiative and, therefore, sensing nascent gaps in new markets, identifying customer needs, and emergent trend patterns through human sensor networks with rapid feedback cycles will enable higher probabilities of innovation success.

Managing Different Aspects of Innovation Uncertainty

Innovation activities inherently entail higher levels of uncertainty and complexity, as developing new ideas and service offerings is often an unstructured and fuzzy process which requires creativity and divergent thinking. A perfectly predictable system produces no novelty except through external perturbations and black swan events. Uncertainty results from the fact that, on the one hand, all the necessary information is rarely available or expensive to acquire, and, on the other, knowledge about the future is always incomplete.

Different types of uncertainty may exist during the innovation process (than the production process) such as technological uncertainty, market uncertainty, regulatory uncertainty, managerial uncertainty, and social-political uncertainty. Managing through these different classes of uncertainty remains a key responsibility in innovation leadership in order to improve the chances for success. The most important aspects of uncertainty include:

• Market Uncertainty (Market/Customer Segment): The idea of innovation implies that it serves the needs of a specific target market. The market environment for innovation includes the needs of customers, the actions of competitors, the prices of substitute offerings, and consumer taste influenced at times by global social media. The uncertainty regarding the shape of demand for the innovative offering, the unknown consumer behavior, and the inchoate customer needs are the main sources of market uncertainty.
• Technological Uncertainty (Customer Problem/Solution): Innovation teams encounter technological uncertainty, both in product design and production execution. The uncertainty related to product design is dependent on the newness of the technology while product development uncertainty refers to a diverse collection of processes, techniques, and know-how required to create products and services. Technology produces uncertainty associated with the skills and knowledge required to succeed in using new technology to create new value, but also technical dependency uncertainty between different IT systems.
• Managerial Uncertainty: Managing innovation differs from managing routine, and standardized work. Routine (predictable) processes usually imply high level of standardization and stability whereas innovations requires lighter constraints, unstructured tasks, and abductive or designerly ways of thinking. In practice, there can be managerial uncertainties about the staffing the innovation team, the required resources and competencies, the management of relationships and dependencies with the rest of the organization, and co-operation or competition with other teams vying for a limited funding pool. Therefore, it can be argued that managing the process of innovation challenges the traditional way of thinking and requires effective leadership on behalf of the program team.
• Socio-Political Uncertainty: The innovation process does not occur in a vacuum, but instead is rooted in the social interactions among different stakeholders and innovation actors with different cultural backgrounds, various viewpoints and often conflicting priorities. These interactions are a significant source of uncertainty, as the diversity of interests amongst members of an organization are revealed. Equally important, most decisions relating to the development of new products take place under high levels of ambiguity. Although decisions can be improved with more effective information exchange and use, they are often influenced by political antagonism, conflict over budget allocation and subjective value judgements between different functions or departmental leaders maximizing their own probabilities of promotion within their spheres of control and influence. Therefore, managing innovation projects is often beholden to political antagonisms and may prove a significant challenge to overcome.
• Regulatory Uncertainty: Regulations play an important role in innovations. An unclear regulatory environment creates fields of opportunity in which entrepreneurs can create their own rules before governments are able to apply governing constraints. On the other hand, a highly regulated environment (for instance housing development in San Francisco) consists of laws and policies that have been developed in order to constrain and/or enable certain types of investment activities. Constraining regulations are typical in contexts like healthcare and finance. Enabling constraints refer to governance patterns that supports or directs the trajectory of investment and innovation processes themselves. One type of uncertainty in the innovation process relates to the issue of whether the innovation qualifies for intellectual property protection, such as a patent or trademark. On the other hand, those same governance patterns may have unintended effects on the innovation process, as changes in regulations may be seen as factors that increase environmental complexity and turbulence.

Enable Sustainable Innovation through Constraints

Many successful innovators have long recognized that constraints can both spur and guide innovation. Attempting to innovate, that is, create novel options to existing or emergent gaps, or the intentional creation of new gaps in the market or value networks. Without processes or boundaries, innovation teams become overwhelmed with options without a framework for transitioning from divergent thinking (generating options) to making choices.

Creating options is inherently generative, expansive, unbounded, while making choices is convergent. Without guidelines, constraints, and boundaries to structure the interactions or transaction points, innovation teams struggle to coordinate their innovative activities, generate testable solution prototypes, or align the activities to strategic objectives through convergence which is dependent on choice architectures. In such cases, projects funded to deliver innovations can seem random or incoherent and rarely produce sustainable competitive advantage.

How, then, can organizations embrace a more structured approach to innovation?

One productive approach is to apply a few simple constraints (rules or guidelines) to key steps or transaction points in the innovation process as well as to the boundaries of the system (or the boundaries between sub-systems). Simple rules and constraints act together to reduce or absorb complexity and uncertainty. One example of a constraint is a time-box sprint. Simple constraints add just enough structure to help organizations avoid the stifling bureaucracy of too many policies (or the wrong policies!) and the chaos of none at all. It is through the application of these constraints that meaningful interactions or value exchanges can occur.

By imposing enabling or governing constraints on themselves, individuals, teams, and organizations can catalyze creativity and then channel it along the desired trajectory coherent with the organizations strategy. Instead of trying to think-make-check inside the wrong box, you can use simple rules to draw the right boundaries for new boxes within which to explore potentially exploitable options.

"Constraints can either be governing or enabling. Governing constraints hinder actors to do something or only allow them to do it in a certain way. Enabling constraints make it possible for actors to do something that would not be possible otherwise (Juarrero, 1999). An example of a governing constraint would be a law that prevents companies from colluding, while an example of an enabling constraint would be legislation that enables people to establish companies which have certain rights and privileges. Governing constraints can also be physical, like walls or fences that prevent people from going somewhere; or they can be social like norms and taboos. An enabling constraint is for example kinship, as it enables humans to trust each other by binding them together." – Marcus Jenal, Constraints and Emergence

Simple rules cannot, of course, guarantee successful innovation—certainly not under conditions of complexity and ambiguity. Innovation activities that create novel products, processes, or business models which generate economic value either through organic growth or operational cost reduction are inherently uncertain. Trying anything new inevitably entails experimentation and failure. Simple enabling constraints, however, add discipline to the process to boost efficacy and increase the probability that the resulting innovations may generate sustainable value.

Enabling constraints are most commonly applied to the sustaining, incremental kinds of innovation, often viewed as less important than major breakthroughs (certainly less splashy!). The current fascination with disruption obscures an important reality. For many established companies, incremental product improvements, advances in existing business models, and moves into adjacent possible markets remain critical sources of value-creation pulling the organization into the near future.

What matters from a systems perspective, is the appropriate application of enabling constraints also at the portfolio level to fund innovation activities across different components of the system including not just new product discovery, but also on the value propositions, value streams, and value chains. For example, the turnaround of toymaker LEGO over the past decade, has depended at least as much on restructuring the core systems of value exchange as it has on the introduction of customer-centric agile and design thinking practices into the company’s new-product development value stream. All three were necessary to realize new outcomes from the system. The whole was greater than the sum of any individual improvement, and all three needed to be coherent with the strategic intent of the organization. 

Balancing Act of Governing and Enabling Constraints

Within organizations there is a fundamental tension between the need for stability (through the application of governing constraints) in core operations, and the need for explorations of unknowns with high degrees of uncertainty requiring a different set of policies to guide action. On the one hand, enterprises are required to maintain some levels of localized stability and predictability as well as to develop standardized processes with the aim of accomplishing daily tasks in a more efficient and effective manner. For example, the processing of credit cards by transactional firms every day or the maintenance of supply chains demand high levels of compliance, efficiency, and predictability. On the other hand, new product development needs lighter constraints and a focus on multiple, concurrent, often contradictory bets with different types of objectives and measures. An organization’s success and survival then require context-dependent governance policies for different parts of the organization as well as different constraints on the interactions between the various operating models if they are going to work well together as one value network. However, it’s not just managing multiple operating models that is the challenge, but continuously managing the interactions and boundaries between these different systems and cultures. 

Due to limited resources and other intra-organizational contingencies, serving incompatible priorities such as aiming at both cost efficiency as well as enhancing the organization’s knowledge base through experimentation is even more challenging. Intentional coordination and orchestration across the boundaries between the different subsystems leveraging different operating models while ensuring proper diffusion of information and know-how necessitates leadership presence and attention. I worked with one organization that failed to see the conflict arising from two competing strategic initiatives, an agile transformation positioned as part of a growth strategy, and a cost-cutting initiative ironically or not named the “Lean Agile Program.” The latter succeeded (in the short term) while the former became a battle of hearts and minds yielding little measurable, sustainable impact while costing the organization millions. One order of concern higher than these two programs, was the failure by leadership to address the black elephant in the room – their business model faced existential threat on multiple fronts and would not survive the decade, and their strategy developed by another Big 4 consulting firm failed to address the black elephant in the room.

The concurrent achievement of conflicting goals poses a fundamental challenge for enterprise innovation today. Without the emphasis on cost control, a firm’s expenses might simply spiral upwards and the firms’ products and services would become uncompetitive. On the other hand, organizational learning through innovation activities is a basis for gaining a sustainable competitive advantage. Firms that are able to learn stand a better chance of sensing events and trends in the marketplace and feeding those insights into the investment decision-making processes. As a consequence, learning organizations are usually more flexible and faster to respond to new challenges which enables them to maintain long-term competitive advantage. For example, think of any company from P&G to Ethicon (a medical device innovator owned by Johnson and Johnson). They have to ensure that their products are carefully designed and manufactured to precise specifications and that they are delivered to consumers in a consistent way. Additionally, if companies do not make room for creativity and innovation it is very likely that their competitors will out-perform them, or they will be disrupted by nimble startups with more compelling offerings.

Hence, here is the dilemma:

How do organizations optimize for resource efficiency on the one hand and also expand their innovation, and therefore learning capabilities on the other?

As usual with dilemmas the answer is difficult and has to do with balancing activities and strategic orientations through the requisite application of enabling constraints, and intentional management of the interactions between the various systems. The firm needs to ensure there is a constant pressure to improve effectiveness in its operations (standardizing what can or must be standardized) while at the same time creating the space with well-defined boundaries for new product development and nurture a learning orientation through continuous experimentation.

Companies that have the capabilities to achieve an optimal mix between these different orientations such as exploration and exploitation, or novelty and predictability, tend then to actively manage the transitions between continuous experimentation and continuous improvement across different sub-systems though effective the use of contextual constraints. This then requires more interactions by cross-functional leadership teams with greater interaction frequencies to reduce and absorb the increased complexity of managing across the various systems. It’s not just greater interactions between the innovation teams, or between innovation and delivery teams. It also requires greater interactions between these different constituencies and customers. Too often it is the case that agile teams, innovation teams, and even leadership teams are so busy creating alignment, making decisions, and managing the system that they don’t create the space to listen to customers and engage with the market.

Customer Co-Creation and Enabling Constraints

 "You've got to start with the customer experience and work back toward the technology - not the other way around." - Steve Jobs

During the previous decades, companies were mostly focused on productivity gains through economies of scale or economies of scope (transitioning tasks within their value stream from the firm to the consumer) and investing in higher-order products and services. Historically, innovation was a job exclusively for the research and development department of an organization and competitive advantage was protected by high barriers to entry. Researchers and engineers may have been inspired or triggered by new technology without having adequate access to complete knowledge of the external market, contexts, customers, and cultures. The need for exploring customer preferences and emerging trends as well as the introduction of online platforms has led to a wider range of interaction possibilities between customers, producers, and innovators. These new platforms have reduced the cost of knowledge diffusion across the firm, and between the firm and the external environment.

A shift in the early 90s to an active co-evolving collaboration with customers in the innovation process began as customers became key partners in the value discovery process. New innovation methods, including the emergence of design thinking, lean startup, and agile methods acknowledges the importance of integrating directed customer research and feedback during new product development. These methods rely increasingly on the contributions of customer’s ideas, insights, and knowledge particularly during the earlier innovation stages. These methods have failed to produce results in situations where new practices were not combined with tight feedback loops with the customer and a change in the constraints preventing constant, iterative interactions with customers. Two week sprints may allow organization to ship product faster, but if it's not combined with customer interactions and feedback loops to validate that the right product is being shipped, its waste.

The inclusion of customers in new product development is becoming a trend for many organizations and is often referred to as “customer co-creation” or “customer centricity”. Organizations aim to discover assumptions, needs, preferences, and challenges or limitations in the customer experience that may help them develop and commercialize new service concepts aligning the “things which are built” with validated customer problems to be solved, or “jobs-to-be-done.” Co-creating new products with customers’ participation may provide companies with a competitive advantage since not only are they able to meet needs more successfully but also may enhance customers’ perceived value by establishing them as co-creators of the new service offering.

This process of co-creation with tight feedback loops between innovation teams and customers may yield repeatable patterns of incremental innovation in the near-term contingent on the relative pace of change in the market. As innovation teams look out across longer time horizons, uncertainty increases requiring a shift to more generative activities for producing more options and hedges to run as concurrent experiments until patterns of potential value emerge and can be tested. This too requires the easing of constraints which govern what kinds of problems to solve for which customers. What is often lacking, however, is a commitment at the executive level to increased customer centricity to guide strategy and policy decisions – it’s not just innovation teams that need to “get out of the building”. Valuable insights about emergent trends, customer needs, and shifting markets rarely come from the boardroom or investor relations call, which is why it’s equally important for executives to get out of the conference room if leaders want to lead and guide innovation activities. 

Integration Impediments and Structural Obstacles

“A company's success no longer depends primarily on its ability to raise investment capital. Success depends on the ability of its people to learn together and produce new ideas.” - Arie de Geus

The existing structures and policies of an organization can clearly influence the way new products and services are developed. First, as innovation success requires high cooperation among different competing functions, the extent that cross-functional discovery teams are used is positively associated with innovation success. Second, given the dominant role of technology systems for monitoring customer feedback and ensuring effective information exchange, integrating IT systems during exploration and development efforts to increase shared understanding and communication also significantly contribute to innovation success. Third, the organization’s power structure affects innovation activities, as various structural characteristics may enable, amplify, or inhibit creativity. One consistent area of concern is the role that shared IT services has on innovation activities. The purpose and intent of structuring the provisioning of IT services has been to minimize costs and reduce risk, not to enable new capability development for competitive advantage. This friction between competing regimes (cost reduction and opportunity enablement) invariably acts as an impediment for innovation teams because it deprives them of the use of the right information, tools, and networks required for new forms of collaboration and knowledge diffusion. A case in point was a digital innovation team that was prevented from using cloud-based collaboration tools without spending 6 months traversing the complex and ever-shifting political landscape of the IT shared services organization to gain approval for such tools.

Managing successfully the integration of employees from different functions during innovation efforts is critically important as innovation activities create high task interdependencies and require intense information exchange – the primary reason methods such as agile software development favor co-location. Mediation and mediated environments tend to place filters decreasing the transparency and intensity of the information exchanging activities. Inter- and cross-functional teams tend to be more effective when they have a shared common goal, shared language and rules of engagement, since without it, each function develops their own language, perceptions, interpretations and sub-cultures which lead to interpretive barriers among them during the innovation process of exploration, synthesis, and experimentation.

Another common impediment relates to functional silos and ownership over the relationship with the customer. In all cases, innovation activities require frequent interactions with customers to test and validate assumptions about problems, prices, and solutions options. In one organization, the global brand marketing team, reporting into the CMO of the business unit, claimed to have sole ownership over customer experiences and interactions. This prevented agile software development teams and product owners from talking to customers directly and validating the features in their product roadmap backlog. Eventually, those agile teams got the feedback they needed, but the resources had already been spent developing features nobody wanted or was willing to pay for. What was required was building more collaboration points between the marketing and product organization to increase shared understanding of customer problems and solution options – before – the product was built, tested, and released into production.

Cross-functional teams facilitate the diffusion of novel market and customer information across social networks, offering significant advantage for the innovation process. Also, information exchange within cross-functional teams helps employees to achieve shared understanding of the problem space enhancing decision making and risk identification throughout the process.

Finally, the use of cross-functional teams can lead to a more effective and more efficient use of resources, as it can reduce coordination challenges between different organizational units in the innovation pipeline and reduce hand-offs (which invariable introduce queues in the process and therefore lead-times). On the other hand, the lack of cross-functional collaboration will hinder the diffusion of new knowledge produced during innovation activities, undermining the extension of the organizational knowledge base and the effective use of scarce organizational resources – the most finite of which is time.

---

In the next section, I will continue with the structural, cultural, and leadership elements contributing to innovation success in large enterprises. The series will continue with a exploration of how organizational leaders can focus on creating sustainability through organizational learning necessary for competitive advantage. - WE