Agile Complexity: A Technology Service Portfolio for the Age of Disruption

This article asks how Technology Service Providers, both IT departments and software consultancies, can remain relevant as the scope of challenges facing their business clients grow.  When business disruption becomes business as usual, what new services do technology providers need to offer?  

The answer lies in a new level of business-technology partnership, one that delivers complex transformational innovations capable of overturning the organization’s status quo.  We must go beyond well designed mobile apps and smart business systems to solve harder strategic problems.  We need to embrace complexity while remaining agile.  

Eight new offerings show how a service portfolio based on “Agile Complexity” can address today’s bigger and more complex business challenges.  

Business Titanics and the Age of Disruption

A favorite movie quote comes from James Cameron’s movie Titanic.  The crew of the salvage vessel has come together on the bridge to listen to a description of the choices the captain of the Titanic made on the night of the great ship’s sinking.  “He's standing there with the iceberg warning in his hand ... and he's ordering more speed, 26 years of experience working against him.  He figures anything big enough to sink the ship they're going to see in time to turn. But the ship's too big, with too small a rudder... it can't corner.  Everything he knows is wrong.”

This is a disturbingly apt description of the reality facing today’s organizational leaders.  We are entering an era of ubiquitous business disruption, a time that has been variously called The New Renaissance by authors Ian Goldin and Chris Kutarna, or The Fourth Industrial Revolution by World Economic Forum chair Klaus Schwab.  

Commercial, public sector, and non profit organizations are finding that their secure long standing positions in the world are thrown into disarray. Schwab warns that this deep systemic change is “evolving at an exponential rather than a linear pace … disrupting almost every industry in every country. The breadth and depth of these changes herald the transformation of entire systems of production, management, and governance.”

This has forced enterprise leaders to embrace transformational change, or face irrelevance and obsolescence.  In this turbulent world it won’t be enough to incrementally improve existing business operations or dabble in small creative endeavors.  A new mobile app won’t save the day.  Nor will adding Artificial Intelligence or other shiny new tech to existing business systems.   

New competitors disrupt these markets with big bang interventions, entirely new approaches to creating value, that Larry Downes and Paul Nunes describe as “simultaneously better and cheaper, and more customized than the products and services of incumbents.”  These big bang disruptors don’t slice off a bit of the market, they pursue a winner take all strategy by making existing actors obsolete or commodifying status quo offerings.  

Entire organizational ships are going down unless they can develop the ability to envision and deliver far more impactful change.  

 (Un)Fit for Purpose Tech Services

Responding to this level of creative turmoil requires a seemingly impossible combination of big ambition, speed, and agility.  Technology service providers, both internal IT shops and software consultancies, should be rising to this challenge.  Unfortunately, the current portfolio of software development methodologies and services were designed to serve fundamentally different creative needs.   

Today, digital service providers have two primary approaches to creating software.  The first is an engineering approach rooted in the detailed analysis and planning of the traditional software factory.  While it’s waterfall stage gates and heavy documentation have been criticized, it is nonetheless still applied to the task of delivering big systems with well understood functionality.  

Lean innovators offer a much different approach to technology development. They experiment rather than relying on analysis and planning, leveraging fail fast iterations to create original but smaller applications. This light weight user centered methodology has successfully filled mobile phone libraries with millions of apps, each hoping to “delight the customer”.   

These methodologies are fit for purpose when applied to the appropriate business challenge.  As a result, CIO’s can confidently promise that they will deliver both big business operating systems and compelling new customer facing apps.  

Agile Complexity – The Next Service Portfolio

Unfortunately, neither of these approaches is well suited to forming a bold response to the disruption of an organization’s broader value proposition.  Every day, organizations build new mobile apps or apply trendy new tech to existing operations, only to find that their improvements are rapidly copied by other incumbents.  These safe investments are a danger to the organization.  They waste valuable resources, and even more importantly squander time as others rush to pursue more comprehensive visions for change.  

Strategies for fundamental business disruption must be both big and original, complex systems innovations which fall in the “Messy Middle” between existing software delivery models.  There is by definition no best practice for a proven business solution to fall back on for detailed planning and the scale of the response is much greater than is possible with neatly bounded lean product innovation. 

These disruptive opportunities require bold strategies that weave together business and technology. Both the problem and the solution involve multiple actors, tangled interactions, and unavoidable uncertainty.  Creating new systems in this space requires a willing embrace of complexity, as well as a nimble response that evolves as the world rushes forward with more disruptive change. 

In this exciting new creative space, Technology service providers need a new fit-for-purpose service portfolio, one capable of delivering both agility and complexity.   Emerging experience with this work indicates that a practice based on Agile Complexity requires three interrelated shifts in thinking.

(1) Seeing that disruptive opportunities lie in complex systems

The source of creative opportunities comes from collaborative systems that span the organization.  Disruptive threats demand more than incremental improvement or small extensions of the status quo.  Complex systems provide a big toolkit with the creative power to imagine and deliver game changing ideas.  

(2) … with deep business and technology collaborations  

These complex systems will take advantage Fourth Industrial Revolution technologies like IOT, big data, cloud computing, AI, and robotics, weaving technology and business into a much tighter collaboration. Increasingly every aspect of a business’ value proposition will be a dynamic combination of business and technology, bringing the entire enterprise into an unprecedented dynamic partnership.    

(3) … that evolve through an extended enterprise level journey.

These powerful but complex initiatives must continuously evolve in a changing world.  The agile lessons of lean startups must be applied on a wide scale, with Organizations building agility into their fabric, nimbly pivoting in response to new learning and insights.  

Eight ‘Agile Complexity’ Offerings

This is new ground for most technology service providers.  Based on the past sometimes rocky reception of agile practices into the software factory, many technologists will approach these new challenges with the same old no longer fit-for-purpose offerings.  This will be a mistake.  

In the same way that business clients need to recognize fundamental shifts in their markets no longer allow for business as usual, the leaders of technology service firms need to see their own industry’s disruption.  The challenges ahead are not simply tied to adopting new technologies like machine learning or big data.  These new technologies must be used in fundamentally new ways, if they are to deliver the disruptive business impact that clients will increasingly demand. 

The following eight services are based on emerging experience with delivering Agile Complexity as an offering.  The services life cycle flows from an initial expansion of the problem definition (1,2), through the development of complex system solutions (3,4,5), and ultimately to a shared business + technology journey where ambitious ideas are implemented in the real world (6,7).  Finally, broadly enabling this work requires creating agile capabilities at an enterprise level (8).     

Each of these eight services is briefly described below.  Because these offerings must ultimately be explained and sold to business clients, each service description begins with a “Pitch” that briefly makes the case for why the service is needed.  This is followed by a short section on “Practice” which provides an overview of how the work would be done.

1) Claim a Harder Problem

The Pitch:  The threat (and opportunity) of disruption requires initiatives that are transformational.  To rise to this level of change, it will be necessary to have a bigger and richer view of the problem space.  Harder problems open the door to bigger solutions.  

Yet, few organizations or their technology partners invest in creating a rich view of the problem that extends beyond the needs of immediate projects.  This creates an opportunity for organizations to create a competitive advantage by creating a proprietary model of the problem space that embraces additional complexity.  This model can them become a living asset to the organization, a unique view of how the world works that can enable the ongoing creation of transformational ideas.

The Practice:  Conventional problem definition is often done at either a very high executive level or at the level of a specific project.  These show up as static PowerPoint decks from management consultants, or detailed needs analysis conducted by researchers and analysts.  Both practices tend to produce outputs that are generated, reviewed, and then put into a drawer.     

A living visual model of a complex problem space model seeks to highlight the varied actors involved in an endeavor and then explore how they interact to create value.  This view of how the world works is unbounded by a specific project or the status quo practices of the organization.  It doesn’t make judgements on what needs to be fixed first.  Instead, it shows in broad terms just how the world works. 

This is an ideal initiative to deepen the creative relationship between business and technology thought leaders. Instead of enumerating a long inventory of facts, this shared big picture view encourages thinking that goes beyond individual pain points.  As a visual model it can distill powerful insights into a single page, a resource that can be shared across the enterprise with multiple interlocking initiatives.   

2) Workshops that Distill Complexity 

The Pitch:  Workshops have become a staple for lean product innovation, pursuing admirable goals like fostering open ideation and the broad inclusion of participants.  In the traditional structure, they begin with an enthusiastic generation of ideas followed by a series of activities to sort, filter, and prioritize the results.  This “Diamond” model is well suited for challenges that have a specific project goal must be select a path forward from among a variety of mutually exclusive options. 

Unfortunately, this workshop model is poorly suited to understanding and shaping more complex challenges.  It often results in tangled mess of ideas that can’t be acted upon, or forces participants to embrace an oversimplified view of a genuinely difficult issue.  

In contrast, workshop designed for working with real world systems distills the complexity of a domain while still preserving the structure of its interconnected features and diverse actors.  This provides a shared view of the whole problem or opportunity, untangling and structuring the many different ideas people bring to the table.  The resulting system models lay the foundation for shaping complex ideas for transformational change.   

Practice:  As with conventional lean product workshops, the first step in distilling complexity is bringing together a diverse group with multiple perspectives and encouraging them to think expansively about the space.  With a rich trove of insights in hand, the focus then shifts away from the classic ideation practices of sorting and prioritizing individual post it notes.  

Instead, this workshop framework seeks to distill an understanding of the complex systems that underlie the flood of ideas contributed by participants. Because multiple systems are often discussed at once, this work begins by partitioning the different systems under discussion.  Then the structure of each system can be modeled, connecting actors, resources, and actions that relate to one another.  This system views of how the world works are turned into a visual models, which the group can iteratively edit until a consensus view of real world complexity is achieved.     

3) Unify the Innovation Portfolio

The Pitch:  Organizational leaders often react to the threat of disruption by investing in a scattershot portfolio of individual projects.  Many of these projects seem revolutionary, but lacking a unifying vision or strategy for building upon one another, they lack the concerted power to drive substantial new opportunities.  Often they are short lived examples of an organization’s commitment to “original thinking” or simply become incremental improvements to the status quo.   

Despite its obvious shortcomings, a portfolio of mixed “bets” gains credibility from the Silicon Valley venture capitalist models of technology investments.  What is missed in this comparison is the important differences between the challenges faced by an organizational executive and venture capitalist. A venture capitalist only needs a few big wins from a collection of independent investments to declare success.  An organizational leader has an existing organization, assets tied to a domain. Facing obsolescence and commodification they must redefine their entire organization’s place in the market.  They lead a business not a innovation incubator, and as such need an integrated system of new capabilities, not merely the hope for a homerun.      

The Practice:  This service takes a step back and looks at the overall portfolio of new ideas and investments as an integrated whole. This is not simply a prioritization of work or review the business cases behind individual projects.  Rather, the goal is to see the project level work in a broader systems view of complex opportunities.  

The work begins by understanding the various future focused projects within the current portfolio.  These initiatives are mapped against a broader view of the threats and opportunities on the horizon (see Own a Big Problem).  A long term vision for how the organization will drive a transformative change is developed and the projects are overlaid on top.  This unified view makes it possible to identify where projects in the portfolio need to work together, where key gaps lie, and which projects fail to contribute to the future transformation.

4) Architect a Disruptive Vision

Pitch: To be an agile disruptor, rather than be disrupted, it is necessary to drive fundamental changes to the competitive ecosystem, circumventing barriers and changing the equations that govern the current marketplace.  These are ambitious opportunities that necessarily involve many interlocking elements.    

Innovating with these complex business + technologies requires a crucial shift in thinking.   A disruptive new strategy for the future is an interwoven system.   It is unrealistic to hope that a series of individual initiatives will eventually come together to realize a complex integrated original strategy where all the parts tie together. 

Opportunistically poking at flaws or gaps in the current approach may make the status quo better, but it is unlikely to produce a revolutionary new system.  To invent a new way of generating value there must be guidance that shows how the overall future system will work, a big picture vision of how the future will work.  a

Practice:   This offering creates an end state vision for how an original system of business + technology will create value.  It provides a simple view of a complex future system.  While this end future system may evolve over years, it can be guided along the way by a view of how all the pieces will come together.   

Architecting a holistic systems view provides an opportunity to embrace the creative power of complexity.  It can make use of multiple actors, diverse resources, and interwoven collaborations that blend business and technology.  This disruptive systems vision has the freedom to leverage conflicting incentives, ambiguous tradeoffs, and shifting connections. 

This work has few natural boundaries and will often lack definite answers for key design choices, yet this is not a barrier for developing an initial model for the future.  The vision is developed with the expectation that there will be ongoing evolution and deep pivots in the approach.  The goal is not to get all the details right, but rather to imagine a holistic end state where the elements make sense with each other.  

5) Take Pilots to Scale (Embrace Complexity)

Pitch:  Proposing to partner on a bold reimagining of an organizations key value propositions may be seen as a big ask.  Fortunately, there are other low hanging fruits where complexity can be applied, deepening the relationships between technology and business.  

Many new products and service innovations are generated by teams working with conventional technology delivery practices.   These initiatives create pilots that show promise, but then become stuck, finding it impossible to go further on the journey to scale.  This is not a sign that the pilots were flawed.  Rather it reflects the reality that scaling an innovation is a far different challenge than validating the initial idea.    

Ideas that hope to thrive in the messiness of the real world must surrounded by a network of services and systems. They must integrate with other technology and business systems, overcoming barriers that were conveniently overlooked during the fast moving pilot, and establish an enabling ecosystem of support.  In short, even small pilot projects will often need complex systems around them to scale.

Practice:  This service identifies multiple factors that will be necessary for the pilot to become a complete and sustainable solution to a real world problem.   This requires action across a number of varied domains; deepening the value created by the initial pilot, addressing issues required to make the solution feasible and complete, and establishing a sustainable operating model.   Even when the idea is finally sustainable in one context, there will be subsequent challenges associated with adapting it to new users or situations.

There are a number of challenges in each of these areas.  Some will be technical, some will be primarily business focused, and a growing number will combine both business and technology. Building these complex capabilities creates the need for an extended multi-disciplinary collaboration between business and technology.   

6) Collaboratively Evolve Complexity 

Pitch:  Many organizations and their technology providers fall back on reductionist project planning for large initiatives with lots of moving parts.  Even so called Agile projects are often embedded in more traditional waterfall planning (water-scrum-fall).   

Complex transformations, the kind that produce disruptive opportunities or avoid disruptive threats, are poorly suited to this approach.  They necessarily involve exceptionally diverse elements, untested collaborations, and unavoidable unknowns.  This fundamental complexity means they cannot be planned out in advance and delivered as a classic enterprise business systems project. 

Fortunately, there are opportunities to apply the lessons from agile project teams to these sophisticated business + technology strategies.  While the complexity can’t be planned out in advance, it can be iteratively discovered and shaped. This requires an extended journey where business and technology collaboratively architect, learn, and pivot across the entire system of change. 

Practice:  The key challenge when working with large complex systems is that they cannot be arbitrarily simplified to fit with either of the existing software delivery models.  However, it is possible to manage risks and track performance toward a clear vision by iteratively delivering a a series thin system slices. Each thin slice integrates business and tech to provide an end to end capability.  The slice performs a dual job, both testing the vision for the way the future system will work and laying a functional brick that contributes to the ultimate system.   

The guiding direction for each slice is provided by the target system vision (see service #4), the current best estimate of how all the proposed change will come together.  This is not a rigid specification.  It remains a flexible objective, allowing the business and technology collaborators to extend and pivot as they proceed.   The progress of the work is not tracked simply based on the number of tasks completed, but rather on how much of the ultimate system approach has been validated and the success of the emerging system slices in creating value.     

7) Evangelize Big Ideas  

Pitch: Historically, technology teams have executed projects and funded by business leaders.  In theory, top down control over investments and formal delivery responsibilities assured that once a project was approved the entire organization would fall in line.  While there might be disputes and grumbling along the way, support for a sanctioned project can generally be assumed.   

Unless of course the project threatens to blow up the entire framework of the organization and the way the industry works.  Genuine disruption challenges accepted culture and principles that have grounded an organization for years, so many different people will find their work and world view upended. Tradeoffs will be necessary, and conflicts will arise.  Some will win, and others will not.  

This institutional and personal upheaval will be conducted against a backdrop of ongoing uncertainty and limited proof for future success.  Simply instructing people to sign up for change will not be sufficient to keep the transformation moving forward.  Someone will need the bring the many diverse participants along on the weaving journey of change. 

Practice:  Being an evangelist for deep structural change requires far more than providing conventional stakeholder management.  An evangelist for change must create strong narratives, providing each person with a compelling reason to engage which recognizes their unique challenges, concerns, and motivations.  Ultimately, “Everyone gets a pony” if they join the adventure.

This is a big picture role.  The service must build and leverage an extensive network of relationships.  It also requires a robust systems view of the technology involved as well as a deep understanding of the business change.  

As unexpected barriers arise or new opportunities present themselves, the evangelist must help varied participants pivot and adjust. Throughout the journey of system transformation, they reach across traditional silos and often engage outside the bounds of the organization itself.  

8) Enable an Agile Enterprise

Pitch: This final Agile Complexity service is directed at building the flexibility and responsiveness of the organization itself.   Organizations structured on 20th century models of command and control and a clear division of labor are very good at optimizing the performance of well defined tasks.  Ironically being able to predictably deliver familiar activities, works against continuous learning and adaptation.  

In Technology departments this challenge was addressed by replacing rigid Software Factory practices with Agile software teams.  These practices have made a real difference in technology development but have largely been limited to the confines of software teams.   

As organizations confront disruptive new threats and opportunities, their success will increasingly be driven by how quickly they can adapt to substantive change. This requires an Agile Enterprise that responds boldly and flexibly to market signals, applying fundamentals like iterative learning and adaptative pivots across the broader business organization.    

Practice:  Technology teams are well positioned to be the advocates of agile enterprise practices.  Years of experience using iterative feedback and learning to manage difficult creative challenges provides a foundation for reimagining agile at the enterprise level.  Yet, it is not possible to simple give business operations a standard course in agile software practices.  

Business enterprises interweave bigger and more diverse operations, with many more layers of responsibility and control.  These are not “two pizza teams.”  A genuinely agile enterprise needs to nest multiple levels of learning and feedback, so that insights flow up and down as well as across the organization.  This enterprise form of agile will require structural changes in how strategies are formed and shared, how portfolios are formed and how success is measured. 

This internal transformation is itself a complex system change.  If successful, it will enable the organization to more effectively pursue its own disruptive responses to a changing world.    

Real Change for Technology Service Providers

These eight services provide technology service providers with a tangible offering that is designed to address the challenges business leaders face in an age of ubiquitous disruption.  They can be sold in order, providing a lifecycle of engagement that begins by claiming a hard problem, then moves to imagining big ideas, and finally evolves an on the ground system transformation.  

These are long journeys which offer the potential for an unprecedented level of collaboration between business and technology.  Or course, it may be hard to sell such an ambitious partnership right out of the gate.  Fortunately, the service portfolio offers multiple points of entry.  It may be easier to begin with a simple workshop that distills complexity or meet business clients where they already by unifying their existing innovation portfolio. 

Technology Service Providers need to embrace this opportunity for change.  The New Renaissance makes disruptive demands on technologists as well as business leaders.  Sparkly new tech done with the same old service models won’t be good enough.  We must be prepared to support increasingly complex challenges with agility at scale if Tech is to deepen its partnership with business.  It’s an exciting opportunity, one that Technology Service Providers will embrace, or find that they too face the risk of obsolescence and commodification.   

About the Author - Dan McClure

Dan is practitioner-theorist of systems innovation, developing and applying advanced innovation methodologies based on over three decades of hands on work with enterprise level system disruption and change.  Today, as the principal innovation choreographer of Practical Clarity, LLC , he works with complex problem spaces where traditional innovation methodologies prove to be too limited.