Complex, Complexity, Complicated

The popular notion that?Cynefin?can be applied in the software development domain to discuss the problems involved in writing software for money needs to include the profession of?Systems Engineering. From Wikipedia, Cynefin is...

The?framework?provides a typology of contexts that guide what sort of explanations or solutions might apply. It draws on research into?complex adaptive systems?theory, cognitive science, anthropology, and narrative patterns, as well as evolutionary psychology, to describe problems, situations, and systems.

While Cynefin uses the term Complexity and Complex Adaptive System, it is applied from the?observational?point of view. That is, the system exists outside of our influence on the system to control its behavior - we are observers of the systems, not?engineers?of the?solutions?in the form of a?system that provides needed capabilities to solve a problem.

Read carefully the original paper on Cynefin,?The New Dynamics of Strategy: Sense Making in a Complex and Complicated World.?This newsletter is NOT about those types of systems but the conjecture that software development is, by nature, Chaotic. This argument is used by many in the agile world to avoid the?engineering?disciplines of?INCOSE-style Systems?Engineering.??

There are certainly?engineered?systems that transform into complex adaptive systems with emergent behaviors that cause the system to fail. Example below. This is likely to be different when?engineering?principles are applied in the domains of Complex and Complicated.

A good starting point for the complex, complicated, and chaotic view of?engineered systems?is?Complexity and Chaos -?State?of the Art: List of Works, Experts, Organizations,?Projects, Journals, Conferences, and Tools?There is a reference to Cynefin as organization modeling. While organizational modeling is important - I suspect Cynefin advocates would suggest the only important item - the?engineered aspects??of applying Systems Engineering to complex, complicated, and emergent systems is mandatory for any organization to?get the product out the door on time, on budget, and on specification.

For another view of the complex systems problem,?Principles?of?Complex?Systems for Systems Engineering?is a good place to start, along with the resources from INCOSE and AIAA like?Complexity Primer for Systems Engineers,?Engineering Complex Systems,?Complex System Classification, and many others.

So Let's Look At the Agile Point of View

In the agile community, it is popular to use the terms?complex, complexity, complicated, complex?adaptive?system?many times interchangeably and many times wrongly - to assert?we can't possibly plan ahead,?know?what we're going to need, and establish a cost and schedule because the?systems?complex, and emergent.

These terms are often?overloaded?with an agenda used to push a process or even a method. In the agile community, claiming that we have no control over the system is popular, so we must adapt to its emerging behavior. This is likely the case in one condition - the chaotic behaviors of Complex Adaptive Systems. But this is only the case when we fail to establish the basis for how the CAS was formed, what sub-systems are driving those behaviors, and most importantly, what are the dynamics of the interfaces between those subsystems - the System of Systems architecture - that create the chaotic behaviors.?

It is highly unlikely those working in the agile community actually work on complex systems that evolve AND at the same time are?engineered?at the lower levels to meet specific capabilities and resulting requirements of the system owner. They've simply let the work and the resulting outcomes?emerge?and become?Complex, Complicated,?and create?Complexity. They are?observers??of the outcomes, not?engineers?of the outcomes.

Here's one example of an engineered system that actually did become a CAS because of poor efforts of the Systems Engineers. I worked on Class I and II sensor platforms. Eventually, FCS was canceled for all the right reasons. But for small teams of agile developers, the outcomes become?complex?when the Systems Engineering processes are missing. Cynefin partitions beyond obvious?emerge?mostly when Systems Engineering is missing.

First Some Definitions

• Complex?- consisting of many different and connected parts. Needs to be easier to analyze or understand. Complicated or intricate. When a system or problem is considered complex, analytical approaches, like dividing it into parts to make the problem tractable, are insufficient because the interactions of the parts make the system complex. Without these interconnections, the system no longer functions.
• Complex System?-?is a functional whole consisting of interdependent and variable parts. Unlike conventional systems, the parts need not have fixed relationships, fixed behaviors, or fixed quantities, and their individual functions may be undefined in traditional terms.
• Complicated?- containing a number of hidden parts, which must be revealed separately because they do not interact. The mutual interaction of the components creates nonlinear behaviors in the system. In principle, all systems are complex. The number of parts or components is irrelevant n the definition of complexity. There can be complexity - nonlinear behavior - in small systems or large systems.?
• Complexity?- there is no standard definition of complexity is a view of systems that suggests simple causes result in complex effects. Complexity, as a term,?is generally used to characterize a system with many parts whose interactions with each other occur in multiple ways. Complexity can occur in a variety of forms:

– Complex behavior

– Complex mechanisms

– Complex Situations

– Complex systems

– Complex data

• Complexity Theory?-?states that critically interacting components self-organize to form potentially evolving structures exhibiting a hierarchy of emergent system properties.?This theory takes the view that systems are best regarded as wholes and studied as such, rejecting the traditional emphasis on simplification and reduction as inadequate techniques on which to base this sort of scientific work.

One more item we need is the types of Complexity.

• Type 1 - fixed systems, where the structure doesn't change as a function of time.
• Type 2 - systems where time causes changes. This can be repetitive cycles or change with time.
• Type 3 - moves beyond repetitive systems into organic, where change is extensive and non-cyclic in nature.
• Type 4 - is self-organizing, where we can?combine the internal constraints of closed systems, like machines, with the creative evolution of open systems, like people.

And Now To The Point

When we hear complex, complexity, complex systems, complex adaptive system, pause to ask what kind of?complex?are you talking about. What?Type?of complex system. In what system are you applying the term?complex. Have you classified that system in a way that actually matches a real system. Don't take anyone saying?well the system is emerging and becoming too complex for us to manage?Unless in fact that is the case after all the Systems Engineering activities have been exhausted. It's a cheap excuse for simply not doing the hard work of?engineering?the outcomes.

Using the terms complex, complicated, and complexity interchanged is common. And software development is classified or misclassified as one, both, or all three. It is also common to need to understand their meaning and application to toss around these terms.

We need to move beyond buzzwords. Words like?Systems Thinking.?Building software is part of a system. There are interacting parts that, when assembled, produce an outcome. Hopefully, the desired outcome. In the case of software, the interacting parts are more than just the parts. The software has emergent properties. A Type 4 system, built from Type 1, 2, and 3 systems. With changes in time and uncertainty, modeling these systems requires stochastic processes. These processes depend on estimating behaviors as a starting point.?

The understanding that software development is an uncertain process (stochastic) is well known, starting in the 1980s [1] with COCOMO. Later models, like?Cone of Uncertainty, clarified that these uncertainties evolve with time. The current predictive models based on stochastic processes include Monte Carlo Simulation of networks of activities, Real Options, and Bayesian Networks. Each is directly applicable to modeling software development projects.

[1]?Software Engineering Economics,?Barry Boehm, Prentice-Hall, 1981.