Addressing Complexity in Megaprojects

In a series of articles I discussed the results of research conducted by MIGSO-PCUBED and others which shows that there is a consensus that there are 4 recurring reasons for the increasing failure rate of major projects in the 21st Century. These reasons, or Secrets, are:

Secret No. 1: The quality of business to business relationships

Secret No. 2: The speed of decision-making

Secret No. 3: The modularity of design and agile delivery

Secret No. 4: The extent of digital integration.

We have also identified that the increasing complexity of modern projects is a key reason for the increasing rate of project failure. This is a reasonably well understood concept. Terry Williams suggested in his seminal paper in 2005[1] that we needed to move on from traditional explanations for project failure and accept that current techniques can't cope with increasing levels of complexity. My articles led to an interesting debate on LinkedIn and other forums. The main challenge to me was could I show that the 4 secrets are linked to this increasing complexity. Challenged accepted, and with thanks to Dr. Ilias Krystallis at UCL I think I have the answer to that challenge. The starting point is: what do we mean by project complexity?

We lack a common definition of complexity in projects resulting in a number of complexity theories and assessment models which cover a range of different approaches. However, Geraldi et al tried to bring this together into a common theory in their 2011 paper[2].

The understanding of complexity in projects in academic literature can be said to have developed since 1996 with the publication of Baccarini’s paper in 1996[3]. Geraldi identified that this was one “dimension” of project complexity, known as structural complexity. This is close to the mathematical definition of complexity and relates to the project size (number of independent elements), variety of elements and interdependence between the elements. Classical project management techniques address this dimension of complexity, but if structural complexity is high, modularity and agile techniques (Secret No. 3) are required to reduce the complexity to a more manageable level.

Geraldi et al identify a total of 5 dimensions of project complexity. Structural Complexity is the first and Uncertainty is the second dimension. This is related to the potential for the elements within the project to change and the ability of the management system to react to these changes. This is compounded by two factors: the information available about the element (e.g. ground conditions for a civil engineering project) and the interaction between the elements, e.g. soft ground conditions affecting the seismic response of the structure requiring changes to a crane suspended from the superstructure. In classical project management theory uncertainty is measured and (attempted to be) managed using risk management techniques. However, it is essential that the decision making system can react fast enough to mitigate the risk before the decision-making window closes. This is Secret No. 2: The speed of decision making.

As shown by Michael Cavanagh in his book[4] “Project Complexity Assessment” many of the complexity assessment models available only address structural complexity and uncertainty in a 2x2 matrix and therefore ignore the other 3 dimensions of complexity.

The 3rd dimension, Dynamic complexity, relates to the stability of the enabling system. The enabling system is the approach being used to deliver the project (system). Projects with high dynamic complexity are vulnerable to constant change. These changes are generally related to the actors and their constancy and certainty. If a client is unsure of what they really need they may change the specification frequently. A project team may be subject to constant changes in personal or roles. For example in a joint venture between four companies it is not unusual for the companies to agree that each will provide the project leader on a cyclical basis, say every 18 months. Dynamic complexity can be thought of as the state of disorder of the project environment and in modern projects this is generally quite high. Attempting to manage this with classical project management techniques, which are based on the assumption of constancy and predictability, is unlikely to succeed. Classical project management does not address this dimension. Secret No. 3: Modularity and agile is the approach required in this situation.

The fourth dimension of project complexity is Pace. Some projects require rapid decision making in order to meet their timescale challenges (Secret No. 2), i.e. they are urgent. Some projects become urgent over time due to delays and commitments made by other projects based on an understanding of when the project will be complete. Some projects suffer artificial pace constraints, often from political sources. The loss of the space shuttle Challenger is an example of this when the NASA project team felt compelled to launch because a senior politician was in attendance and the future of the programme was politically uncertain. The key factor in enabling rapid decisions within the project (as opposed to governance) is the timely availability of data (better expressed as information). This requires highly effective data processing solutions and is closely related to the information management systems on the project. Classical project tools are poor at this, because of the split created between the project management information, engineering information and asset information systems. Integrated digital systems are required to manage this. This is Secret No. 4: Integrated digital systems.

The final dimension of project complexity is socio-political complexity. This is sometimes described as a “wicked problem” because this dimension is often hidden. It concerns relationships between people and organisations at all levels of the project. Personal aspirations and organisational commercial interests are often (always?) in conflict with the project goals and this leads to relationships which are hard or impossible to predict. They also become more influential as problems arise, often leading to the complete breakdown of the relationships and the subsequent failure of the project. Staughton et al[5] examined this in 2011 and found this dimension alone had a further seven dimensions, at least 5 of which are not addressed by classical project management theory. This is Secret No. 1: The quality of business to business relationships.

The dimensions of complexity are clearly not mutually exclusive and one will affect others. In particular the socio-political dimension if often dominant. If relationships break down the other dimensions will be harder or impossible to manage. This is key reason why classical project management techniques find it harder to control modern projects because internal and external relationships have become more complex. The number of stakeholders has increased (following the breakup of the vertically integrated public bodies in the latter quarter of the 20th century) and the increase in regulation and social interests.

Geraldi’s description of complexity as a set of dimensions is useful. Carl Sagan famously described the world of Flatland (first introduced by Edwin Abbott in 1884) where the 2-dimensional citizens aren’t aware of the third dimension and so never look up. It’s similar with complexity in projects. Classical project management at best only addresses two of the dimensions with any level of comprehensiveness. Project managers therefore live in a form of Flatland where they simply don’t see some of the dimensions.

In particular they don’t see (or their systems don’t) socio-political complexity. They are aware of external (to the project team) relationships, especially political and regulatory relationships, but have no tools to manage them (other than being wildly optimistic to get the project approved). Bent Flyvbjerg[6] has much to say on this topic. B2B relationships inside the project, however, are simply not understood by the project leadership and definitely not measured or managed. I recently spoke to a very senior industry leader, who understood that relationships were important, but expressed a view that they weren’t worth putting on the risk register because they couldn’t be managed, other than by great leadership. So, this is where I would start. This dimension trumps all the others and if not actively managed will become toxic for the project. Even if relationships start off well (and this is only possible if everyone is happy with the contract) they will not survive the first problem unless the relationships are measured and actively managed.

There is strong alignment between the 4 secrets of projects success (5 if you include the conventional project management techniques we need to retain) and Geraldi’s complexity model. Why might this be? In fact it’s not that surprising. My research is partly based on a review of the academic studies of project success and failure over the last 25 years and my conclusion is that complexity is a driving factor in this success and failure. Geraldi et al have reviewed the same body of knowledge from the other end of the telescope and concluded that there are five dimensions of complexity which align to my paradigm. We also both conclude that most of the dimensions/secrets are not currently actively managed.

Geraldi et all stated in 2011 that project management needs a paradigm shift to address complexity and reverse the current trend of project failure. This has not happened and until it does our projects will continue to run aground on the rocks of complexity. We need different navigation and manoeuvring solutions to avoid these rocks. I propose that the 4 Secrets plus retained elements of traditional project management embodied in a bespoke project enabling system using a systems thinking approach comprise this new paradigm. I call this PM5.0 or System Delivery.

So how should project leaders implement this new paradigm? The starting point is assessment of project complexity and it needs to recognise the 5 dimensions discussed above. Michael Cavanagh and the International Centre for Complex Project Management reviewed the available project complexity assessment tools. He identified 12 available tools in the public domain and proprietary tools. He identifies the NCTP tool developed by Shenar et al (known as the diamond model) as one which addresses 3 complexity dimensions (excluding dynamic and socio-political complexity) and is probably the most comprehensive and trusted. Using other, more limited solutions such as the APM complexity assessment (which is targeted at the selection of project managers) will give a false impression of how complex your project is, leading to the appointment of project managers who are not skilled in the relevant aspects of the 4 Secrets, especially the measurement and management of B2B relationships.

More recently the National Audit Office in the UK has published the DECA project complexity assessment model[7]. This post-dates Cavanagh’s research, but to me it appears to cover all 5 dimensions and I would recommend its use. Importantly it stresses it should be used to give the project team insight into the key risks on the project. They can then take action to ensure the team has the capability required to address these risks (including ensuring the team has the competence in the 4 Secrets). They may then be able to influence the delivery environment to reduce the level of complexity.

Importantly when undertaking a complexity assessment the factors should not be aggregated and averaged to create a “score” for the project. This creates a false impression of the risks on the project. For example if a project has a very complex stakeholder map and challenging timescales but relatively low complexity in all other factors it may appear to be low complexity when averaged. But it isn’t. It is still highly complex due to its stakeholder environment. This is a major risk and requires a team highly competent in stakeholder management and B2B relationships to manage this risk.

So, I think I have shown that there is coherence between complexity and the 4 Secrets and I hope I've provided some food for thought on how to take this forward for your project. I look forward to the continued debate!

[1] Williams, T. (2005), “Assessing and moving from the dominant project management discourse in the light of project overruns”, IEEE Transactions on Engineering Management, Vol. 52 No. 4, pp. 497-508.

[2] Geraldi, Maylor, Williams, Now, let’s make it really complex (complicated). A systematic review of the complexities of projects, International Journal of Operations & Production Management Vol. 31 No. 9, 2011 pp. 966-990 DOI 10.1108/01443571111165848

[3] David Baccarini, The concept of project complexity—a review, International Journal of Project Management, Volume 14, Issue 4, 1996, Pages 201-204, DOI 10.1016/0263-7863(95)00093-3

[4] https://www.amazon.co.uk/Project-Complexity-Assessment-Michael-Cavanagh-ebook/dp/B00C2HXX58

[5] Johnston, R. and Staughton, R. (2009), "Establishing and developing strategic relationships – the role for operations managers", International Journal of Operations & Production Management, Vol. 29 No. 6, pp. 564-590. DOI 10.1108/01443570910957564

[6] Flyvbjerg, Bent. (2008). Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice. European Planning Studies. 16. 3-21. DOI 10.1080/09654310701747936.

?[7] https://www.nao.org.uk/insights/deca-understanding-challenges-delivering-project-objectives/