The Application of Lean Principles within the Software Delivery Industry

This article will cover a number of key points regarding the current state of lean principle adoption within the software delivery industry.

The state of the art application of lean principles within the software industry today exists in a suite of principles known as DevOps.

Implemented and advocated by industry leaders such as Netflix and Amazon, DevOps was established by Allspaw et al. in 2009 (John Allspaw and Paul Hammond 2009) and it draws heavily from lean thinking (Fig. 1)

There are five fundamental lean principles and what follows is (1) a description of these principles as they apply to DevOps, along with (2) a brief outline of how six sigma methodologies currently apply to the mainstream software delivery industry and (3) an introduction to the concept of software delivery waste.

Fundamental lean principle #1 - Value

In order to deliver a valuable software product, it has become standard practice for the software delivery organisation to work with the customer to define the desired outcome. This ensures that the customer will pay for the product, seeks to clarify the new functionality (the transformation) and ensures that appropriate planning steps are considered to ensure the solution works first time (Right-First-Time). An internal product owner (the product owner is typically an embedded member of the agile delivery team which delivers the product/service) liaises directly with the customer to achieve these goals before implementation begins. High degrees of customer interaction continues throughout the delivery process, through, for example, regular demonstrations and backlog prioritisation discussions. This approach supports the value concept, the first of the lean principles by specifying value in the customer’s terms.

Fundamental lean principle #2 - Value Stream

A fundamental lean concept, now pervasive within DevOps, is the concept of the value stream, the sum of all the activities necessary to create, order and produce a product or service. The software delivery value stream defines the steps and actions necessary to efficiently deliver high quality software artefacts to the production environment (the location from which customers access the product or service) and do so with the least disruption possible to materially add most value to the customer.

Fundamental lean principle #3 - Fast Flow

Achieving Fast Flow within DevOps means striving to attain the fastest delivery of the software artefacts to minimize lead-time (thereby minimizing waiting waste) to the customer at the appropriate level of quality (minimizing defect waste, potential re-work and provide quality benefits across several other dimensions (Ishikawa 1985)). This has traditionally been a challenge as software deployment events were infrequent and the pipeline (the technical realization of the value stream) frequently broke, causing flow friction. Borrowing from the concept of reducing batch size, software delivery teams now minimise the units of work to the most atomic but sensibly sized units (or batches) possible, striving for single piece flow and therefore lower lead-time.

Fundamental lean principle #4 - Pull

Pull has also been adopted as a fundamental principle of DevOps regardless of the flow management framework; typically a flavour of agile software development (Beck et al. 2001) such as Scrum (Sutherland 2015) or an adaptation from Taiichi Ohnos’ Kanban (Sugimori et al. 1977). The Pull principle mandates that tasks are managed in such a way as to respect capacity constraints, promote flow and ultimately avoid task bottlenecks.

To enable the Pull principle, visualization of the workflow which underpins the value stream is critical, allowing teams to manage flow between states. Agile scrum pulls batches of work per iteration whereas individual tasks are only pulled when capacity is available within the Kanban model. It is now standard practice within software delivery teams to visualize work and manage tasks using an agile board such as a Scrum Board or a Kanban Board.

Another highly utilized concept in DevOps is that of Work In Progress (WIP). Once again borrowing from Taiichi Ohnos’ Kanban concept, a WIP limit, used as a capacity management tool, defines the maximum number of work units which can be in a particular state.

Fundamental lean principle #5 - Perfection

Continuous and evolutionary improvement, i.e. Kaizen, is baked into DevOps culture. It is achieved by creating an organisation wide culture of trust (supporting Demings principle number 8), by overt encouragement of shared responsibility, by coaching and learning (supporting Demings principle numbers 1, 2, 6, 9, 13, 14 and the Toyota Way principle number 14), by encouraging experimentation and by rapidly performing evidence based corrective action informed by short feedback loops (supporting Demings principle number 5; Toyota Way principle number 13; Kaizen culture). Continuous improvement opportunities are regularly identified through measurement and reporting, also an integral aspect of DevOps.

Six Sigma in Software Delivery Environments

Six sigma methodologies (such as DMAIC (Define, measure, analyse, improve, control) and DFSS (Design for Six Sigma)) are not widely practiced within software delivery environments. While applying DMAIC has been proven to reduce errors in the software development process (Karout and Awasthi 2017), there exists fundamental characteristics of software development, such as application design, team experience and tooling, which profoundly impact on software delivery success. Karout et al. make the point that the application of six sigma may be premature in many software organisations where more basic implementation issues exist. Heston and Phifer echo this, considering six sigma a being ‘particularly useful as an addition to an improvement program that is already using another framework to address basic issues’ (Heston and Phifer 2011), i.e. don’t run before you walk.

Waste in Software Delivery

The 8 Wastes associated with lean thinking are typically described in the language of physical manufacturing however there are equivalent wastes in the software delivery industry. A significant type of waste experienced is, for example, that of waiting waste; typically associated with a local team waiting for a remote team to perform a necessary task. These remote teams typically queue their work thereby inhibiting the local teams capability to achieve fast flow and causing delays in value reaching the customer. Some further examples of wastes as they apply to the software delivery industry are as follows:

Lean Waste Category #1 - Transport

Examples of transport waste in a software delivery context include the transportation of tasks and software artefacts. Reliable and rapid task management and source code and repository tools reduce transport waste as the transport event happens reliably and in in real-time. Transport risk is somewhat mitigated by procuring and utilizing industry best task and artefact management tools.

Lean Waste Category #2 - Inventory         

An example of inventory waste is that of a software developer beginning a task, i.e. creating code, and then not completing that task. This partially done work (Kim et al. 2016), adding no value unless it is fully completed, can be classified as inventory waste. It is necessary that the product owner constantly refine and prioritise the backlog and that the developer complete all tasks which they have committed to deliver as part of the iteration (i.e. the Scrum ‘sprint’).

Lean Waste Category #3 - Motion

To avoid unnecessary movement of people and information, individuals must ideally be dedicated to a single team and be physically co-located with that team (tough to achieve in the current Covid-19 situation). Also, the number and complexity of inter-person handoffs of software artefacts and information must be minimized to avoid motion waste.

Lean Waste Category #4 - Waiting

All elements of the value stream must ideally be automated and this will lead to the reduction of waiting waste by reducing lead time.

Lean Waste Category #5 - Overproduction

The risk of developing products that are not directly responding to customer demand is mitigated by, for example, conducting a feasibility phase before implementation work begins on any product. This feasibility effort must be driven by senior team members and it must ensure that there is a compelling market-driven reason for developing the product.

Lean Waste Category #6 - Overprocessing 

Overprocessing waste, i.e. developing either processes, services or product features that are not required by internal or external customers (extra processes and extra features (Kim et al. 2016)), is mitigated by close customer engagement throughout the delivery lifecycle and continuous backlog refinement. Also waste associated with an overly complicated toolchain can be categorized as overprocessing waste.

Lean Waste Category #7 - Defects

Defect resolution cost is correlated with speed of detection, i.e. the faster defects can be detected after their introduction, the cheaper it is to fix them. Significant functional and non-functional validation must be built into the automated pipelines to mitigate defect waste as much as possible.

Lean Waste Category #8 - Skills    

An example of skills waste is having highly skilled humans perform repetitive tasks which could be effectively automated.

Application of Lean Thinking in Practice

The culture of lean thinking has only recently become established within many large organisations. Early-stage DevOps implementations are typically based on some (usually not all) of the 5 Lean Principles as outlined above.

Significant effort has been invested in creating cultures of trust, learning, problem solving and transparency. This cultural nirvana aspires to support an ‘ethos of ownership, pride, and trust’ (Randy Cook and Alison Jenkins 2014). The true challenge has been to ensure that in the experimentation phases of organizational adoption of DevOps, a failure tolerant work environment is fully supported by management. This has not always been achieved.

Another key concept which must be achieved (even in a basic way) is that of building quality into the product (Supporting Demings principle number 3; Toyota Way principle number 5), i.e. performing software validation as soon as possible in order to ensure conformance to requirements (Suarez 1992). Once again, attempting to run before one can walk in this regard can result in confusion and sub-optimal organizational performance.

The concept of whole team ownership and swarming on incidents must also be achieved to quickly analyse incident root cause, and permanently fix the identified problems; a principle borrowed from Taiichi Ohnos Jidoka.

Definition of the software delivery value stream with any DevOps implementation is also critical whereby dedicated processes are mapped out to enable the movement of software artefacts from the developers environment to the target product environments (example below in Fig. 2).

The implementation of a value stream to move artefacts from development to production has revealed itself to be a completely different type of activity than the activity of developing software artefacts in the first instance. The Scrum framework often proves to be fit for purpose as a framework for development activity but not appropriate for the software delivery value stream which aligns more closely with task-wise Kanban principles. This key learning has led to the implementation of Kanban (as opposed to Scrum) as a management framework for several stable (One of the six rules of Kanban is that Kanban can be successful only for stable and rationalised processes) delivery processes across the industry.

Another key learning has been that overprocessing waste manifests itself within an overly complicated toolchain, i.e. aiming for an overly complex toolchain before the fundamentals of the technical pipeline are working well. This points to the requirement to evolve the capabilities thoughtfully over time and to strive for effective simplicity as opposed to ineffective complexity.

Conclusion

The emergence of agile delivery philosophies was met with substantial resistance twenty years ago. The cultural shift from waterfall to agile has taken a generation to take hold, and pockets of resistance are still evident. The introduction of lean principles to the software delivery domain (DevOps) remains a work-in-progress across most of the industry. Notable however is that the rate of DevOps adoption has been significantly faster than was the case with agile adoption; we are accelerating.

Clover Networks Inc.

Clover Networks Inc. are a leader in continuously improving software delivery efficiency, in focusing on increasing speed to market while simultaneously improving software quality. This remains the situational value driven purpose for implementing lean thinking within Clover Networks Inc.

We approach problems in the right way, with a failure tolerant ‘whole-team’ philosophy (which IS fully supported by management). We understand software delivery value streams and we build quality into our products from the get-go.

Please reach out to me directly if you are interested in learning more about Clover Networks Inc. and the new Clover team we are building right now @ Fiserv Nenagh Technology Centre. Thanks for reading, Fergal ([email protected])

Bibliography

Beck, K., Beedle, M., Bennekum, A. Van, Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., Thomas, D. (2001) Manifesto for Agile Software Development [online], The Agile Alliance.

Heston, K.M., Phifer, W. (2011) ‘The multiple quality models paradox: How much “best practice” is just enough?’, Journal of Software Maintenance and Evolution.

Ishikawa, K. (1985) ‘What is Total Quality Control? The Japanese Way. Total Quality Management.’, in Reading 11.

John Allspaw, Paul Hammond (2009) ‘Velocity 09: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr’, 23/06/2009.

Karout, R., Awasthi, A. (2017) ‘Improving software quality using Six Sigma DMAIC-based approach: a case study’, Business Process Management Journal.

Kim, G., Humble, J., Debois, P., Willis, J. (2016) ‘The DevOps Handbook : How to Create World-Class Agility, Reliability, and Security in Technology Organizations’, The DevOps handbook.

MIDDLE, J. (1987) ‘ A review of: “ Zero Quality Control: Source Inspection by the Poka-Yoke System ”. By SHIGEO SHINGO (translated by A. P. DILLON). (Productivity Press, 1986.) [Pp. 305] Price: US $65.00. ’, International Journal of Production Research.

Randy Cook and Alison Jenkins (2014) Building a Problem-Solving Culture That Lasts, available: https://www.mckinsey.com/~/media/mckinsey/business functions/operations/our insights/the lean management enterprise/building a problem solving culture that lasts.ashx.

Suarez, J.G. (1992) ‘Three Experts on Quality Management: Philip B. Crosby W. Edwards Deming Joseph M. Juran’, Tqlo, available: https://apps.dtic.mil/dtic/tr/fulltext/u2/a256399.pdf.

Sugimori, Y., Kusunoki, K., Cho, F., Uchikawa, S. (1977) ‘Toyota production system and kanban system materialization of just-in-time and respect-for-human system’, International Journal of Production Research.

Sutherland, J. (2015) Scrum.Org | The Home of Scrum [online], scrum.org.