Continuous Integration and DevOps

This article explores Continuous Integration, what it is, how it fits a software development life-cycle and within what context it should be comprehended. It connects to DevOPs and how this philosophy is the glue between functions that are automated. Automation being the foundation of engineering software within and iterative and agile methodology to systematically check and test code with little to no human intervention in real-time development fashion.

Continuous Integration

Continuous Integration is a software engineering practice that makes use of the DevOPs philosophy, honed on an iterative development methodology that ensures frequent code checking. Automation is fundamentally required to build the code and perform continuous testing to detect problems early (ThoughtWorks, 2019). Big-bang integration tends to uncover new problems with uncertain locality causing a larger surface area over which to trouble-shoot.  Incremental integration yields early detection of errors by iterating over smaller and more frequent assemblages into the integration (McConnell, Steve;, 2004).

Continuous Integration therefore has it to heart to improve developer productivity, identify and fix bugs early in the process and deliver product iterations faster. It keeps the master code branch clean and achieves an iterative, fail-fast workflow. Due to the nature of the Agile methodology, i.e. that it creates a much faster feedback-loop, Continuous Integration and Test Automation are required to ensure product feedback is the same. This allows for early risk mitigation that also ensures input to quality by avoiding product drift through a protracted development cycle.

Test Automation

Test Automation is a prerequisite for Continuous Integration. Test must cover all code branches in the central repository.

1. Unit tests – verification of individual units/ modules of code
2. Integration tests – group of multiple components function correctly together, e.g. integration with peripheral services
3. Acceptance tests – looks at business flows, but with integration as the foundation
4. UI tests – user work-flow and interaction testing

Unit tests are the easiest, cheapest, and fastest to implement with UI tests on the other end of the spectrum (Pittet, 2019). Tests must execute automatically on every code push.

Various tools are available to implement test automation, e.g. Postman, Apache JMeter, TestComplete, Selenium, Katalon Studio, Microsoft TestManager with their DevOps feature.

Iterative Development

Iterative development reduces the amount of back-tracking required to discover where a problem manifests such as when doing a big-bang integration. That time can be better spent on developing features, not messing around to find where the bug is.

Continuous Integration is not cheap, but not doing so is expensive. It does not negate bugs but makes it much easier to find and remove. It positions itself to Reduce Waste.

Making the Agile development methodology successful, little promotes this as much as continuous integration. Few things can dowse Agile as much as not automating the iterations of build, deploy, and testing.

Mobilising a Continuous Integration Process (Figure 1) requires appropriate architectural enterprise principles. A shift to an engineering practice is more a people shift than technology. It requires oneness of mind, i.e. a broad stakeholder or executive stakeholder buy-in.

Good principles are founded in the beliefs and values of the organisation and expressed in a language that the business understands and uses. These principles are a firm foundation for decision making, framing policies, procedures, and standards, and for resolving contradictory situations (The Open Group, 2019).

Automation Motivation and Principles

Automation is the heart of process optimisation. Through implementing an autonomous repetition of tasks, in an unfailing and unvarying system, automation achieves this critical improvement to the SDLC. It removes humans from having to engage in these repetitive tasks, operating as highly variant entities, making mistakes. By nature, computing is a type of automation that allows the execution of applications and tasks, in a rigid, unvarying manner that has the same outcome every time it is executed.

The Continuous Integration Motivation diagram provides a rationale for automation. At the hand of a few principles, goals are realised to achieve a set of outcomes, viz. economic, high quality software through short-path error resolution.

As guidance, architecture principles govern any software engineering activities by prescribing that software builds should be automated. Coupled with a committing to an integration computer, having automated builds and continuous testing, automated deployments (into test environments) reduce the development cycle significantly. It also allows the automation to repeat the CI process mechanically without waver.

A variety of goals are achieved through CI. Team morale being a huge accomplishment. Reducing defects through early detection not only saves time, yield quality products, but fosters good customer relationships. All of which makes the development team proud and thereby boosting their confidence and morale. A motivated team is capable of innovation and usually tends to deliver high quality products at a more efficient pace.

Software Engineering

Software development is an engineering practice by which software products are manufactured in a systematic fashion. Much of the physical manufacturing realm applies. Primarily, manufacturing methods that produce quality products economically that meet the requirements of users with the ability to deliver those over an acceptable life-span at an acceptable price.

Such an enterprise requires appropriate work-methods, i.e. sound engineering principles through which to design, implement, test, and document software products.

DevOps

DevOps Specialises Software Engineering as a set of engineering practices through which to automate the processes between development and other Information Technology (IT) teams (Atlassian, 2019).

These software development practices straddle development and operations to shorten the Software Development Life-cycle (SDLC) by delivering products in short delivery cycles conforming to business objectives (WikiPedia, 2019).

The objective of DevOps is time reduction, on the path between introducing a system change to it being in a production environment, being used by users, while ensuring high quality. It removes disconnects in understanding and influence by encouraging a domain-centric perspective. It takes the principles of Agile and expands their scope, recognising that ensuring high quality development requires continued engagement and feedback from a variety of technical experts, including QA and operations specialists (Cois, Aaron C.;, 2015)

Quality being the aspect that should not be diluted while achieving the objective of time dilation relative to the SDLC. By fostering collaboration between the development and operations teams (with QA attached) the historic silo operations are merged into one. In principle, this type of integration is akin to automation and all its trappings. Tighter integration is key and of most importance. Such affinity emulates Agile through a tighter relationship. It suits the short iterative stints of Agile software development. Faster feedback within this relationship is comparable to early bug detection and resolution forthcoming from CI. Both imbibe and require the same culture.

Striving for a holistic appreciation of software engineering is what CI and DevOps are collaborating towards.

Question: How do the actions of a team affect others and the product being released?

Misalignment results in finger-wagging and resentment with retracement and fraternalization of teams into their respective silos. There they are festering and descend into a blame “scape goat” culture.

Release faster and work smarter, “improve constantly and forever the system of production and service to improve quality and reduce waste” (Hessing, Ted;, 2013). Further, “Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” (The Deming Institute, 2019). All of this is based on reducing waste.

Continuous Integration, as with the real-time operational monitoring of systems, provide a real-time window into the actual state of quality associated measurements for the system under development. It allows immediate and constant engagement from all team members, inclusive of QA and operations, throughout the SDLC with real-time reports issued to stakeholders.

Reduce Waste

The goal of waste reduction is quality improvement.

Automation, because it is consistently repeating a well-designed and optimised task, is a good tool for eliminating waste.  Lean manufacturing is achieved by optimising and simplifying work methods. These, converted into machine execution, remove the variability of human execution. Thereby, consistently yielding an optimised process.

Waste is eliminated by attaching value to the product. Therefore, implement customer requirements by tightly defining them as unambiguously and simple as possible to reduce misunderstandings and thereby the possibility of creating waste.  Produce a product that customers are willing to pay for is manufactured by a process within which work-methods are optimised to produce the desired quality and reduce waste to a minimum during the process.

The primary waste contributors in software development are:

1. Overproduction/ Over processing, in the sense of over-engineering, i.e. making the requirements too complex by producing superfluous features that take inordinate amounts of time to implement
2. Manpower and time waste, making mistakes, improper integration methods and/ or schedules, producing difficult to find-and-identify-bugs, unclear requirements placing non-wanted and useless features, etc.
3. Process-waste, where software processes and methodologies are not utilised, tests are manually created and executed, tests are inadequate or scheduled too late in the SDLC, and inter-team disconnections break knowledge-sharing and work-flow enablement
4. Bugs, being latent and not identified early in the development process or not detected at all

Existing wasteful practices must be eliminated. 

Target the following areas:

1. Time
2. People time
3. Tasks
4. Processes not used effectively

These can include humans, excessive time spent on tasks, lack of tooling and process contraints.