Continuous Integration

Continuous Integration (CI) is to optimize development capabilities of the team. Scope of implementing CI is much beyond the installation and configuration of build and integration tools. The key purpose is to produce feedback on an integration build and make improvement on continuous basis by fixing the problems quickly and at the early stages of the release. Continuous Integration practices enable project delivery teams to implement frequent integration builds, so that it becomes a regular activity that includes:

• Compilation
• Refactoring
• Rebuilding database
• Executing automated tests and inspections
• Deploying software
• Receiving feedback

Challenges

• Multiple Check-in Instances by Distributed Teams

When distributed Sprint team works on the same code base, handling multiple check-in instances (multiple branches, code merges and conflicts) is a challenge. Sometimes, it is an overhead to get efficiency and benefit out of round-the-clock development activities due to the check-in limitations.

• Maintenance of Multiple Source Code Files
  • Properties file
  • DB schema
  • .SQL files/patches
  • Automation test scripts
• Each development activity comprises not only source code creation, but also build-related resources such as:

It is essential to maintain these critical resources in the respective code repository. Missing any of these resources may lead to compilation or build error.

• Long Build Time

Traditionally, it takes many hours for the completion of each build which makes it more difficult to run ‘build’ every day. Another reason for this limitation is that a large amount of manual effort is required to complete the build process.

• Identifying and Fixing Check-in and Build Failures

Many times, check-in issues and build failure go unnoticed during the build creation process and eventually, they are captured by customers.

• Lead Time to Integrate and Prepare Deployable Software

One of the major challenges faced by the team is amount of time required to integrate the changes and prepare package that can be deployed to the target system.

Concepts

Continuous Integration with automation of test execution has seen wide adoption in recent years. The concepts of Continuous Integration have transformed how companies think about and implement Build and Configuration Management, Release Management, Deployment Automation and Test Orchestration.

Continuous Integration is a software development practice of continuously integrating source code into a central repository wherein each instance of integration is verified by the Automated Build process to ensure the stability of the project (working software) at all times.

This process takes place throughout the day and ideally should trigger the Automated Build process after every check-in. But the frequency of the Automated Build can be decided by the team, based on the project need (e.g. every hour).

The advantages of Automated Build and Continuous Integration are:

• Automated Build integrates the tools for code review, unit tests, and code coverage and thus, ensures the quality at source.
• Continuous Integration eliminates last-minute integration issues and thus ensures cohesive build throughout the development lifecycle.
• It is an engineering process that completely rebuilds and tests an application frequently. Although the essential concepts existed earlier, Continuous Integration normally refers to the Extreme Programming practice.
• Large projects often require a longer build and test cycle. So Continuous Integration is automated in such projects to ensure precision and reduction in effort and time.

Stages in Continuous Integration

• Stage for Build, Unit and Integration Testing and Quality Check

1. Build gets initiated when the developers commit their individual work/code to the version control repository system. The CI server which is installed as part of project environment, continually polls this repository (e.g. frequency could be every few hours based on the need of the project) for detecting any changes made by the development team
2. The CI server then detects changes, if any, in the version control repository, and initiates the process of retrieving the latest copy of source code from the repository and executes the build script using build automation tools to perform integration of the software
3. Once a successful build is generated, CI server triggers the process of validation of test cases/scenarios and code quality inspection. The process of validation and inspection is automated using the automation tools for unit and integration testing and inspection
4. Though these steps are automated, there would be some manual intervention required in these stages. Post this activity, feedback is generated by the CI server and is communicated to the team members through various mechanisms such as email and text message

• Stage for System/Functional/Non-functional Requirement Testing

  Different categories of tests can run from a CI system. This includes unit, component, system, load or performance, security, and various other types of tests.

  ? Component Tests: These tests verify portions of a system such as database, file systems, or network end points

  ? System Tests: These tests exercise a complete software system including external interfaces

  ? Functional Tests: These tests are executed to test the functionality of the application or product from the client perspective

Component tests take a bit more time to run when compared with Unit testing because of multiple dependencies whereas the System and Functional tests take the longest time to run.

The CI system would be configured to automate and run all these types of tests. These tests can be categorized into distinct buckets and run the slower running tests (example, Component, System) at different intervals than the faster running tests (example, Unit). The CI system Configuration would run each of these test categories with different staged builds. The slower running tests (example, System and Functional) can be automated through the tools interfaced with the CI server.

• Stage for Automated Deployment Continuous Integration needs Continuous Deployment for the culmination of all engineering practices which enables the release of working software at any time and any place with minimal effort. Continuous Deployment should include the capability to automatically roll back all changes applied in the deployment.
• Continuous deployment enables to deliver working, deployable software at any point in time. By running a fully automated build including compilation, all tests, inspections, packaging, and deployment, the team acquires the capability to release working software at any time and in any known environment.

Benefits of Implementing Continuous Integration

• Reduced Overall Risk fixed as and when they occur. Also, it becomes much easier to create complete builds of the project very often as they are done on a daily basis rather than once in a fortnight or so. Most developers have their own project set-ups on their machines. Because the build is created on a centralized CI (or build) environment, it reduces dependencies on individual hardware set-ups.
• As the project is continuously built and integrated; defects in the code are found and fixed at the early stages of the development. This helps in reducing risk and rework because the majority of the problems are Continuous Integration Concepts
• Reduced Human Error by Automating Repetitive Tasks
• Converting the process of manual repetitive tasks (wherever applicable) into an automated system eliminates probabilities of missing out on certain steps and reduces human error. It also allows developers to focus on what they are good at in the project and making the tasks/work a better experience.
• Continuous Creation of Usable Software
• At any point of time, usable build can be created; meaning, designers or developers do not have to depend on other teams for a new version of software. Depending on the type of CI environment, a new and working build should always be available and has to be up-to-date.
• Improved Project Visibility
  • Time required for creating stable builds
  • Current version of the build
  • Status of the last build instance and deploy jobs
  • Current problems and issue areas

By using CI, stakeholders will have a better idea of the existing status of the project. They will get information about:

• Build Failure Detection Using Automated Developer Testing
• Developers who write tests more often use some x-unit based framework such as Junit or Nunit. These tests can be automatically executed from the build scripts. CI enables automated regression tests to be run on the entire code base whenever a change is applied to the software.
• Implicit Enforcement of Coding Standards
• Coding Standards are a set of guidelines that developers must adhere to while working on a project. In many projects, it is a manual process that is performed through code review. CI can run the build scripts to report on adherence to the coding standards by running a suite of automated Static Analysis Tools that inspect the source code against the established standards whenever a change is applied.
• Refactoring for Easy-to-maintain Code
• It is the process of changing the software system in such a way that it does not modify the external behavior of the code and yet improves its internal structure. It makes the code easy to maintain and identifies potential problem areas at every build.
• Frequent Releases
• It allows the testers and users to get working software to use and review as often as required. CI works well with this practice. Once a CI system is implemented, release can be created with minimum effort.

Application in Projects

Application of Continuous Integration is necessary to plan at the early stages of the projects. If the implementation of CI is initiated in the later stages of the project, it is important to start small and plan the coverage incrementally. CI is not just about the technical implementation but is also an organizational and cultural implementation.

Step-by-step Process of Implementing Continuous Integration

1. Use Version Control Repository

This is the first step and is a must to start implementing CI. The main objective of using a version control repository is to manage all the source code changes and other software artifacts using a controlled access repository. This provides the team with a “single source point”, so that all source code is available from one primary location.

1. Configure the CI Server

The CI server has to be configured within the project environment to continuously check for any changes in the version control repository. The main purpose of CI server is to run an integration build by retrieving the source files whenever a change is committed to the version control repository. The CI server should support hard-scheduling of the builds on regular frequency, that is, every hour and should provide a dashboard and feedback mechanisms where the results of the build get published.

1. Build Script for Automation

Next step is to perform build automatically. Build script must be developed and implemented (which could be a single script or a set of scripts) that will automate the software build cycle which includes compiling, testing, inspecting, and deployment processes. Developers from the teams that implement CI system should run different categories of tests which include Unit, Component, System, Functional, and Regression tests to speed up the build process.

1. Use Code Quality Analysis

This step is very critical in large projects. It helps in stabilizing coding conventions and constantly monitors code for:

• Average code complexity
• Coverage of code
• Duplication of code
• Performance requirements
• If any suspicious code is committed, server should generate and publish warnings.

1. Configure CI with Automated Deployment

After the creation of each build, new version of binaries should be automatically deployed to test servers. It helps in improving integration with customer systems. After each fix, customer may verify new versions.

Best Practices

To reap maximum benefits of implementing Continuous Integration, it is advisable to follow the best practices listed here:

• Commit Code Frequently
• One of the major advantages of CI is integrating early and often. So to utilize the advantages of CI, developers must commit the latest version of the code frequently. Waiting for more than a day to commit code may not be beneficial and prevent developers from being able to use the latest changes.
• Don’t Commit Broken Code
• A precarious assumption in any project is that everyone knows not to commit code that would not work on the version control repository system. The eventual mitigation of such risk is to have a well-factored build script that compiles and tests source code in a repeatable manner. Team members should always run a private build before committing code to the version control repository. It is required to make this process a part of the development culture.
• Fix Broken Builds Immediately
• Broken build is anything that prevents the build from reporting success. It may be a compilation error or a failed test. When operating in CI environment, these problems should be fixed immediately. Fixing a broken build should be done on high priority.
• Write Automated Developer Tests
• Build should be automated in order to run tests for a CI system. It is essential to verify that the software works using automated developer tests.
• All Tests and Inspections Must Pass
• In CI environment, 100% of the project’s automated tests must pass. For any build to pass, automated tests are as important as compilation. Codes that have error will not work and if we accept such codes, it will lead to lower quality software.
• Run Private Builds
• To prevent integration failure, developer should run private build in local environment before integrating changes into repository. Any defects found in private build should be fixed before committing changes. This ensures less broken builds.
• Avoid Getting Broken Code 
• Failure of build will lead to wastage of time if developers get broken code from the repository. Developers should get latest code only when the build failure is fixed.