Eliminating Configuration Drift for Oracle SOA and BPM Projects

During the lifetime of a project, code will be built and promoted to various environments such as Development, System Integration and Test (SIT), User Acceptance and Test (UAT), Pre-Prod as it makes its journey to Production.

Specific testing is performed in each of these environments, with any defect resulting in the impacted code being modified to “hopefully” address the issue, and then re-start the journey from Development to Production. This route the deployments take is sometimes referred to as the "path to production".

What is rarely appreciated is that we are not just testing the code as we move it through each environment, but the release process of building, deploying and configuring the code, and the configuration of the infrastructure to run that code.

The problem with Configuration Drift

It is unfortunately all too common for each new release to fail the first time it is deployed to each environment, in many cases the deployment itself will fail due to configuration issues. In these cases, developers will often make a manual “quick fix” to complete the deployment process so that testing can begin.

This can quickly lead to small discrepancies between environments, such as disparities in the configuration of deployed SOA/BPM composites and OSB services, adapter configurations, WebLogic resources, or applied patches.

Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted months of man effort to resolve

These discrepancies, referred to as configuration drift, often impact the behavior of the code and start to surface during testing or worse in production. Typically these issues are reported as code defects. The standard process to try and resolve any defect is to re-produce it in an earlier environment, so that the root cause can be diagnosed, a fix can be produced and validated before promoting to upstream environments.

But often we face a catch-22 scenario, where we cannot re-produce the issue in other environments due to the configuration differences! As a result, issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted months of man effort to resolve; resulting in project delays, cost blow out and missed milestones.

Worse still, if these issues surface in Production, it can result in all sorts of problems including security, performance and availability issues that can have a material impact on the business. For example, disaster recovery failures and HA system failures are frequently a result of configuration drift.

Minimize differences between Environments

To avoid this, we need to ensure that each environment in the software delivery pipeline should have an equivalent configuration to production.

The first step in enabling this is to start off with equivalent Oracle Middleware environments at all stages of the SDLC. This on its own can be virtually impossible to achieve when provisioning these environments manually.

Rubicon Red MyST allows users to fully automate the end-to-end process of designing and provisioning Oracle Middleware environments, on-premise and on-cloud. Enabling users to deliver a consistent and reliable platform in minutes, NOT weeks or months.

Central to MyST is the Platform Blueprint, which provides an abstraction layer over the underlying infrastructure, meaning that you can use the same Platform Blueprint to provision production-like environments to your staging environments.

A Platform Model is then used to capture all the environment-specific configuration information required to provision an instance of an Oracle Middleware platform into the selected environment. With this approach, we create a Platform Model for each staging environment, all based on the same Platform Blueprint.

The Platform Blueprint and Model are placed under version control, allowing us to treat configuration as code. This gives us the flexibility to provision a consistent middleware platform across all environments, as well as having the ability to roll forward / backward to a different version of the platform and hence eliminate the possibility of configuration drift.

Once we have automated the provisioning Next is to fully automate the build and deployment of code to those environments, as well as any additional configuration changes required to execute the code. Again with MyST, this is straight forward (see Automating the Build & Deployment of Oracle BPM and SOA Projects for further details).

Putting in place automated platform provisioning and continuous delivery for Oracle Middleware, will allow us to make significant progress towards preventing configuration drift. However whilst this addresses many of the root causes of configuration drift, it does not address all of them.

The reason for this, is each time we make a deployment to any environment, we are making changes to that environment, which means that it is no longer in-alignment with production. If the release passes, and the code gets successfully promoted through all the remaining stages and into production then this is not an issue. As eventually the same release will be promoted to production and our environments will be re-aligned.

But if the release fails, we will need to create a new release, in order to address the defects reported in the previously failed release.

As a result, what is deployed to any environment is the culmination of previous failed releases and configuration changes (i.e. releases that have not made it into production) plus the latest build. In addition it’s quite common when releases fail, for developers to make quick fixes outside the release process, so that testing can continue until the new release is available.

This leads to the environment becoming less like production each time we deploy a new build; causing configuration drift. This commonly leads to releases failing when promoted to the next environment, or worse still, when promoted into Production.

Teardown / Provision Oracle Middleware Platform… Automated

To prevent this, then prior to making the next release, we need to restore the environment back to its pre-deployment state, so that it is re-aligned with Production.

Typically, the easiest way to guarantee this is to tear down our middleware platform and re-provision it to a state consistent with production, prior to each release. This includes re-provisioning the Middleware environment from scratch, and then re-deploying the current version of applications that are deployed in production.

For our CI environment, where we are often making several releases a day, it’s not practical to re-provision the environment prior to every release. Rather, it is better to schedule a re-build of the CI environment nightly.

As we have already automated the provisioning of an environment, as well as the build and deployment of applications to an environment, then automating the re-provisioning of an environment is very straight forward.

As MyST integrates with popular Continuous Integration tools, including Jenkins, Hudson, TeamCity and Atlassian Bamboo, it is very easy to integrate the re-provisioning of an environment into our delivery pipeline.

An additional benefit of this approach is that it helps to encourage developer best practice when making configuration changes. As we have previously observed, deployments will often fail due to configuration issues. In these cases, developers will often make a manual “quick fix”; this will fix the release in that environment, but those changes are often forgotten, and the same issues are encountered at the next stage. Tearing down our middleware platform and re-provisioning will quickly discourage this unproductive behavior and encourage developers to make configuration changes as part of a controlled process.

Reduce Risk, Decrease Costs, and Speed Up Time to Market

Organizations are in a digital race, where the speed at which IT can reliably deliver new features and innovations is what sets them apart from their competition.

Eliminating configuration drift during the development of Oracle Middleware projects can significant reduce the amount of time wasted troubleshooting and resolving configuration issues, significantly reducing the overall project development time and cost, as well as significantly reduced the risk of a major issue at go live.

In short, the benefit of adopting these techniques as part of an overall Continuous Delivery strategy will provide the business with a strategic advantage in its ability to be more responsive in delivering new solutions faster, cheaper, and more often.

Download White Paper

Click here download a white paper on Best Practice for Implementing Continuous Delivery for Oracle Middleware.