Network Comparison Application Security Testing (NCAST) – A Crucial Tool for Modern DevOps

Abstract

Network Comparison Application Security Testing (NCAST) is a useful and necessary addition to the DevOps tools organizations use. NCAST can both speed the creation of unit, API, regression, and other tests and significantly improve the security and quality of software. More importantly, it can detect unexpected vulnerabilities (including zero-day vulnerabilities), bugs or performance problems, and supply chain issues—such as from a new or updated version of open-source code. Conveniently and unlike certain other types of testing, NCAST does not require changes to the source code. NCAST is especially useful for detecting problems introduced by code changes or environmental changes like moving the code from on-premises to cloud.

Embracing NCAST will result in increased velocity, quality, and security for software.

Background

Many software and IT organizations have embraced DevOps to increase the velocity and flexibility with which new features and applications can be developed and deployed, while at the same time improving quality. Automation is the key to doing this and organizations put together automated Continuous Integration (CI) and Continuous Deployment (CD) pipelines to implement DevOps.

A key DevOps tenet is to shift left as many tests as possible earlier into the pipeline. This includes shifting left security testing so that rather than trying to bolt-on security after the fact, security is designed into the application and automatically tested in the pipeline. To emphasize the importance of shifting security left with DevOps, the term DevSecOps has become popular.

Too often rather than shifting testing left to earlier in the pipeline, organizations try to make up for the lack of early testing, like unit testing, by spending tremendous amounts of time and effort building additional late-stage automated QA tests or, sadly, still relying on manual QA testing late in the release process. One reason for this is that often developers have not implemented unit tests for most of the code. Another reason is that the organization has insufficiently invested in application security and other automated testing tools.

Perversely, organizations which make these classic DevOps mistakes, do so with the excuse that they need to hit deadlines and don’t have sufficient resources or budgets to do DevOps right. Yet these shortcuts incur technical debt in the form of code without sufficient unit tests, which are difficult to build after the software project is completed, and which are insufficient to support the necessary quality and security of the resulting software.

Inevitably, these shortfalls result in bugs and vulnerabilities in production that cost far more than doing DevOps right in the first place would have cost. Jones Capers determined in Applied Software Measurement: Global Analysis of Productivity and Quality that finding a defect after software is released costs 16x more than finding it during final system test, 160x more than during unit testing, and 640x more than finding it during development.

Creating automated unit tests, application tests, API tests, acceptance test, performance tests, security tests, etc. is time consuming without investing in appropriate tools. For example, Static Application Security Testing (SAST) and other code scanning tools should be run as part of the build process to verify that code has been written well and doesn’t contain common programmer mistakes. SAST is passive code scanning that doesn’t require executing the software.

Dynamic Application Security Testing (DAST) can be used when the code is running to analyze the behavior of the software and look for suspicious or vulnerable activity. Interactive Application Security Testing (IAST) usually requires specific changes to the code, but can go even further to examine that code performs as expected. At a minimum organization should use SAST and strongly consider DAST. Due to the amount of work involved in changing source code, IAST may not be appropriate for some DevOps situations.

Introduction to NCAST

NCAST compares the traffic responses generated by a newly upgraded version of software to those generated by the current version of the software.

What has been missing for most DevOps organizations, is a way to easily determine if software is performing in an expected fashion relative to the network traffic it generates. This is crucial since most of today’s software uses network services to receive requests and generate responses. Network Comparison Application Security Testing (NCAST) gives a way to easily do the necessary network traffic analysis comparison. NCAST solutions, available from companies like Curtail, analyze the traffic responses generated by a newly upgraded version of software and compare it to traffic responses generated by the current version of the software—in lockstep against the same input requests.

For example, see Figure 1 where NCAST is deployed so that production traffic being used by the current version in production of the software (blue box) is also sent and compared to the release candidate (upgraded) version of software (green box). In this example, development will have already indicated to the NCAST tool what the expected network traffic differences are due to feature changes. Other differences that are found by NCAST are unexpected and must be accounted for. They might be bugs, vulnerabilities, environmental differences between staging and production, or differences due to feature changes that should be redefined as expected.

Figure 1 – NCAST Compares results from current version to new version

Since NCAST does not require any changes to the software, adding it into a CI/CD pipeline is straightforward. It also does not impact user experience at all, even if using production traffic as an input for comparing upgrade candidates.?Using production traffic with NCAST can provide an easy way to have broader User Acceptance Testing (UAT) coverage safely and detect regressions that would be triggered by unanticipated user and systems interactions “as if” in production.

NCAST Deployment

Unlike certain other types of testing methods like IAST, no source code changes are required to use NCAST. Once NCAST is deployed it can immediately detect zero-day vulnerabilities, bugs, performance problems or any other issues that might be introduced from code changes, especially regressions.

NCAST can be deployed as a container or VM either inline or passively, often behind a load balancer. When deployed actively inline, NCAST acts as a proxy examining all traffic requests and responses to both servers. When deployed passively as a tap into network traffic between a client and server, NCAST can then duplicate the network traffic to the upgraded server software.

Another method for using NCAST passively is to create a traffic recording of a known good version of the software and then have NCAST replay that traffic to the upgraded version under test and see if it produces unexpected network traffic differences.

If TCP mirroring has already been deployed, NCAST could take advantage of that capability. For high traffic situations, NCAST could also be deployed in a fashion to use sampled traffic and still provide the necessary comparison tests to look for unexpected differences.

If the traffic NCAST is seeing is encrypted, only some tests will be possible since deeper packet analysis will not be available. To get the full value of NCAST, consider a deployment option that lets it see more unencrypted traffic. For example, NCAST can be configured to proxy the TLS traffic by configuring it with the appropriate TLS keys, or NCAST can be deployed within the same pod where unencrypted traffic may be available. Another method would be to log a complete or partial set of network requests and responses and provide those to NCAST.

NCAST Deep Protocol Analysis and Mapping

NCAST should be capable of looking deeply within the packets and mapping and analyzing various protocols. Each instance of the software, both the current version and the candidate upgrade version, should be sent identical requests and produce identical responses, unless the responses are expected to be different because of the upgrade.

The comparison should be done intelligently so that attributes, parameters, authentication data, session or instance-specific values are mapped seamlessly. Mappings might include http headers or complex JSON mappings. These can then be used to highlight unexpected behavior at a granular level. This allows NCAST to be used to test APIs and pinpoint behavioral differences in a highly deterministic fashion. NCAST also must translate or map IDs since they are likely to be session or server specific.

Use NCAST to Identify Expected Differences

Software changes often include improvements that are expected to change the network responses. Developers and testers should be able to use NCAST to quickly identify expected differences, leaving only unexpected issues that must be investigated.

NCAST As Part of the CI/CD Pipeline

A common place to use NCAST is in the “Test” phase in the Continuous Integration Pipeline as shown in Figure 2. It can be used both as part of unit testing and part of other system testing as shown in Figure 3. If NCAST discovers unexpected differences, it can automatically trigger a pipeline break, forcing development to account for the difference before the updated software can be shipped or deployed. This enables automated network behavior change control for software—something that has long been needed.

Figure 2 - NCAST in CI/CD Pipeline

Figure 3 - in Scanning and Test Phase of CI Pipeline

NCAST Use Cases

Supply Chain Testing

Today’s software usually relies on many 3rd party sources, including open source as part of the supply chain. Even a minor change to software may be built or dynamically linked at run-time with an updated version of a piece of 3rd party software. This can result in the introduction of unexpected vulnerabilities, bugs, and performance problems from the updated 3rd party software. While Software Composition Analysis (SCA) can identify open source and other 3rd party software that is used, it doesn’t assure that such software is performing as expected.

Including NCAST in your CI/CD pipelines can help you detect problems from 3rd party and open source software updates. This is because such problems often cause the network traffic to be unexpectedly different after the newly updated 3rd party software is included (see Figure 4). Moreover, if the supply chain has been maliciously attacked to introduce a vulnerability or malicious code into software—such as occurred with SolarWinds in 2020, NCAST will detect when the malicious software attempts to communicate since that will generate an unexpected difference in the network traffic comparison.?In the future, as hackers increase their dwell times after attacks, looking at the behavior changes in the software may be the only way to know whether or not the software has been compromised by a third party.

Figure 4 – Using NCAST to detect supply chain attacks

Unit Testing

Many developers can run unit tests for microservices or other components they build in their own development environment. Deploying NCAST in that environment can make it easy for developers to create unit tests that ensure that the microservice produces expected responses and behavior when fed different requests. This can be done by either testing two versions (one of which is known to be good) or by using NCAST to replay a known good traffic recording.

Different developers working on the same microservice using NCAST unit tests will have the assurance that their updated code did not generate any unexpected behavior. Now, many expensive-to-fix problems that are not found until in production or late in the test cycle can be found and inexpensively fixed early in the development phase.

As mentioned in the background, many organizations struggle to implement DevOps due to the lack of unit tests. NCAST can be used to reduce the need for so many unit tests or easily create missing unit tests. Shifting defect detection left to unit testing, improves development velocity, raises quality, and decreases the cost to fix defects and respond to security vulnerabilities.

API Testing

Software services, including microservices, commonly provide and use APIs. NCAST is a powerful tool that can be used to analyze and compare the API network traffic response differences for a release candidate with a known good version of the software. This can be live traffic or replayed API traffic.

Regression Testing

NCAST can stop the Whack-a-Mole cycle.

Sometimes bugs occur in production that must be quickly fixed. The danger with a quick fix, sometimes called a hot fix, is that while it might solve the problem, it could introduce unexpected new problems that then also have to be fixed. This constant cycle of hot fix—new bug—hot fix is derisively called, “Playing Whack-a-Mole” in the software industry. Regression testing can help ensure that the hot-fixed version still performs in all other ways like the original version. In addition, when bugs are fixed the organizations might build a regression test to ensure that the same bug does not reappear in a later version.?

Whenever new features are added to software, organizations also use regression tests to try to show that the software still performs as expected even with the updated features. Creating regression tests is time consuming. Moreover, the set of regression tests that are run as part of the CI/CD often grows continuously since obsolete tests are rarely culled, resulting in regression testing taking ever more time and slowing the release process to the point that it becomes impossible to run the set of regression tests for a quick hot fix—and the Whack-a-Mole cycle starts all over again.

This cycle is particularly problematic when a patch is developed to fix a security vulnerability. Due to legitimate concerns about causing other potential problems, a software vendor may delay distributing a crucial patch until after the full suite of regression tests are run. Meanwhile attackers might successfully exploit this vulnerability and attack customers. This is what happened with Kaseya in July 2021 (resulting in over $70M in Ransomware demands) and is a common dilemma for vendors and customers alike: do you patch and pray that everything will work without sufficient testing, or do you wait for the testing and hope no one exploits the vulnerability while you wait?

NCAST makes the process of doing regression testing easier since it can be used to quickly identify any unexpected differences in network traffic and ensure the development team fixes or accounts for them prior to release (see Figure 5). NCAST can eliminate the need for many traditionally built regression tests allowing the overall release process to maintain acceptable velocity and quality so that the tests can be run even with a hot fix, stopping the Whack-a-Mole cycle.

Figure 5 – NCAST can reduce patch validation from weeks to hours

Performance Testing

NCAST can also discover differences in performance between a release candidate and the current release. Acceptance criteria for the release candidate often includes statements like, “must perform at least as well as the current release” or “10% better than the current release”. NCAST can automatically trigger a break in the pipeline if the performance difference does not meet the desired metric. Without NCAST performance problems are often not discovered until long after the release is in production.

Environmental Testing

NCAST can be useful when you move software from one environment to another. For example, when you move software from on-premises to cloud, you could use NCAST to ensure that the code is behaving in the cloud the same way that it did on-premises. Not only can NCAST be used to discover vulnerabilities and bugs, but as mentioned above, it can also discover unacceptable differences in performance.

Blue/Green Deployment Testing

Some DevOps deployments have two deployments, commonly called blue and green. The live deployment is alternated between the two. Whichever deployment is not the live environment at any given time is updated with proposed changes from the CI server. Then at a certain point, that environment goes live and the other becomes non-live. If a problem occurs shortly after going live, then the organization can roll back to the previous version. With NCAST the non-live environment can also be tested with the same network traffic going to the live environment, automatically flagging unexpected differences that must be accounted for before going live reducing the need for roll backs.

Root Cause Analysis

NCAST can also be an important tool for root cause analysis of problems. When NCAST finds an undesired difference, developers can see the exact request that triggered the undesired behavior, the exact field and protocol message that had improper values, and a full history of the network session including a packet capture.?

Moreover, developers can use NCAST to replay the network capture against their development environment providing a quick and reliable way to recreate the problem. NCAST allows developers to quickly identify the code causing the problem and create a fix. In addition, developers can use NCAST to prove the fix is correct.

No More Dead Canaries

Some organizations like to deploy a canary release to a subset of production environments or customers to see if any problems occur in a live environment. This type of progressive release strategy allows an organization to see if the canary release has early problems or if it is safe to continue releasing to a larger subset. NCAST can be used to compare a candidate release’s network responses to those of the previous product version in live environments, improving detection of problems and confidence in moving forward and (unlike normal canary testing) without exposing users to any negative effects of the new software version. Use of NCAST can also reduce or eliminate the need for canary testing since it increases confidence in the quality and security of the candidate release. Since live customers and users are generally the ones using canary releases and experiencing the pain when one fails, they would appreciate having no more dead canaries.

Conclusion

Include NCAST in DevOps CI/CD pipeline.

NCAST is a useful tool that should be part of DevOps CI/CD pipelines. It lowers the cost for developing tests and raises the likelihood of discovering zero-day vulnerabilities and bugs. Embracing NCAST will result in increased velocity, quality, and security for software. It can also deal with vexing problems that many DevOps efforts face like the lack of sufficient unit tests, efficient regression testing, confidence in switching between blue/green live environments, confidence in cloud migration, and avoiding bugs and vulnerabilities being introduced via the supply chain.