The Agony of Dependency Management

Dependency management is one of the shared challenges that companies that build complex software spanning many teams deal with.

To be clear about what “dependency management” is, consider an example whereby Team A changes component A, but A is used by component B, which is maintained by team B, and the change to A necessitates a change to B. How do you coordinate those changes? And where do you test them? If you test them in a shared multi-team test environment, then all of the other teams (C, D, E…) will be affected, because they are less far along and still need the old versions of A and B, so if you put a new A and B in there, you will break testing for teams C, D, E, and so on.

To deploy A and B into a shared test environment before other teams are ready for it is what teams mean when they complain that a test environment is “unstable”.

Also, how do you coordinate or specify the changes that are needed to A and B, while at the same time permitting other teams to make changes that require additional changes to, say, B?

Tools like git enable a single team to coordinate multiple concurrent changes to a single component. However, git does not solve the problem of how to coordinate changes to multiple components. To address that problem, other techniques are needed.

Techniques for Managing Dependencies Between Components

Dependencies between components can exist in two forms:

1. Intra-runtime dependencies - One component links with another to form a larger component. E.g., a component might require two jar files. If one of the jar files is later modified and a new version is created, it must be tested for compatibility with the other jar file—otherwise the link will fail, or it might succeed but there could be runtime errors that occur when the combined component is used.

2. Cross-runtime dependencies - A dependency between two components that span runtimes, e.g., two microservices that interact via a REST API, or two apps that each use the same Kafka topic or the same database tables.

The following techniques are useful for dealing with the above types of dependencies. Below, I’ll explain each.

Table 1: Techniques for managing dependencies between software components.

Technique: Feature flags/toggles

A feature flag (aka “toggle”) is a configuration parameter inserted by a programmer, enabling portions of code to be disabled. This enables a component to support two different and potentially incompatible versions of functionality—the feature flag chooses either the old behavior or the new behavior. Thus, if a client of the component requires one version, it can select that version during its testing or operation; another client can select the other version.

Feature flags are usually set at deployment time, rather than runtime. However, API versions (see below) are essentially feature flags, and they are a runtime selection.

One disadvantage of a feature flag is that it makes the code complex: the code has to support two different behaviors, and this complexity can create bugs. Another disadvantage is that if the feature flag is a deployment configuration, then a shared test environment can only support testing with one version of the feature at a time. (This is a good argument for not using shared test environments.) Feature flags make a build and testing process very complex if they are used too much.

An advantage of feature flags is that one is able to immediately merge a new feature into a component’s shared development branch. Clients of the component that are not yet ready to use the new behavior can simply disable the feature. However, it is often the case that a new feature in one component is dependent on a new feature in another component, and so the right flags need to be selected for each when deploying the components. Keeping track of the feature dependencies can be complex.

Technique: Feature branch builds

A feature branch build is an alternative to feature flags. Instead of layering new behavior into a component so that both old behavior and new behavior is supported, a feature branch build implements only the new behavior. (This is related to the “gitflow” approach.) A build (binary) is generated for the component, and it is stored in a shared artifact repository with a label that states the name of the feature branch that it was created from.

Feature branch builds are useful for pre-merge integration testing, because clients of a new feature can pull the desired built artifact from the artifact repository, deploy it in a private test environment, and then test with it. There is no need for a feature flag when deploying the component. The code is also not cluttered with support of multiple behaviors.

The disadvantage of feature branch builds is that one must keep track of feature branches. This can be simplified by naming a feature branch according to the Agile story that requires the change: in theory, each affected repo should have a feature branch named after that story, with those branches merged in only when they pass the integration tests (see below). However, stories sometimes affect the same code, and so it is essential to do merges to the various affected branches in the same order. In practice, this tends to not be much of an issue.

Technique: Pre-merge integration regression testing

In this approach, feature branch builds are integration tested before the feature branches are merged into their respective repo’s shared development or master branch. For example, suppose an Agile story called 123 requires changes to component A and changes to component B. One can create a feature branch in repo A called “Feature 123”, and a feature branch in repo B called “Feature 123”. One can build A from feature branch “Feature 123”, creating a binary (e.g., jar file) which is then stored in an artifact repository and labeled “Feature 123”. One can do the same for B, storing a binary for B labeled “Feature 123”.

To test these together, one can pull those two binaries, deploy them into a private test environment and then run the integration tests. If—and only if—the tests pass, one can then merge A’a “Feature 123” branch into A’s shared dev or master branch, and merge B’s “Feature 123” branch into B’s shared dev or master branch.

The advantage of this approach is that other programmers are not exposed to the changes until the integration errors have been worked out. Using this approach tends to reduce your system-wide “integration phase” toward zero.

Technique: API versioning

API versioning is a runtime technique, and is commonly used for protocols, including REST services. In the REST version, it is common practice to specify the API version in the request URL immediately prior to the endpoint name. The REST service receives this request and if it still supports that version of its API, it returns the result using that API version, otherwise it returns an error.

API versioning is a way to defer requiring clients of an API from upgrading to a new API until they are ready. However, sometimes a new business feature requires new APIs to be adopted, and so API versioning will not help in that case. Also, API versioning can be over-used: supporting old API versions makes code complex, and continual new versions can be burdensome for API clients.

Technique: Database schema versioning

Database schema versioning is a build time technique, and it is important for Continuous Delivery (CD) settings that treat database schema as code. A database cannot have more than one schema at a time for the same tables: the schema versions are historical versions. Each successive version is implemented as a “migration” from the prior schema to the new one—essentially a delta change. This works because in a CD process, test databases are created from scratch each time a test suite is run, database migrations applied, test data loaded, tests run, and then the database is torn down. The only time that process is not followed is when a test is performed using a very large (production-scale) dataset.

Technique: Monorepo

A monorepo enables you to create a feature branch that spans multiple components. That saves some of the coordination that is needed for making related changes to multiple components, but it does not solve the fundamental problem of deploying and testing compatible versions of components: one must still reach out to other individuals or teams who maintain those components and agree to work on a feature branch together; and one should build from the feature branch to run integration tests prior to merging the feature branch.

Some monorepo systems—like Google’s—are really hybrid systems that allow one to branch on a folder—at least, that is my understanding, although I have not personally used Google’s system. Such a system is not really a monorepo. In a true monorepo system, a branch is a branch of the entire repo—the entire system.

If you have lots of teams, and you branch the entire system, you can expect hundreds—maybe thousands—of merges to occur while you work on your feature branch, and you will have to pull from your upstream to obtain all of those changes to get up to date. In practice, most of the changes will not conflict and the merges will be done automatically. Still, it is important to understand what is going on here: you have branched the entire system and others continue to work on the version that you branched from.

Thus, a monorepo does not really solve the problem of coordinating changes to many different parts of a system. What it does, which is slightly useful, is tell you which components have been changed (although you often know which components you are affecting)—that is the important task of determining dependencies—but you still need to reconcile those changes. It is not foolproof, because with loosely coupled message oriented systems, the version control system cannot always determine which components are affected, because it does not know which components consume each message.

All this is not really that much of an advantage if you have contracts between your components, and you do integration testing, which you need to do anyway with a monorepo.

To use a mono repo, one must use a source repository system that can handle that. Google developed its own. Out of the box git cannot handle extremely large repositories. Microsoft, which maintains all of Windows in a single git repo, has made modifications to git, known as GVFS, to enable it to efficiently handle large repos. Github plans to support GVFS in the near future.

Technique: Share-little

In a share-little approach, dependencies are avoided by never using other things: instead of using an existing module or component, one re-creates it where it is needed—a so-called “copy-paste”—and maintains the new version separately. This is a problematic strategy for an enterprise because it results in massive redundant code, making it nearly impossible to ensure that a bug that is fixed in one place is fixed in other places as well.

Technique: Dependency analysis

Dependency analysis is simply determining the downstream dependencies on a component (things that use the component or use its output), as well as the upstream dependencies (the things that a component uses or that it uses artifacts from).

Dependency analysis is crucial because it enables one to run integration tests more often: it takes less long to run fewer tests, and if one can determine that only a few components are affected, one can run tests that only span those.

Dependency analysis usually entails judgment. While Google has created a tool that performs automated dependency analysis, in practice it is a hard problem, because dependencies are often not explicit in code. For example, a Java program can load classes using reflection, bypassing static declaration of dependencies.

For Java, Maven build files declare dependencies, but those dependencies are at a project level, which is so large grained that if one relies on that, one ends up testing way more than necessary for each change. (Project level binding is also why code is so bloated today—most applications use only a tiny percent of what they get bound to at link time or runtime.) Indeed, the trend toward dynamic rather that static binding has made dependency analysis very difficult.

In practice, it often makes sense to perform a full system regression test on a regular basis, but for on-demand integration tests one should specify a subset of tests to run, based on judgment of the impacted components.

Technique: Pull request

A git “pull request” is a manual workflow step that ensures that a human is in the loop for a merge. It is up to that human to decide if enough tests have been run, using production-like test data. It is highly recommended that integration tests be included in the criteria for what is “enough tests”.

Technique: Collaboration and coordination

It is possible for a few teams to collaborate and coordinate sufficiently that they do not need any automated or systematic help for changing multiple components that have interdependencies. However, if one relies only on collaboration and coordination, things break down quickly as complexity increases and as the number of teams increases.

Summary

There are many techniques for dealing with dependencies between components. A simple strategy of "put it all in the Integration environment" is old school, and can be greatly improved by using the available techniques.