The Invisible Process

Recently, I was in a conversation with a colleague regarding when to implement processes within a company versus when not to. This discussion then branched off into the various ways a process can be implemented well or not, and what makes that difference. One suggested idea was of processes being invisible, in the sense of a process being built such that the majority of individuals involved in the work do not have to understand the process as a whole. That was the jumping off point for this article which looks to explore the idea of invisible processes.

Let's start by trying to define what both constitutes a process, and what it means to be invisible in the context of this thought experiment.

A Process

1. A defined, repeatable sequence of actions that combine to produce a desired result.
2. Actions enacted independently by multiple individuals.

An Invisible Process

1. Individual actors of process steps are aware, only, of their own responsibilities.
2. The collective group of people who are not charged with any actions pertaining to the process are aware of the expected input and output without requiring any knowledge of the steps in between.

My knee jerk reaction is that such a system would induce chaos to such an extent that any process would fall apart. If people don't understand the whole picture they won't see the value in their own actions within it. This would lead to the process not being followed with people opting for easier routes that fulfill their own needs but break the invisible system which they cannot see.

In debating my internal monologue, this initial argument starts to unravel by pulling at the thread of automation. If the steps between input and output are done by a machine then I am far more willing to accept it as a black box. A concrete example comes from one of my development teams who are working on a suite of android applications. The CI/CD pipeline performs numerous tasks that the team readily accept without the need to really understand how it works.

The development team are aware of the responsibilities they have:

1. To use conventional commits in git with the feat: and fix: tags being human readable accounts the changes.
2. To make a pull request against development for work that they consider ready for deployment.
3. To add tests in the codebase for their changes.

The team member responsible for deployments is aware of the outputs and actions they are charged with:

1. Checking tests ran successfully (output from CI/CD - visible in pull request).
2. Referring failed tests back to developers to fix issues (logs outputted by CI/CD).
3. Merging the PRs with a squash commit, ensuring the wording in the fix: and feat: descriptions are customer friendly.
4. Merging development into the release branch in order to trigger a deployment.

The company as a whole is aware only of the expected output from this process:

1. Correctly versioned applications.
2. Release notes generated and published.
3. Customer specific application versions built and deployed to customers.
4. Test reports published alongside release notes and app downloads.
5. Slack notifications to customer facing staff alerting them to the deployment and informing of the release content.

This process has served us well, allowing for smooth, regular deployments for the past year. By our chosen definition this would qualify as an invisible process, the only caveat being that the majority of actors are machines.

The natural next step is to then consider whether there are fundamental differences, between human actors and machine actors, that would render invisible processes inviable. If the input and output of any particular step within a process is known to the actor, then it should be just as workable a solution regardless of whether a person or a computer performs the action. The difference lies in the reliability of the output, a programmed machine will perform the task the same way every time, a human will not always do so. They may change how they perform a task in order to optimise their own workflow, which could inadvertently affect other parts of the process. They may omit certain outputs which they believe are superfluous and unnecessary from the perspective they have.

In the case of a process that is explicitly visible to everyone, an individual can be expected to understand the process as a whole. In that case they would know what aspects of their own workflow and output are essential or not. For an invisible process to work, the inputs and outputs for each individual need to be clear enough that they gain the knowledge of what is essential without the context of the larger picture.

Is it worth it?

Having concluded in my own head that invisible process should indeed be workable, there is the question of whether doing so is worthwhile. What do we achieve by implementing an invisible process as opposed to trying to spread company wide awareness of a process? The answer as I see it is quite simple but also significant: less noise. A company will have countless processes, in many cases there will be variations of those processes for specific circumstances, customers, projects, teams. Asking everyone to try and hold all that in their head, or writing a million pages of documentation pages for everyone to search through each time they need something is a lot of noise and more often than not it only creates confusion.

I had long been an advocate of always giving the context of decisions to my teams so that they can understand and appreciate the reasoning. This was based on a projection of my own feelings, I struggle greatly with working on something without understanding the whole picture. One of the phrases that frustrates me to hear more than almost any other is "you don't need to worry about that". It is the easiests of traps to fall in, to assume that everyone thinks like us, but it is a false premise and led me astray in this case. By really observing and listening to my team members I realised that everyone has a different level of interest in the details beyond their own tasks, and in fact I was the extreme case.

With the realisation that in pretty much every case, my team care significantly less about the wider picture, I have come to see that providing that larger context is not always helpful and can often even be counterproductive. I believe that many of our day to day processes in development would fall into this category. So while invisible processes will not be the best choice everywhere, I think that there are many instances where they would be a better way to go.

Tangent

In writing this article a notion came to me, that I think bears a closer look. In reflecting on the variance of how much people want and need to know about the context and background decisions behind the work they are doing, I started to consider who cares least and most. From a quick anecdotal survey, and hastily jumped to conclusion, I believe that there may be a correlation between people who excel as individual contributors wanting to reduce the noise of things outside what they need to do. On the other side, people who excel at glue roles (thinking of the archetypal servant leader roles) are likely to have an innate need to understand all the context of what is happening. I'd like to explore this idea further, and in order to do so would need to speak to more people. So, if you have thoughts on the subject, if you follow this trend or buck it, please comment or reach out to me to discuss.