How do we gain confidence in our software systems?

Mastering the Interplay of Empiricism and Theory in Software Development

"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald E. Knuth

“Nonfunctional requirements should not be an afterthought in a distributed architecture.”?

We can confidently predict that the sun will shine tomorrow. How can we make such a prediction? It is because the sun has consistently shone each previous day. At some point, we believed that all swans were white because that’s what we observed. This belief was shattered when black swans were discovered in Australia. This type of reasoning, forming a belief based on repeated observations, is a cornerstone of scientific empiricism known as inductive reasoning. However, it only takes one counterexample to render such an argument false.

We build and test software systems. We subject them to various automated tests to gain confidence in their reliability. There are many types of testing, and the simplest tries randomly to break a system. If a test breaks the system within its expected production behavior boundaries, then we have found a counterexample or, in other words, a bug that requires rectification. And it is important to maintain those boundaries. For example, it is unfair to overload a transactional system with a million transactions in 24 hours when we know that the expected daily peak load is a thousand transactions. Still, it might be acceptable to test it with two thousand transactions.

We can prove a bug and know that the system doesn't work. The reverse isn't so cut and dry. Testing can only give us some level of confidence in the proper behavior of our system. Such confidence derives from repetitively executing the test suite with minor scenario alterations and at its boundaries. Consider the transactional system designed to handle one thousand daily transactions (for instance, some small e-commerce online company). You would perform multiple tests under varied loads within 500 to 2000 transactions and under various stressors, such as the failure of specific components. You will also test in the extreme cases of zero or one transaction. You will gain confidence in the system's resilience after a series of tests, say 50 to 100.

As in nature, inductive reasoning is our only method to comprehend the universe. We extract mathematical theories from observed physical phenomena around us. For example, Einstein postulated the special theory of relativity in response to the invariant speed of light, regardless of the observer's frame of reference. This foundational insight was based on empirical observations and corresponding experiments.

Digital computers are like the universe in the sense that both compute and are based on mathematical rules. However, they are different in that we know these rules, and we built them based on these rules. This leads to the second form of reasoning: deductive reasoning.

At its simplest, deductive reasoning involves premises and a conclusion. The argument is valid if the premises logically lead to the conclusion. If the conclusion is true, then the argument is sound. Deductive arguments address facts that remain consistent in any universe. For instance, the circumference of a circle being twice the radius times Pi is such a fact. The fastest running time of a sorting algorithm is "Nlog(N)", is a second one. And the requirement to choose between consistency and availability in the event of a network partition of a distributed system is a third.

While developing software systems, particularly distributed ones, it is essential to understand the theoretical underpinnings of their behavior. Automated tests can certainly boost confidence in a system's expected operations, but understanding the system from a deductive reasoning perspective is equally important. Many practitioners shy away from theory, but it's worth noting that the world's largest distributed systems, such as Google, wouldn't have been possible without it. For instance, Google engineers initially noticed random computing node failures. These observations led to the design of MapReduce and failure-tolerant distributed computing, which powered Google for years before the advent of the deep learning era.

As we build increasingly complex systems, our understanding must evolve to match their intricacies. Our belief in the robustness of these systems hinges not only on empirical testing and observation but also on sound theoretical knowledge grounded in mathematical principles. This blend of inductive and deductive reasoning forms the crux of efficient system design. Practitioners must pay attention to the theoretical foundations that inform their work, despite the often-tempting lure of pure empiricism. The world's most powerful distributed systems remind us that theory and practice are two sides of the same coin. As we navigate the challenges of system failures and aim for resiliency, we must remember that the interaction between observed data and mathematical theory is at the heart of every system we create and every problem we solve. The black swan awaits us in every unexplored corner, urging us to challenge our assumptions, test our systems, and seek a deeper understanding of the principles that govern them.

The present article was revised using ChatGPT.