The Complexity of Software Development

One may argue that Software Development is a relatively recent field and hence hasn't reached the same level of maturity as say civil engineering. True, but notwithstanding that programming languages haven't changed a whole lot in the last few decades. At least the basic concepts and constructs have remained the same. Software Development was and remains to be a complex business.

Why is that? Given that most programs can be written with relatively simple constructs such as if and for loops, functions, records/structures? Most algorithms can be written with the same (or slightly modified renditions) of the data structures that were conceived decades ago.

Most programmers don't find standalone programs of medium complexity hard to write. In fact, working in a silo, many programmers can perform quite well. But as more features are added over time and more programmers start working on various parts of the same codebase/product, the complexity increases significantly.

Writing software is a human problem. The human brain in it's natural state is capable of focusing on one thing at a time. It could possibly give a semblance of multitasking, but that is more akin to a processor time-slicing threads (threads of attention in case of our brains).

As programmers develop new features by changing let's say a function or a class (the proposed change) they make an attempt to think reasonably about how the change in the behavior of a software component will affect any other piece of software that uses it. They need to consider several aspects:

• Who else is using this component?
• What behaviors do they expect from it and does this change break those behaviors?
• Even if the change doesn't break the behavior today, do the other programmers expect the proposed changes to be made in these components or consider these to be reasonable changes in behavior?
• How does this proposed change change the behavior of other components that rely on or use this component?
• Even if the proposed change is acceptable in current and future expectations of this component, has the proposed change been implemented correctly?

To address some of these issues the software industry uses certain best practices, the common ones being:

Code Reviews - A process where one or more programmers read the changes made by the author of the change and discuss what they agree or disagree with. The result of the code review is usually a set of changes for the author to incorporate. Code reviews are definitely useful but have their own set of challenges, namely:

• Most teams are pressed for time and don't actually do the code reviews very thoroughly but more as a best effort.
• Most of the time the code reviewer is looking at the code in isolation rather than as a part of a larger code base. In other words, the code reviewer isn't necessarily aware of all the various ways in which the component is actually being used by other parts of the codebase than the one that he or she is actually reviewing (unless the reviewer wrote a piece of code that uses this component, in which case it is reasonable to expect the reviewer to take into account this usage).

In many cases the new logic may be complex (for example time dependent, race-condition prone, multithreaded code) that even an accomplished programmer may not be able to account for all the possible conditions in which the code being reviewed may fail. The perspective of the reviewer is very different from that of the programmer as the programmer is actually implementing the component. In some ways, the programmer is a lot more involved (and probably vested) in the correct workings of the component. To understand why consider who understands a book better, the writer of the book or the critic who reads it thoroughly (even if the critic may be an author of a related but different book)? Also, traditionally and typically programmers are more responsible for the correct working of their software than the reviewers that review it.

Testing - Software testing is the single most important practice in software development. Testing can be of several kinds but broadly their are two categories, unit testing and integration (or system) testing.

Unit testing tests a single self-contained piece of software while system testing tests several pieces of loosely (or tightly) connected software components as a single system. Testing, although invaluable and essential for anything that isn't frivolous software development, isn't a perfect practice. Some of the inherent problems are:

• The correctness of test cases - Who shaves the barber? Or who tests the tests themselves for correctness. A logically inconsistent test may prove that a software component with a similar logical flaw works and result in a catastrophe.
• The completeness of the test suite - It is impossible (or at least impractical in most industries) to tests all possible combinations of inputs to a software component and hence programmers (or more ideally, testers) choose the inputs that are representative of a 'class' of inputs. Ideally tests should be written for every decision point in a piece of software and for every possible class of input and for all boundary conditions. But most of the time this isn't what happens. It takes an enormous amount of time and discipline as well as enforcing of best practices to ensure this. Hence most test suites aren't complete.
• System Testing => NOT EASY - System testing is HARD. Many a time components aren't written in such a way that they can be easily tested as a whole system. Some components don't lend themselves to testing and need to be changed to be made more testable.
• 3rd Party components - 3rd party components aren't always easily testable as the vendor may not have made them so. As a result, a system that uses them is also not fully testable.

An unpredictable environment - Many UAT and TEST environments aren't as stable as production environments. This is partly due to resource constraints and partly due to the inherent nature of such environments (since systems are being deployed to such environments more frequently than production environments). Hence they provide an additional hurdle to system testing.

Formal Methods, Software Components & Contract by Design: These methods are less widely used but are areas of theoretical computer science research.

Formal methods are an automated way of testing a software component by testing the state of the system after the execution of each statement. This is obviously impractical in many cases, especially with the untenable complexity of modern languages. Also, the time needed to test software components (not to mention 3rd party libraries) probably makes this method limited to theory.

Design by Contract: This is the practice of meticulously specifying the inputs, outputs, pre-conditions and post-conditions of units of software components (such as functions, classes) and having the execution environment verify these at runtime. Surprisingly this practice isn't as widely used as I would have expected. One of the reasons I can think of is that such checks at runtime would make time-sensitive systems slow. The other reason is probably the lack of built-in support for Design by Contract in most popular languages. Lastly, time taken to meticulously think about these contracts, not to mention the ongoing maintenance of the pre/post-conditions as behavior evolves is a major factor as well. TDD or test driven development is often used as a substitute to specify and enforce behavioral constraints.

The field of software development is complex and the development of new systems, platforms, tools, devices and languages is increasing complexity on an hourly basis. Countless hours are spent in diagnosing bugs after they cause production issues and affect clients. It remains to be seen whether the standard practices of testing software will prove to be sufficient for rapid software development. It suffices to say that we aren't there yet.