The Software Entropy

In this article we will talk about broken windows, boiled frogs, comments in code, and software entropy.

Last week an event happened to me that made me decide to write this article.

I already told you in previous articles that I have been developing software for exercise and rehabilitation equipment for several years

So, last week I had to reopen the source code of an equipment called "Stair Climber" which is a sort of escalator which makes you climb its steps by keeping the same pace with which the steps descend, trying to stay at the same height.

It was necessary to adapt the software of our Stair Climber to a hardware modification made to increase the safety of the equipment and therefore I had to analyze and estimate the development times for the software adaptation.

figure 1 : Stair Climber

I remembered that the software that managed the drive of the Climber had a good code structure. In fact it was a state machine implemented with the state (1.) pattern.

The main idea is that, at any given moment, there’s a finite number of states which a software can be in. Within any unique state, the program behaves differently, and the software can be switched from one state to another instantaneously. However, depending on a current state, the software may or may not switch to certain other states. These switching rules (transitions) are finite and predetermined. There are classes for all possible states of our state machine object, called "ClimbStateMachine" (CLBSM) , and we extracted all state-specific behaviors into these classes. The CLBSM stores a reference to one of the concrete state objects and delegates to it all state-specific work. The CLBSM communicates with the state object via the state interface.?To transition the CLBSM into another state, it has to replace the active state object with another object that represents that new state. This is possible because all state classes followed the same interface and the CLBSM itself works with these objects through that interface.

I had not looked at this code for over a year and when I reopened it to estimate the cost of adapting the software to the new hardware component, I found it as a jumble of hard-to-understand code. So I start looking on "Git" for the history of the changes made to that code in the last year. Going backwards, I saw that some adjustments had been made, one worse than the other, but with great disappointment I realized that the first doing a bad coding was myself. I did it to implement the "UserUndetect" feature which allowed to stop the movement of the steps when the user leaves the equipment without having finished or stopped the exercise. In this way, if another user arrives before the exercise ends for "unmovement timeout", he will still find the climb braked.

This error of mine was a bad design mistake. Instead of considering the "UserUndected" as a new state of the "ClimbStateMachine" I had handled this event within the pre-existing states. I had therefore complicated the logic of existing states with a duplication of code between states. I realized I was wrong but there was no more time, I had to close the version for release. So I thought I'd write some comments on the code to make it more understandable, since it was very complicated. The purpose was still to go back over it to remedy that technical debt, but then I forgot.

So I left a "ugly broken window" in the code, broken not only by the bad design and code duplication but also by the comments I put in.

figure 2 : class diagram and state diagram of "ClimbStateMachine"

Let me give you a piece of advice: when you think you have to write a comment to make the code understandable, it means that you have written poor code. Good code does not need comments to be understood. Robert Martin in "Clean Code" says:?"Don’t comment bad code, rewrite it." (2.).

Nothing can be quite so damaging as an old comment that propagates lies and misinformation. The use of comments is to compensate for our failure to express ourselves in code. Indeed, comments are, at best, a necessary evil. If we had the talent to express our intent with just the programming language and identifier names (3.), we would never need comments. "So when you find yourself in a position where you need to write a comment, think it through and see whether there isn’t some way to turn the tables and express yourself in code".

One of the more common motivations for writing comments is bad code. We write a module and we know it is confusing and?disorganized, as I did in the example I told you. Clear and expressive code with few comments is much better than messy and complex code with lots of comments. "Rather than spend our time writing the comments that explain the mess it would be better to spend it cleaning that mess".

It is sometimes reasonable to leave “To do” notes in the form of //TODO comments. This is exactly what I should have done, instead of leaving a comment explaining the mess I had made. I should have left this comment:

// TODO : REVRITE IT WELL ASAP -> Stefano

What happened then was that I had left a broken window with the intention of repairing it soon, which I then didn't do, because I forgot to do it.

Later my collaborators continued to develop code on that broken window without realizing that the structure of the module was deteriorating and the entropy of the code was exploding. They also didn't add new states to the state machine but, trusting what I had done, they complicated the existing states.

So I broke a window and they acted like boiled frogs.

Therefore broken windows and boiled frogs are serious and dangerous things, we must learn to know them to avoid them.

Let's start with the broken window theory.

The "Broken Window Theory" is based on an Atlantic Monthly article published in 1982.

Social psychologists and police officers tend to agree that if a window in a building is broken and is left unrepaired, all the rest of the windows may soon be broken. This is as true in nice neighborhoods as in seedy neighborhoods. One unrepaired broken window is a signal that no one cares, this instills in the inhabitants of the building a sense of abandonment so another window gets broken.

People start getting dirty and graffiti appears. Serious structural damage begins. In a relatively short space of time, the building becomes damaged and the sense of abandonment becomes reality.

The "Broken Window Theory" has inspired police department in New York to suppress the small stuff in order to keep out the big stuff. It works: keeping on top of broken windows, graffiti, and other small infractions has reduced the serious crime level.

So we programmers don't have to leave "broken windows" unrepaired: bad designs, wrong?decisions, or poor code. Let's fix them as soon as we discover them. If there is not enough time to fix it properly then we have to comment the code with an indication that we need to fix that code as soon as possible. If we are good programmers this is the only kind of comment we can use.

I've seen clean, functional systems deteriorate pretty quickly once windows start breaking. Neglect in us programmers accelerates code going bad very fast.

But we all work in a team, and in the team there is another important danger, that of the "Boiled Frog Syndrome".

The 'Boiling Frog' syndrome is based on an urban legend describing a frog being slowly boiled alive. The premise is simple: if a frog is suddenly put into a pot of boiling water, it will jump out and save itself from impending death. But, if we place the frog in a pan of cold water, then gradually heat it, the frog won't notice the slow increase in temperature and will stay put until cooked.

This principle can be used to describe the behavior of inert, immobile, submissive, renouncing, careless people, who take no responsibility.

Note that the frog syndrome is different from the broken windows. In the Broken Window Theory, people lose the will to fight entropy because they think no one else cares. The frog just doesn't notice the change.

Don't be like the frog. Keep an eye on the big picture. Constantly review what's happening around you, not just what you personally are doing.

Therefore we have seen two psychological aspects often triggered by technical debts present in the source code, that of broken windows and that of the boiled frog syndrome, which gradually lead to the ruin of a system's software. This perhaps slow but inexorable trend towards the deterioration of the software is interpreted as an increase in entropy.

The "Entropy" also refers to the second law of thermodynamic in Physics. It states that, for a closed, independent system, the amount of disorder doesn’t decrease overtime. It can stay stable or increase. The idea of software entropy for Software Engineering was influenced by the scientific definition. Basically, more a software system change, more its disorder, its entropy, increases.

We all need to understand, us, poor developers, this tragic truth. We can fight the amount of entropy in our software, but we can’t entirely remove it.

We will add more and more stuff: features, code, packages, etc. How do we create complexity? We just need to scale your project. More elements often bring more coupling, then of complexity. I wrote already an article about component coupling management (4.).

But it is also true that if we do not create a certain amount (right amount) of components we risk creating components with too many responsibilities and entropy ends up in there anyway as "Cyclomatic Complexity".

Additionally, if we need to change something in our software after enough time, we won’t remember everything about it. Even the best documentation won’t help us. The world changed, and what was true yesterday might not be so true anymore, on the technological level as well as on the business level. These changes will introduce a lot of entropy.

The real question is not about removing entropy, but slowing it down. That’s the real fight we are thrown into, as developers.

Despite these difficulties, we can still define guidelines that will enable us to understand sources of software entropy, measure the risk it represents for our objectives and, if necessary, identify what steps can be taken to limit its growth.

Metrics for the evaluation of software entropy have also been proposed. Personally I don't find them too reliable as they are based on factors adapted to a specific development process and on known quantities that are often monitored in a system like JIRA. They do not consider complexity in the source code, such as cohesion, coupling and cyclomatic complexity. However, for those of you interested, I report an interesting article where a software entropy metric is proposed for an agile development process.

Personally if I had to work to find a quantifier for entropy in systems developed with the object oriented paradigm I would try to derive a metric from the average and maximum values of the cohesion and coupling coefficient (4.) , (5.), (6.) from the average and maximum cyclomatic complexity and from the depth of average and maximum call.

The "Kiviat" graph printed during the life of the software system can certainly give us a very interesting visual indication of how the entropy of the code is evolving. I have been using Kiwiat graph for many years and I can say that it has often helped me to identify anomalous situations in the code.

figure 3: Kiwiat graph

So how can we decrease the entropy in the code when we notice that it is degrading?

We have said that entropy cannot decrease. True but entropy can go from one system to another.

The "Refactoring" of the code is that activity that allows us to lower the entropy of the code by transforming it into the heat of the environment. The medium is us who, through our intellectual work, burn calories that we release into the environment to heat the gas molecules present in the air and thus transferring entropy from the source code to the air.

Martin Fowler gives the following definition of Refactoring: "a change made to the internal structure of the software to make it easier to understand and cheaper to modify without affecting its observable behavior" (7.).

One of the reasons that software entropy increase is that when we add a new function to a program, we build on top of the existing program, often in a way that the existing program was not intended to support. In such a situation, we can either redesign the existing program to better support our changes, or we can work around those changes in our additions.

Although in theory it is better to redesign our program, this usually results in extra work because any rewriting of our existing program will introduce new bugs and problems. However, if we don't redesign our program, the additions will be more complex than they should be.

Gradually, this extra complexity will exact a stiff penalty. Therefore, there is a trade-off: Redesigning causes short-term pain for longer-term gain. Schedule pressure being what it is, most people prefer to put their pain off to the future.

Refactoring describe techniques that reduce the short-term pain of redesigning. When we refactor, do not change the functionality of our program; rather, we change its internal structure in order to make it easier to understand and work with.

Refactoring changes are usually small steps: renaming a method, moving a field from one class to another, consolidating two similar methods into a superclass. Each step is tiny, yet a couple of hours' worth of performing these small steps can do a world of good to a program.

Also for refactoring there are rules to follow if you want to get a good result:

-) Do not refactor a program and add functionality to it at the same time.

-) Make sure you have good tests before you begin refactoring.

-) Take short, deliberate steps. Move a field from one class to another. Fuse two similar methods into a superclass. Test after each step. This may sound slow, but it avoids debugging and thus speeds you up.

-) You should refactor when you are adding a new function or fixing a bug. Do not reserve specific time for refactoring; instead, do a little every day.

If you follow these rules you can always keep the code clean.

Instead, it is more complicated to use refactoring to try to fix already very compromised systems. It has happened to me several times, and I assure you that it is exciting to see a big software system reborn while before we were afraid even to make simple changes to it.

Personally I consider refactoring an art, it needs a lot of study and a lot of practice. Creating a new design is not easy but fixing a compromised design assured you that it is much more complex.

After the series of articles that I am going to publish on design patterns, I promised myself to write articles on the "Art of Refactoring", I am already collecting the code to take as an example.

I hope you found interesting these topics : broken windows, boiled frogs, bad comments, software entropy and refactoring, I give you an appointment to the next article where instead we will continue to talk about design patterns.

I remind you my newsletter?"Sw Design & Clean Architecture" :?https://lnkd.in/eUzYBuEX?where you can find my previous articles and where you can register, if you have not already done so, to be notified every time I publish new articles.

?

thanks for reading my article, and I hope you have found the topic useful,

Feel free to leave any feedback.

Your feedback is very appreciated.

?Thanks again.

Stefano

References:

1.?Gamma, Helm, Johnson, Vlissides, “Design Patterns” Addison Wesley (2° Edition October 2002).

2.?Robert Martin Series, “Clean Code : A Handbook of Agile Software Craftsmanship” Prentice Hall (1° Edition 2009).

3. S.Santilli, "Coding style: The meaningful names": https://www.dhirubhai.net/pulse/coding-style-meaningful-names-stefano-santilli/

4. S.Santilli, "Component Coupling": https://www.dhirubhai.net/pulse/component-coupling-stefano-santilli/

5. S.Santilli, "Cohesion and Coupling": https://www.dhirubhai.net/pulse/cohesion-coupling-stefano-santilli/

6. S.Santilli, "Component Cohesion Principles": https://www.dhirubhai.net/pulse/component-cohesion-principles-stefano-santilli/

7. Martin Fowler, "Refactoring: Improving the Design of Existing Code" Addison-Wesley Professional (2° Edition 2019)