Process and Product – Two Sides of the Software Coin

The way we have developed software over the years has followed a zig-zag path. Early on, we had no prescribed way of working, but we created code. In the 1970s, structured methods became popular, followed by object/component methods from the mid-1980s through 2000. These were technical practices. After that, we adopted Agile methods which focused on human practices or social engineering. Now we are in the Scaling Agile phase, which includes both human and technical practices.?

In parallel, attempts have been made to assess structural quality of the software product itself and the quality of the way we work. During the late 1990s and until recently, the world’s focus was on doing things right - a trend personified by CMMI, Capability Maturity Model Integration. However, CMMI only helped us understand how well we knew how to do what we did; there was no measurement of the quality of the product. Around 2010, CISQ (Consortium for Information and Software Quality) was born, designed to focus on addressing the structural quality of the software product itself.

The fact is, you need a high-quality development process, and a high-quality end product. This requires more than just knowing how to do it according to your organization’s culture and standards. It requires assessing and applying practices that produce the highest quality software – software that is resilient, safe, resource efficient, and easy to modify. It is more than simply having a process; it is about having a?good?process.

This article addresses both process quality and product quality. An imbalance of the two leads to many unpredictable outcomes, which simply won’t work in a world increasingly reliant on complex software systems. A balanced approach is here to stay.

Good Process

Behind every?process?there is a?method. In other words, a process is a scaled-up method. These two terms are so closely related that for this paper we’ll treat them as synonyms. A process or a method is?good?if it fits its context, which includes application area, technology applied, and competence level of the team.

There are more than 20 million software developers on the planet, with numerous methods to develop software. Over the years the methods have improved, and we have gone through several paradigm shifts: structured methods in the 1970s, object/component methods, agile methods, and now?several scaling agile methods are competing for hegemony – we are in 50 years of method wars.?

Even if the methods we use have improved significantly,?the way we?work with?our methods – our production ‘machines’ – has not changed significantly over the last 50 years.??We still learn about the new selected method the same way – through papers, books, or web sites, that is through what we can call descriptions. We still don’t get any guidance from the selected method just in time when we need it – it may be on the shelf, but it’s not handy. We still are attracted by the method most in fashion.

How can we dramatically improve the way we work with methods? The art of our discipline is in such chaos, characterized by a multitude of competing methods to choose from. We need to fundamentally change the way we interact with methods; how we teach, learn, change, and use them. The Object Management Group (OMG)?Essence?standard leads the way: a new way of thinking that promises many things, never before applied to software engineering or system engineering.

Essence was born with the hypothesis that?all?methods share a common ground.

On top of that common ground, practices are defined. A practice is a very specific element of what is usually considered a method. Examples of practices are use cases, user stories, component-based development, DevOps, scrum – technical practices, and self-organizing teams, pair programming, release train, PI planning – human practices. A method is created by selecting and using a set of practices. The difference between different methods is usually just a small number of practices.

Essence prescribes new values to methods which apply to a range of stakeholders:

Individuals?gain the ability to see the big picture of software engineering that will always be useful to them throughout their career, even as specific practices change and evolve around them. They will have an eco-system of proven practices to find what they need; human practices such as self-organizing teams, pair programming; technical practices such as user stories or use cases, components, micro-services; and last but not the least?architecture.

Teams?can flexibly mix and match practices from several sources that suit their situation and needs. Crucially, this guidance focuses the team on the essential items and provides actionable advice in the form of simple checklists to guide and assess progress. The card-based nature enables the whole team to be involved in bringing their practices to life and reasoning and improving their way of working through serious games. Teams get live guidance instead of relying on descriptions (papers, books, or web sites).

Teams of Teams?can improve their overall performance by having clear and shared practices for coordinating teams where it matters, while still allowing the flexibility for local practices within the teams. New practices that have been pioneered and proven in some of their teams can then be shared and used by other teams. Having a common way of seeing and measuring progress gives a clear view of status across all the dimensions that are important for success. Using standards for end-to-end analysis of software across teams, as described in the next section, is core to coordination. This more systematic way of developing software moves us from being a craft to being more of an engineering discipline with expected improvements in quality, speed, and cost.

Organizations?have the opportunity for lean and agile governance, where the teams transparently apply the states and checklists to their work, avoiding much overhead and non-value-add bureaucracy. The common language and way of thinking about method makes it easier to form teams and have them coordinate with others. The organization can still have common practices but not constrained within a one-size fits all method. This approach is future-proof rather than blindly following the latest ‘fashion’ in industry standard method frameworks. They can evolve on their own over time, practice by practice, as the organization learns from within and from the outside world.

Coaches and Service Providers?gain common ways of expressing, teaching, and putting into practice their ideas. Essence enables this knowledge to be plugged into an industry standard framework for context and for composing with other practices. For coaches, the practice techniques and games foster team understanding and application, giving structure to their services and ensuring the knowledge sticks long after they are gone.

The software industry, for the first time,?benefits?from having a common ground of elements to use and a common language to express practices. This allows us to build ecosystems of proven practices which organizations and their teams can compose into practical methods. In return, they can give back new or improved practices back to the broader community. This also makes it easier to compare, mix, and improve practices from various sources and make them available to others.?

The Essence approach will accelerate the learning and sharing cycles across organizations and liberate the knowledge from around the world.

Thought leaders are adopting Essence to give their methods a common ground, for instance for Scrum, Scrum at Scale, Disciplined Agile. Universities are increasingly using Essence to form the basis for their teaching. The software industry, both major software houses such as Infosys, Cognizant, Accenture, and large product vendors are finding ways to significantly increase the quality of their multitude of methods. New tools are emerging, for instance Practice WorkBench, and Essence in Practice from Ivar Jacobson International, that allow organizations to work with methods comprised of such best practices. Students of the future will hit the ground running in their new jobs, holistically understanding what is important, and speaking and thinking with the same common language.

Good Product

Embracing a good foundation such as Essence is mandatory for any?industry to develop better, faster, cheaper software. While Agile or DevOps and, in general terms, the move from a traditional V cycle to a more agile world, is a better way to deliver products that are?functionally?good, the “non-functional”, structural, engineering quality has been increasingly suffering. A product is good when it does what it is supposed to do, and when it does it well, all the time. In software, instability, mediocre performance, security issues, abnormal CPU, memory, or cloud resources consumption can have a devastating effect on the best functional design ever. At the end of the day, the user experience is based on both the functional and structural quality of the software.?

Over the last 10 years, the emergence of the?CISQ, now an OMG standard , has defined the long-awaited engineering quality rules, or a set of best practices that ensure Reliability, Efficiency, Security and Maintainability of an entire software system, not just individual components. See?it-cisq.org/standards/code-quality-standards/. This has been a silent revolution, but a revolution nevertheless. The preceding ISO 25000 standard has been all about defining a proper framework to capture software characteristics, but falls short of defining the technical, engineering how to. The work done by CISQ and the Software Engineering Institute of Carnegie Mellon University, has focused on complementing the ISO standard with the proper definition of what to assess, based on real life: the best coding and architecture practices that need to be applied to build safe and agile software.

Does the software industry use those practices?

Most software practitioners are still only focusing on a narrow yet easy to grasp part of the challenge: they focus on the quality of their piece of the code. They almost always end up disappointed with the actual, real benefits of their work. Why??The exact same piece of code can be safe or highly dangerous, depending on the context in which it operates. Checking the quality of the code within a file or a program, outside of its architectural context, is akin to building a sophisticated 20-story building, hoping that the quality of the bricks alone will make the whole construction sound, secure, efficient or easy to change. In software engineering, the structural quality depends on understanding what the brick of code truly does when dealing with the other bricks. The CISQ rules address system-level quality and highlights empirical correlations?between architectural programming mistakes and production defects. There are many coding errors, 92% of which are within a component (the brick), but account for only 10% of all defects seen in production. However,?system-level bad practices account for 8% of all coding errors yet lead to 90% of the most serious production issues.?

So, what do we do??

We cannot blame developers, who are typically under pressure and being paid to produce good quality code within their own individual bricks. They rarely have a holistic, practical understanding of the entire software system. Developers should have a clear and comprehensive view of the entire software system and understand the?context?in which they have to create, develop, and enhance the bricks. They need a holistic, contextual understanding of the architecture, in order to be able to do their job properly. As the software industry advances along the lines prescribed by Essence, the standardization of contextual understanding will become more commonplace.

A few vendors, such as Synopsis for embedded real-time systems, and CAST for complex business applications, have followed CISQ and OMG guidelines and have produced automated software analysis platforms that check these rules with a focus on architecture. Such platforms enable developers with a growth mindset to improve every day, while the others can be explicitly shown which critical flaws to address. These platforms provide feedback about the work being done and its impact on the entire system, from user entry or automated data entry, all the way down to data access and multiple transaction chains. It also helps individuals and teams learn more about the system they work on, and bring in new architectural best practices to sharpen their expertise in building sound, safe, and resilient software.?

The latest advances in such software analysis platforms even provide the ability to automatically reverse-engineer and?visualize the actual structure?of the entire software system. This further helps architects and developers understand the flows through the layers, all component inter-dependencies, and the impact of changing a ‘brick’ on the rest of the ‘building’.

Conclusion

Process and product go hand in hand. They are intertwined, and you cannot just focus on one or the other. The process you select determines the product you will get; for instance, the architecture practice you select determines what architecture you get. And vice versa, the product you want specifies what process you need to apply; for instance, if you want a component-based system you need to apply a practice resulting in components.

All experts predict a massive shortage of trained software developers in the next three to five years. In some high-demand areas, we are already starting to feel the effects of this shortage. Every large company, all system integrators, and large ISVs report already having too many “freshers” (developers with less than 2 years’ experience)?developing software in their IT factories and R&D labs. The combination of this shortage and the world becoming software dependent is putting enormous pressure on software factories, Global 2000 IT departments, and all Global SIs.?

To avoid a chaotic software world, teams need to be responsive. For that, they need to adopt frameworks like Essence that support today’s practices and methodologies as well as offer foundations to quickly and effortlessly adopt new ideas and practices. They also need to combine it with?rock-solid engineering best practices and?apply standards such as?Essence?and?CISQ?to ensure they continuously deliver high quality software.