From Continuous Desaster to Continuous Delivery

Being late in Automotive

My apologies: This edition of #SWdriven is late. But as I'm writing about SW development in Automotive, I thought this would be somehow appropriate... just joking. Being on time was always one of the key paradigms of Automotive engineering. Missing the SOP (Start of Production) was never an option, looking at the impacts on customers, revenues and costs.

While a complete ecosystem was drilled to achieve the development and manufacturing milestones, SW development in the Automotive context was different at the beginning. Two decades ago, many managers thought: "Hey, it's not iron casting. It's just typing code lines into a computer and compiling". So SW came very late to the party, was integrated just before the vehicle hit the production line and the customer. And much too often, SW hit the vehicle even after it had been delivered to the customer, when being fixed at the next dealer visit.

Banana development: The product matures at the customer.

The consequences on customer satisfaction were massive. The launch of vehicle such as the ID-3 were commonly perceived, but many more launches were late, without the full feature range or with poor quality. Internally, almost all of the vehicle launches were creating enormous stress, extra efforts and highest pressure on people and suppliers.

But what is "late"?

Then Tesla and many other automotive startups entered the stage. Their startig point was different. If most of the innovation in the Automotive world was now driven by SW code, they asked why the Automotive paradigms needed still to be dominated by a mechanical, HW oriented mindset.

The focus on the SW platform ignored many of the complexities of the automotive products, not only regarding the size and interdependence of the features and systems, but also external requirements on safety, security and quality. But those were initial problems, and the learning curve was fast.

It's not about adding SW to the vehicle, but to integrate the vehicle into SW platforms.

The main benefit came from decoupling two domains with different speeds: A HW/mechanical domain with slow speed, as much longevity as possible and long-lasting assets. This domain obeys all most of the inherited and optimized rules of Automotive engineering excellence. Looking at the vehicle development processes of most Automotive startups, you won't find many differences: waterfall, "Lastenhefte" and task forces to reach the SOP. If you wonder why it's so easy to identify a vehicle on the street as a "Tesla" - it's because you have around a decade to get used to the design before it changes significantly.

The Continuous Mindset

The SW domain, however, is run in a quite different way. Instead of a 6-8 years product cycle with some updates in between, it aims to develop SW based functions in a continuous flow, with a much higher release rate and different processes to permanently code, integrate and validate.

Looking at your Windows laptop, you can easily understand the concept: While in the early days, Windows 3.1 was followed by Windows 95 followed by Windows XT (let's please ignore Vista...) and so on, Microsoft switched to Windows 10 - any nothing more. Each 6 months, a new release delivered a major set of updates. But it remained Windows 10 for several years, until the architecture behind got an update.

The concept of a permanent, non-is known as "Continuous Development", or respectively, Continuous Deployment (CD), Continuous Integration (CI) or Continuous Testing (CT). Also, DevOps as a joint vision of IT Development and IT operation is a commonly used concept, which brings in extra focus on the operation phase ("after SOP" in Automotive speech). Both, CI/CD and DevOps, are not describing a tool chain or specific process. They describe a paradigm, including processes, methods, tools and a mindset how to do things when developing SW. They are supported by a vast ecosystem of tool chains, as shown in the overview I found here.

The long road from SW to Vehicle

But a car is not a piece of SW. It's easy to paint a continuous, fully integrated and automated target picture for pure SW products. But it's a much longer and difficult way to go when physical products and legislation and regulation are involved, when legacy systems have to be maintained and a green field approach is not feasible.

Let's have a look on the main success factors which need to be reflected in a successful implementation.

• Architecture & Interfaces: Put architecture in the lead. Architecture is the foundation of everything. Enable interoperability by consistent and stable interfaces. Learn how to run APIs and SDKs.
• Decoupling: Get away from binding specific HW to specific SW. Make SW interchangeable, create abstraction layers where feasible (nothing comes at zero cost). Protect decoupled solutions from specific, one-time problem solving.
• System of Systems: Stop thinking in components or vehicles. User experience is not crippled by bad components, but by bad interplay between systems. Take care of external systems if your customer does. Implement systems engineering.
• E2E Functions: Define functions or features (some different wordings here) end2end. Handle architecture, functions and their realization in vehicle projects separately. Implement processes to manage "Verbünde" over all features and over the life cycle.
• Automization & Virtualiziation: Invest much more into automization and virtualization. Take benefit from the new architectures, decoupling and more stable interfaces. Invest more into environments than into features.
• Agile & DevOps: Refine your specific way of working. Measure your development efficiency: Is it Software world class? Or only Automotive median? Define clear principles - and execute. Kill anything that is in contradiction to speed, your new risk profile and hindering effectiveness.

Sound all reasonable? Yes, but look at the details. Virtualization of a vehicle is a huge project. Emulation of HW layers needs to be on your screen before assigning your suppliers. Regarding functions end2end is different when you don't own the system, but you have to follow Android's rules. Agile is nice, but how to meet the expectations of your manufacturing colleagues? And decoupling yes, but at what acceptable cost?

But it's possible to do: It needs a masterplan that brings structure and order into the many loose ends you have to bind together. First things first...

Speed is your friend

Another aspect is important. I once visited startups in Tel Aviv and Beer Sheva. In one of the meetings, we were talking about the impressive energy and speed about those startups. Their response was quick:

Be the fastest for the second best solution. While others try to aim perfect, you already have the feedback from your second generation solution.

In a nutshell: There is no perfect blueprint you can rely on. Setting up clear principles, implementing them with an experienced network of partners, and being as fast as possible to improve is key.

Big thanks to...

As speaking of expert partners, I'm very grateful to work with such a broad and excellent network of colleagues. Naming just a few of them here to say thx!

Stefan Beer Malte Becker Altan Yamak Frank Nies Dr. Gabriel Seiberth Juergen Reers Hans Loes Dominic Craciunescu Janos Fichter Marcel F. Bettina Blum Liam Friel Raffaele Menolascino