DevOps you say? No Problemo - We’re SAFe?!

SAFe? The Scaled Agile Framework, that supposedly makes large organisations agile and lean is on the rise among late-majority and laggards, as it plays bingo with all the contemporary software industry buzz-words and gathers them all into one shiny graphical poster.

Actually, some of us are so old, that we recognize history repeating itself - do you remember this one?

The favorite poster of any software team at the turn of the century. The RUP humpback whale chart. It was born out of the Object Oriented (OO) hype in the late 90’s. It advocated iterative and incremental software development. An early approach to today's agile and a rage against waterfalls.

How is this related to SAFe?

Stay with me, we’re taking a trip down memory lane, we will arrive there eventually.

Back then, around the turn of the millennium, Rational Software managed to end the great blob war among Object Oriented (OO) celebrities like Ivar Jacobson, Jim Rumbaugh and Grady Booch who publicly argued how to draw a class before you code it. Rational employed all of the three amigos and organized their collaboration - ending the war - and creating UML; The Unified Modelling language. The three amigos were not the only generals in the great blob war, but everyone back then agreed that Rational Software was representing the good side of the war. They seemed to be on a mission to make software development holistic. 

Rational also fought a war against waterfall methods, which manifested itself in the Rational Unified Process, which was led by another of that time’s big celebrities; Philipe Krutchen.

No Silver Bullet

But alas - RUP wasn’t a silver bullet - essentially it showed 11 small waterfalls - overlapping ("overlapping? brilliant!"). The whole shebang was chopped up in 4 phases - of different size ("different sizes? brilliant!") But ultimately, like any other process wich is chopped up in phases - it is founded on the imagination that at some point, we’re coming out in the end with software that is done, finished, wrapped up, handed over. All waterfalls reaches the ocean - in the end.

It’s imagined as construction - as if we were building a skyskraper.

Like today’s SAFe? RUP also went through various releases and version numbers. Which - when you think of it - is kind of contradictory to the concept of construction. I mean; Empire State Building doesn’t exist in version 2.0, 2.1 etc. But then again - neither RUP nor SAFe? would probably admit to advocate for a construction paradigme:

“Iterative and incremental ... does that sound like we’re building a bridge or a skyskraper to you?”

No, it doesn’t - so what is it with the phases?

Me and Dean

Back then, I was the biggest advocate of RUP and UML that you could possibly imagine - I dare say I was an innovator but if not, then definitely an early adopter.

I worked for a consulting bureau in Denmark; Enator (it’s long gone today). Enator was a partner and representative of Rational in our region. My first public speech to a 200 people audience was at a Rational User Conference in Denver - 2001 - No doubt - I was a true believer in Rational’s agenda to bring new approaches to the software industry.

At that time I was also a Rational certified trainer - one of the classes I conducted was a 3-day course with the not-so-easy-on-the-tongue name ”Rational Requirements Management - with Use Cases”.

Besides the three amigos and Krutchen, yet another celebrity at the time was directly involved in Rational - Dean Leffingwell. Yes him, the founding father of SAFe?. Besides a book that shared the title with the training class, he was also the author of the material used in Rational’s requirements management 3-day class. Dean had been brought into Rational when they acquired his company Requisite Inc. which were the creators of the tool RequsitePro.

After each customer training, I would evaluate and pass on notes on the slides - to Dean in Colorado - I was star-struck; Me, passin’ in notes to Dean OMG!

The buzz - back then

Back then the contemporary buzz words were Object Oriented, Use Cases, Design Patterns, Architecture (a word we lend from construction of buildings) even agile was introduced and emerging. And the strongest advocates for agile, you would find among the OO believers. However, at some point in time OO became mainstream in the industry - Use Cases became old news, replaced by the much more nimble and poetic User Stories (As a blah - I would like to bla ha - so that I can blah ha ha) - while Agile most certainly still was a bit of an infant terrible. 

Rational was acquired by IBM in 2003, It was IBM’ified and so the magic was gone. 

Dean left Rational even before IBM acquired it. But surely he made a name for himself - remember - one of the humpback whales in the RUP diagram was his contribution - quite an effort. So being the requirements management expert that he is, he published his come-back book “Agile Requirements Management” in 2010.

Concepts were much the same, after all Dean always had a bias towards lean and agile, even in his early days, and the name was great - agile was a great buzz in 2010.

And my oh my - what a come back.

The book is the de facto version 1.0 of SAFe? although it was only a bit later that the clever name was introduced.

Really! Who doesn’t want to be safe?

So from being just one small tiny bump - 1 out of 11 - in Rationals humpback whale diagram at the end of the millennium, through a book release in 2010 - we fast forward to 2013 where Dean co-founded “Scaled Agile, Inc. - Provider of SAFe?, the world's leading framework for business agility” and onward to today, where every single SAFe? certification in the world pays royalties to Scaled Inc.

Today SAFe? is probably more widespread than RUP ever was.

Huh! - eat dirt Rational and IBM! - what a come back!

Is agile a hamster wheel?

Today we’re all wiser, we know now, that it’s wrong to draw these phases as humpback whales swimming in parallel towards a sunset - software never ends. Today we know that these phases should be seen as an endless hamster wheel.

Really - there’s something wrong with this picture! Agile is not a hamster wheel of phases biting each other's tail. Agile defies phases!

We will correct this later.

Diffusion of Innovation Theory - applied to agile

Here’s another famous humpback whale - the Diffusion of Innovation Theory which seeks to explain how the spread of new technologies and innovations spread (blue curve) - and consequently the accumulated market shares (yellow curve).

One thing to notice is that the late majority and the laggards are - well - late.

Example: When late majority and laggards start buying DVDs, then - somewhere in the world - others have already begun streaming their movies straight off the internet.

In terms of agile - It’s obvious that large scale organizations can not easily adopt agile. They can never be among the early majority. Agile means nimble, light-footed, able to move quickly and easily.

A deer is agile - an elephant is not.

large scale ≠ agile

If I was an ignorant C-level manager

So what do I do - If I’m a CEO or a CTO in a large business, that is anything but nimble and light-footed? I look out the window and see a world that’s being eaten by software. And I’m responsible for the success of my business and yet I know close to nothing about software - The solution came to me in my dreams (or was it incepted by management consultants?) We need to initiate a Digital Transformation.

So a framework - or a recipe - that despite common sense, magically makes your large corporation agile, tech savvy and safe sounds like a good deal. Especially for a C-level management who hasn’t got a clue about tech and software.

What we see today is a lot of companies, who again and again fail in their large system endeavours. While facebook, Netflix and infinite number of tech startups seem to come out of nowhere and while they don’t even bother to put version numbers on their software - they are moving inconceivably fast. The large companies know that they need to read and speak the contemporary buzz words, if for nothing else, then for the sake of attracting talent.

And SAFe? is great! Name me one contemporary buzz word, that isn’t on this slide - what’s not to like?

I see Continuous this and that, Kanban, Scrum, Agile, Pipeline, Build-in quality, DevOps, Architects even Design Thinking. Aaaah - I Love it!

Change management consultants

A true fact (as they say in Washington D.C.) is that a very large portion of the people who get certified to train SAFe? and certify downward themselves - are change management consultants - not techies. And an increasingly large portion of the people who get a SAFe? certificate are also not necessarily related to software development.

We’re converting “manager” titles to “owners” titles. We put posters on the walls to force the culture and the vocabulary upon the company serfs. We take everyone through 3-day training classes - to understand their particular role in the great hand-over-to-the-next scheme. We learn to talk the talk.

Change management consultants would probably argue - “Who said anything about SAFe? being specifically related to Software? - It’s about the Digital Transforming of an entire organization”.

“Ehhh, Right, Digital transformation!” and then let’s visit the etymological meaning and origin of the two terms digital and software. They are the same:

Digital = Software

Software Is Eating The World.

Agile is software related

Agile is software related - it’s not a change management consultant scam. Agile derives from the lean concepts that Toyota and Honda developed in the early 80’ies. 

Toyota and Honda knew that in order to compete with the international market, they needed to bring down the batch sizes considerably - while maintaining or even improving the cost-efficiency - not that everything should be hand-made, after all it was still a factory, but they needed to optimize the flow.

The Toyota Production System (TPS), which is referred to as lean, when it used outside Toyota is about clearly identifying the value stream and continuously improving it by getting rid of waste - defined as anything that doesn't contribute directly to the value stream.

Single Minute Exchange of Die

An iconic lean story is the one about the “Single-minute exchange of die”. Problem: At a car factory you use a die to press the metal plates into the form of perhaps a left-front fender to a Toyota Corolla. It takes days - if not weeks, to change a die and calibrate it. Consequently it’s not easy to change the produktion from Toyota Corollas to Toyota Carinas. But what if we could change the die in just one minute? Then we could make 10 Corollas, followed by 3 Carinas and the 25 Camrys - if that’s the demand, then we shouldn’t make 2000 Corollas - and pile them up until someone needs them - how does that create value?

So Toyota optimized the flow and learned how to change die in a minute as opposed to a week.

It’s not a method or a framework, it’s more like a way of thinking. A different perspective, putting people first, by acknowledging, that everyone involved in a process, has something to offer in the continuous improvement of that very process. 

At some point, when lean, kaizen and kanban started to diffuse out of the manufacturing industry - people in software thought “What if - we were to see the software development less, as if we were constructing a building - and more as if we were optimizing a factory floor. See it as a flow, like they do in lean; Get rid of the phases from the construction paradigm, and just continuously improve - in small batches - eliminate waste.”

The agile manifesto is from 2001.

When we apply lean to software - we call it agile. It’s software related, end of story.

But how can - think small - become large systems?

Avoiding large scale bureaucratic organizations, doesn’t mean that you can not achieve large software endeavours - maybe even as big as an entire Linux distribution or a self-driving car.

You can think big - easily! 

But not by utilizing a standardized framework like SAFe? - or RUP, or DSDM og LeSS or Scrum of Scrums or...

Historically large and successful software endeavours consequently organized themselves differently that what SAFe? recommends.

It’s a paradox, that In order to succeed big-time in software you need to think in very small - but whole - increments. Think “Flow” not “Phases” - Think “Startup” not “Organisation”. Know the “Value Stream” forget about the “Hierarki”, Think "Team" not "Roles". Acknowledge that there is only one backlog to the Value Stream. One joint backlog in the entire enterprise - no matter how big it is.

As an alternative to a large Scale Agile Framework - that will promise to make you safe - consider the concept of a Lean Enterprise understanding that concept doesn’t promise you anything in terms of software success, but it will most likely make you more operational - adaptive to change.

In other words; You don’t need a framework - but you do need a thorough understanding of what software is, and how it behaves. 

In the future, responsible C-level managers can not not know about software development. Then they are not safe.

In the introduction of the book “Lean Enterprise - How High Performance Organizations Innovate at Scale” by Joanne Molesky, Jez Humble and Barry O’Reilly. There’s an adorable story about General Motor (GM) getting a peak at how Toyota conducts TPS (“lean”) and how, despite the fact that they have all the information they need - the framework so to speak - they fail big time in implementing it - simple because they don’t understand the culture.

Toyota is teasing GM: You can study and copy our factory floor lay-out, it won't help you much. First of all, your cars are different from ours, and even if you wanted to build the same cars, then by the time you’ve implemented it, a couple of years from now, we’ve learned a bunch of learnings and we will have changed the flow into something entirely different. As we Continuously Improve the flow - to the way we work.

Think “Philosophy” not “Framework”.

So what’s new today - what buzz should we listen to?

So if agile frameworks today are for late majority and laggards - what are the young kids looking at then? What comes after?

I’ve made a google trend analysis of four buzzes:

• Scaled Agile Framework
• DevOps
• Infrastructure as a Service
• Rational Unified Process

I’ve started the graph in 2008, around the time when the buzz “DevOps” was used for the first time. It’s clear, that at that time, the buzz “Rational Unified Process” still had a lot of traction. The Google trend data only goes back to 2004 - but back then it was possibly even bigger - than DevOps is today. Today RUP is dead and buried.

“Infrastructure as a Service” took off around the same time - 2008 - and from 2010 and until 2016 it remains the biggest buzz - and after 2016 only superseded by “DevOps”.

Although almost every large business in Denmark and Sweden is currently inviting armies of Change Management Consultants to help them implement agile and become SAFe? it’s really not much of a buzz when measured in a worldwide context.

“DevOps” is off the chart - well more literally it defines the chart. The index 100 in the chart is defined by the topic “DevOps” in September 2020 - that is now.

I’ve also been looking at a few tools that are on the rise, and should be interesting for anyone in software, who aren’t there yet, to start utilizing - if for nothing else, then to attract talent. “Docker”, “Kubernetes”, “Node.js”, “GitHub” - and I’ve kept “DevOps” there just for reference. Have a look for yourself.

DevOps - what is it?

Well, it’s closely related to the single-minute exchange of die story from Toyota.

Some things are so overly complex, that we don’t even dare to try to solve them, this is the case for modern software development too.

Toyotas alternative to the single-minute exchange of die was to make a costly overproduction of Corollas, store them somewhere for a long time, only to learn that the market wanted someting else - or even worse, they all have the cars suffered from the same design flaw - repeated over and over again, because they never bothered to get feedback from the end-users.

In system development a great portion of the problema lies with the infrastructure, the servers, the machines, the installations, the network. These things that historically have been seen as the domain of IT operations - and therefore they have been considered outside the scope of the software project. When we can't easily (single-minute) deploy and test to see if it's working, then we just produce tons of software, piling it up somewhere, until one day, we may start deploying and testing - or shut down the project.

In large organisations - even the ones that are completely SAFe? - it may very well still take 14 days, to get an extra machine from the IT department, so that the team can do more deploys and more tests.

So DevOps is about attacking this, and all other infrastructural problems.

Understanding the challenge in System Development

The problem in creating IT systems seldom lies in the actual writing of the code, but in maturing it to the market. Presenting a software developer for a wicked problem and waiting until she heroically throws her arms in the air shouting “Yaaah - it works on my machine” is not going to cause grey hairs on anybody. Software is very plastic, you can easily model, mould, fold, tweak it to do what you want - if you are good.

But releasing that code snippet - that is making sure that it will not just run on the developer's machine, but on any machine and then physically moving it from where the developer created it and onto the server - or thousands of clients - where it’s supposed to work. And in the process making absolutely sure, that nothing could possibly go wrong - That takes a frustrating long time - for many software teams it’s a complete unknown how to even reach that point.

Some organizations only do this entire process once - at the end of the software development cycle, just before release. It’s awkward - considering that the software will be updated many times in the future, why not practise the release process itself? Similar to the exchange of the die. Keeping it a manual - largely unknown process - means that it can take many months to carry out.

Or years.

The other day, the Danish minister of taxation postponed a system that was started back in 2013, it had grown to a budget of 630 million EUR, and the message was that “it probably works”, it’s just that they never tried to actually release it and test it - so they don’t know. On that note the minister postponed if for another 12 months! “So we have time to test it” he said.

(Here's a post on the topic that i wrote - it's in Danish)

Wha does lean software development look like then?

Let’s hear it from the softwaredeveloper who just raised her hands and victoriously announced that now the software works on her machine:

“Yaaah - it works on my machine” 

“Great, but in lean thinking, you need to deliver - not to the next in line, but to the value stream - so please verify that it will run on any machine”

“How do I know if it will run on any machine?”

"Well try it!"

"Wait - so lean means, that every time I finish just a small snippet of code I should try it on another machine?"

"Yep!"

"Why!"

"To build quality in - as opposed to glueing it on - when you're done."

"But that’s inefficient - I need to build it, and deploy it to another machine in order to test verify that. It takes a lot of time."

"Then automate it."

"You mean turning the entire build and deploy process into software?"

"Yep!"

"But! Even if I did that me and all the colleagues in the team, would need a lot of machines!"

"Yep!"

"No, you don’t get it - I mean kinda an almost infinite number of machines!"

"Exactly!"

"Seriously? These machines are expensive, it takes our IT department 14 days to deliver just one of them."

"Hmm - but what if the machines were virtual or services themselves."

"OK, Then maybe IT could do it in 3 days - but still - I would need infinite numbers!"

"Let's fix that first then - maybe you should bring the colleagues in IT operation closer - think of infrastructure as part of the same value stream as the software."

"Yeah! Wise guy! Don’t patronize me - I know of DevOps. How do you think IT manages to deliver in 3 days as opposed to 14 - of course we have a DevOps team - we're not stupid!"

"What if DevOps wasn’t a team - a demilitarized zone between the software project and the IT department - what if it was more of a concept - maybe real capable Ops wiz - right here next to you - in your team."

"Converting all hardware to software - as a joint effort? Me and one of the guys from the basement?"

"Yep!"

"Naah - I don’t know infrastructure and the guys from the basement don’t know software, it would be the blind leading the blind."

"Or you would gain understanding of each others work - I’m sure you would get there in the end."

"Maybe in a decade from now - and then what?"

"Then the dull repetitive work of providing this infrastructure would simply be to write a computer program and execute it a lot of times."

"OK, so essentially turning the entire infrastructure into software you mean?"

"Yep - Infrastructure as Code!"

"But even then - according to you, I still need to test it 10 times a day - currently we only test at the end of each 14 day sprint - and that's already more often than most"

"Yep - 10 times a day sounds right, for you - and 10 times for each of your team mates."

"No, we're 30 developers in the team, so that would be 300 test executions a day - No way"

"Why not?"

"Well, you know: read instructions, click, point, check, read next instruction, click point check. I’m telling you - it takes a lot of time - like forever"

"Sounds a bit like dull repetitive work to me?"

"Exactly!"

"What if it was software? Software is good at dull repetitive tasks!"

"The tests? - You mean turning the test into software and executing it?"

"Yep!"

"But you clearly don’t get it! Part of the test is to feed the system with test data - to load it - simply preparing it for test that takes forever, and the test data isn’t always sophisticated enough to cover all corner cases of all issues." 

"Your point?"

"My point is, that even if we automate the preparation of the test environment - there would still be a lot of things we wouldn't see."

"OK - due to the quality of the test data - So you need more test data!"

"Yep - exactly - you see? It wont work!"

"What if you turned that into software too?"

"Come on! ...Turning the creation of test data into software?"

"Yep!"

"No no, even if we did that, some of the tests run for many hours, and I can't wait for that - 10 times a day."

"What if you made the test smarter or faster?"

"Some kind of Artificially Intelligent test?"

"Maybe AI yes, and maybe combined with running in parallel - I mean you have enough machines now, and then maybe even turning some parts of the system into emulators - software based simulators."

"Program a simulator of the system?"

"Yep!"

"Why on earth would I program an emulator of the system - when I have the system itself?"

"To get access to infinite numbers of the system, as you said, and eventually to speed up your test."

"Oh my god - you really don’t get it. The software system is one system - it’s huge, you can’t split it up like that."

"Yep - you’re probably right, You'll need to break up the monolithic structure, and stitch it together again as small individually releasable components - you need to manage dependencies - using semantic versioning - and more software."

"Aaaaah - no no no! Now I see what you are doing! No matter what I say, your answer will be, 'Turn it into software!' - right?"

"Yep!"

"HA! ...and you would go on and on with turning everything into software - the deploy, the release, the documentation, even the simulation of the outside world like network including DNS servers, installing, uninstalling, updating database schemes - everything - you would turn into software?"

"Yep - anything as code! whatever you do, write it as code. In this context code refers to anything being stored in files and version controlled - code can be executed and it's immutable."

"OK - you’re crazy! That won’t work!"

"Why not?"

"Well first of all - We don’t know how to do that, and second of all - it sounds overly complex and third... ...eeeeeh We’re special!"

"Well. I’ll let you in on a few secrets. First of all, there are a lot of new tools out there that are emerging, and which can help you in turning anything into code - you need to look up. And second: software is just software, if you are coding the logic of a washing machine, a rule-machine for calculating real-estate evaluations, assessing real-time risk in FX-trades or if you are coding an emulator or an automated test - it’s all just software. Software always behaves like software. So you're definitely not special."

"OK - but it’s difficult and complex and in our case it’s also not needed!"

"Why not?"

"Well because we're agile! - we’re SAFe?"