This week, we cover System Design, Firefighting, Containers, Product Led Strategy, Modernization and Racial Equity in Media
?Quote: System Design and Results
“Every system is perfectly designed to get the result that it does.”
Tweet: Firefighting
Technical Article/Presentation: How a Hotel Company Ran $30B of Revenue in Containers
?I love this story from Dwayne Holmes about how he enabled a major hotel chain to adopt containerization and massively lower the friction of how to deliver apps in containers. It all started with his vision of Container workloads + Developer tools + Pipelines + Platform + Base Images?= Modern Operations.?It’s a great story and vision of what we all should be working towards.
LinkedIn: Note that this video is only available by subscribing to the DevOps Enterprise Summit Video Library. A free membership(10 videos/month) is available as well as individual and corporate memberships.
FYI: IT Revolution announced 2022 Conference Dates. I'm happy to say that the flagship event will be back in Las Vegas this year and in person! Additionally, registration and CFPs for the May Europe Event are now open!?DevOps Enterprise Summit Virtual –?Europe: Call for Speakers/Papers @ Sessionize.com
DevOps Enterprise Summit Virtual - Europe
?DevOps Enterprise Summit Virtual - US
DevOps Enterprise Summit?US Flagship Event?
The Cosmopolitan of?Las?Vegas
Podcast: Product Thinking - Melissa Perri?/ Creating Product Led Strategy with Oji Udezue
This is my favorite podcast for information about Product Management. Melissa is one of the foremost leaders on Product Management practices.?She writes quite a bit(see the Build Trap), runs The Product Institute(online training for Product Managers) as well as producing this great podcast.
The Product Thinking Podcast has two different formats.?One format features leaders from the industry who Melissa Interviews.?The other format is called "Dear Melissa" where Melissa fields questions from listeners.?I find her insight and approach enlightening and refreshing.
In this Podcast, Melissa talks with Oji Udezue about product led strategies.?Oji is an experienced product leader who worked at Microsoft, Atlassian and Parsable to name a few.?I loved how he laid out the role of the CPO and his draft strategy framework: vision, mission, strategy, objectives.
Some other links to check out:
Books: Kill It with Fire / 6: COMING IN MIDSTREAM
We build our computer systems the way we build our cities: over time, without a plan, on top of ruins. —Ellen Ullman
In this chapter, Marianne discusses techniques to deal with coming into a modernization that is midstream and in trouble.?Specifically, how do you disrupt failure modes that are in compounding cycles.
She hits on the following:
- Recovery modernization projects are much more common than ground floor modernization efforts.?This occurs for a variety of reasons.
- Remember that a medics job is to stop the bleeding first.?Then, do long term planning.
Find the Bleed
- Find Responsibility Gaps: Find the places where responsibilities are unclear, unowned or have high-handoff costs.?Look for Conway's law.
- Study the cadence: Look for meetings that are maladapted attempts to solve problems.
- Pay attention to leader rhetoric and look for people who run towards the problem.
- Look for compounding problems. There is unlikely more than one issue that is trending the project towards failure.
MESS: Fixing things that are not broken
- Look for solutions chasing a problem: assuming new technology is better than old, artificial consistency, confusing success with quality, over optimization, chasing micro-services/breaking up monoliths.?I have been calling this "micro-services theater."?To quote Marianne: "Treating monoliths as inherently bad pushes organizations into fixing them when they’re not broken."
- COMPOUNDING PROBLEM: Diminishing Trust: Large, expensive programs to fix things that are not break trust with business stakeholders and executives. This can create a downward cycle of mistrust
- SOLUTION: Engineering Practices(my emphasis) and Formal Methods
- Fixing the wrong thing is only made worse but not finishing it.?Get the initiative to a place where it's not a "Frankenstein monster".
- If you are already in the "fixing something not broken mode" use it as an opportunity to experiment and/or improve engineering practices: testing, CI, monitoring.
- If you are already good at the engineering practices look at formal specification methods like TLA+ or Alloy to create a blueprint for your applications.
MESS: Forgotten and Lost Systems
- Overtime, systems get lost and abandoned.?These lost systems become risk and cause drag on the company.
- COMPOUNDING PROBLEM: Crippling Risk Avoidance
- Long term risk avoidance eventually escalates to create a Catch-22: 1) changing things can create an outage vs 2) not improving things will eventually create an outage
- SOLUTION: Chaos Testing
- If you have systems that are unclaimed or you can't track down dependencies consider injecting a fault via chaos testing or shutting them off to see what breaks.
- It is better to do this with people alert and awake and able to react then a 2AM outage from this lost system that no one understands.
MESS: Institutional Failures
- If you have problems in one piece of software no one has put much thought into maintaining it is likely that those problems are systemic.
- This can take the form of repeated poor patterns or technologies that have permeated the landscape.
- Look to differentiate mistakes vs patterns.
- COMPOUNDING PROBLEM: No Owners
- The issue with systemic problems is that no one actually owns them.?If they affect large groups of folks the only people with the authority to fix them are far removed from the daily realities.
- SOLUTION: Code Yellow
- Code Yellow is a set of techniques and processes at Google to handle issues beyond the scope of what any one part of an organization owns.
- The purpose of Code Yellow is to create momentum to fix a problem where ownership and clarity is unclear.
- Code Yellow has the following critical features
- The Code Yellow leader has escalated privileges
- The leader servers as the central point of contact
- The team to fix is small
- The team is focused
- The Code Yellow is temporary.?Months not years.
- Code Yellows are called by the lowest level leader that is close enough to the program and has enough influence over the affected parts of the organization.
- Code Yellows end when the issue is stabilized
MESS: Leadership Has Lost the Room
- "Losing the room is a sports term. It means a coach has lost the respect of his or her players. The team, instead of following orders and working together, struggles to self-organize."
- When an engineering leader has lost the room it is often because the organization has pushed the engineering team back into a place where it is impossible to succeed.
- Incentives have a very strong impact on effectiveness. Conflicting incentives create no-win situations where people feel trapped.
- Sometimes organizations refuse (or can't figure out) how to take responsibility for the situations they have put themselves and their employees in.
- Teams and people without confidence find ways to self-sabotage and create self-fulfilling prophecies displaying signs of learned helplessness.
- Restoring trust may require changing the scenery and replacing old leaders with new ones.?Doing this may be a good first step but fail to restore teams to excellence.
- SOLUTION: Murder Boards
- Murder boards work by convening a panel of experts who challenge assumptions.?It is supposed to be combative.?The experts are trying to outright murder the ideas.
- Murder boards have two goals
- Prepare candidates for a stressful event
- Build a candidates confidence that their plan is battle tested
- The technique of murder boards is only appropriate in special circumstances.?They can help a team see that everyone wants them to succeed, their colleagues respect them and a new go forward path has been painted.
- "To accomplish those goals, it is essential that both sides of the murder board know that the purpose of the exercise is to make the candidate stronger."
Something Else: CBS This Week On Sunday Morning / Byron Allen on Increasing Minority Ownership & Racial Equity
?I loved this story about media mogul Byron Allen.?His vision from a young age was to increase minority ownership and create racial equity.?He learned the business side of show business and even played executive at 6 years old in his basement.?He bought the Weather Channel in 2018 in an all cash deal.?And he just prepared a bid to buy the Denver Broncos and if successful will make him the NFL's first Black majority owner.?He is passionate about creating economic inclusion. His energy is infectious.
