Technology adoption does not have emotion. But the emotion from Technology failure brings to fore the missed opportunity, disappointment and often outburst. Social media has become the avenue to vent both frustration and provide positive reinforcement.
When you have an existing challenge with technology every good aspect of the existing platform and service gets wrapped in an aura of disappointment. The only way forward is that you always looking for a solution to fix it and not replace it. The need to replace rises when the challenges in regards to maintainability and the platform are repetitive and the impact of operations requires heavy intervention.
We had arrived at the statement that showed that the limitations on the existing solutions were not going to address the challenges we were facing. In every chaos there is always need for someone to chalk out a path. The path was to prioritize and funnel.
We setup some foundation rules for release
- We will release a change every week on Monday
- We should vote if it ready / tested by Wednesday
- We review release elements on Friday and release on Monday
- No Adhoc release until absolutely necessary (We never hit adhoc at all)
The funnel – Day 0
- All changes were halted – Bold but needed
- Everything big, medium, urgent, critical, feature, demand, etc was put on the list
- A scoring was done, Bugs were given a monetary loss value, a crash was automatically labelled as critical, Anything that had potential revenue generation was given a higher priority
- Business Teams were give the final authority to move things up in priority
- Competition launches were automatically added to the list as low priority items – This allowed us to see how far back we were
- If a priority was interrupted, it was automatically completely dropped and was relegated to lowest priority – Hence business had to be committed to what they needed first
We then gave ourselves and technology teams 2 weeks before the start – Mostly who Follow the Agile Practices would call this the Scrum 0 or PI Planning or Story Writing phase.
The rules and funnel created Chaos and Argument, which came from perception. We already had slow delivery and now we were giving a pause and time loss was the understanding. What no one was willing to understand that to take a leap forward sometime we need to take a few steps back.
For folks planning to adopt the approach, what we did during the 2 weeks is as follows
- Sessions were organized with all teams, technology and operations, to explain how the participation will work
- Everyone, Designers, Engineers, Infrastructure, operations, customer support, call centre, business, were given an opportunity to make comments on the top 10 priorities.
- Small groups were created who were focusing on designing the right solution and were given an open hand and guiding principles by the banks branding team
- All criticism was to be accepted and it was written down anonymously
- Communication to customers became a part of the design process
- Capacity planning was done to talk about the technology capacity and the operations, how would they like to handle failures and what data would help them.
The outcome of the Chaos was
- Failure was not treated as a disaster, it was understood as an eventuality. The example I used was Netflix Engineers using Chaos Monkey – A tool that invoked failures randomly and hence the need to build resilience as a part of solution design. In our case we still relied on Operations Teams Human Resource
- Prioritizing the map of delivery helped bring structure. The debates on what needs to be done first ended.
- Design process included the business team, who were describing the requirement from all aspects
- The 2 week break gave technology teams time to clear up the critical items they had been putting off – Maintenance was at the top of the list, building observability to all aspects was necessary to measure performance.
- We created Dashboards, data points were high level and were needed to provide us with visibility at 30,000 ft on what was happening. We knew the areas that were getting traction and the ones which were failing.
So far we had not even discussed the new mobile app, as it was not even on the horizon. There were problems that were more pressing that was not helping. What was really needed before we leapfrogged and innovated, was Stability. And from the Chaos of the problem we created a path to build outcomes.
#buildvsbuy #bank #mobileapp #customerexperience #chaos #engineering
