journey to #SLSP WAY OF WORK (4/?) - Automation

How to get from 2 weeks to 3 days? From 2 hours to 15 minutes? And finally, from 30 minutes down to 0?

Once upon a time, there was a bank. IT and business were together working on achieving business goals, expectations were managed, relevant stakeholders understood the rules of the game, and everyone was quite ok with the situation. The next big challenge was to improve efficiency, minimise human errors and make planned outages shorter. Soon it was clear that DevOps was the way for us.

With efficient factory, working change management and high-level planning via action plan and platforms, it was clear that IT Delivery, with its cadence of changes, would either make OPS admins give up or burn out. We decided to help them and started an activity to automate as much as possible. We had a simple vision:

Anyone with proper rights can?deploy changes to any environment with a single click

We had three major goals (yes, this was the initial list for us ??):

1. Deployment automation
2. Test automation
3. Reduce the amount of human work to avoid human errors

In the back of our heads, there was a much bigger picture - automation of full SDLC - configuration automation, orchestrated deployments, automated upgrade of whole environments, infrastructure provisioning etc. (for your better understanding - container platform was a forbidden word at that time in part of IT).

We expected to reduce human errors, speed up development and testing, reduce overtime and work in non-working hours, and other nice things used when pitching to the Board. And we believed we could achieve it.

We built a specialised department to handle the whole DevSecOps topic. Here came the first hurdle - not all admins wanted to join this new wave, as it was more convenient for them to work in the same old way. Never mind, we started with those willing to change their WAY OF WORK, drew the first version of pipelines and toolset landscape, and slowly built the first real automation for deployments.

We were able to demonstrate the vitality of the new concepts quickly. The first installations were buggy, but the team improved and reworked the concept. Then came the first successful release. More apps used DevSecOps pipelines, and the DevOps approach slowly became the default way of work for most of the admins.

It does not make much sense to describe the whole DevSecOps pipeline, as it is custom built for us. There are the common names you would expect - like Jenkins, Gitlab, Nexus, Sonarcube, Config server and OpenShift.

Testing automation

It was not just about admins - testing automation was ongoing as well - again, using open-source (Robot framework), the team was able to automate smoke tests executed after the release to verify everything ran smoothly. This was the point in time when one very visible milestone happened.

We were used to a common practice - deployment of the release during the weekend meant that quite a big group of people had to be onsite in the office - either because of installation or because of testing duties. With ongoing automation, the group's size was shrinking to a minimal level.

Regression tests are part of our routine executed in the final stage of release preparation. In the old way, it was a two-week-long process of verifying the correct functionality of applications. These are nowadays automated to a large extent as well - it's about 1k test cases that can be executed with a single click. Its major benefit is shortening the duration of regression tests from days to hours. This was a critical task in order to achieve a higher cadence of releases - more on that later on (a little bit of teasing ??). Achievement - reduction of regression test window from 2 weeks to 3 days.

Part of our business processes are running not just on PCs but tablets and mobile phones as well. We are using tablets for better advisory in branches, signature pads to minimise paperwork and, of course, modern mobile application George. It was natural for the testing team to find a solution to automate testing on these platforms, again using open-source technologies. As a result, we now have a farm for mobile and tablet testing and also with the capability to verify application behaviour remotely on specific device - we call it rental service for mobile phones.

What came next? Robotic Process Automation, a.k.a. RPA. The idea behind sounds great, right? You take a platform, teach it how to do some work, and magic happens - you can lay off a few employees. The fun begins when you start to calculate the business case in our environment, where business processes are streamlined quite a lot from the design. The only way to make it work in our environment was to cut out the costs of any licences. And here comes the open-source platform for testing - we are using it for RPA across the bank now, with specific RPA FTEs created based on the savings already delivered.

Current state

DevOps pipelines exist for all build and deploy processes. They are executed either by admins or teams (depending on team maturity). Deployments are much less time-consuming, with the additional benefit of mitigation of human errors. Everything under our control is treated as a code. And benefits?

• Shorter installation windows (all installations are faster now, for some applications, it went down from 2 hours to 15 minutes)
• Fewer human errors during installation procedures - more stable environment post deployments
• Increased ownership of applications by teams/labs
• Real 0 downtime deployment working in the production environment (30 minutes to 0 minutes) using the micro-service architecture running in the OpenShift cluster - more on that later
• Test automation is the new normal, with 70% of all test cases automated
• RPA is part of our company saving tens of FTEs
• Admins cannot imagine returning to the old way of work, even though they were not the happiest people about the change
• Automation is natural for many people, and additional opportunities to improve are ongoing - for example, automation of the creation of installation procedures across our application portfolio. Imagine you have to deploy a new version of 40 different applications in one night, with various dependencies between them. Identification of ideal flow was made manually, but we are piloting our home-brew tool to assist, which will be connected to our Jira process and to installation pipelines soon.

The next chapter will be about Switch from 3 to 10 (releases per year)