Lessons Learnt from our Cyber Attack

This month we celebrate our one-year anniversary…

Last year, City of Stonnington experienced a major crisis when we were the victims of a #cyberattack, forcing us to shut down our systems for 11 days.

12 months on, it’s time to reflect on that experience and share learnings. I’ve been asked by my colleagues to provide insights in the hope we can all share, improve, and pro-actively protect our customers and ourselves.

I had never managed a crisis like this, and as the designated Business Continuity Lead for this event, there wasn’t any training that could have prepare me for what was needed.

In late August 2021, we were fortunate that the Victorian Department of Premier and Cabinet alerted us of a possible cyber-attack on several city councils.?Upon investigation, evidence of initial stages of a cyber-attack which would lead to a complete compromise was confirmed in our infrastructure.?The investigation also revealed the presence of malwares. To prevent further compromises, we substantially locked down and isolated ourselves from the internet.

We then embarked on a response and recovery tactical operation to contain and treat the cyber incident, and to expediently restore systems and services to minimise disruptions.?All IT and project resources were re-allocated to work with the business to restore progressively, based on priority of needs.

Towards the end of our response and recovery, we began a Cyber Remediation and Essentials operation which concluded in December 2021 – significantly lifting our Essential 8 compliance and tripling our Microsoft Secure Score.

Response and Restoration Timeline and Action:

1. August 28 (Saturday) to August 29 (Sunday) – detection, root-cause, and isolation…removed Stonnington from the internet
2. August 29 (Sunday) – all employee communications and direction to work from home, updated customer notice on website
3. August 30 – executive meeting to decide and enact our Business Continuity Plan, assign roles, and agree on communications & engagement
4. August 30 – an All-People Leader meeting to brief out the problem and what to do…deal with PR/Media with Channel 7 story breaking in that evening’s news
5. August 30 (evening) – technical problem management meeting that fed into the next day’s communications for the employees (what they can / can’t do)
6. Continued technical, approval, communication, and notification process to employees
7. September 3 – containment and eradication complete. Restoration plan to executive team for approval, including nomination of business users for acceptance testing
8. September 5 – user acceptance testing conducted and signed-off
9. September 6 – priority 1 systems restored (Customer, Finance, Property & Rating, Payroll, and others), plus, front facing customer services and website
10. Rinse and repeat Steps 7 to 9 across three phases

Post-Incident Cyber Remediation and Essentials Program:

Post the event we conducted a range of reviews; however, it was the remediation program that really accelerated improvements to our cyber security posture. We delivered a range of improvements over the proceeding 13 weeks:

• Implemented cyber awareness training for all employees and Multi-Factor Authentication
• Implemented compliance with Victorian Protective Data Security Standards
• Reduced cyber security risks through deployment of critical updates across the enterprise
• Significantly improved sensitive information and password management
• Significantly improved ability to detect, respond to, and deflect cyber security threats
• Implemented safer ways for 3rd party application server providers and system administrators to access Council’s information and systems
• Improved ability to visualise and manage our data network
• Significantly improved processes to roll out critical updates across the enterprise
• Developed security and system knowledge documentation to underpin support processes and reduce delivery risks for future projects

Customer Benefits

• Protection from identify theft which can lead to privacy breach, and financial fraud
• Customer trust and confidence
• Customer satisfaction of quality of services underpinned by technology efficiency
• Protection from service disruptions which can lead to customer dissatisfaction and delayed access to services

What did we do well?

• Leveraged individual and team expertise across the whole organisation
• Timely response and continuous communication
• Team cohesion and consistent ways of working providing certainty for the team
• Appropriate reaction with limited information, with continued decision making
• Managed through the limitations of remote working
• Implementation of Council procedures

What didn’t we do well?

• Captured detailed time spent by individuals on specific tasks. We did this at a high-level theme but needed to do this at a task level.
• Tried to manage too many smaller requests for document access. We needed to stop doing this earlier and focus the team on root cause fix.
• Some communications delivered didn’t take into consideration how people were feeling e.g., feedback was provided that the comms was too positive in a time when people were struggling.

Post Incident Review Summary:

• An updated Data Breach Response Plan and Business Continuity Plan with specific protocols for cyber incidents with regular “fire drills”.
• Clear prioritisation of business processes and connected systems to restore.
• Documented business processes should have both “happy paths” and “work-arounds”.
• A framework for involving and escalating matters to legal.
• Review third party contracts to confirm key resource providers and ensure understanding of own obligations to report an incident, and the third party’s reasonable response action.
• Review cyber insurance policy.

Closing remarks and reflections:

We could have made life easier for ourselves if we had at least half of those improvements, however, you can only plan for so much.

For us, the success was all down to our people.

Our amazingly dedicated people at City of Stonnington care deeply about the community and came together to work the problem and fix it. A real can-do attitude, calmness under pressure and supportiveness to get each other through a difficult time.

Thank goodness for the great culture we have at Stonnington.

You can’t write that into a Business Continuity Plan.