Part 3: Monitor Your Controls, But Take a Risk-Based Approach!

In this concluding part of the series, I share more lessons learned from a recent deployment of a SaaS solution for continuous controls monitoring (CCM). Follow these URLs to catch up on Part 1 and Part 2 .

Mistake 6 - Assuming all data sources can be integrated

As well-meaning as they are, CCM vendor claims that their products can integrate with most technologies are not always true in reality.

Embarking on CCM automation without assessing the integration capabilities of the technologies generating the telemetry data of interest (e.g., directory services, threat, and vulnerability scanners, network availability measurement tools, service management platforms, DLP tools, etc) will lead to false starts and cause significant delays.

Thought should be given to constraints on both on-premise and cloud-based technologies and potential challenges arising from data egress.

Key questions to ask include: What logs are available from the target systems? Do they contain personal data? What is the quality and utility of the data? What is the ability of the target systems to allow data to be consumed via APIs, web services, or CSV file formats? How will secrets, keys, and other access credentials be stored?

Mistake 7 - Use of data without validation or context

It is one thing to be able to collect data from disparate technologies. It is a different thing altogether to be able to actually rely on the data that they provide.?

When collecting data for controls monitoring, challenge your technology teams to provide context.

For example, was the higher number of failed backups reported this month linked to any major infrastructure changes? Does the number of redundant firewall rules include both on-premise and cloud-based firewalls?

Controls monitoring (manual or automated) needs complete and contextual data for it to be effective. If you’re not asking the right questions about the data you collect, you will basically get the wrong answers. Automation will only exacerbate this problem.

Mistake 8: Some things can be automated, others cannot

In reality, controls monitoring relies on obtaining and correlating data from various sources, essentially combining human-generated data and data sourced directly from technology platforms.

For example, measuring a technology-related risk such as ‘the number of leavers whose systems access was not removed within 30 days of departure’ may rely on correlating data collected from manual HR leaver records, centralised IDP platforms like Active Directory, and a disparate number of other systems where local user accounts were created.?

In my experience, human dependencies involved in most joiners, movers, and leavers (JML) workflows and decentralised user access to other systems can introduce complexities when considering automation.

Take another common security risk measurement example where penetration testing is outsourced to an external vendor and test findings are manually reported to in-house teams.?

For many organisations, ingesting findings into incident management software (e.g., ServiceNow) for tracking and correlation purposes would be ideal. However, this raises significant questions about compatible data formats, ingestion methods, and data sensitivity. While it would be great to be able to automate a metric such as “number of high-risk pen test findings resolved within 30 days”, this may not be the easiest thing to do in reality.

One idea to explore when seeking to generate qualitative data is the use of robotic process automation (RPA) methods in human-driven workflows. However, recognising the limitations of what is possible with automation will save time and effort.

Mistake 9 - Overreliance on the vendor

In my opinion, the aim of CCM automation should be to improve the efficiency of existing controls monitoring processes and workflows. Risk and compliance teams should be able to enhance their ways of working using CCM technologies.

Making changes to control frameworks, metrics and KRIs, monitoring thresholds and alerting and reporting are ‘par for the course’ in controls monitoring.?

While CCM vendors can add considerable value to the building of monitoring logic and data visualisation, in-house teams using CCM technologies should be able to carry out many of the more operational tasks on their own without relying heavily on vendors.

Mistake 10 - Poor delegation and lack of oversight

CCM automation is not the sort of project you want to delegate to the newest member of the risk and compliance team. A considerable amount of institutional knowledge, experience, and goodwill is required to help projects like these succeed.

Consider your project staffing very carefully and ensure you have the right blend of skills and experience.

My suggestion is to set up a steering committee made up of senior stakeholders from IT, risk and compliance, cybersecurity, and vendor representatives (if using external platforms). This committee should agree on core deliverables, critical success factors, and project plans and provide delivery oversight and leadership.

In summary…

Auditing, compliance, and the reporting requirements that come with these functions place a considerable time, effort, and cost burden on IT and security operations teams. Easing their workload makes a strong case for continuous controls monitoring and automation of associated workflows.?

Additionally, organisations that are subject to multiple regulations and that have to demonstrate compliance frequently can benefit from having a ‘single source of truth’ or a ‘single pane of glass’ for reporting. CCM can add value here.

However, before spending loads of money on a CCM technology that may not be a good fit for your organisation, invest time upfront on identifying potential pitfalls and loopholes that could sink your CCM implementation project.

I'd be happy to discuss your current and future CCM plans and share further insights from my experience. Drop me a DM. :-)

Useful Resources

• NIST Special Publication 800-137 - Information Security Continuous Monitoring for Federal Information Systems and Organizations
• Useful article about leveraging Continuous Controls Monitoring to reduce cyber risk from KPMG