Will we pass the audit?

Key compliance indicators provide assurance

As an auditor and a consultant, the question, “Will we pass the audit”, is not uncommon. That question gets asked whether I work with RTOs with robust ongoing compliance programs, as well as with RTOs that have less formal programs; though, it happens more often in the latter.?

After an internal audit, compliance managers or members of the executive team ask me the same question, “Will we pass the audit”. My answer is always along the lines: “Well… you’ll only pass if you fix the following issues”. Most organisations follow the recommendation or rectifications suggested in my audit report, and consequently they pass their regulatory audit. They usually call me again just before their next regulatory audit to conduct another internal audit, and it is not uncommon for me to find similar issues as my previous audit.?

How can the RTO improve its quality control process and avoid recurring non-compliances? The RTO can use key compliance indicators (KCIs). KCIs can be monitored to drive changes that reduce the risk of audit findings, incrementally mature the overall compliance program and provide some peace of mind.

What are key compliance indicators?

KCIs, as the name suggests, are the operational and organisational clues that provide assurance that the activities required for meeting specific compliance requirements are happening the way they should be. Those clues can be related to operational tasks, resourcing or any of the activities necessary to achieve compliance. Specifically, a key compliance indicator is a metric or measurement that provides a quantitative description of an organisation’s adherence to a stated compliance objective.

While not necessary for understanding KCIs, those familiar with key performance indicators (KPIs) may see some parallels. Both KPIs and KCIs are ways to measure success against objectives. For KPIs, those objectives are related to business goals. For KCIs, those objectives are related to compliance. Compliance objectives must be very specific. For example, having a compliance objective to “pass the audit” is not going to yield much value without further exploration and definition. Success with KCIs lies in specificity. The answer to the question, “Will we pass the audit” should be an aggregate of the answers to more specific compliance objectives like, “Can we demonstrate that our assessment system meets the Principles of Assessment” and “Do we have sufficient assessment records to show compliance with the Rules of Evidence and training package requirement” or “Can we demonstrate that our assessment system is systematically validated” and “Do we have sufficient records of validations to meet clauses 1.9 – 1.11”.

KCIs may vary from RTO to RTO, even where the same compliance objectives exist. As we drill down through the levels of specificity to get to our KCIs, the specific tasks that comprise each implemented control will guide the identification of the KCIs. While some KCIs can be tied to automations related to continuous auditing, it’s also just as likely that the indicator could be derived from an implied requirement or even from a control implemented as a remediation after gaps were identified.

How do we identify compliance objectives?

As we’ve already discussed, our compliance objectives determine our KCIs. For RTOs the compliance objective usually comes directly from the Standards, themselves. For example, requirements around marketing practices, or managing complaints and appeals are basically, an open-book test, as the Standards provide a common explicit criterion. For other areas, the compliance objectives may be a little less prescriptive, for example, requirements around industry currency of trainers, or amount of training. By examining the implied control requirements, you can still identify the KCIs. For example, monitoring the effective implementation of a Continuous Professional Development system for trainers, or the rationale provided to support the course duration considering the learner cohort and delivery mode.

I always include compliance objectives for rectification actions within the audit report. However, during follow up meetings, the question, “How will we know when we’ve succeeded or failed at meeting a compliance objective” is always asked.

Ask yourself the “how” questions

For any stated compliance objective, measurable indicators can be derived by asking the question, “How will we know when we’ve done it”. In other words, what are our symptoms of success? Let’s walk through an example.

Compliance objective: Maintain industry currency of trainers and assessors.

In this state, the objective is very broad and not very measurable. I tend to use a root cause analysis, asking myself, “How?”

“How do we ensure that trainers and assessors have access to activities that will meet the required industry currency requirement?” The first set of “how” answers might include:

• Having a PD plan approved by the RTO that meets industry currency requirements
• Having time to complete the required PD activities
• Commit to complete the PD activities.?

“How do we ensure that trainers and assessors completed the required PD activities?” Answers might include:

• Checking completed individual PD register
• Checking supporting evidence recorded for PD activities
• Checking that trainers maintain PD activities as per schedule (i.e. PD points).

“How do we ensure that only trainers and assessors, that maintain industry currency, are allocated to classes?” Answers might include:

• Only trainers and assessors with industry currency status as “current” can be allocated to classes
• Only trainers and assessors with industry currency status as “current” have access to a trainer stamp (issued yearly by the RTO)
• All assessment records and other relevant documents must have a valid trainer.?

What should we monitor?

For RTOs, the number of KCIs can be around 80-120 (considering only the Standards for RTOs). Is it necessary, or even desirable, to monitor every possible compliance indicator? That answer is a clear and resounding, “No”. Just like any use of metrics, reporting or dashboards, too much data can distract from the important issues and will facilitate the feeling that the organisation is collecting metrics for metrics’ sake. Instead, KCIs should be focused on areas that are likely to cause compliance failures, and they should be tailored to the specific success criteria for the requirement.

Good candidates for KCIs include those compliance objectives for which:

• Automation is not possible or feasible such that adherence relies on manually executed tasks. KCIs provide that visibility and opportunity for quick issue resolution
• Non-compliances that have been previously identified by auditors. In these cases, the KCI might be derived successfully from the objective of whatever remediation was implemented, as well as from the original compliance objective
• Risk. Areas that can significantly damage the RTO, and where the remediation can compromise the business viability.

KCIs can also be used to insert a “buffer zone” where exceeding an internal threshold will generate an alert and action to prevent breaching the actual compliance deadline. KCIs can be used in any of these areas to provide early warning systems, evidence that can be used across multiple frameworks with similar compliance objectives, or even just the peace of mind that comes from knowing that compliance objectives are being met. Remember, as your compliance program changes, the requirements or solutions change, or your areas of concern change, so will your KCIs. Even if your organisation only implements one KCI, it’s useful to revisit that metric regularly to make sure that it’s still the important metric and it’s still using the right data to predict audit success.

So, will you pass the audit? If that answer causes any hesitation, the measurement and reporting of one or more KCIs could help your organisation breathe a little easier. Key compliance indicators (KCIs) are to regulatory obligations as key performance indicators are to the bottom line.

Javier Amaro - www.insources.edu.au