Do not trust AI unless you have audited AI?
Arif Zaman FCCA, CIA, CISA, CPA, CFE, CCSA, CRMA, CRBA
Head Internal Audit & Risk | Consultant | Trainer | Speaker | Author | YouTuber I xEY I xEmaar I xTelenor
"Algorithms are still not strong enough to cancel out all noise completely and give a correct probable answer every time."
Artificial Intelligence (AI) is the concept by which we allow the machine to make a predictive (probable) intelligent decision. It is a tool which might facilitate decision making by using algorithm and techniques on data sets.
I try to give a summary overview of what AI means to an internal auditor and how internal auditors should see AI application from risk and audit perspective.
1. USE OF PROBABILITY RATHER CORRECTNESS OF THE RESULT
The very nature of AI using probability rather than the absolute correctness of the result posses the very first risk in the context of a business. AI uses the data to see the hidden pattern (data point) to predict the future trend and likely give the most probable answer. On the contrary, auditors generally use a rule-based approach by using the techniques of re-performance to see the certainty of the result.
E.g. if the rule-based formula of a system is 2+2=4, the internal auditor will re-perform the task to ascertain the same results appear every time they add two numerical numbers. Whereas, in the AI, if the data sets have the highest result indicating data point as 2+2=3 and fewer data sets indicating 2+2=4, than it might give the higher percentage of likelihood let’s say 70% chances of 3 being the correct answer and 30% chances 4 being the correct answer.
Such instances may lead to the wrong prediction and thus wrong business decision. In this simple illustration, the AI results are based on popularity (higher percentage) than the correctness of the result (lower percentage).
2. SOURCE OF THE DATA
The quality and quantity of data affect the results of the machine learning system.
To know the very basic of Machine Learning refer to my earlier post "What is Machine Learning (ML)?" Click Here
If AI is the body, the data is the soul. AI uses the data to give a probable answer to the problem and to find a hidden pattern. Having said, this pose another big challenge due to several challenges associated with the data:
- Is the data sourced of the right quantity and quality to be used in AI?
- Is the data sourced are from legal means?
- What is the relevance of the data? E.g. if the data sets are only provided with American people dietary habit, will it be able to predict the Asian people dietary habit?
- Is the source data properly collected, cleaned and processed?
- Is the data set being biased to achieve malicious intent through AI predictive model?
- Is the unstructured data sourced (data fish) from credible sources (data lakes*)?
* Data lakes are established to serve as a repository for unstructured documents.
3. AI RESULTS MAY CHANGE OVER THE TIME
AI uses data to make a decision and as the data evolve and more data sets are available either from a different source or through its own learning, the machine learning may behave differently and provide different results. E.g. as we stated in the above example if the majority of the data reflects 2+2=5 than the results might occur as 60% probability of 5 is the correct answer, 30% probability of 2+2=3 being the correct answer and 10% probability of 2+2=4 being the correct answer.
As an auditor, the same AI application will give different results over a period of time, which again poses the risk of volatility of the business decision based on the data set.
Based on the above only three major issues, it is important to have an oversight on the governance of the AI application and it can only happen when there is a high level of coordination between executives, data scientists, programmers, attorneys, and auditors, among other players.
The internal auditor can play a vital role when a business intent to explore the use of AI by partnering in identifying the key risks and providing assurance on the data source, quality and the methodology deployed in AI to use that data for predictive results. The level of risk post-implementation can be mitigated by a thoughtful and thorough risk assessment during system development.
#ArtificialIntelligence #AI #InternalAudit #Audit #Auditing #MachineLearning #ML #SpreadKnowledge
If you find it useful, don't forget to Like and Share to impart knowledge and if you have something to add, comment below!
ABOUT THE AUTHOR
Arif Zaman brings with more than a decade of proven experience in internal audit, risk management and fraud investigation. He is the Head of Internal Audit at Private Joint Stock Company based in Dubai, UAE. He holds a MSc in Professional Accountancy from University of London and BSc Hons in Applied Accounting from Oxford Brookes University along with an impressive set of professional certification including ACCA, CIA, CISA, CFE, CCSA, CRMA, CRBA, CPA and CGA etc.
For more immediate reading, here are some other posts I have written:
Technical Article
How to become Internal Auditor? How to Gauge Audit Department . Do not trust Artificial Intelligence . How to establish Internal Audit Department in 8 simple steps . Corporate Governance . Risk Appetite . Road Map to Data Analytics . Political pressure on CAE . Difference between the role of internal control, compliance, risk management and audit? . Internal audit is a dying career? . Internal audit - Innovate or stagnate . Internal audit insight from IIA President . Auditing business ethics . Business email compromise . Create a risk register in 4 steps . Cloud computing - Internal audit perspective . Annual risk assessment (4 steps) . Annual audit planning process (5 steps) . Role of internal audit in risk management . The impact of emerging technology on auditing . Family business governance . New IPPF 2015 (summary) . Internal audit function maturity curve . Real story - Ponzi scheme
Others
Feel like you are falling apart . My most vivid childhood memories . I think of my failure as a gift . Life changing story - From admin staff to TV anchor . Remove toxic people from your life . Africa is not a country . The best time of the day to do things at work . Build your personal brand . Pass the 6 second CV scan test
Member, Global Advocacy Advisory Committee, IIA Global; Former CEO, The Institute of Internal Auditors-India (IIA India),
5 年This a very good though provoking article & just as your other articles worth referring to in research papers & training material?
Deputy General Manager Control and Compliance (SOX)
5 年Very good article
VP (Head) Internal Audit in a PIF Company | Cybersecurity Committee | I Help Develop and Improve Governance, Risk, and Controls| (x)IA Head in a Listed Co. in KSA, Alkhorayef, Zamil Groups, KPMG | Cyber, CIA Trainer ??
5 年Nice read. As usual simplified and to the point. Arif Zaman What do you say about python, SQL, deep data analytics skills and other like stuff. Do we, as auditors, need to acquire such skills now, because otherwise, we may not be able to proactively identify and highlight the risk in system development (as you have mentioned in this article). Needless to mention the technicalities involved in all above programs or skills. OR Just having basic understanding of AI and MLand their working mechanism, we can manage risk based auditing effectively?
Experienced legal, eDiscovery and Investigations professional. Certified Fraud Examiner (CFE) with the demonstrated ability to organize and manage teams working in government, large, unstructured relational databases
5 年This is a great article. I wrote the fraud analytics and algorithms for the fraud review of the BP Deepwater Horizon Settlement. Those analytics were initially created so I could audit the results of my own qualitative review. While that was the opposite scenario, I agree that to be thorough there should always be an audit of the AI processes. Always. This is also necessary to provide proper context to the data. Fraud and money laundering activities by their nature are intended to deceive and misdirect. The crimes themselves are often very creative so a general, blanket application simply will not account for all permutations and evolutions of fraud/AML activity. I conduct an audit of the analytics as part of any review I do. I consider employment of the analytics and audit of the analytics complimentary facets of a single process.