Pragmatic Algorithmic Risk Management: Scoping an Algorithmic Assessment
Sri Krishnamurthy, CFA, CAP
CEO, QuantUniversity | Entrepreneur | AI Innovation | Education | Author | Speaker
I recently met with a client who wanted a proposal for an algorithmic assessment. For regulatory reasons, they wanted a third-party audit. We asked what the scope was and got into a whole discussion on audit scoping.?
The biggest question one needs to answer when structuring Algorithmic assessments is to define what is in scope!?
Typically, a scope-of-work document is developed prior to starting an assessment and the parties agree on what’s in scope and what is not during the assessment. A statement of deliverables is listed and bounds are drawn on what is to be accomplished in the timeline.
Here are some best practices to think about. When you intend to get a third party independent audit, the first step is to scope the audit internally. We advocate initiating a scoping process where all stakeholders come together to define the scope of the assessment.Typical questions that are addressed include:
When structuring these audits, the more specific the better. You could take a top-down or bottoms-up approach to scoping. A top-down approach would list all the aspects that need to be done. This would include:
and then choosing what is to be done internally and what should be done by external auditors.?
An alternate approach is to go for a bottoms-up approach where you start with the use-case and context and define how the model would be incorporated into the use case and the aspects that needs to be audited by internal teams and aspects that need third-party audits. The scoping here would factor the regulatory requirements and/or any specific areas that needs to be audited to obtain a statement of compliance/ assessment of a particular aspect (like security/privacy/fairness) in the use-case where the model would be used.
I favor the bottoms-up approach as it focuses on the model’s use-case and factors the risks and aspects that are relevant for audit based on the use-case. When we do discovery workshops, we?usually lead with the bottoms up approach to scope and structure audits and have found this is much better than top-down cookie-cutter approaches.
We will discuss three types of assessments.
领英推荐
and how to scope them in the next letter.
??Concept of the day: ?
The New York law talks (https://news.bloomberglaw.com/daily-labor-report/nyc-targets-artificial-intelligence-bias-in-hiring-under-new-law) about Bias audits where an auditor assesses the automated tool’s "disparate impact" on people of particular gender/race/ethnicity.?
What is disparate impact?
“Disparate impact in United States labor law refers to practices in employment, housing, and other areas that adversely affect one group of people of a protected characteristic more than another, even though rules applied by employers or landlords are formally neutral. ” Source: https://en.wikipedia.org/wiki/Disparate_impact
??Keep on learning!
?? Want to learn more formally? Join the AI Risk Management Certificate program developed in partnership with PRMIA ->?https://lnkd.in/eVEhyNSQ
??Many of these topics will be elaborated in the AI Risk Management Book published by Wiley. Check updates here ->?https://lnkd.in/gAcUPf_m
??Subscribe to this newsletter/share it with your network -> https://www.dhirubhai.net/newsletters/ai-risk-management-newsletter-6951868127286636544/
I am constantly learning too :) Please share your feedback and reach out if you have any interesting product news, updates or requests so we can add it to our pipeline.
Sri Krishnamurthy?