Will Automation Reduce Modeling Skills ?

An article in The Toronto Star on how recent airplane crashes are making some people re-think autopilot makes interesting reading. Through interviews with experts, the article makes a few points :

• new pilots are less comfortable with taking manual control of planes, as they spend more time learning how to use automated systems
• increased usage of automated flying has resulted in them not knowing how to react, or reacting too slowly to changes/unexpected events
• a shortage of pilots globally has led airlines to rely on less experienced pilots, and more on automated flying
• the industry is not adequately informing and training the new pilots on manual flying, and what to do when things go wrong

Sound familiar ?

The world of data mining and predictive modeling is struggling with similar challenges. In speaking with risk executives from dozens of global banks, I find that :

• Initiatives such as Basel II/III, IFRS9, CECL, DFAST, the rise of non-traditional players (Fintech), and the increased usage of analytics in banks to make better decisions has resulted in a shortage of qualified modelers.
• The use of 'Big Data' such as information from social media, IoT, deeper transactional data, Telco data and others have increased reliance on automated analytics. Modelers simply don't have the time to properly study the data.
• Newer modelers are hesitant to question the data, apply manual adjustments to it when needed and question the model results from a business perspective.
• The increasing usage of AI and Machine Learning is causing concerns around transparency and control.

Risk executives tell me they are concerned that the combination of very large datasets and complex algorithms mean their staff simply cannot perform exhaustive analysis of the data, or explain what is going on in simple business terms. They worry that less vigilance may leave them exposed to using variables that would cause legal and ethical issues in decisions, or use models where spurious variables are used without being known. We are moving from a world where each model variable and its causality is expected to be explained, to one where the model and its contents can be unknown. In many institutions, there is also a culture clash between the 'old guard' i.e. people who relied on heavy business expertise/intuition, were skeptical of complex methods, used simpler models and placed a high premium on explainability/causality, and the new generation of modelers who are eager to use more complex 'black box' methods, are skeptical of intuition and place a higher emphasis on results than causality or explainability.

The reality is that the amount of data available to financial institutions will only increase, more complex modeling algorithms will be used to deal with the larger and more sophisticated data, and the amount of analytics performed in banks will applied on expanded use cases. In particular, the use of unstructured data will require new thinking and analytical skills.

So how do we equip new modelers to produce results that are not only statistically robust, but are also infused with business sense and skepticism ?

Business Training/Mentoring : Better models in business are built when you utilise robust statistical methods, and interpret the results from both qualitative and quantitative perspectives. New modelers should be given mentoring by experienced business people on business issues, historically known and accepted causalities, the basics of lending money, when to recognise that the data may be wrong due to biases, interpreting model results based on business requirements etc. This will enable new modelers to overlay business judgment on newer, more complex techniques.

Data Audit : The single biggest issue banks face in any data based activity is in dealing with the data itself. In almost all situations, the data is dirty, biased or incomplete. As such, you cannot merely perform statistical analysis on it and rely on the results. All data in the firm should be analysed on a variable by variable basis at least once. Identify variables that are biased, dirty/unreliable, illegal, unethical (and those which may be correlated with them, for example media 'likes' in Facebook), traditionally strong and so on. This will prevent suspect variables from being analysed or entering into black box models. It will also alert new users when traditionally strong variables don't enter models (this is usually due to data issues).

Business Analysis Assignment : In my experience, the better modelers have been those who have a robust numerical qualifications, but also have some experience in lending, risk strategy development (for example originations, credit limit management or collections) or performing other business analysis for lending and customer management. They have a better understanding of the underlying data and its provenance, decision making and business thinking. Full time modelers could be given periodic assignments to perform such work. This will enable them to get a broader understanding of how their models are used, practical realities of business decision making and perhaps understand the point of view of their business colleagues.

Regular manual modeling/Benchmarking : In cases where modelers are heavily involved in using automated techniques for feature engineering and modeling, it would be useful to build a simpler model such as a grouped variable scorecard. This can additionally serve as a benchmarking exercise, and will enable them to practice modeling techniques that are more business oriented. In particular, 'grouping' or 'binning' variables and explaining relationships in business terms is a valuable exercise for this.

The actions suggested above can result in closer cooperation between modeling teams and their business counterparts. They will also equip modelers with solid statistical backgrounds in applying business judgment in their work, learn to question data and apply appropriate skepticism to results.