Quantifying your ML Efforts - (how to get more money from the CFO)

I talk to alot of people in the advanced analytics discipline in the largest enterprises on earth; Data Scientists, ML Engineers, DevOps, Architects, and Executives from VP’s up to CIO, CDO. Something I’ve been looking for over the past 18 months is how the practice is being metric’d. What are the measurements of success for the individual contributor, or the executive responsible for the outcomes? The answer has been elusive. Technicians tend to look at things like uptime, # of API calls, models in production, models in development, etc. Executives tend to look for operational efficiencies, like faster turnaround times on applications (loan docs, for example), reducing labor for back office tasks, etc. However, more often than not I find that people are not able to quantify the work being done. There’s many reasons for this, not the least of which is that the advanced analytics supply chain is hugely complex and can span many departments in an organization.  

We have a saying here at Algorithmia: ROI happens at the API call. Everything prior to that is an expense - and one that we’re not seeing being addressed very often. This is one of the reasons ML efforts lose momentum, or worse: die on the vine.  

Models do not add financial value until operationalized and acted upon 

The very first step any organization should take is to make c-level executives aware of the power of analytics. Most of them do, but generally at a ‘lookback’ level and not a predictive level. Finding a common language, sponsorship, and collaboration are vital to having a successful and healthy Data Science practice.  

From there, you should take the mindset of understanding how to track your ‘model to the bank’, or the cash-to-cash cycle of your ML efforts. Understanding how your work affects the P&L, and the time it will take to have that effect will help open doors to very senior level conversations that can remove obstacles and make alliances that will speed things up.  

Most organizations have some type of framework for identifying business needs, quantifying it, exploring the technical feasibility, and then developing the solution. What we’ve seen is that these frameworks are used in a one-off manner, and each project becomes bespoke (the silo'd nature of Data Science in very large organizations adds to this challenge). A better way to approach this is to identify change agents in your organization, and work with them to develop a project portfolio. Individual business needs can then be organized and quantified using the framework above, and then prioritized. One customer of ours has an internal dashboard that anyone in the company can access. It shows each project, the stakeholder, the expected value, the budget associated with it, and where it is in the queue (idea, started, in progress, finished, etc). The visibility this provides ensures that each business stakeholder is in the know at any time of not only where their project is, but also what other projects are on the board and what value each is providing to the company. This transparency is absolutely essential in keeping the trust and relationship with the business healthy.

For the Advanced Analytics leader, this planning and prioritization also allows them to show the bottom line dollar amount that their efforts are giving back to the enterprise. This number can help CFO types understand the importance of not burning your time, money, and focus.  

Align with the Strategic goals of the organization

Along with the above, it’s important to make sure you’re prioritizing projects that are important not only to the C-level execs, but also the board, and the shareholders. Sometimes dollars and cents aren’t the main driver; for example one of our clients places high priority on customer experience. This can show up on a P&L in a number of ways, but most important to them is that the customer has a seamless experience with minimal disruptions and can transact easily. If machine learning can help get ahead of issues for that customer before they even know it, then that’s a win.   

Some areas that we’ve seen particularly important: commercialization (product pricing, etc), supply chain, and customer experience.  

Optimize your resources

If you’ve adopted the ‘model to cash’ mindset, and recognize that anything prior to the API call is an expense, then you can see the importance of balancing your resources to optimize your efforts. I often ask clients what their ratio of developers to operations is. The ratio is all over the board. Most are 50/50 give or take 10%. One client is 10:1! They feel that they don’t have enough operational support for their Scientists, and therefore can’t meet the needs of the organization and their business partners.  

Here’s a better way to look at it: If you’ve worked closely with all of your business partners, identified and quantified needs, aligned them with the strategic goals of the company, then you’ve likely prioritized them and now have a pipeline of WIP and a number of projects on the bench waiting to go. Why not focus all of your resources on providing more value by getting to work on the backlog? Use technology and partners to do the operational work for much less cost, and put your energy on delivering projects that accelerate your total value to the company. Speed to value. 

Closing thoughts  

At the end of the day, models don’t make business run better. People do. Without creating the alliances and buy in from the business, you risk not having trust and therefore having projects that never see the light of day.  

The current Covid environment has proven just how important it is for companies to have a digital strategy. Unfortunately, many have succumbed: J. Crew, Hertz, Pier 1, Golds Gym, JC Penney, 24 hour fitness, GNC, Brooks Brothers, Sur La Table, Guitar Center, and numerous others have either filed for bankruptcy or closed their doors in 2020. Yet others like Best Buy and Home Depot posted amazing growth numbers for one of the most challenging years in modern history.  Smart strategy and investments in technology have always been talked about as being vital, but 2020 proved just how critical it really is.