How Explainable A.I. Can Boost Revenue

The Explainable AI Problem & the Black Box Dilemma

If you enjoy articles about A.I. at the intersection of breaking news join AiSupremacy?here. I cannot continue to write without community support. (follow the link below).

https://aisupremacy.substack.com/subscribe

I bow to LinkedIn Engineering's evolution to Empower Sales Reps to be more efficient.

This is a pretty crazy enterprise testimony to the power of Engineering, Datascience and evolution of talent to improve revenue. This is not a sponsored post.

Click on the image to learn more about LinkedIn's Engineering. Blog.

https://engineering.linkedin.com/blog

This article is slightly easier to read with images.

A.I. Explainability and the Black Box Problem

The Black Box Problem is traditionally said to arise when the computing systems that are used used to solve problems in AI are opaque. Think of it when engineers and sales managers have power struggles at companies, how is an Engineer supposed to explain to a Sales manager how A.I. can solve their issues, when it’s not overtly clear how the A.I. arrives at the answer in the first place?

One of the biggest hurdles that AI faces?today is?public?trust and acceptance.?For Sales managers that want ROI, they need some measure of opt-in in the Enterprise setting to adopt new data tools and A.I. systems. Meanwhile customers and consumers who are just ordinary people often?struggle to?trust the decisions and answers that?AI-powered tools provide.??

Now let’s say an A.I. system is not good at explaining to you how it solves or arrives at a conclusion? The AI black box problem feeds?this hurdle?further.?AI doesn’t show?its workings.?It doesn’t explicitly share?how and why it reaches its conclusions.?All we know is that some omniscient algorithm has spoken.?This can be a real problem even in a society that has become rather data-centric and wants to actively adopt A.I. to boost revenue.

A.I. Explainability?is a super interesting issue for various reason. This is also because as A.I. gets more advanced and if AGI or some stage of AGI manifests it will be doing this many people won’t be able to understand.

While it’s fun to speculate about it, in the real world Explainable AI is a set of tools and frameworks to help you understand and interpret predictions made by your machine learning models, that is likely a perquisite for the design of?interpretable and inclusive AI.

In the last few years?Microsoft Research?has become one of the top AI R&D teams in the world.

I think A.I. explainability and interpretable and inclusive A.I. are basic tenets in any A.I. for good mission statement.

Sales Augmentation can be an A.I. For Good Stepping Stone

This so happens to be one of the?goals of LinkedIn?and its parent company, Microsoft. So let’s look at our case study today related to this. This is regarding my snooping on LinkedIn’s Engineering blog.

Co-authors:?Jilei Yang,?Parvez Ahammmad,?Fangfang Tan,?Rodrigo Aramayo,?Suvendu Jena,?Jessica Li

Read LinkedIn Engineering Blog

The journey to build an explainable AI-driven recommendation system to help scale sales efficiency across LinkedIn

Reference: https://arxiv.org/abs/2105.12941

The LinkedIn Engineering blog has a lot of interesting work. And this topic really stood out to me. Below is from their the?LinkedIn story:

• When the pandemic hit LinkedIn Sales teams needed to invest their time to have a?deep understanding of their customers’ goals?(Enterprise clients) and be able to create highly personalized product and solutions recommendations, which can be a time-consuming process.?

So, the question became: how could they help their sales team effectively identify the best LinkedIn solutions or products to fit customers’ needs in a scalable and accurate manner?

To meet this challenge, the data teams leveraged machine learning (ML) models to better segment, prioritize, and help target accounts for our sales representatives.

How Can Teams Bridge the Gap from Engineering to Sales

While this ML-based approach was very useful, LinkedIn found from focus group studies that ML-based model scores alone weren’t the most helpful tool for their sales representatives.

Rather, they wanted to understand the underlying?reasons behind the scores—such as why the model score was higher for Customer A but lower for Customer B—and they also wanted to be able to double check the reasoning with their domain knowledge.

The LinkedIn Engineers were able to device a system where they expanded this tool to leverage the state-of-the-art, user-facing explainable AI system?CrystalCandle?(previously named Intellige) to create narrative-driven insights for each account-level recommendation.

Read Paper on Crystal Candle

CrystalCandle?plays an important role in Project Account Prioritizer, how data teams leveraged machine learning (ML) models to better segment, prioritize, and help target accounts for our sales representatives. So by helping their sales team understand and trust the modeling results because they understand the key facts that influenced the model’s score.

In short, LinkedIn was able to create a way for its Engineers and Datascientists to explain the ML models and tools to the Sales managers and business development account managers. This was really interesting to me as this exact issue is occurring in most companies and especially bigger Enterprises where data scientists and A.I. tools are leveraged to increase sales and improve accounts. In the real world there are even more pure and more applied data engineers with their various preferences.

Enjoy this Newsletter, recommend it to a friend:

Share AI Supremacy

How A.I. Explainability can Boost Sales

Anyways, once AI Explainaiblity was achieved in this context at LinkedIn for this particular tool more ROI occurred, that is, the combination of Project Account Prioritizer and CrystalCandle?deepened LinkedIn’s customer value by increasing the information and speed?with which our sales teams can reach out to customers having poor experience with the products, or offer additional support to those growing quickly.

Fundamentally this is fascinating since LinkedIn on many levels with its Sales Navigator and other things is a B2B sales driven architecture. Of course this is diversified increasingly with advertising revenue, but this was not always the case.

Many of the computing systems programmed using Machine Learning are opaque but companies like LinkedIn are evolving to find way around those classical engineering vs. sales issues.

The genius of how?Jilei Yang?and these data scientists at LinkedIn explains it is thus:

Project Account Prioritizer: Predicting upsell and churn for our SaaS products

• Before Project Account Prioritizer, sales representatives relied on a combination of human intelligence and spending huge amounts of time sifting through offline data to identify which accounts were likely to continue doing business with us and what products they might be interested in during the next contract renewal.
• Similarly, identifying accounts that were likely to churn and proactively addressing them constituted a huge time draw for the sales teams as well.

Now at LinkedIn with Account Prioritizer, Sales Reps were essentially being augmented with A.I. But to be truly good at their job, they needed to understand why and how the tool worked.

• With Project Account Prioritizer, LinkedIn was able to greatly improve the efficiency for these sales teams.
• They provided?accurate predictions to differentiate the accounts that were likely to upsell/churn, and predict to what degree, and quantify the number of products they were likely to buy/churn in the upcoming renewal.
• Technically they did this by training a set of?XGBoost regressors?on historical purchases and renewals of our customers across the globe.
• Some of the patterns that the models look for include historical bookings (e.g., field vs. online spend), trends in product engagement and usage (such as utilization of our Recruiter or Jobs products), hiring patterns (such as growth in employees, hiring seniority levels, and talent flows), company firmographics (such as industry and its performance during COVID-19), macro trends, and most importantly, delivered customer value (e.g., hires delivered by LinkedIn).?You get the idea!

Two challenges that make this modeling exercise complex are :

1. Generating accurate labels for churn/upsell: As with other SaaS products, while churn happens during renewal, upsell can happen throughout the year—e.g., an add on opportunity mid-cycle.
2. ~ They solved this by designing overlapping time periods of label generation throughout the renewal cycle, instead of one label assigned to a particular renewal.
3. Generating predictions in advance of actual renewal: For our sales teams to act on our predictions (mitigate churn or justify upsell), we need to generate them in advance of actual renewal, so the most recent product engagement and usage data would not be available for predictions.
4. ~ They solved this by creating various time series features across historical data (e.g., Inmail Response Rate or Number of Job Applications in the last three months, last six months, last nine months, etc.) to capture the evolving trends. They also scored the accounts monthly to provide the most recent predictions to sales teams. In particular, features from the?LinkedIn Economic Graph?such as talent flows and industry macro trends have been very helpful.

You can visualize this like this:

See image here.

• Figure 1. Label generation across overlapping time periods within the same renewal cycle. Blue square dots represent the beginning of a contract (new business, add-on, renewal), while gray square dots represent the ending of a contract. Green up arrow means an upsell label, while red down arrow means a churn label.

The Tool Can Often Confirm Field Knowledge and Intuition

Performance of these models has reached a range between 0.73-0.81 on metrics Precision and Recall. Qualitative feedback gathered from our sales teams also showed that the?models match closely with their field knowledge and intuition?(~80% - 85% accuracy across individual sales books from field surveys).

• All of these scores are shown to sales teams by integrating directly into the CRM system (e.g.,?Microsoft Dynamics), enabling them to decide the best course of action.
• In the near future, LinkedIn Engineering is exploring multi-task learning to combine these separate models into a?single consolidated framework?to provide more unified recommendations and further simplified end user experience.

CrystalCandle overview

So what the heck is?CrystalCandle? It’s the User-Facing Model explainer, but we’ll get to that.

For a Sales Rep, data isn’t enough, they need to trust and understand the origins of the data.

• A key thing they learned from a focus group study with sales representatives is that the scores alone may not be the most helpful. For sales representatives to take action, they need to know the?underlying reasons behind these scores, and they also want to double check these reasons with their domain knowledge.
• Even though some state-of-the-art model interpretation approaches (e.g.,?LIME,?SHAP) can help create an important feature list to help users to interpret the ML-model provided scores, the feature names in these lists are often not very intuitive?to a non-technical audience.
• Sales managers and Sales Reps are usually fall into the category of a non-technical audience.

LinkedIn Engineers Solved an Important A.I. Explainability Gap in their Sales Teams

Like Engineers the LinkedIn team sought of a model to Explain the AI behind how the scores were calculated and I think that’s pretty brilliant.

To deal with the above challenges, they built and implemented?a user-facing model explainer?called?CrystalCandle, which is a key part of developing?transparent and explainable AI systems at LinkedIn.

The output of CrystalCandle is a list of top narrative insights for each customer account (shown in Figure 2), which reflects the rationale behind the ML-model provided scores. These narrative insights are much more user-friendly, bring important metrics to sales representatives’ attention, and are clear and concise. These narratives give more support for sales teams to trust the prediction results and better extract meaningful insights. I find this truly fascinating, and I’m not even a data scientist.

Here is a mock example of an account:

So the Engineers essentially created a language guide to the scores. Obviously this would help Sales Reps confirm their field knowledge and sales intuition. This is truly phenomenal and an example of the?AI-human hybrid workforce?in an Enterprise context in business development at LinkedIn. We have to be clear this already happening.

No wonder LinkedIn’s sales are growing when they have Engineers like this.

The solution is really elegant:

CrystalCandle serves as a bridge between the machine learning models, such as the upsell propensity model in the Project Account Prioritizer, and the end users (i.e., the sales representatives).

Let’s take a look:

See image here.

• Figure 3 shows the pipeline of CrystalCandle.

1. The Model Importer consumes the model output from a set of major machine learning platforms (e.g.,?ProML), and converts it into a standardized machine learning model output.
2. Then in Model Interpreter, we implement model interpretation approaches onto the standardized machine learning model output and generate the important feature list for each sample.
3. They also feed some additional inputs into CrystalCandle at this stage, including the additional feature information and narrative templates.
4. They then conduct narrative template imputation in Narrative Generator and produce top narrative insights for each sample.
5. Finally, we surface these narrative insights onto a variety of end-user platforms via Narrative Exporter. The entire CrystalCandle product is built on?Apache Spark?to achieve high computational power and high compatibility with upstream platforms such as?ProML?and downstream platforms such as MyBook in?Microsoft Dynamics. Next, they dive deeper into the major components of CrystalCandle.

Narrative Generator and Insights Design deep dive

The goal of Narrative Generator is to produce the top narrative insights based on model output and model interpretation results. Some insights that were helpful in designing the Narrative Generator include:

1. They need to incorporate feature descriptions into narratives to make feature names readable. This requires input from domain experts such as the model builders.
2. They do not want to overwhelm end users by producing one narrative for each feature, since models can have hundreds of features. Therefore, feature clustering based on semantic meaning is important.
3. There may exist a set of narratives that are consistent (e.g., XX changes from A to B), and so constructing reusable narrative templates can be helpful.
4. They need to select top narratives in a scalable way, where we can leverage the feature importance scores from the Model Interpreter.

[skipped a bunch of steps you can read in the original blog]

Results and in Practice

For several quarters so far, CrystalCandle has assisted LinkedIn data scientists in converting machine intelligence from business predictive models into sales recommendations on these sales intelligence platforms. As you can imagine this has enabled LinkedIn sales to function better.

• The models resulted in +8% lift in Renewal Incremental Growth (a measure of bookings growth during contract renewal) for the business.
• Feedback from the sales team has also been highly positive: “The predictive models [are] a game changer! The time saved on researching accounts for growth opportunities has been cut down with the data provided in the report which has allowed me to focus on other areas.”

What’s so valuable with what the LinkedIn Engineers created is their methodology could be scaled to other applications. CrystalCandle-based sales recommendations have also been surfaced onto other sales intelligence platforms for different audiences.

I hope this case study illustrated how companies like?Microsoft are upgrading A.I. explainability?in their specific teams like LinkedIn Engineering empowering their Sales professionals using Microsoft software, now more effective than ever.

Major Takeaway

So many Executives know the importance of not just being data-centric but having Machine Learning models can that can drive revenue, but its with A.I. explainaiblity where the true magic can happen in sales. Datascience is amazing!

Thank you for reading AI Supremacy . This post is public so feel free to share it.

Share

Read the original:

https://engineering.linkedin.com/blog/2022/the-journey-to-build-an-explainable-ai-driven-recommendation-sys

If you want to support me?so I can keep writing, please don’t hesitate to give me tips, a paid subscription or some donation. With a conversion rate of less than two percent, this Newsletter exists mostly by the grace of my goodwill (passion for A.I.) & my own experience of material poverty as I try to pivot into the Creator Economy.

Anyways I hope you enjoyed the topic, that’s all for today.

If you enjoy articles about A.I. at the intersection of breaking news join AiSupremacy?here. I cannot continue to write without community support. (follow the link below).

https://aisupremacy.substack.com/subscribe