Record (a loooong post)

A brief recap from the last post:

• CS Ops is rapidly coming to be seen as an essential part of any high-performing Customer Success organization. However, it’s still not widely understood.?
• I'd like to provide some day to day insight about the role by focusing on a specific real world example.?
• I believe that Customer Risk Forecasting (our example) is very much like finding threats within the Cybersecurity world (yes, likely because I work at a Red Canary).?
• Following the above, the first step to understanding Customer Risk Forecasting is to RECORD Customer "Telemetry"

I first encountered “telemetry” while working as a chemist. My assignment was to evaluate the effectiveness of different rocket propellants. There was no way to get close to the rockets in flight, so any evaluation had to be made via remote capture of diagnostics - rocket telemetry. The small, subtle variations in pitch, thrust, roll, burn, etc., etc., all combined to provide a clear picture of what was happening - and allowed us to do our job.?

At Red Canary telemetry took on a whole new meaning. Here, we focus primarily on endpoint telemetry - file modifications, logs, process start/end data, registry changes, network connections, module loads, cross-process events, etc., etc., etc.. But the concept still holds:?

With the right data/tools you can observe a system/object at a distance and use diagnostic information to assess the status or direction of the system - or the threats inherent within.

How does the definition of telemetry tie into the question: How does CS Ops operationalize risk??

Over time I’ve come to think of customer churn risk as the same thing as threats - be they from a rocket heading towards you or in the form of a customer leaving your business. In all cases, you need the right kinds of telemetry before you can assess the trajectory.?

When we think of CUSTOMER TELEMETRY we tend to automatically jump to those things that are measurable by instrumentation - logins, pageviews, click path, license utilization, etc.. Highly quantitative, typically automated, and easily analyzed (if you’ve defined what success looks like) - these types of metrics are tempting to utilize as a sole source of customer risk behavior.?

However, there are other types of more qualitative customer telemetry that are useful for understanding customer churn risk. They can include (certainly not an exhaustive list) things like:

• Sentiment - from survey responses or CSM assessment
• Maturity - product, industry, etc.
• Payment history
• Meeting notes
• Relationship mapping scores

In sort, we live in a world of RICH customer telemetry - much too much to list here and we’re discovering more all the time. As an ops person, I tend to think of all these bits of information relative to how they’re obtained (manually or automatically) as well as whether they are quantitative or qualitative in nature. This can often help to categorize the telemetry in terms of utility but often helps to same-page internal teams as well. Here’s an example of how this mental map might look:

In general… we mature our craft by moving the needle on these bits toward more automated capture and quantitative, leading indicators over time. Again, this view might help you select targets within your business to advance.?

I think you get the point that there are many types of customer telemetry. The question is:?

What customer telemetry is useful for predicting customer churn?

The answer here is likely unhelpful: it depends on your business… Ultimately, the right mix of predictive customer telemetry will be determined by your customers, product, platform, and strategy. Thus, it’s impossible to describe a universal. However, there are steps you can take to determine what telemetry is best.

Starting small and manually

While it’s tempting to jump right into HOW, it’s important to emphasize that you can get started today with a single, simple but reliable metric: Customer Success Manager Sentiment. Over the past few years I keep coming back to this bit of insight as being the most reliable predictor of threats to retention. It’s smart and inherently sophisticated, easy to capture, and you can do it for free.?

For our very first go at this approach, I built a simple Google form that we provided to CSMs in order to capture their sentiment at each customer interaction. The form captured all notes in an accompanying spreadsheet which we could use to track sentiment (on a scale of 1-5). This allowed us to do many things simply:

• We have search-able notes for all our meetings - easily shareable to the rest of the company
• We have early warning for churn events based on CSM read of Customer sentiment
• We have detailed interaction data - useful for Business Reviews, and project management
• We have training fodder (record of conversations) for new CSMs
• We have lots of detail to share with Sales and Marketing teams

The following is an example of how we tracked notes and sentiment (as telemetry) over time in the beginning. This is a screenshot of our early Google-Sheet based dashboard that captured our earliest types of customer telemetry. CSMs would complete a form after each engagement to record the telemetry and sheets did the rest (under the redacted portions you’d find detailed searchable notes):

Widening the aperture

Strategy 1. Use the Journey

Deepening your approach to customer telemetry should take place over time. Essential to the effort will be a well defined customer journey that defines success (from the customer perspective) at each stage. With this map in hand you can determine what signals to look for that indicate desired outcomes or indicators at each step. For example, if a major element of a successful onboarding is to have > “X” % of licenses in-use within a specific timeframe - you should be able to measure license usage accurately, ideally in an automated way. Without this piece of customer telemetry you are lacking in your ability to see the trajectory of the account.?

Looking at the success criteria at each stage - and importantly the lack of these signals will be the first step quickly followed by a prioritization of each of these metrics. Obviously, some types of data will be more valuable than others, and might be useful across multiple stages (in-app customer behavior for example). Prioritizing these indicators by their predictive ability for renewal is key to presenting the case to go after each in a formalized/instrumented way - whether they require additional tooling or not.?

Pros: customer-first perspective, clear cross-functional definitions of success

Cons: time consuming, possibly harder to implement in mature organizations, much consensus building

Strategy 2. Collect all the things

Another approach might be akin to the process of 19th century gold miner. That is, scoop up everything that you possibly can (customer telemetry) and extract the “gold bits” from the “sand.”?

There are lots of tools on the market that offer this type of collection out of the box. Some organizations go the rout of instrumenting this collection in house. Whether? you decide to purchase this capability or build it, gathering the data seems to be the easy part. Often, making sense of this kind/amount of data tends beyond the skill set of most Customer Success organizations. Ultimately, this need is typically addressed by partnering with your organizational data science team(s) or through the continued specialization of people on the CS team.

To cut down the volume of data at the beginning, consider that you might be able to narrow down where you look for these insights. In some cases it’s possible to find your best hunting grounds by observing commonalities among your most successful customers.?

Pros: deep insights about customers, comprehensive understanding of customer behaviors

Cons: expensive, special sills required, time consuming

Cautionary tales about too much data: https://www.dhirubhai.net/posts/jaynathan_csms-customersuccess-activity-6861113370263732224-NWQo/

Strategy 3. Analyzing past churn

Similar to looking for gold in your best customers you can find indications and warnings of impending churn by studying the customer telemetry of the ones who have left.?

Sometimes the reasons for churn are known before the event. On other occasions they are not. Either way, customers leave they leave behind a treasure trove of telemetry leading up to the event. This behavior often yields warnings for churn amongst customers who have not left.?

However, you need to ensure that you’re not leading your team astray by looking at too few events or those that are so unrelated that they are meaningless in aggregate. Sometimes customers will churn for odd or indeterminate reasons and it’s important not to adjust your strategy based on a single customer. The “right amount” differs for all businesses but there are guidelines. I think Ed Powers does a great job at explaining how easy it is to misinterpret too few data points:

CSMath: Make Data-Driven Decisions in Customer Success

If you have enough churn events to analyze, it is useful to group like customers (by market segment or some other means) in order to determine common causes by customer type. This will allow you to pinpoint the most valuable data moving forward.?

You may already have access to this telemetry, especially if you’ve instrumented your Customer Journey or whether you’re “collecting all the things.” In other cases you will have to enlist the help of a third party to ask the tough questions of former customers in order to get usable leads for the best predictive telemetry. In any case, a rich repository of this data can be extremely valuable especially as your business changes over time. Understanding how you successfully addressed past challenges is often key to overcoming those in the present.

Pros: understanding why customers leave is fundamental to your business and this approach is grounded in this understanding

Cons: you have to have churn to do this, it can be costly

Wrap-up

When it comes to customer telemetry and determining your best strategy (1, 2, or 3) the answer will lie somewhere in the combination and determined over time. Remember, it’s best to start simple and build out as you go. Time is going to be a recurrent theme moving forward, as iterating on what customer telemetry, how you detect events driven by telemetry, and how you respond to those events need to be under constant scrutiny - you need to have a plan for how to evaluate that data and process on a recurring basis. More on that next time.

References:

@Carl Gold’s amazing book/webcast: Fighting Churn with Data

Multiple talks/posts/writings by @keith: https://redcanary.com/authors/keith-mccammon/

@Sean Lane of @Drift does amazing work in the ops field. This specific talk (that I recently found) is sales related but it totally hits the nail on the head with how I think about customer telemetry: Get Reps Out of the Data Entry Business with InsightSquared CEO Todd Abbott