The Scientific Method and Community Metrics

Reposting this for reference, as my blog went down a while back but lots of folks found this post interesting. Thank you, Wayback Machine for your excellent archiving, as always.

The Scientific Method and Community Metrics

One of the recurring topics in the community manager network is how to measure success and “prove value.” As always, I have Thoughts?? on this topic.

TL;DR

Here’s the TL;DR version:

1) Align executive expectations on outcomes.

2) Write your hypothesis on how best to support those objectives.

3) Define a limited set of measurable and actionable key performance indicators (KPIs) that will help you course correct, if needed.

4) Develop an action plan for when things go unexpectedly.

The Grand Experiment of Community Building

Successful community building is, in many ways, a scientific experiment. While communities of humans share many broad characteristics, each community has its own inner culture of acceptable behaviors; shared interests, needs, and desires; and expectations. As a result, community building is a new experiment each time you join or start a new community. With experience, we develop a framework of community building that we are familiar with, and we may know what generally what works and what doesn’t for a particular ?type? of community, but we can never operate with 100% certainty that a given program or action will impact our community in the way we expect.

Setting Up the Experiment

With your executive team’s input, start by developing a hypothesis about the impact of a successful community on the business. This often takes the form of a vision or mission statement. Here are a few examples:

• “Our community will facilitate peer-to-peer knowledge sharing that will?reduce our support costs.”
• “Our community content will be valuable and SEO optimized to?improve our brand reach and recognition amongst developers.”
• “Our community will host customer-only exclusive events that connect our customers with our internal experts,?increasing product adoption amongst our customers.”
• “Our community will help our end users find answers quickly,?increasing overall customer satisfaction and renewal rates.”

Note that these hypotheses focus on the community value add that will impact a particular business metric that is important to the company and that the leadership team expects to see to consider the community a success. This hypothesis should be a result of discussions with that same leadership team in which you ask them to answer the questions: “Why do we want a community?” “Why did we choose to pursue this?” “What do we expect to see from a successful community?”

Do not move forward until you have reached consensus amongst your executive stakeholders on the purpose of the community.?Trying to develop goals and plans without the agreement amongst your leadership team on what the intent of the community is will set you up for failure and frustration in the future.

?? If you are struggling to achieve consensus, this is also a good time to identify your?executive sponsor?or the person who you can go to to rally their peers and subordinates in support of your project.

Document this consensus via your hypothesis statement. This is a good time to start a centralized repository of information about your project that is visible publicly within your company.

Designing Your Methodology

Once you have your hypothesis, this will serve as your Objective (the “O” in “OKR”). Think about how you will quantify your progress towards, or achievement of, this objective. The “how will you quantify this?” component is the Key Result (“KR” in “OKR”) that you will use to communicate to your leadership team on the success of your program. The Key Result should align with the?business outcomes?you identified in your hypothesis, these might look like:

• “Decrease support ticket volume by X%” (aka “ticket deflection”)
• “Increase new developer members in our community by X%”
• “Increase feature adoption by X% of existing customers” or “increase user licenses by X% at existing customers”
• “Increase NPS by X% year-over-year” or “Increase customer satisfaction survey ratings by X% month-over-month”

From your Key Results, you can start to develop your working metrics. These are the core key performance indicators (KPIs) that you will build a reporting dashboard around and that will help you monitor the pulse of your community program. Your KPIs must give you a specific course of action, should they trend in the wrong direction or should the community respond to your actions unexpectedly. KPIs are focused on establishing the cause-and-effect of explicit actions that you’ve taken to stimulate a certain result from your community. This might look something like:

• Time to first response, percentage of questions with answers, rate of return by new members
• New member growth, increased mentions of the company/product in tracked channels, increased positive sentiment amongst developer channels, increased referrals to trial downloads
• Increased number of new questions, increased percentage of questions answered by community members/non-employees, increased percentage of product ideas submitted by community members, increased attendance at product-focused activities (events, meetups, talks), increased participation in beta testing/collaboration opportunities between the community and the product team(s)
• Increased rate of return for new members, increased ratio of daily active users:monthly active users, increased positive user sentiment

Of course, we all know that the community often does several or even all of these things, so how do you prioritize? Again, focus back to your hypothesis. What is the priority for the business leadership team? What is the priority for your executive sponsor? You should look to your leadership team to help you determine what the priorities are for the business.

?? Select?no more than ten?KPIs to track and report on.

Remember that your KPIs should be actionable and should give you direction. This means that you should be able to?directly?influence those measurements by taking action. And, if you have too many signals, they become noise, so hone in on your hypothesis and keep your scope tight.

???You should always be able to answer the question: “Why are we tracking this and what will we do with the information we gather?”

Run Your Experiment

Set up your executive team and stakeholders with the appropriate expectations for the amount of time it will take to see results. Again, this should be addressed early on in defining and getting buy-in for your hypothesis. Any community builder will tell you that community building is a long-game. This means it will take at least a year to establish the foundational elements of a new community. You’ll need to give yourself and your community time to respond to any initiatives that you implement.

Using your KPIs as a guide, formulate a plan for how to influence the KPIs that you’ve identified as critical for impacting your success metrics (Key Results). Your roadmap should directly tie to the KPIs you’ve established, with each action and each task designed to influence them. One way to ensure that this happens is by using a project management tool like Jira, creating Epics for each KPI and assigning your tasks and initiatives to those Epics.

Telling the Story of Your Success

Check in on the community program’s performance against your KPIs regularly – but not too often. Remember that community building is a long game and we’re looking for trends over time, not transactional and short-term blips. I’d recommend reporting on your KPIs once a month. More often than that and you’re likely to make deductions and assumptions based on limited data. Less often and you may end up wasting time and resources on tactics that simply do not work for your community.

Plot your events and activities across your trend lines to see how what you’ve been working on impacts your KPIs. This will help you write the narrative of your successes (where the trends and events intersect) and your learnings (where the trends and events don’t seem to have any correlation).

Wherever your activities correspond to a spike in your trend lines, you may choose to dig in deeper (if you want to be extra fancy, you could do a?regression analysis ?if you have a skeptical leadership team or you just want to understand more of the factors that might influence your outcomes for better or worse. But at a minimum,?do more of the activities that make your trend lines go the right direction.

Interpreting Unexpected Results

So, what do you do when you see an outcome in your trend line that you don’t expect? Maybe the trend goes opposite the direction than you expected or maybe there is no measurable impact at all.?What if we added a few anomalous events to these metrics?

If you see an unexpected result from a planned activity or event, there are a couple of possible causes that you should explore:

• Not enough time for the impact to be clear.?If your KPI is actually a lagging indicator, there may not be a linear correlation or immediate impact from your activities. For example, is it reasonable to expect immediate increases in your SEO ranking compared to a well-established site, like Stack Overflow, if you’ve made some changes to your own site? Probably not. So, you shouldn’t expect to see a major increase in referrals from search results for particular keywords in the same month that you’ve made the UI changes designed to impact SEO ranking.
• Your tactics are not aligned with your community’s motivations.?I explain a bit about intrinsic vs. extrinsic motivators in?my talk about psychological safety , but this topic is worth exploring more deeply if you’re unclear about what motivates your community members’ behavior. A lack of understanding here will create bigger problems for you in the long run and should be explored as soon as possible in your planning of any new initiative.
• Other factors influencing your outcomes.?Whether your activity or event is drowned out by an even bigger news story or announcement, your members are dissatisfied with your company for reasons outside of your control – like a bad customer support experience, or something as simple as a regional holiday or cultural event (e.g. summer vacation in western Europe) could cause a disconnect between a tactic that would otherwise resonate with your members and the results you’re looking for. When researching anomalies, be sure to rule out any external factors that could have skewed your results.

???In the chart above, the easiest anomalies to eliminate would be the summer months and end-of-year holiday season. You could validate this by checking the year-over-year trends during that same time period. The next step would be to compare the trends against any known external factors, such as company or industry events which may have brought more attention to your community than usual. Lastly, check against your own initiatives to determine which tactics had the most desired (or most undesirable) impact on your KPIs.

??Remember that your KPIs should be providing you with actionable intelligence, meaning that there should be clear and explicit actions to take whether the metrics prove?or disprove?your hypothesis.

Adapting the Roadmap as a Living Document

The most important element of the scientific approach to community strategy planning is the willingness to adapt based on new information. Your strategy should evolve as you begin to compile more and more data to ensure that you are not wasting time and resources on tactics that are not strategically beneficial to your community and your company. Keep going back to your hypothesis to check and see if:

1. it still reflects the expectations and needs of the business
2. it is supported by your evidence

If not, it’s time to revisit and revise with your stakeholders. Admitting that your hypothesis was wrong, your plans didn’t work out, or your assumptions were inaccurate can be a tough conversation to have, especially with your management team, but ultimately you are working in the best interests of your company and your community by leaning on data to make smart, educated decisions about how to best serve your members.