Knowledge Management: Analytics

Modern methods of storing and reusing intellectual property are very sophisticated. Knowledge Management (KM) portals are created and enhanced every year in all manner of businesses and government organizations. Mostly, they seek to make the stored IP generated every year easily searchable and organized so it can be reused to create additional value. These systems measure submission, review, and consumption of internal IP. These measurements can sometimes even be incorporated into performance management systems, acting as metrics for advancement or promotion in the organization.

I will use a series of posts to address two knowledge management issues using the KM example from a large pharmaceutical company:

1. Active Searching versus Passive Exposure to KM IP
2. Analytical Measurement of KM Activities

This post will focus on (2), Analytical Measurement of KM Activities. See other post for (1).

Pharmaceutical Project KM Problem Statement

The company spends $100M+ on marketing research every year, divided among the global markets it operates. As you would expect, these research efforts are coordinated by a global team of analysts. They don’t sit together when planning research investments. Drug branding varies by country but the pharmaceutical chemicals and ailments they treat are generally common across countries. The complexity of operating analysts globally and performing marketing research globally leads to research happening concurrently on the same drugs in different countries. The company believes that redundant research could be reduced and some research could be enhanced if analysts (i) reuse more prior market research or (ii) collaborate on the creation of new market research more effectively.

Implications: This pharmaceutical company needed a central portal for knowledge management with reporting capabilities on portal activities to monitor collaboration. However, we noted early on that simply requiring analyst to submit materials to the portal and make those materials searchable was not necessarily going to trigger a massive wave of reuse by itself. Analysts in their organization were busy as it is, working long hours. Random curious searches of the portal were not necessarily going to happen on their own. And the portal reporting on content creation and reuse would usually be limited to counts on portal usage, document uploads, document downloads, likes, comments etc.

Knowledge Management Analytics

If your goal is to trigger reuse of company IP through Knowledge Management, all roads tend to lead to two specific measurement objectives which create accountability for performance.

1. Measurement of individual Knowledge Management contributions
2. Measurement of the value that Knowledge Management is bringing to the organization

“What gets measured is what gets done” they often say. To this end, (1) offers a way of measuring if people are behaving the way the organization wants them to behave, specifically that they upload project assets to the repository and that they reuse the assets of others. On the other hand, (2) involves evaluating the degree of collaboration and how much productivity that may be generating, specifically to identify what kinds of IP related activity are leading to downstream value and which are not.

Individual Measures of Knowledge Management Performance

Some KPIs for knowledge management are simple and some are more complex. The simple metrics are very easy to fake by individual in order to inflate their performance scores so be very cautious about using them. These simple metrics measure the things you submit and the things users touch. And by extension, the documents users submit may each track the number of touches those documents receive and possible the reviews those documents receive, such as ratings, likes etc. The more complex measures require more data collection from users and customized configuration of your portal. The best way to track knowledge management most rigorously is to track the association between the producer and the consumer of IP. For example, if I do a project on liver function medication impressions by doctors in the UK and I use documents from a similar study of doctors in France, I would cite that research in my deliverables. Doing this allows us to distinguish between documents/projects that are viewed from documents/projects that really created downstream value. Advanced metrics can take advantage of that link allowing a very accurate view of who in the organization is producing the most influential research and who is most effective at using organizational IP. Note that these are both valuable types of people to have in your organization and they may or may not be the same people.

Simple Individual Performance Metrics

• # of Projects
• # of Document Submissions
• # of Views of Documents Submitted
• # of Document Views
• # of Document Reviews

Complex Individual Performance Metrics

• # of times work cited in other projects
• Value of projects in which work cited
• # of times cited other projects in own projects

Applications of Knowledge Management Analytics

Provided that you can effectively identify the origin and destination of IP being generated in the organization, you can create reporting and ultimately analytical insights on the path that IP takes. Given 2 years of project data and some basic master data on the analysts (like country they work, division, etc) you could effectively report on how much IP in a given period is generated from reuse, what areas is most reused IP being applied, what areas the IP is originating from (country, division, drug, etc). This measurement encourages the organization to have some practical conversations about which research is worth conducting. Research with significant reuse can be done more cheaply or can be benefit from “prior art” in study designs. Research that is new and groundbreaking may not involve reuse of other “prior art”, but it could significantly reused by other analysts in future research. On the other hand, research that is neither reusing prior IP nor likely to be reused by the analyst community in the future should be treated skeptically and should require better business case to justify.

For those of you that know Web Analytics, specifically Adobe Analytics pathing reports, this is the kind of reporting and analytical insight that can be created for the governance of Knowledge Management portals. For any given collection of filtered IP assets (documents), you can identify the properties of the IP that created it and the properties of the IP that it was used in. For Web Analytics, reports like this are done on specific web site URLs. You can report on the sites or site groups that preceded visiting a site and you can identify the site or site groups that they led to.

Specific Example: Let’s say I want to determine how market research KM differs from larger vs smaller European countries. So I look at the origin and destination of IP for Germany vs the Czech Republic. We might find that Germany shows a lot of reuse from studies in France, UK, and US markets with few small country research being reused. And we might show that German market research is being reused by UK and some smaller European countries in their market research, but France and US analysts are not using the German research. In contrast, the same report for the Czech Republic might show reuse of German and other Eastern European countries research materials, but that Czech studies were only reused by a single Hungarian study. Some countries may have better ROI on their research because of similarities with other markets. Some market present contradictions: The US market is for pharmaceuticals is large, but its system and regulatory environment differ substantially from that of other developed countries. The implication could be that market research is less reusable there. It is a hypothesis that could be easily tested if the Knowledge Management systems is setup to track origin and destination of IP.

Practical Consideration for KM Analytics

• Simple metrics for users, projects, and documents should be tracked
• Simple metrics need to be time stamp logged so they can be summarized by date range
• When submitting documents to the portal, users should be prompted to provide citations
• KM portal should stamp the user, project and document that was reused
• Citations should be optional, but if analysts are measured positively for reuse, they have an incentive to provide it
• Citing material that was not used should be considered unethical (to avoid false citations)

In my other post, I discuss Active versus Passive exposure to IP in KM portals.