Why rating your doctor is bad

My conclusion [1] is that rating healthcare professionals as it occurs right now, does not yield any value, or poor value at best.

Why?

1. Ratings do not have a clear meaning, and as such do not hold any authority;
2. ratings are one-dimensional, whereas data in order to justify beliefs should cover multiple dimensions;
3. ratings are a measure, and they will become a target which makes them inadequate to be a measure;
4. ratings fuel extrinsic motivation and reduce intrinsic motivation;
5. moreover, ratings serve advertising purposes which may be considered unethical and may even be found illegitimate – especially when it would concern comparative advertising based on flawed data.

Feedback in whatever form is key to learn. Without feedback you won’t learn anything. And the sole purpose of giving and receiving feedback should be to learn, not top-down or bottom-up but horizontally in both directions. People should be both teacher and learner, and should engage in a virtuous and continuous teaching cycle (Tichy, 2002).

Patients should learn from their doctors on how they can lead healthy lives, what they should do to recover from a disease or how they can maximize the quality of the rest of their lives. And healthcare professionals should learn from their patients how their diagnosis turned out to be, how patients experience their treatment and whether they put trust in their physician.

In ideal circumstances, feedback occurs one-to-one and in two directions. In a lot of cases, many-to-one feedback may prove convenient as well – e.g., in software development, testing of pharmaceuticals, etc. Important, both patients and healthcare professionals should be able to learn something from it. Only then will feedback, reviews or ratings have value.

However, feedback often occurs many-to-many. This is the case when feedback you give, is made publicly available – e.g., to influence others. But as I will argue, in many-to-many feedback settings the data will be unreliable, and will lose all of its value leaving patients and healthcare professionals in a bubble and self-fulfilling prophecies.

[1] This contribution should be regarded as a hypothesis. If there would be scientific or evidence-based research on this topic validating or on the contrary invalidating the arguments I put forward, I would be happy to learn about them.

Context

There appears to be a tendency to rate healthcare professionals. And it’s a bad idea, at least how it is approached right now.

What if all of a sudden LinkedIn would introduce ratings? What if anyone whatever his background or relation with you could rate you and make it publicly available? Would you enable ‘receiving recommendations’ in your settings, if you would not have the possibility to modify or discard the recommendation?

How would you feel if a heart surgeon would rate you as a lawyer? What if the surgeon would not recommend working with you? What would your reaction be? My take on this – and I don’t know you –, is that you will question what her feedback is based upon, how on earth she could say anything of value as she lacks any legal knowledge, and so on. Or the other way around, what if you a doctor is rated great, because he just gives his patients what they ask for?

Yet, we are perfectly fine with seeking feedback, reviews and ratings on certain products and services. In the travel and restaurant industry this is a well-established modus operandi. And before you buy a book, electronics or toys in an online shop, people will first check the ratings, and possibly also the reviews – although the latter is a bridge further as it requires more energy, time and focus.

Often, reading feedback from other buyers may be very helpful, especially when you can’t hold the product. And for services it’s even more helpful, as you can only determine its quality after the service has been delivered. Reviews and ratings are very a convenient means to prevent or at least narrow down the risk of a bad purchase.

Measuring is knowing. But measuring people or the performance of people is very dangerous and ambiguous, let alone make those data publicly available and create the perception that those data or aggregation of data hold the truth.

Why this is important right now

 Verba volant, scripta manent. Words fly away, what is written remains. Despite European GDPR and other privacy measures, what you put on the internet sticks, and will be used and analysed. Accordingly, in a world where data will become increasingly important, it will be key to preserve the quality of that data. If you do not, you will not get accurate answers in the best scenario, or a perverse outcome or self-fulfilling prophecies in the worst scenario.

We have entered the data era a while ago. This is not new. So why should we care right now? Simple. Because the amount of data and the amount of data-generating sources will keep on increasing. It will be harder to fix the data and its consequences attached to the data tomorrow, harder than it is today.

The question we have to ask ourselves, is whether we are willing to take the risk of acting blindly and run our healthcare based on unreliable, non-justifiable data.

And why do we need ratings anyway?

Well, data is information, lots and lots of information. Information leads to knowledge. Knowledge is power. And therefore, knowledge is money, big money.

Yet, data does not equal knowledge. Information does not equal knowledge. Knowledge is traditionally defined as “justified true belief” (Hislop, 2013). According to Nonaka and Takeuchi (1995) knowledge should be regarded as “a dynamic human process of justifying personal belief toward the ‘truth’”.

Information on the other hand is “commodity capable of yielding knowledge” (Dretske, 1981). Information or data can be viewed from either a syntactic perspective (i.e. volume-based) or semantic perspective (i.e. meaning-based) (Nonaka & Takeuchi, 1995). As we will highlight below, the meaning attached to data should be the prevailing perspective. Kian Bakhtiari wrote a great article on this data trap.

Data are not per se capable of yielding knowledge. The essential question is to what extent we can learn something from data. Data should enable us to justify our beliefs – be if through deduction or induction. My statement is that ratings often, if not always, lack the very meaning in order to justify our beliefs, and thus to create knowledge. As such, it is difficult to attach any value to ratings.

Why ratings are bad

Ratings don’t hold any authority

If you would trust a review on an oncologist’s performance, which would you trust more: the review given by a patient or the review given by one of his peers? Probably you would trust the review of a fellow oncologist more.

Yet, besides the fact that the fellow oncologist seems better equipped to give a review, based on his common background, this assumption may be skewed. The fellow oncologist may give a poor rating, because they are competitors, or they may know each other well and they hold a grudge toward one another. Or exactly the opposite may occr, the fellow oncologist writes a good review, because they are friends, or perhaps because he wants a good review as well.

Ratings don’t have authority, because they are hardly justifiable as you simply cannot know where they come from. Ratings are not seldom based on an intuition or feeling. And as Daniel Kahneman and Amos Tversky (Kahneman, 2011) have taught us, a hunch is often ill-founded.

Yet, feelings are valuable. But relying on feelings as the sole dimension of data, is very tricky. Data should be multidimensional.

Ratings are one-dimensional

When you would rate your doctor on a scale from one to five, what specific question would you ask yourself? And are you confident that your readers will ask the same question, and truly understand all of the nuances of your feedback? Measuring people is very complex, and complex things never have a simple or single answer.

Would you consider Dr. House a good or a bad doctor? He’s very unfriendly and eager to solve the medical conundrum often at the cost of the patient. Yet, he mostly succeeds in curing his patients.

In my opinion, you cannot consider Dr. House a good nor a bad doctor. There are multiple dimensions to consider, and you will weigh the dimensions differently compared to how I would weigh them. In other words, we have other definitions on what a good doctor would look like.

Edward De Bono’s theory of lateral thinking can be applied here (De Bono, 1985). When people are asked what to think of, let’s say, a house, you cannot describe the house and argue over it if each person is on another side of the house. You would see something different and find yourselves at cross-purposes. As a result, no one will truly understand what the other person’s talking about. You have to talk from the same perspective.

The same applies to ratings. Ratings will only be valuable if you’re approaching them from the same dimension. The difficulty is that, in regard to ratings, we cannot know these dimensions upfront as people don’t know each other and therefore cannot assess the underlying parameters of the rating. Therefore, in my opinion ratings are only valuable if they hold multiple dimensions where people can weigh the different dimensions as they see fit.

And a feeling may be one of them. But so are positive qualities and negative qualities. And the cold hard facts may be the most important in healthcare – i.e. how good was the diagnosis. But even then.

Ratings will become a target

Relying on the hard, objective data (such as the average length of recovery of a healthcare professional’s patients) is far from beatifical. How would you rate the diagnosis or treatment of a patient who is terminal? Once they see it as a target, it is rather likely that healthcare professionals will act upon the rating and turn it into their favour.

Say you’re a lawyer. And how you’re measured, how good you perform, how good a lawyer you are, will mainly depend on the success rate of your cases. What cases will you take? Would you take the cases out of sympathy, but which you most likely are going to lose? If you’re concerned with your status – and most lawyers are – you will take only the cases, which you think you’d win, which would benefit you, and not necessarily your client. And so successful lawyers will take the successful cases, and in turn the successful cases will come to the successful lawyers. This will create a virtuous cycle for the successful lawyers (or better who are perceived successful), and a vicious cycle for the unsuccessful lawyers.

And healthcare professionals will do the same. And I’m not saying they won’t abide by Hippocrates’ oath. But they automatically will attract the successful cases as they are already successful, and will not have any time left for the unsuccessful cases or would have long waiting lists.

This is an application of Charles Goodhart’s law, stating that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” Or put in more understandable words by Marilyn Strathern: “whenever a measure becomes a target, it ceases to be a good measure.”

To prevent a measure from becoming a target, it is key to not attach any consequence, good or bad, to the measure. As soon as you would attach consequences, the measure will become a target.

Of course, easier said than done. During our entire life, since we were a toddler we received rewards and faced punishments in order to steer us toward the desired behaviour. And it’s no different in corporate structures. The ones getting to the top are not always the best. They are the best in playing along, in conforming their behaviour to certain standards. Then again, this may make them the most suited to thrive in such corporate environments.

Nonetheless, what we know from scientific research is that rewarding and punishing affect the level of intrinsic motivation.

Ratings may reduce intrinsic motivation

In general, there are three kinds of motivation, three reasons why people do what they do.

The first is biological: we eat, drink, have sex because we are biologically driven by it. The second is extrinsic: we do something because we are rewarded or punished for doing it. The third is intrinsic: we take pleasure solely from performing the task itself or we are driven by inner beliefs or perceptions.

Since the experiments of Pavlov and B.F. Skinner extrinsic motivators have widely been accepted as adequate drivers to steer us toward desired behaviour. However, research over the last couple of decades have shown that extrinsic motivators may lead to adverse effects, and more importantly it may harm our intrinsic motivation.

A study I would like to quote took place at several day-care centres for young children in Haifa, Israel (Gneezy & Rustichini, 2000, volume XXIX). In order to reduce the number of parents picking up their children past closing time, they introduced a monetary fine. You’d expect that the number of parents coming in too late was reduced. Yet, the opposite happened. The number of parents coming in too late actually doubled compared to when no fine at all was applied! And when the fine was removed, the number of parents coming in late remained at a higher level than before the fine was introduced.

What happened is that the perceptions and beliefs of the parents changed. Before the introduction of the fine, they had a moral obligation toward the teachers, and no clear view on what the consequences would be of coming in late. Afterwards, the parents perceived the fine as a price for either their time or the service of taking care of their children past closing time provided by the teachers.

In analogy with this study, do we really want our healthcare professionals to be more occupied with how many likes they are getting, how well they are rated, how much status they have acquired, etc. than with the health and lives of their patients?

Ratings serve advertising purposes

As I have argued, rating your doctor does not make patient care better. It does not improve the life and work of healthcare professionals. And it sure does not have a positive impact on healthcare on a macrolevel.

When we look at the purpose of rating healthcare professionals nowadays, it is to distinguish oneself from a fellow healthcare professional. In other words, ratings are just a form of advertising. You may be pro or con advertising for healthcare professionals, but you can’t deny that it’s a form of advertising.

In my opinion, advertising for healthcare professionals could be allowed insofar it concerns educating people and making them aware of certain risks, unhealthy lifestyles, etc.

But the ratings as they occur right now don’t have the objective of making healthcare better, for patients, healthcare professionals, government or society. Serving only advertising purposes, as a cheap form of commerce, can hardly be considered ethical and may be found illegitimate – especially when it would concern comparative advertising based on flawed data.

Works cited and further reading

De Bono, E. (1985). Six thinking hats. New York: Little, Brown and Company.

Dretske, F. (1981). Knowledge and the flow of information. Cambridge: MIT Press.

Gneezy, U., & Rustichini, A. (2000, volume XXIX). A fine is a price. Journal of Legal Studies, 1-17.

Hislop, D. (2013). Knowledge managent in organizations. A critical introduction. Oxford: Oxford University Press.

Kahneman, D. (2011). Thinking. Fast and slow. New York: Farrar, Strauss & Giroux.

Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating companies. How Japanese companies create the dynamics of innovation. New York: Oxford University Press.

Tichy, N. (2002). The cycle of leadership. How great leaders teach their companies to win. New York: HarperCollins Publishers.