Why ESG ratings are so inconsistent

Nearly 5,000 investors with over $120 trillion in assets have committed to integrating ESG information into investment decisions. Many of these investors rely on third-party ESG ratings to assess corporations' ESG performance. There are several good reasons reasons for seeking out these ratings:

• It is often more cost-effective, especially for small companies, to buy third-party datasets than to hire employees to collect and analyze proprietary datasets.
• Sustainability is a multi-dimensional concept, and some investors do not have the confidence or competence to evaluate all of these dimensions. ESG ratings are simple summaries that facilitate quick comparisons by those with little or no background in sustainability.
• Third-party scores are meant to provide an objective perspective, which should reduce the perception of greenwashing. This important because finance is a trust-based industry, and greenwashing may undermine public trust.

ESG ratings for both companies and mutual funds are also presented to retail investors, including by online distributors like Nordnet Bank AB and Avanza Bank , as well as by the Pensionsmyndigheten as part of the Premium Pension. Studies find that these ratings influence investor decisions - well rated mutual funds experience inflows after ratings are announced, whereas poorly rated funds experience outflows.

The problem: massive inconsistency among ESG raters

A few weeks ago, I joined Humle Fonder managers Petter L?fqvist and Elin Wiker for a company meeting where one of the executives mentioned their very good Sustainalytics rating as evidence of good governance and low sustainability risk. After the meeting, I looked up the company's Refinitiv rating. It was poor. If ESG ratings are supposed to provide objective and decision-ready sources of information on ESG characteristics, why do they produce such divergent views of the company we are trying to decide on?

Looking at the companies in Humle's small-cap portfolio, the correlation between Refinitive and Sustainalytics ratings is 0.21 (the correlation is actually negative because the raters score in opposite directions - a higher Refinitiv rating is better, whereas a lower Sustainalytics rating is better - but what is important here is the absolute value). This correlation indicates that companies with an above average Sustainalytics rating have a 43% chance of having a below average Refinitive rating. Random chance would be 50%.

This inconsistency is the norm with ESG ratings. A number of studies have examined ESG ratings from different providers, all finding weak correlations. One recent study of ratings for 924 companies from 6 providers found that correlations among providers ranged from 0.38 to 0.71, with an average of 0.54. This average correlation indicates that 32% of companies that are above average according to one ratings provider will be below average according to another. Another study, which included different providers, reported correlations from 0.03 to 0.64, with an average of 0.32. This average correlation indicates that 40% of companies with above average ratings from one provider will have below average ratings from another provider. A third study reported correlations from 0.12 to 0.75, with an average of 0.45. This average correlation indicates that 35% of companies with above average ratings from one provider will have below average ratings from another provider. For comparison, correlations for credit ratings are typically 0.9+ (86+% chance that a company with an above average credit rating from one provider will have an above average credit rating with another). The very consistent result is that ESG ratings are very inconsistent.

Why ESG ratings are inconsistent

There are three main reasons why ESG ratings vary among providers:

• Weight: Providers have different opinions about the relative importance of sustainability issues, weighing them differently both overall and among industries when calculating ratings.

• Scope: Providers create ratings based on different corporate attributes. For example, one provider may include lobbying activities in its rating while another does not.
• Measurement: Providers operationaize the same sustainability characteristics in different ways (ie. they use different indicators when measuring the same corporate attributes).

Each of these are problematic. Ratings differences due to weight and scope (these are somewhat related in the sense that excluding a topic from scope is equivalent to giving it zero weight) imply lack of consensus about what is important to measure. Ratings differences due to measurement are even worse because they reflect fundamental disagreements about the underlying data. Minor differences in the indicators used to operationalize important sustainability issues should not lead to widely diverging ratings. These measurement problems are highly problematic for investors that believe that corporate sustainability characteristics are quantifiable and who are expecting to make decisions based on objective information.

In this month's issue of Review of Finance, researchers at MIT's Sloan School of Management describe a new quantitative framework for diagnosing the relative contribution of weight, scope, and measurement to divergent ESG ratings. They report that 56% of the inconsistency is due to measurement, 38% due to scope, and 6% due to weighting.

Implications for retail investors and asset managers

These results are a big problem for retail investors. Distributors typically provide ESG ratings from just one provider, but the inconsistency of ratings among providers indicates that this is not sufficient for making investment decisions. Even if distributors provided multiple ratings, it would be unreasonable to expect the average retail investor to analyze this divergence and form an investment decision based on the result. How can retail investors trust the industry if they are being fed potentially unreliable information? Hopefully the new sustainability disclosures required by SFDR are sufficiently reliable and easily understood to provide an alternate basis for decision making. Nordnet Bank AB , for example, already provides SFDR classifications (Article 6, 8, or 9) in their fund list (although there is still substantial variation in the level of ambition among funds within each classification that requires detailed readings of fund policies).

For asset managers, the results indicate there is no easy solution to sustainability analysis and ESG integration. Because of the measurement issues, simply buying more datasets, which is hugely expensive, is not the solution. Inconsistencies are particularly pronounced for governance ratings (often with no or even inverse correlations among providers), which is problematic given that fund managers must evaluate good governance for all SFDR Article 8 and Article 9 funds. One option is for asset managers to recruit staff with strong competency in sustainability to guide decisions instead of relying on opaque, over-simplified, and inconsistent scores. Another option is to focus on obtaining raw sustainability data, where the measurements are well understood, and constructing proprietary scores relevant to and trusted by portfolio managers.

Looking forward, some of the data problems will be solved by the EU taxonomy, SFDR, and the CSRD, at least for companies and based in the EU and for EU-focused funds. Comparable regulations in the USA would be particularly valuable given that American holdings figure prominently in global funds, for example in the tech and healthcare sectors.

References

Berg F, K?lbel JF, Rigobon R (2022) Aggregate confusion: The divergence of ESG ratings. Review of Finance 26: 1315-1344.

Brandon RG, Krueger P, Schmidt PT (2021) ESG rating disagreement and stock returns. Financial Analysts Journal 77:104-127.

Capizzi V, Gioia E, Giudici G, Tenca F (2021) The divergence of ESG ratings: An analysis of Italian listed companies. Journal of Financial Management, Markets and Institutions 9: art2150006.

Chatterji AK, et al. (2016) Do ratings firms converge? Implications for managers, investors and strategy researchers. Strategic Management Journal 37: 1597-1614.

Christensen DM, Serafeim G, Sikochi A (2022) Why is corporate virtue in the eye of the beholder? The case of ESG ratings. The Accounting Review 97: 147-175.

Hartzmark SM, Sussman AB (2019) Do investors value sustainability? A natural experiment examining ranking and fund flows. Journal of Finance 74: 2789-2837.

Rotblut C, Swedroe L, Sus A (2022) Addressing the challenges ESG investors face. American Association of Individual Investors (AAII) Journal 44: 7-10.