The problem with sustainability scores

Three weeks ago, Lubomila Jordanova (the Co-Founder of Greentech Alliance) posted an image of an Allbirds shoe with carbon footprint data on the label. Ms. Jordanova then states the simple opinion, “This is what every product should have on its product's label.”?

I love this post. More specifically, I love the comments on this post. I love them so much, I scraped them all and analyzed the results in R. Why? Because I’ve been trying to figure out a way to illustrate some of the challenges with product sustainability scores. Then this bundle of joy fell in my lap.?

I want to be clear up front. I love that Allbirds is doing this. Not because customers will understand the footprint numbers (they won’t), or because their methodology is perfect (none are), but because they’re starting somewhere with the full knowledge of their imperfections and aware that they’ll be mercilessly critiqued by the sustainability community. You only do that if you care. Well done Allbirds! I’m sure this is the start of the journey and not the end.?

All that being said, I do want to talk about product sustainability scores. This is such a common approach to building a sustainable shopping experience that it needs to be addressed.?

But before going into my own thoughts on the matter, let’s try to understand what Lubomila’s audience thinks - a non-representative sample, surely, but I don’t think it’s a crazy proxy for the sustainability community.?

You can see the full details of my analysis (including the raw data and analysis script!) on GitHub , but in short, I ended up scraping all 1,248 comments available at the time and then manually categorized the first 100 in order to get an idea of the sentiment of “highly relevant” comments. LinkedIn ranks comments by default according to “Relevancy” (some black box algorithm), so while this isn’t a random sample, it’s probably a good approximation for what the average reader would take away from this post’s comment stream.?

I categorized each comment as positive, negative, or neutral, and then further tagged comments on their content. Let’s start with the overall sentiment: 55% of sampled comments expressed a negative sentiment about the footprint information. Only 19% were positive, and the remaining 26% were neutral.?

I find this fascinating. The sustainability community, at least from my point of view, really wants “sustainability scores” on products. I get pitched this idea constantly. Yet here is the simplest of possible scores, the carbon footprint, and a majority are unhappy about it. It includes only a single attribute, and the score is already opaque. The average shopper will have no idea how it was calculated or what it really means. So, let’s assume they do try to dig deeper and see if it’s a good idea. What’s the average shopper supposed to think if they peek behind the curtain and find over half the folks in the community griping about the score? They’ll hear it’s great from some, others will say you need to be careful and it doesn’t include everything, and some will just hurl invective or reflexively shout “greenwashing!” Is there any hope that customer will leave trusting the score? Will they tell their friends to trust the score? I doubt it.

Ok, aside from the overall sentiment, what did the content of comments look like? Here’s a high-level breakdown. I’ve included an example comment for each (note that many comments were given more than one tag, so the total percentages add up to more than 100).

(Here’s the raw coding file where you can see the comments and how I tagged them.)

Again, I want to stress that this is the simplest of sustainability scores. Does the problem go away if we include more than carbon? Perhaps the score will get less contentious as we add more components. I’m very confident the answer is a resounding no.

Carbon is a wicked problem, but at least we all agree on the units and there isn’t a geographic component to it. Do we all agree on the units for social responsibility? What about animal cruelty? Water usage? What about non-carbon environmental issues like biodiversity loss, waste, toxics, and so much more? In truth, carbon is easy to measure compared to these attributes. Sustainability scores that claim to combine carbon with wildly different values like those listed above are doomed to magnify disagreements, not reduce them. It also makes the job of calculating such scores exponentially harder. Not only do you need entirely new data sets, you generally need the social and geographic context of that data as well.

Even more fundamentally, the heart of the issue is that there is no scientifically objective way to combine different elements of sustainability. It’s not just difficult. It’s impossible. Combining elements of sustainability requires inherently subjective evaluations between values. I think this should be an uncontentious statement, but in practice I’ve found it isn’t. I’m tempted to go into great depths to make this point. My original draft did, but it gets technical, and frankly boring. I’m not a good enough writer to make the material interesting to all but the wonkiest of readers out there, so I’ve put that material in an appendix (your welcome to both readers and wonks).?

So what’s the alternative? If sustainability scores are inherently subjective and doomed to endless debate, how do we help people make informed choices? This is a topic I have much more to write about, but I’ll start by presenting an analogy. Think about sustainability like health. Both have a single important number that doesn’t paint a complete picture (calories and carbon). Both are umbrella terms that include a lot of important topics. Both are contentious. Ask ten people to define a healthy product and you’ll probably get twelve answers and three non-responses. Ask a vegan and someone on the paleo diet and you’ll get nearly opposite answers! And here’s the key, we don’t tell those people they’re wrong. There’s a lot of uncertainty in what makes a healthy food. Genetically, peoples’ bodies are different and need different things. Lifestyle matters: the body builder needs a different diet than the marathon runner.?

In the world of health, we don’t decide up front what makes a food healthy, then only help customers find products that match our definition. We definitely don’t create a numeric “product health score” and tell everyone it’s scientific, objective, and indisputable. Instead, we ask a customer what health means to them and work backwards from that. For ideas on how to help customers shop more sustainably, look for inspiration in the world of health. At least there we do people the honor of respecting their opinions and preferences. Sustainability should learn from that.

I’ll have more to say on the topic in a future article. Until then, thanks for reading! If you liked this, check out my first article in this series, “Virtue signaling with my sustainable underwear: the intention-behavior gap and why it matters.” Also, follow me to stay up to date with future articles.

P.S. This article was definitely not a “proof-of-work” style attempt to chat with Lubomila! But, if you know her and can provide an introduction, I’d much appreciate it!

Appendix A: More than you wanted to know about the futility of a scientific, objective sustainability score

For years, quite literally, it drove me nuts because I couldn’t precisely define what a sustainable product was. I could point to examples of more sustainable products, but when pressed for a definition, I came up empty. And people always wanted a definition. Something scientific and objective. Something based purely on reason that could withstand all would-be attackers because god forbid we ever be accused of greenwashing (a term I have a love-hate relationship with, though that’s a topic for another post).

I tried to rely on definitions of sustainability. The most common is from the Brundtland Report , “sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs.” So a sustainable product would be one that meets the needs of the present without compromising the ability of future generations to meet their own needs. Simple, right?

But wait, what are “needs”? Are party balloons ever needed? And what are the needs of the future? Are they the same as mine, or higher, or maybe even lower if you’re a fan of the degrowth movement ? If the majority of future Earthings are actually Spacelings living in orbit can we do as we please today? I get it, that’s crazy talk (sorry, I used to be a wildly optimistic aerospace engineer), but how about something less crazy. If one assumes technology progresses at 3% vs 2% per year does that change what makes a sustainable product today? It should because faster technological progress will allow future generations to meet their needs using fewer resources, so we can consume more today, right? Let’s not even get started (yet) on intertemporal discount rates.

Don’t get me wrong - the Brundtland definition is great. It’s high level, it’s aspirational, it points in the right direction. Like a platonic ideal, the Brundtland definition is beautiful, intuitive, and yet at the same time absolutely useless as the basis for a definition of a sustainable product. It simply can’t be used to separate products into “sustainable” and “convention” (pro tip, “conventional” is a great, non-judgemental way of saying “not sustainable” to people who don’t like their products being called that).?

What about other definitions? Here’s what Wikipedia says, “Sustainable products are those products that provide environmental, social and economic benefits while protecting public health and environment over their whole life cycle, from the extraction of raw materials until the final disposal.” I’m quoting Wikipedia not because I like getting yelled at for citing Wikipedia but because so many people do in this case! Here’s the World Economic Forum doing it. Wikipedia doesn’t actually cite where this definition comes from, and it’s there all the way back to June 2012 when the page was first created .?

This definition seems to avoid the need for a crystal ball to predict the future, so does this solve our problem? Sadly, no. How are we to measure; and, more importantly trade-off, environmental, social, and economic benefits? And why the hell did economic benefits suddenly enter the picture? What does that even mean? Let’s ignore that one for now and further assume we could actually measure the first two. The real scientific deal-breaker comes in assuming we can objectively trade them off against each other.?

This is actually where intertemporal issues sneak back in. Assume you wanted to objectively combine environmental and social impacts. You’d need to translate one into the other (or both to a common unit). Let’s first try by transforming environmental benefits to social ones because it’s less contentious sounding (e.g. “the environment matters because it impacts people” is an easier argument to make than “people matter because they impact the environment” even though from the point of view of the underlying math it doesn’t matter which direction we go). Human suffering exists today in innumerable forms while environmental degradation causes human suffering largely in the future. How should we make trade-offs between known suffering today and inherently uncertain suffering in the future? Bring back that crystal ball and then endlessly debate the right discount rate! (Here’s an approachable introduction to the topic of discount rates, complete with cute pictures of otters!).

And if all that isn’t confusing enough, let’s complicate it further by adding in animal suffering! This is more a semantic argument but animal welfare typically is included in the sustainability umbrella. Let’s also add biodiversity loss for good measure, and insect welfare; because, why not, we’ve gone this far (yes, some people care about this ).?

It quickly becomes apparent that there is no value-neutral (i.e. objective) way to combine these very different aspects of sustainability into a single number. But this doesn’t mean people don’t try. Most sustainable shopping concepts I see attempt to create a “Sustainability Score” because they think that’s the best way to help customers shop for such products. Every single one of these scoring systems makes subjective trade-offs that try very hard to look like science. Just because something uses math to produce a number doesn’t make it science, and it certainly doesn’t make it objective.?

A common attempt at working around this problem is to create a formula with weights and then let shoppers personalize the score by implicitly or explicitly setting the weights. I’ll admit this is an improvement, but it’s not nearly good enough. For starters, the formula is usually some type of linear combination of terms: think X1*environment+X2*humans+X3*... where you get to choose the X’s in some way. But if we’re really talking about being scientific and objective then there needs to be a reason to believe that sustainability is a linear weighting of relevant terms. Why not nonlinear components? Why not interaction terms? Fine, let’s ignore that, an even bigger problem with such schemes is that they are generally limited to the data they have available. If the person calculating the score doesn’t have good data on animal testing, guess what gets left out of the equation? Same for living wage, or biodiversity loss, or ocean-bound plastic waste, or any other sustainable value with poor data coverage. If your shopping experience is based on a sustainability score, you’re straight-jacketed by the lowest common denominator among your data inputs. Letting shoppers pick their own weights gives them some wiggle room in that jacket.

Another approach is to keep the science and throw out the objectivity. The argument goes, we’ll use scientific methods and models of human behavior to estimate how the average person trades off these different sustainable values. So we’re using science to estimate average subjective values. Great, that’s a fun exercise and even has a lot of value outside the context of a shopping experience, but why should any individual person be subjected to the values of a mythical average person? This approach adds more science, and more math that looks like science, but it doesn’t result in a sustainability score that is objective - it can still be endlessly debated. Science hasn’t settled the matter, it just tells us (given the assumed model of human behavior) what the average person’s response looks like.?

To be clear, none of this is intentional trickery. It’s unavoidable. The right formula isn’t waiting to be discovered. There’s simply no way to combine wildly disparate concepts like protecting the environment and ensuring human welfare without relying on subjective evaluations.