Founder’s Choice VC Firm Ranking

There have been many different approaches to ranking VC Firms over the years I've been an entrepreneur and a VC. Each approach I've seen has issues. Most are easily gamed or have statistical bias issues.

I got the following note from Roy Bahat at Bloomberg Beta a while ago about a new approach called Founder's Choice.

We and a few other firms sponsored a "founders choice" version of the Midas List, with a legit (IMHO) rating methodology, built by two Penn students. No vitriol possible (unlike The Funded, etc.). We've wanted this to exist for a long time -- NPS of us as a firm is too forgiving a metric, everyone scores well.

My first question was:

How are they dealing with sampling bias on this one? For example, we send to all our founders and say "please fill this out and give us high scores." Mostly just curious on methodology.

Roy had a thoughtful answer that made me a believer after a few more questions.

You are literally the only one (and I'm relieved someone did) to ask on sampling bias. For context, the general way it works is founders auth with LinkedIn and then the product tosses away their identity (or, more accurately, only keeps a hash and disconnects it from their ratings). Then the founders get asked for pairwise comparisons of only the VC firms who have backed them (so this is about who founders like as investors, not who has sour grapes from a pitch). How this addresses, to a degree, sampling bias:

1. Dampens outliers: because it only asks for pairwise comparisons between firms (like an ELO rating in chess, if you're familiar), one very un/happy respondent can only affect so much, and same for a sample. (As opposed to giving one firm a 10 and everyone else 2's or something.)

2. At the same time, it forces comparisons. A firm can ask founders to rate them highly, but ultimately founders have to choose who gave them more value. Can't rate everyone a 10.

3. This is why we're looking for as broad participation as possible, because the sampling bias will actually probably most show up in which firms even have enough ratings to count. (Like ELO in chess, more ratings doesn't necessarily help you -- you get more "points" if a founder rates you as better than a highly-rated firm. More ratings can just as easily hurt as help.)

If you are a founder, go spend five minutes and anonymously rank your VCs on Founder's Choice.