Who will watch the forecaster?

In 2015, before I decided to ride the NewSpace wave, I had a short-lived project in the form of an IoT (Internet-of-Things) startup. The overall idea was to design a hardware thingy capable of interfacing to some sensors, and use it as a bit of a sensing platform, which would eventually be hooked to the cloud. Nothing very different to what Particle or Tessel did and (I think) still do today, but of course with shallow pockets. The design was quite rudimentary, and simple. Here’s an actual picture of the board (PCB assembly quality was questionable, I know, but that’s what the budget allowed, oh and by the way, the domain has expired so some other company is using the name now, it wasn't so pretty):

Back in 2013-2015, Internet-of-Things was *the* thing. There was this massive excitement about everything becoming connected, including very ordinary objects such as fridges, toilets, water boilers, TVs, toothbrushes; it would all be hooked to the cloud and our lives would change for the very best, since data would make our existence much more efficient. I must admit, I bought some of the hype. It’s not that I was hyped about having my toilet connected to a server (I am particularly protective of my toilet digital footprint), but I was somewhat optimistic about some hypothetical industrial use cases of IoT, where connected sensors could be used for tracking assets, for optimizing supply chains or adapting manufacturing in real time. So, you could say I was a bit of a (cringe alert) believer that there was some technology shift of sorts approaching. Which, well, never materialized. Or, it did not materialize the way it was forecast.?And here lies the key of this article: forecasts, and the role they play in hype cycles.

Note that I am not dodging my naivety in all this. But, when you have data and numbers in front of you, you have two choices: a. question them and go down a rabbit hole to understand who came up with them and how they were generated, b. trust them and start to make some conclusions on top of it. Or a bit of a limbo between both. I was closer to b.

Perhaps the best approach in general is the one that Ronald Reagan used to recommend, which in fact is (ironically) a Russian proverb: “Trust, but verify”

Hype cycles happen, and the IoT is an interesting one for me because I experienced it from within. But it seems I have a thing for hype cycles, because the NewSpace industry has been going through something similar. Still, I will use the IoT case as a case study which applies to many other industries which undergo these type of waves. Since I remember that 2020 and 2021 were very typically chosen years for projections back in those days, I find it very interesting I can now observe how the whole thing turned out.

When I was preparing the pitch to impress investors for my ill-fated company, I wanted them to see the massive numbers forecasts were showing. Like, come on, there will be tens of billions of devices connected by 2020! (remember this was 2015). The total addressable market (TAM) is humongous, you gotta be crazy *not* to invest in this!

So, what were the forecasts saying back in those days? Let’s see.

The CEO of Ericsson was hyping investors with these figures in 2010:

A report from Cisco from April 2011, showed the following, pretty aligned with Ericsson’s forecast:

(Cisco did not even get right the world population projection, which is now ~7.8 billion, so, 200 million people off, roughly the whole population of Brazil, but ok let’s give it to them that population projection is not their area of knowledge…).?

And they also added a some weird visuals in the report:

I am not saying cows can’t wear sensors (in fact they do), but that TV antenna in the head was a bit too much.

Cisco even launched a counter of IoT connections. See what they wrote:

“By the way, what are all of these “things”??Mobile devices, parking meters, thermostats, cardiac monitors, tires, roads, cars, supermarket shelves, and yes, even cattle.?The list is endless, and it just keeps getting longer and more interesting.?Literally, by the second.

Even more exciting is when all of these things are combined with people, process and data via the network to deliver transformational value to the world by improving the way we make decisions, saving us time and money, and so much more.?That’s the Internet of Everything, and its value increases every time we connect the unconnected.

So we’re paying close attention.?The connections counter will help us keep track of exactly where we are in this journey, starting now and continuing through 2020.

We encourage you to keep track as well.?Cisco invites journalists, analysts and other interested parties to check out the IoE Connections Counter and to feature it in your own content.

Let the countdown to 2020 and 50 billion connections begin!”

Well, here I am Karen, keeping track as you suggested, and it does not seem the 50 billion connections are happening. By the way, the connection counter is down, bummer:?

Now, look at this one from IBM from 2012, it’s...uh...interesting: 1 trillion connected devices by 2015. It makes you think of the sanity of whoever was behind these slides.

In this slide, the trillion remark is not even the worst thing. Look at the projected figure of 450 billion business internet transactions over the internet per day in 2020. Let’s unpack this one for a moment: this means that every day, every single soul on the planet would, every hour, perform almost 4 business transactions (because around 5 billion people in the world have an internet connection). But ok let’s assume they meant something broader with business transactions over the internet and not just a purchase made by a human, and they included all trading (including algorithmic trading) and any other “robotic” business transaction, and let’s also assume they had to somehow justify the trillion nonsense in some way.?Hyped nonsense on top of nonsense.

Then, Ericsson started to tone it down a bit in 2015, because well, reality was perhaps hinting a thing or two, so they settled to 28 billion connected devices by 2021, which is almost half of what they had said 5 years before, now predicting ~15.3 billion IoT objects connected:

Gartner did their bit, of course, projecting (in 2015) 20 billion IoT units by 2020:

Also Juniper Research (from 2016) says that because they had "new data" (?), they forecast 46 billion devices by 2021, in a report which in 2016 would have cost 4000 pounds:

And last by definitely not least, McKinsey had to say something, predicting ~30 billion IoT objects:

NOTE: Please have in mind I have purposely avoided talking about market sizes nor CAGRs or similar, because the numbers are more obscure and bullshit-esque, if that’s possible. But yes back in 2015 I would have used those numbers extensively with my investors, oops.

So, let's recap. We got 1 trillion (!), 50 billions, 46 billion, 30 billion, 20 billion, 15 billion. Some nice variance. As forecasts get closer in time, there is a fair decrease in the projected values. For example, in 2018, Gartner was already observing things somewhat differently and calibrating:?

Now we stop the forensics a bit and we fast-forward to our very real July 2021: how many IoT devices are actually connected to the Internet, happily sending bits back and forth?

Well, there are not very solid figures. You can so easily find projected forecasts but not so many actual numbers. So, hard to say. But let’s use this as the best we can find: we seem to have between 7 and 8 billion IoT “thingies” or devices connected.

What?

***

I can’t help reflecting a bit and see my 2015 version and my 2021 version having a conversation here. Besides the extra kilograms and gray hairs in the beard I have gained during these years, it feels I have grown better skepticism skills I didn't seem to have those years back, or at least I have strengthened them. I feel a bit ashamed to have accepted such shitty forecasts, but again, that is because my 2021 version now *knows* they are horse crap. So, yes, there is definitely plenty of hindsight at play in this article.?

Industry forecasting is, and perhaps a bit ironically, an industry itself. I have to hand it to the forecasters: they put their names and their reputations at stake to provide numbers which they can only guess. Would you sign a prediction about the chances it will rain in two months? Probably not. That being said, you can also see that there are forecasts which tend to serve a purpose. Such as, hyping. Or, put in a different way, forecasts meant to say what an audience is expecting to hear. An investor is most likely eager to see that a market is ballooning, not to see a market that is boringly growing a few points per year. But, manufacturing a forecast to hype someone comes with clear risk as well: reality eventually kicks in. It may take time, but its pace is ruthless. Reality always ends up here's Johnny-ing the door.

A century ago, as medicine was slowly becoming scientific (after many centuries of being more or less witchcraft) a Boston doctor named Ernest Amory Codman had an idea: scorekeeping medical practice. He called it the End Result System. Hospitals should record what ailments incoming patients had, how they were treated, and—most important—the end result of each case. These records should be compiled and statistics released so consumers could choose hospitals on the basis of good evidence. Hospitals would respond to consumer pressure by hiring and promoting doctors on the same basis. Of course, everyone hated it. Doctors and hospitals seeing their reputations being affected by their performance? Outrageous. But today, hospitals do much of what Codman demanded, and more, and physicians would find it flabbergasting if anyone suggested they stop.

Industry forecasting needs to cut the crap.

But can it? There is this research (done by Philip E. Tetlock and Dan Gardner) which (in)famously showed that the average forecaster performs as well as a dart-throwing chimpanzee. Is it impossible to forecast accurately at all? Are we just tossing coins? The key here is the word average. Forecasts can be accurate, if done properly, which requires lots of researching, analysis, a good dose of doubt, and more importantly, constant adjustment.

It's about time a forecasting score-keeping of sorts is established. This would have a double benefit: we would get to know how good or bad a forecaster is (for example, by means of knowing their Brier score), but also forecasters would stop enjoying a life of open-loop conjecturing and think twice before pumping up their numbers. Then, for a forecaster, keeping a good score could mean the difference between more business or having to switch to a less rigorous industry such as astrology. As Codman's idea, perhaps this will be hated by many, but eventually would, in time, become the norm.

***

Oh, yeah, and since we as society are not ready just yet for the idea of having a microchip in our toilets and our toasters, and because sometimes IoT devices act up (because they run software and software has bugs), there is a great recollection of IoT things malfunctioning, with amusing consequences.