What's Up With These Forecasts?

January’s Hiring Boom Caught Economists by Surprise. Why Forecasts Often Miss the Mark - WSJ

The jobs report on 2/3/23 was vastly different from what economists had forecast. The attached article comments on reasons for this disparity. One of the most important aspects is somewhat buried. Putting a forecast out that is vastly different from others putting out forecasts is a risky proposition. If you are right, there is some glory but if you are wrong, you are a laughing stock. If you put a forecast out with a similar result as others then you are either all right or all wrong. You don't stand out in the crowd. And that's ok to most. So there is modification to model based results to be similar to others. This is a deep dark secret of many forecasts. And it needs to stop in my mind. Why have experts develop models if they are just trying to figure out what others are going to say.

?

When I left academia, I got the pleasure of working

in polling, well for part of my job, for Harris. My very first election was

1999. It was only a few governor's races that year so not a lot of pressure. We

got to election day, really the day before, and we were preparing to release

our final estimates. Two weeks before we released a poll that had the

Republican up substantially over the Democrat in the Mississippi race. This is

what one might expect, even in 1999. Other pollster doing the race had a

similarly large lead for the Republican. For the final poll, I produced results

that called it a dead heat. In fact the Dem was up in our data by 0.2 points. I

had checked everything and nothing looked wrong. Not the partisan split, not

the underlying demos, nothing. But I agreed not to release a final estimate due

to the fact we were so different from the other polls in a state where it just

made sense that the Republican would win. Well guess what happened... The Dem

won by 0.2 percentage points. I had the exact result in my hands and we chose

not to go with it. To be sure I went along with the decision but it was a

learning moment for me. As long as I was certain in my data, I haven't been

afraid to be the outlier. In almost all cases, not all though, I have been

right. (2000 NY Senate race and most famously the Scottish Referendum where I

was the only pollster to call the result correctly and to the percentage point.)

?

A second thing happened which calls to mind the

clustering / manipulation of results. I went to a small conference on the use

of cell phones in survey research late in 2008. Strangely enough all roads lead

to Mississippi...it was at Mississippi State University. Remember that cell

phones weren't as all encompassing as they are now then. Some firms doing

survey research by telephone were including cell phone samples and some were

not. There are different rules about calling cell phones than the rules for calling

landlines. I put up a chart of the results produced by polling firms for the

2008 election the day before the final prediction and the final prediction.

Most polling firms, back then, showed where the race was each day for several

days out. But the sample was largely comprised of the same data plus one new

day of polling each subsequent day so the results weren't likely to change day

to day.

?

The results were striking when graphed out.?Every firm with a cell phone sample showed a large lead for Obama and they were all +/- 2 points of each other.?Every firm without a cell phone sample was showing a small Obama lead and were all within +/- 2 points.?The point of the graph was that there was a different clustering of results depending on whether you had a cell phone sample or not and the results were striking.?

I took it a step further though and this is what showed what firms in the space were doing.?I then showed the final predictions pf all firms.?Remember the only difference in these two datasets was an additional day of telephone calling, otherwise it was the same data.?No firm did a brand new sample.?I call it the great coming together.?The cluster that had a large Obama lead previously, had new estimates that were 4 or so points lower than the day before.?The firms that had a small Obama lead in the day before the final estimate, now showed a lead 3-4 point higher.?It is virtually impossible that this great coming together was a result of data driven changes.?

?

Again, this was a small conference but my presentation got passed around.?And it pissed people off.?I even got threated with a lawsuit for misrepresentation by a famous pollster, who is famous for making threats so I wasn’t worried, who had misread the chart, ironically.?I called him back and let him know he had misread the charts, in a voicemail as this guys doesn’t pick up the phone, and I never heard back on the issue.?I had gotten it right.?

?

Once again folks were afraid to be wrong with their forecasts.?So they saw where everyone else was and arranged for their data to get a bit closer to others so they couldn’t be that far off.?I sorta get it, I was convinced in my first election not to release data that was different, although I would never change data to get a specific result.?It is scary to be out there on your own.?But if you are confident in what you have done and checked everything, I cannot make a case for not releasing the unaltered information.?Not only is it the right thing to do from a purist standpoint, if you are just coordinating on where others are, there is absolutely no need for your expertise.?It calls into question your judgments when there aren’t others to coordinate off of.?The jobs report is a huge black eye for the economists making forecasts.?The economy is a very complex thing.?There should be a lot of outliers if economists are using different models.?Folks should use this as a learning example in the same way I used the 1999 election as a learning example.?To this day it is in the back of my mind when my data is questioned.?Have I checked everything and put it into context??If yes, then I stand behind my data.?Others should do that as well.?You won’t always be right but you can grow after those cases where you aren’t.?Please note, we don’t talk about the 2004 election in my household but I grew from it.??