What is Performance, and how do you Measure it?

I was reminded of the difficulty of defining and measuring work performance objectively, by a conversation about hockey. Specifically, about goalkeepers. It turned out that two of the group had been amateur goalkeepers. One played in a top team, and the other in a bottom team.

The keeper for the bottom team let in so many goals that scores of 10-0 against his team were not unusual. His team was always outclassed and outplayed. He let in a lot of goals, and was universally regarded as a dreadful keeper.

Then came the annual grudge match against the next town. The other side's first team had about half the GB Olympic squad in it, and always won - it was just about how wide the margin would be. Well, for various reasons, what was usually a first team match became a second team match. It was an out of season friendly (yeah, right...), so a lot of players were unavailable...including the second team's keeper...and the third team's keeper...and the fourth team's keeper. So, that left the fifth team's keeper – to everyone’s horror.  

It got worse. Some of the other team's players were from the first team, whereas our hero's team were about half and half second and third team players. It was going to be a bloodbath. And it was being played on a well-maintained all-weather pitch, not the Somme-inspired mudpool the keeper usually played on - surely this flat, fast surface would favour the more skilled team.

The match was a victory - by a small margin - to the underdogs, with the keeper unexpectedly making several improbable saves.

Why?

Well, remember that this is a story about the difficulty of measuring performance.

Had the keeper risen to the occasion? Probably.

Did he find it easier because knew where the ball would go? Definitely.

Did he play fundamentally better than usual? Probably not.

So what happened?

Well, to explain that, we need to look at statistics. The National Collegiate Athletic Association of America (NCAA) measures sports stats...of course. So, we have a basis of comparison. Our keeper saved about 3 out of 4 shots on goal. That meant that in a 10-0 defeat, he had probably saved 25-30 shots on goal. According to the NCAA, that would put him in the top 30% of keepers (assuming hockey has broadly similar dynamics both sides of the Pond).

Was he a dreadful keeper?

Yes, because he let a lot of goals in. That’s the results answer.

No, because his save rate was good. That’s the performance answer.

What do you think matters?

You may be wondering if this is a true story, and how I know about it.

Yes, it is true…I was that keeper.