A friend at work told me that whilst giving blood recently, he noticed there were performance indicators on a nearby board showing the number of blood donations, in the dreaded ‘this week vs last week’ format. Nothing’s sacred, is it?
Anyway, this gave me an idea – being as there are so many examples of bad performance measurement around in all walks of life (e.g. binary comparisons, meaningless measures, numerical targets and so on), I thought I’d start sharing a few on here as I come across them. They torment me, so I might as well inflict them on you as well.
I therefore bring you the first installment from the ‘Bad Performance Measurement on Tour’ extravaganza, courtesy of a noticeboard on the wall of my local railway station.
This poster relates to overall performance for train journeys (I think). What does it tell us? Wow, compared to last month, something has changed by over one whole percent. (Note the big ‘down arrow’). Feel the burn, you naughty workers! Is this within the range of normal variation? Who knows. Is it part of a trend? No idea.
It’s certainly below average. So are about half of most things.
Great example of a meaningless poster. It made me chuckle whilst I was waiting for my train, which incidentally was on time. Maybe next month, the percentage might be slightly above average and this will be recognised with a nice big ‘up arrow’. Hurrah!
Here’s some more, from the same ‘noticeboard of shame’.
This chart claims to be able to identify ‘trends’ by comparing two values (i.e. ‘this month’s percentage vs the annual average’). Sorry, a trend can never be established by making a comparison like this. Nil points, as they say at Eurovision. Notice the only row in the top table that is awarded an ‘up arrow’ features the lowest percentage of the four rows, plus it is also below the average. So, is this good, or bad, or indifferent? Is performance getting better or worse? No one knows. The table doesn’t tell us anything useful.
Finally, here’s the third of the ugly sisters…
This table includes another flawed binary comparison made against an average, but it also features the added bonus of a totally arbitrary numerical target of 90%. Why 90%? What science dreamed this one up? Because it’s a nice round figure? Because it was drawn out of a hat? No one knows. Note also that a train is classed as being on time if it is ‘within ten minutes of its scheduled time’. Is this what ‘on time’ means to the customer? If it departs nine minutes early or nine minutes late then is that the same thing as being ‘on time?’ Where does the ‘ten minute either way’ threshold come from anyway? Another arbitrary target.
The funniest thing about this set of measures is that by setting this punctuality target at 90%, the Passenger’s Charter is effectively saying, ‘We plan for 10% of trains to be late”. Think about it. They’re happy with 90% punctuality. It’s in their plan! That’s what they aim for. Not 100%. Not the best they can do. 90%!
The major flaw in the way these performance measures are presented is that they don’t tell us anything about the capability of the system. Neither do they inform the reader of any actual trends, or help predict what performance will look like into the future. The numbers are of no practical use to the customer whatsoever. If the train companies’ management rely on this sort of thing to make decisions they may as well determine strategy based on the National Lottery numbers.
Conversely, by relying on the right measures and presenting them in a format that exposes the extent of variation, along with any trends or signals that might be present (i.e. on my old favourite, a control chart), managers can gain an understanding of how the system is performing and therefore take evidence-based steps to improve it. Measures that are derived from purpose (as defined from the customer’s perspective) can be used to elucidate an evidence base that informs method. Publishing the sort of meaningless pap that passes for ‘performance data’ in the examples above, serves no purpose except to amuse saddos like me whilst I wait for trains.
More cases of bad performance measurement next time, as and when I find them on my travels…
By the way, if you see any similar horrors, please feel free to send me a message on twitter @SimonJGuilfoyle and I’ll post some of the daftest or most outrageous examples.