Question: What have these four things got in common?
1. Choosing a random number generated by a lottery machine.
2. Reading tea leaves.
3. Blindfolding yourself and sticking a pin in a board.
4. Tossing a coin.
Answer: They’re all better methods of informing your decision-making than this:
This table is from an actual performance page within an official document. People get paid a lot of money to read these tables and make decisions about resourcing, funding and operational deployments, based on their assessment of the performance data such tables contain. Imagine it’s you. What would you prioritise?
Well, perhaps vehicle crime isn’t such an issue at the moment as it’s ‘in the green’ (and with a large percentage reduction), so you could ignore that for the time being and put more emphasis on tackling common assault, which seems to be raging out of control. It’s bright red after all, and red means ‘bad’. So, you throw some extra resources at the assault problem and leave the vehicle crime, robbery and the other green stuff to look after itself for a while.
Easy isn’t it? Now you’ve got your priorities sorted what else could you do? Well, one obvious choice is to look at the crime types that are going up and find someone to hold to account. What on earth is the local police commander playing at, by allowing total crime to rise by 6%? Maybe he or she should be replaced with someone who knows what they are actually doing.
So, you shuffle some of your people around, take a bit of funding from here, divert a bit there, have some ‘strong words’ about performance expectations, and hey presto, everything’s hunky dory. Or so it seems, until next time you are shown a similar performance chart and lots of the stuff that was green is now red. Time for more strong words, this time aimed at you.
Ooops! Should have read the tea leaves. It’s a much more stable and scientific method of assessing performance than using one of these silly tables featuring red and green boxes and nice ‘up’ and ‘down’ arrows.
Know why? Well I’ve chuntered about this before – this method of comparing data is known as a ‘binary comparison’, and when used as a technique for judging performance, it always gives false readings. It’s main weakness is that it doesn’t actually tell you anything about performance. Performance data should enable you to assess performance so that you can make informed decisions that hopefully lead to improved performance. The clue is in the name. Comparing just two numbers with each other can never achieve this.
This is what a binary comparison looks like on a chart:
Hopeless, isn’t it? It could be ‘this year vs last year’, ‘this month vs last month’, ‘this week vs last week’, or any other similarly useless comparison between two snapshots in time. The binary approach is rubbish because it ignores all the other data points between (and before) the two points chosen, as well as due to the fact that all the data points are subject to variation, which means you might as well compare today’s figure against any other figure that has ever gone before it. Or a moving object. Or a banana.
The binary comparison approach is commonplace and seen in everyday life; for example, ‘sales are up 5% compared to last Christmas / ‘unemployment is down by 45,000 compared to the same period last year / ‘accidents are down 3.6% compared to the previous quarter’. In all cases it’s meaningless, as the method relies upon scant data, moving variables and unstable assumptions, all of which lead to defective decision-making. Waste is driven into the system because managers react to something that essentially isn’t there; costs go up, people are unfairly held to account and performance gets worse.
So… ditch your tables of numbers, red and green boxes, comfortingly familiar ‘up’ and ‘down’ arrows, and use methods that actually tell you something about performance.
Don’t let this cute, but fundamentally evil, ‘performance banana’ trick you into thinking that you can ever make performance assessments using just two numbers. It’s impossible.
InspGuilfoyle 
Nice post Guv,
Coupled with the fact that the numbers are well and truly fudged, fiddled, cooked and manipulated anyway provides even less justification for reliance on statistics.
000’s of crimes screened out, ooo’s no crimed,000’s more that never hit the books cause they stay as incidents (it’s as bad as the old MOB book for keeping numbers down) … all of which combine to illustrate what a mess the UK Crime Stats are in. Worse still, the cuts to frontline numbers were largely based on the alleged reduction in crime over the last 20 years (confirmed from my conversation with the previous Policing Minister).
So the depleted resources you now have to struggle with are largely the fault of the Chiefs who orchestrated the large scale gaming practices (but hey, why should they care? they got their 15% bonus on top of salary to show crime was reducing… was it ever going in any other direction with so much political (tax payers) cash being paid to Chiefs & SMT’s?
Until crime statistics are completely wiped clean and the books opened again with accurate and honest recording, whatever that may reveal, the job will never have the public confidence and resources it actually needs to justify all of our council tax contributions.
No slight intended against the frontline and response guys, they’re are the poor sods having to cope with dangerously scary levels of cover, and riduculous performance targeting that the Chiefs cant let go, despite Mother Theresa’s instructions of two years ago.
There is so much horrible abuse of statistics amongst senior police officers. Posts like this should be required reading. I’ve always thought there’s a touch of Heisenberg’s Uncertainty Principle in numerical performance targets – if you try to measure the performance using numbers, the performance will change to meet the numbers required, and the very fact that you’re measuring means that you’ll never really get an accurate measurement.
Agree entirely but I can’t help worry that there’s no way out of the numbers game. How can people get promoted if they can’t point to some fudged statistic that proves the pointless radical lurch they instigated led to a performance gain?
I’ve never heard of anyone getting a bump in rank by saying “I joined a good team, I told them to keep it up and sorted their leave requests while protecting them from malicious complaints”.
Changing things would require Acpo rank officers to have a Damascene conversion “my whole career is founded on bullshit it’s time to change course on the oil tanker”
Outstanding lesson. One that should be noted by those with the responsibility of leading our service. Stop burying your heads in the sand, there is another way! Transform the service and change the culture. Systems thinking is the answer.
Reblogged this on Principled Detective and commented:
Outstanding lesson for all the performance junkies out there!
Nice one Simon but don’t you feel a bit like the prophet wandering unheard in the desert? Police Service Bean Counters and Political Masters are so addicted to churning their own garbage they’ve forgotten the quality/human issues associated with policing. Mac J
Most of the comments so far seem to suggest that the premise of this entry is that statistics are bad and cannot be trusted under any circumstances, but such comments seem disingenuous
The main point of the entry seems to me to highlight how dangerous it is to reduce the assessment of the performance of a system to a binary comparison – such comparisons encourage the selective use of statistics to demonstrate support for whatever issue is at hand.
However, the solution to the problem of misuse of statistics isn’t to dismiss all measurements as useless – such an approach would mean the abandonment of objective analysis. The solution is to ensure that analysis is supported by an understanding of the mechanisms and context of a system and that the complexity of a system is reflected in the measures used to monitor its success.
Pingback: This Time Last Year | Policing news | Scoop.it
Lies, damn lies and government statistics.