I’ve been meaning to get around to writing about the issue of using crime figures as an indicator of police performance for a while now. Aside from the risk of mis-recording crime due to target-driven performance management, I believe there is a fundamental argument against judging police performance by using crime figures. It boils down to this:
Crime rates are not the definer of effective police performance; they merely provide information about criminal activity.
What do I mean by this?
Well, we’ve been so used to judging police performance based on whether crime is higher or lower than some previous point in the past (binary comparisons), positions in league tables, or variance from arbitrary numerical targets, that it’s easy to miss the obvious question about whether we’re measuring the right thing in the first place. Even when crime trends are shown in time series format (e.g. control charts), I’d argue that it’s still a case of measuring the wrong things (albeit in the ‘right’ way) if the intention is to assess police performance.
Try these analogies:
- Judging a vehicle repair agency (such as the AA) by the number of breakdowns reported.
- Judging a heart surgeon’s performance by the rate of heart disease cases in a locality.
- Judging the fire service’s performance by the number of car fires.
- Judging Stick Farmer’s performance by the number of lambs born each spring.
(One of these measures is actually real, silly though this may sound).
I’m not saying these datasets are useless for helping these people understand their business (they’d certainly be useful for understanding demand and maybe assisting future planning), but they are not measures of performance. In the policing context, yes of course we want to reduce crime (Peel talked about it, didn’t he?) and we should take reasonable steps to prevent it, but we need to move beyond the simplistic narrative of:
“Crime up = Police bad / Crime down = Police good”.
It’s also necessary to acknowledge that multiple variables affect crime rates; factors such as economic cycles, substance abuse, the weather, societal influences, changes in legislation, and so on. None of these are directly within the gift of the police to influence. Also, what about where the police cause an increase in reported crime by having the temerity to find someone carrying a weapon? Surely proactive problem-solving should not be discouraged on the basis that finding hitherto unreported criminal offences is incongruous with an over-simplified crime reduction narrative.
At the local level, if Stick Burglar is arrested and jailed, and burglaries suddenly stop, it’s probably fair to assume that the police directly affected that particular crime series. Conversely, at force or national levels, if crime happens to go up or down a bit, it’s likely to be as a result of the plethora of external factors that influence the crime rate (as well as normal variation).
It’s time for a shift in thinking about crime rates and police performance. By removing perverse incentives for mis-recording crime, we are hopefully left with a clearer picture of criminal activity. Such data can then act as extremely useful sources of information that assist decision making about how best to tackle reported crime.
But it’s not ‘performance’ data.
In the same way, the AA should not be held accountable for a vehicle breaking down; neither should the fire service be blamed if an electrical fault causes it to catch fire. There may be opportunities to learn about causes and respond to future patterns of demand (where it’s predictable), but that’s all.
Therefore, it doesn’t seem logical to directly equate police effectiveness with crime levels, especially as the true extent of crime is unknown, and what is known is affected by a multitude of factors.
Reported crime, from whatever source, is potentially useful information about criminal activity.
Not performance data.
InspGuilfoyle 
Great blog. Love stick man ! Some thought provoking concepts that make a very interesting point. How would you suggest effective performance ( and I use that word with some reluctance) is measured ? Surely it is prudent to have some type of frame of reference to ensure what your doing is not a wasted effort ?
Good question – for me, it’s all about using the right measures in the right way.We need to measure the things that tell us how the system is performing and help us identify opportunities for improvement. Have a look at this blog post for more – https://inspguilfoyle.wordpress.com/2013/10/12/right-measures-measured-right/
Love this blog, and there’s always lots of what NOT to do…but where’s the what TO do? What should we putting in front of management to prove to them that we’re an effective service (or not)?
Did you read the above link? Try these as well –
https://inspguilfoyle.wordpress.com/2013/10/12/right-measures-measured-right/
https://inspguilfoyle.wordpress.com/2013/01/31/spot-the-difference/
https://inspguilfoyle.wordpress.com/2014/01/23/stick-childs-school-project/
Lots to do there!
I did, yes. The second link you posted is the most useful. I’ve moved from presenting crime data from the Red/Green “Weekly Scorecard” binary measure to control charts. No-one likes it. They want to be able to say, “ooh burglary’s down 25%, aren’t we great”. It’s exceedingly frustrating….
I recognise that. I think control charts (whilst not being the be-all-and-end-all) can be really useful. I point out that a picture is worth a thousand words – if you can visually see trends or step changes that’s a million times better than binary comparisons, as you know. Some people’s refusal to let go of the binary narrative used to frustrate me too, but now I just refer them to this>> https://inspguilfoyle.wordpress.com/2014/05/02/weak-excuses-for-using-binary-comparisons/ Print some off; hand them out, then carry on doing the right thing! 😉
Ha! Will do (then you will have to find me a new job…)
I really like your thinking and it is so frustrating to see so much of police daily business taken up in this form of performance measurement/analysis and “number crunching”. That TV documentary a couple of months ago said it all when someone in a management meeting queried “is that one crime or loads?”
Yep – I cringed all the way through that. “We haven’t been able to hit the target, so we’ll double it”. Classic.
I also have a big problem with numerical & simplistic (not to mention complex) box-ticking performance targets and measurement. I am also instinctively sympathetic with your underlying concerns specifically in connection with policing. However, performance indicators even of the statistical kind can, I think, be useful – provided they are carefully thought through and used sensibly and judiciously. And I have to say that I am not sure the analogies you have used to make your case are entirely sound. For example: whilst the AA most certainly cannot be responsible for an increase in vehicle breakdowns, I do think that an indicator of effective policing and crime detection is the rise or fall of criminal activity. But, yes, it does have to be weighted relative to other ‘external factors’. And the conclusions drawn have to be sound and relevant, not to mention useful. I could go on – as I am sure, could you in debating this but my main points are: analogies have to be carefully thought through, as much as any tool or measure. Thank you for your article.
Thanks for your comment. I’m emphatic that performance indicators are absolutely essential (as long as you mean ‘measures’ and not ‘numerical targets’ – see this>> https://inspguilfoyle.wordpress.com/2013/01/31/spot-the-difference/ )
Regarding your other point, you’ll see from my blog that I don’t agree the overall crime rate describes whether policing is effective or not. At the local level, yes, in some circumstances, as in the burglary example given. Detected offences, possibly to an extent, but again there are external factors that can prevent a crime being solved which means police performance may not necessarily be poor (e.g. there were no witnesses, no CCTV, no forensics, no other lines of enquiry).
I really hope we can make a step change in this. I agree we need to “perform” and show” value for money” but we seem to have lost direction. Not too long ago i was duty officer on the weekend when I was contacted by the local inspector to see how things were going on his patch. I informed him there had been a fatal house fire and a suicidal misper on the go. His reply was along the lines of, “yes but how’s my vehicle crime looking”. Surely that cant be right!
But only last week I was at a conference in London at which a Home Office spokesman outlined the only 3 objectives for Police Forces:
1. To reduce crime
2. To provide vfm for every pound spent
3. To maintain the public’s confidence in policing
I thought I’d recalled there being “only 1 objective” but must have a bad memory.
At least 2 of these would seem to at the very least to imply that a certain numeric level represents “achievement”. And the Home Office are supposed to be in charge of policing so if they say that then surely it must be right!
I would say many of the examples are outcome measures of the system – which is a better measure than is often used.
However, they are often beyond the control of individuals and even police departments overall (many other factors play a part – economic development, social services system, education system…).
Likely more directly relevant the measurement error is often so high that the figures have more to do with measurement than the actual outcome http://management.curiouscatblog.net/2004/08/29/dangers-of-forgetting-proxy-nature-of-data/ And when the figures are being used to blame then it dramatically increases the likelihood the figures will be a poor representation of outcomes.
I would rather see more focus on outcome measures. We should also reduce incentives to misreport data (often blame related).
I think the issues you raise about the system being larger than the police can tackle alone should be a reason to INCREASE the view of the SYSTEM. The important system is LARGER than the police department. When we have institutional walls that break up the system we need to find ways to knock down those walls so the system can work together. Granted this seems nearly impossible given how much difficulty we have even just breaking down barriers inside tiny pieces of our organizations.
Nevertheless that is where the focus should be. We shouldn’t decide the outcome measures are not fair given where we put organizational barriers. We should decide that we need to realize when we cut funding for x it drives bad results in section a. And when we allow y to retain fitfully outdated management practices that doesn’t just impact their ability to succeed it likely creates lots of other problems all over the place.
Pingback: Top of the Table | InspGuilfoyle