Incontrovertible Evidence

red hand1Today, on the 9th April 2014, the House of Commons Public Administration Select Committee (PASC) published their report into allegations of police mis-recording of crime statistics. (The report – Caught Red-Handed: Why We Can’t Count on Police Recorded Crime Statistics – can be viewed here).

During the course of several weeks, the PASC considered written and oral evidence from serving and former police officers, academics, Police and Crime Commissioners (PCCs), statisticians, subject matter experts, and others. (My written submission can be viewed here).

I thought I’d highlight a few salient points from the report, which lays much of the blame for mis-recording of crime stats directly at the feet of performance targets. It highlights:

  • Performance pressures associated with targets acting as perverse incentives. (paragraph 21)
  • An entrenched target culture, which persists to this day. (paragraph 73)
  • A conflict between achievement of targets and core policing values. (paragraph 88)
  • The pernicious effects of target cultures. (paragraph 80)

It also cites a 2010 report by the UK Statistics Authority, which warned:

“The existence of a target may change the behaviour of service providers in ways that have unexpected and unwanted side effects. There may be scope for manipulation or gaming”. (paragraph 80)

But wait, there’s more…

Commenting on the impact upon police officers’ sense of vocation and desire to serve the public as ‘dedicated and courageous professionals’, the Committee concludes targets:

“…tend to affect attitudes, erode data quality and to distort individual and institutional behaviour and priorities”. (paragraph 86)

Finally, in the ‘Conclusions and Recommendations’ section, the document concludes with this unequivocal statement:

“The Home Office… should make clear in its guidance to PCCs that they should not set performance targets based on Police Recorded Crime data as this tends to distort recording practices and to create perverse incentives to misrecord crime. The evidence for this is incontrovertible. In the meantime, we deprecate such target setting in the strongest possible terms. (paragraph 40)

Pretty strong stuff. And that isn’t coming from me, or #StickChild, or someone with an axe to grind or political points to score – they’re the words of a cross-party committee of MPs.

Just saying.

Posted in Systems thinking | Tagged , , , | 12 Comments

The Weather Man

Sometimes when Stick Child and his friends are out playing, they see an old man walking his dog. They call this man ‘The Weather Man’, because he always asks the stick children to help him predict the weather. He asks the children to pick a pebble from the nearby stream, then he looks at it carefully, before declaring whether tomorrow’s weather will be sunny or rainy, hot or cold, windy or calm.

Stick children and weather man

The stick children like the old man, but think his weather forecasting antics are rather odd and amusing, because he’s not usually right. He thinks that if the pebble is larger or smaller, a different colour or a different shape than the one yesterday, this enables him to predict what tomorrow’s weather will be like. Often they see him walking his dog in his T-shirt and Bermuda shorts in torrential rain, or sweltering in a thick duffle coat on a scorching hot day because he got it wrong again.

One day, the stick children asked the old man why he thought he could predict the weather by comparing pebbles and he told them it was because he’d always done it that way. He also said that it must be a good indicator of forthcoming weather patterns because sometimes his predictions had actually been correct.

The stick children felt a bit sorry for him because he believed in what he was doing, but they all knew that his method was no good. Then one of them had an idea – her Mummy had just bought a new television and she didn’t know what to do with the old one. The stick children had a meeting (not like the ones grown-ups have, which go on for several hours) and after two minutes they decided to ask the old man if he wanted the old television, so he could watch real weather forecasts.

Next time they saw the old man, they told him about their plan and he was really excited, because he’d never owned a television before. That afternoon, the stick children (and a stick parent) took the television round to his house and set it up. They showed him how to find the weather forecast and explained how weather experts like Stick Lucy the Weather Lady use real science for predicting what the weather will be like.

Stick children and TV

The old man was very impressed and thanked the stick children for their thoughtfulness. Now he knows there is a better way to understand and forecast the weather than by trying to use pebbles from the stream, which means he no longer gets caught in downpours in his Bermuda shorts. After realising this, he couldn’t believe anyone would ever try and use pebbles at all.

Stick Child’s Thought of the Day

For anyone who knows Stick Child, you’ll probably have already figured out that the weather analogy is all about understanding measures properly. The stones represent binary comparisons (i.e. a completely useless ‘method’ prone to giving wrong signals) whilst the proper weather forecast reflects the use of control charts and associated scientific methods for understanding data properly. Better methods of understanding data lead to better choices – what’s not to like?

Bear in mind that neither control charts nor television weather reports are always 100% accurate, but they use real science, so they’re a whole lot better than pulling pebbles out of a stream.

If the old man was prepared to change his method to something that actually works, so can you.

Posted in Systems thinking | Tagged , , , , , , | 2 Comments

Stick Child’s Guessing Game

Stick Child dice game

Stick Child and his friends sometimes play a game where they take turns to throw dice and everyone tries to guess the number that will come up. It’s an enjoyable game, but the stick children know it’s just a bit of fun – even when one of them guesses the number correctly they know it’s luck, because when two dice are thrown, the total could be any number between two and twelve. None of the children really believe they had genuinely predicted the number, or that a particular number came up because of how the dice were thrown.

One day, Stick Child and his friends were playing the guessing game, when along walked Stick Cop. Now, the ‘Daily Stick’ newspaper may have you believe that most stick cops would treat what the children were doing as as ‘anti-social behaviour’, but really, most of us are not like that at all. Stick Cop just wanted to say hello to the children and make sure they were safe and enjoying themselves.

Stick Cop

After Stick Cop had played a couple of rounds of the guessing game with the stick children, Stick Child told him about his ‘straight lines’ theory. Stick Cop thought it was fascinating because he could see how it applied to his line of work. You see, Stick Cop works with some really clever people called ‘The Stick Analysts’, and although they sound like a 1960s rock band, they are actually a group of very bright folk who understand numbers and use their knowledge to try and help their bosses make good decisions.

One of the things The Stick Analysts do is try to work out what might happen with things like crime rates. They throw lots of numbers into a big machine and it produces charts like the ones Stick Child draws. Sometimes the machine tells them that crime might be going up or down over time, and this makes it possible to predict (to an extent) where crime rates might be heading.

Stick Analysts Machine

The Stick Analysts know, however, that accurate prediction is dependent upon on the overall crime rate trajectory remaining the same as it was at the point when they threw the numbers into the machine. Also, even if it continues to increase or decrease at exactly the same rate, even the cleverest of Stick Analysts could only say that the future crime rate could be anticipated to fall within a certain range. This is just like saying that the dice will produce a number somewhere between two and twelve on each throw.

Unfortunately, what sometimes happens is a Mystery Targets Monster gets hold of the good work done by The Stick Analysts and either swaps it for some indecipherable twoddle based on binary comparisons, or worse still, decides that it’s possible to choose one of those numbers somewhere within the predicted range and turns it into a target. This is because the Mystery Targets Monster refuses to listen to the experts and thinks it knows better.

Targets Monster

The Mystery Targets Monster earned its ‘mysterious’ qualification at college because often no one seems to know who put the targets in, or why. It’s a very cunning, elusive, but fundamentally confused (and grubby) creature that exists in a fantasy world where evidence about the dysfunctional effects of numerical targets is countered by simply being ignored.

Anyway, once the Mystery Targets Monster has ruined the report that The Stick Analysts have produced, it emits a shrill girlish giggle then runs back to its secret lair. The report then goes to the big police bosses who try to do their best with what they’ve been given.

Targets monster and chart

Meanwhile, the Mystery Targets Monster plays the dice guessing game by itself, becoming increasingly angry because its predicted number doesn’t come up every time, like it thinks it should. Whenever its predicted number fails to materialise, the Mystery Targets Monster goes out and roars unintelligibly at the first thing it sees, thinking that this will make the dice behave differently next time. It adopts the same approach with target setting, which of course is also a complete guessing game.

The Mystery Targets Monster just doesn’t get it at all.

Fortunately, Stick Child and his pals do. So do The Stick Analysts. And Stick Cop.

Hopefully you do too.

Posted in Systems thinking | Tagged , , , , , , , , | 3 Comments

Straight Lines

Stick kids and petThe other day, Stick Child was playing with his next-door neighbour, Stick Friend. At first, they were having a great time playing with her pet, ‘Stick Stick Insect’, but soon Stick Child noticed that Stick Friend seemed a bit quieter than usual, so he asked her if anything was wrong.

Stick Friend told Stick Child she felt a bit sad because her teacher had sent home a school report that graded her at level SP3c, when the national average for stick people of her age (8) is level SP3a. The teacher had told Stick Friend that the school would get in trouble if she and her classmates didn’t reach their targets by the end of the year.

Stick Child thought this didn’t make any sense – he supposed that the people who decided on these targets must have never heard of that thing called variation he learnt about in his red cars project. Knowing Stick Friend hadn’t done that project, Stick Child quickly thought of a way to explain the concept at a level that even an eight-year-old would understand.

He picked up a piece of paper and a pen and drew the following diagram:

Stick Childs education chart

Stick Child knew he only had about 15 minutes before dinner, so decided to not even talk about the dys-funct-ion-al behaviour that numerical targets cause, but just to concentrate on why it is totally mathematically silly to try and understand performance like this.

Using his drawing, Stick Child explained that some grown-ups believe things happen in straight lines, as in the diagram. In real life however, he pointed out that stick children learn and progress at different rates and in different ways, and showed Stick Friend that even when a child hits their targets by the time they finish ‘little stick person school’, this progress never ever happens in a perfectly straight line.

As you can see from the drawing, sometimes the child’s work level is a bit below average for the stage they’re at, and other times it’s a bit above average. This happens because of variation and a thing called regression to the mean, which is why the zig-zags tend to fluctuate around the average.

This means it’s pointless trying to compare isolated fixed points on a scale to… the average, a target, a peer, isolated points from the past (binary comparisons give kids nightmares!), or using daily totals or year-to-date figures to see whether performance is ‘on track’.

Stick Friend understood this concept straight away and felt much happier. The friends then played with Stick Stick Insect some more, before Stick Child went home for his dinner.

It took Stick Friend about five minutes to get her head around this stuff.

Even Stick Stick Insect understood most of it.

Stick stick insect

If hand-drawn stick insects can understand it, so can you.

Advanced Learning

At home that evening, Stick Child’s Daddy told him about some research that was done in the old days (1973). It involved two clever people called Kahneman &  Tversky, who discovered that some flight instructors believed praising trainee pilots after a smooth landing was a bad idea because they seemed to be a bit worse next time. The flight instructors also believed that telling the pilots off for a poor landing helped to make them better, because they often performed smoother landings after being shouted at.

The sad thing was that the pilots were actually getting better and better all the time – just not in a straight line – yet because the instructors didn’t understand variation or regression to the mean, they believed the best way to help them improve was to get angry with them when they were ‘failing’.

If you wished, you could draw a similar diagram to Stick Child’s, that shows gradual improvement of trainee pilots. (Or perhaps one that shows crime rates decreasing. Or profits going up. Or people living longer. Or unemployment going down. Or stock market shares increasing. Or anything else that improves or worsens over time).

Stick Child facepalm chart

As Stick Child points out – none of these things ever happen in a straight line. Hopefully, this illustrates why comparing a fixed point of performance (or anything else) against a straight line leads to muddled perceptions, impaired decision-making, and panicked reactions such as:

“We can only afford ‘X’ amount of ‘Y’ per day, otherwise we’ll miss the target!” 

Don’t do it guys! Have some self-respect.

Random Disclaimer

Note: The title of this blog is in no way inspired by the 2013 hit record ‘Blurred Lines’, by Robin #Sticke.

Posted in Systems thinking | Tagged , , , , | 4 Comments

Stick Child Tries to Buy Milk

Stick Child milk 1Stick Child is a healthy little chappie and he loves a nice cold drink of milk at break time. (Do they still do those 1/3 pint bottles for 8p, with the really thin straws? Just wondered…)

Anyway, Stick Child’s usual routine is to go to the hatch in the school dinner hall when the bell rings for morning break, where he hands over his 8p (or whatever it is these days) and collects a nice cold drink of milk. Then he goes out to play with his friends.

Today, however, things were different – the milk counter was closed because the school had adopted a new system for break time milk sales. When Stick Child went into the school hall, he saw a table with a telephone on it and a sign next to it that read…

 ‘Milk-Related Transactions Centre’

Stick Child dialled the number listed in some instructions that were next to the sign, put the phone handset to his ear, and waited to see what would happen…

Then, instead of a human saying “Hello Stick Child, how are you today?” as usual, he encountered an automated voice that said,

“Welcome to our new and more efficient way of dealing with milk-related transactions. Please select from the list of available options”.

Stick Child milk 2Stick Child thought this was odd, but listened to the options, which were:

  • “Press ONE if you wish to discuss your milk account”.
  • “Press TWO for information about how much milk you’ve used so far this term”.
  • “Press THREE if you would like to upgrade to our premier milk account, or to hear about a range of special offers”.

Stick Child waited for Option FOUR, which he hoped might be “Speak to a human being”, or “Buy some milk”, but it never came. This confused him, so he chose Option ONE, which he supposed was the least inaccurate of all the choices, although he didn’t really want to discuss his milk account – he just wanted some milk. To drink.

Upon pressing Option ONE, Stick Child was immediately advised by the automated voice that it would be better for everyone if he just went and used the internet in future. Then it told him to be sure to check out ‘our new range of sweets, crisps and chocolates’. Stick Child wasn’t bothered about any of those things though – he just wanted to buy some milk!

After these messages, Stick Child was put on hold for a few minutes, whilst the phone played Bruno Mars records into his ear. Every so often, Bruno’s velvet tones were interrupted by another robotic voice that advised:

“Your call is important to us and one of our operators will be with you to deal with your query as soon as possible”. 

After a while, another automated voice informed Stick Child:

“We are experiencing unusually high call volumes at this time. You may wish to call back later or use our online milk information service”.

Stick Child milk 3Stick Child thought it was odd that whoever had designed the new system hadn’t anticipated milk-related demand would peak at break time. 

Anyway, he continued to hold, until eventually a human answered and politely informed him that he had come through to the wrong department and would have to dial the number again. This human tried to be helpful, advising Stick Child that he would need to choose Option TWO, then sub-menu Option NINE, followed by sub-sub-menu Option FORTY SIX, then ask for ‘DAVE’.

At that point, the end-of-break bell rang and poor Stick Child had to return to class, without any milk. Stick Child was disappointed. He thinks this new system is rubbish and that the old way of buying milk at break time was better for lots of reasons.

If he understands this, so can you!

Additional Information

In case you’re wondering what would have happened if Stick Child had managed to speak to the right person / voice recognition entity on the phone, then apparently under this new regime he would be allocated a unique reference number, which he must take to a new counter at the opposite end of the school.

There, his reference code will be checked and he will be given a token, before being sent to a third counter somewhere else, where he will hand over the token and collect his milk. The whole process is subject to rigorous service level standards at each stage and audited under a tough inspection regime, of course.


Posted in Systems thinking | Tagged , , , , , , | 3 Comments

Stick Child’s Guide to Systems Thinking

“What is systems thinking?”

This is a question I’ve heard quite a lot recently.

So, with the assistance of our little friend Stick Child (yes he’s back by popular demand!) let’s take a look at the subject in very straightforward terms.

By happy coincidence, Stick Child’s latest piece of homework was to find out about systems thinking, and fortunately his Daddy knows a bit about the subject, so the pair of them sat down together and came up with a simple guide. Here are some excerpts, below:

Stick Child ST pic

Some Systems Thinking Principles

Stick Child ST page 1Firstly, Stick Child learnt that there are lots of different strains of ‘systems thinking’. Some of them are very complex and theoretical, but Stick Child and his Daddy particularly like Deming‘s stuff, as it’s pretty simple and easy to apply in real life.

Deming described a system as, “A network of interdependent components that work together to try and accomplish an aim”. The important thing here is that these interdependent components must cooperate, otherwise everyone looks after their own interests and the system breaks down. This means that the customer or service user loses out, which is bad.

Therefore, it is really important that we are absolutely clear about the aim of the system; for example, a hospital might seek to ‘help people get better’, whilst the police try to ‘prevent crime and catch offenders’. In fact, some systems can have lots of aims – in the case of the police, they do lots of things that have nothing to do with crime; these things are also very important.

Stick Child learnt that there is no single ‘right’ interpretation of systems thinking, because there is lots of common ground between different systems approaches.


Stick Child ST page 2Stick Child discovered that measurement is really important in systems thinking. He learnt that we need to measure things that are happening within the system to understand how it is performing, but that they must be the right measures, measured in the right way. (He found this out from a blog by someone called InspGuilfoyle). Stick Child knows that one of the best ways to measure things is by using control charts.

Remember this one?

Stick Child control chart

Stick Child also found out that some people can’t tell the difference between ‘targets’ and ‘measures’ and this prevents them from being able to understand some really important stuff about how systems work. He wonders if there’s a special medicine they could take that would help their brains.

Bad Performance Management Practices

Stick Child learnt that well-meaning people sometimes use really bad ways of trying to measure things, like binary comparisons. He also found out numerical targets and league tables make people put effort into outdoing each other instead of concentrating on the real aim of the system. Sometimes they even cheat and tell lies. Stick Child’s Daddy says that this is because numerical targets are arbitrary and cause dysfunctional behaviour.

Stick Child ST page 3


Stick Child discovered that waste is activity that slows the system down and doesn’t help to achieve its aim. There are many types of waste – failure demand and rework are types of waste that occur because something wasn’t done properly the first time round. Waste also occurs when demand that shouldn’t be there in the first place enters the system, or when people within the system invent work for others to do, like writing lots of plans, or making them go to meetings where nothing useful occurs.

Stick Child ST page 4Stick Child thinks it would be better to put this effort into actually doing the work, rather than writing about it or talking about it. He does, however, accept that this might be a radical concept for some people.


Stick Child came to realise that the thinking part of systems thinking is really important. He learnt that management thinking needs to change in order for systems-based approaches to succeed. He understands that if we keep doing the same things we will keep getting the same results. That’s why we need to remove the bad things that hurt the system, then redesign it so that all the different bits work together to achieve its aim, whether this be helping people to get better when they are poorly, or catching baddies.

If Stick Child understands this, so can you.

Stick Child ST pic slogan

Posted in Systems thinking | Tagged , , , , , , , , , , | 3 Comments

Stick Child’s School Project

Over the last few weeks, our little friend Stick Child has been doing a really interesting school project about a thing called variation. Not everyone knows about variation. His project is called:

“Understanding Variation (For the Very Young or Pathologically Resistant)”.

Stick Child has learnt some pretty cool stuff during his project, which he thinks might be useful for grown-ups who struggle to use numbers properly when it comes to trying to understand performance information.

Here’s what he’s been up to…

Stick Child and red car

Armed with clip-boards and pencils, Stick Child and his friends have been standing outside their school and counting the number of red cars that drive past on different days. First of all, Stick Child ended up with a nice tally chart like this…

Stick Child tally chart

…then he used the daily totals to make an even nicer control chart like the one below:

Stick Child control chart

As you can see, there were different amounts of red cars each day. Stick Child’s teacher showed the class where the special dashed lines belong on the chart, and explained that in this case, every ‘X’ between them is completely normal because of this thing called ‘variation’. This means it’s a mistake to assume there’s any meaning behind different individual values, as well as a big waste of time trying to find out why one is different from another. (If you have difficulty sleeping and want to know how the lines are calculated, there is a step-by-step guide in Stick Child’s favourite bedtime book).

Next, Stick Child did some experiments with his chart. First of all, he randomly picked one of the numbers between the lines and called it a target. Then he tried to work out why sometimes the number of red cars hit the target and sometimes it didn’t.

Next, he tried to make the target influence the number of red cars by shouting at the chart and / or the cars as they drove past, but that didn’t work either.

Then, he said he would give 10p to one of his friends if she was able to make the target work. At first this didn’t make any difference, but later his friend said the target had been met. Stick Child looked closely at her chart and discovered that she had altered some of the numbers on it, so he told the teacher and kept his 10p.

Next, Stick Child picked random previous days’ totals on the chart and drew arrows between then to try and work out if the number of red cars was increasing or decreasing. Unfortunately, this just caused confusion because he got a different result every time; he quickly determined that making such binary comparisons was rubbish, so stopped doing it.

Stick Child comment with chart

Finally, Stick Child’s teacher timed how quickly the children had drawn their charts, then ranked them in a league table. Then she told the children that half of them were below average. None of the children could understand why she would do this, as they had worked very hard on their projects. It made them feel sad.

After a minute, the teacher told them this was actually just part of the lesson and that really she was very pleased they had done their best, because this is what really matters. Stick Child and his friends were glad that this silly way of assessing performance would never actually happen in real life.

What Stick Child Learnt

The project taught Stick Child and his friends lots of useful things. He learnt that there is no point worrying about why the total number of red cars was different on different days – this happens because of that thing called variation. He found out that unless systems conditions change (e.g. due to a road closure), those little ‘X’s will continue to appear anywhere between the dashed lines.

Stick Child report cardIn addition to this, Stick Child and his friends learnt that targets don’t make any difference to the amount of red cars that drive past his school. This is because variation doesn’t pay any attention to man-made follies, such as numerical targets. He remembered his Dad always says that numerical targets are arbitrary and likely to cause dysfunctional behaviour.

Stick Child also discovered that there is no point drawing arrows between two isolated numbers because it gives the impression of trends that simply do not exist. Finally, he learnt that league tables are a poor way of assessing performance. For his efforts, the teacher sent a nice letter to his parents.

Even though he is only nine, Stick Child knows that control charts do not just apply to red cars or school projects. They can also be used by grown-ups, for things like crime figures, response times, or almost any other set of numbers you might want to learn about. (That’s if the grown-ups really want to learn about these things).

If he understands this, so can you.

Posted in Systems thinking | Tagged , , , , , , , , , | 11 Comments