Why Binary Comparisons are Really Silly

Imagine having a rich resource of useful information at your fingertips, but then deliberately ignoring most of it for no logical reason whatsoever…

Binary conversation

No, I don’t understand either.

Poster: Binary conversation

Posted in Systems thinking | Tagged , , , , , | 2 Comments

Weak Excuses for Using Binary Comparisons

Due to the popularity of my last ‘poster’ blog, here’s another…

Weak excuses for using binary comparisons

You can download a pdf of the poster here: Weak excuses for using binary comparisons.

Enjoy ;-)

Posted in Systems thinking | Tagged , , , , , | 4 Comments

Get Help Now!

After all the heavy news coverage of recent days about the adverse impact of numerical targets within policing (e.g. PASC findings and the Metropolitan Police Federation report), I thought I’d lighten the mood with a #StickChild poster for you to laugh at:

SCSSI poster
You can download a pdf of the poster here: SCSSI poster

Enjoy! ;-)

 

 

 

 

Posted in Systems thinking | Tagged , , , , , , , | 3 Comments

Incontrovertible Evidence

red hand1Today, on the 9th April 2014, the House of Commons Public Administration Select Committee (PASC) published their report into allegations of police mis-recording of crime statistics. (The report – Caught Red-Handed: Why We Can’t Count on Police Recorded Crime Statistics – can be viewed here).

During the course of several weeks, the PASC considered written and oral evidence from serving and former police officers, academics, Police and Crime Commissioners (PCCs), statisticians, subject matter experts, and others. (My written submission can be viewed here).

I thought I’d highlight a few salient points from the report, which lays much of the blame for mis-recording of crime stats directly at the feet of performance targets. It highlights:

  • Performance pressures associated with targets acting as perverse incentives. (paragraph 21)
  • An entrenched target culture, which persists to this day. (paragraph 73)
  • A conflict between achievement of targets and core policing values. (paragraph 88)
  • The pernicious effects of target cultures. (paragraph 80)

It also cites a 2010 report by the UK Statistics Authority, which warned:

“The existence of a target may change the behaviour of service providers in ways that have unexpected and unwanted side effects. There may be scope for manipulation or gaming”. (paragraph 80)

But wait, there’s more…

Commenting on the impact upon police officers’ sense of vocation and desire to serve the public as ‘dedicated and courageous professionals’, the Committee concludes targets:

“…tend to affect attitudes, erode data quality and to distort individual and institutional behaviour and priorities”. (paragraph 86)

Finally, in the ‘Conclusions and Recommendations’ section, the document concludes with this unequivocal statement:

“The Home Office… should make clear in its guidance to PCCs that they should not set performance targets based on Police Recorded Crime data as this tends to distort recording practices and to create perverse incentives to misrecord crime. The evidence for this is incontrovertible. In the meantime, we deprecate such target setting in the strongest possible terms. (paragraph 40)

Pretty strong stuff. And that isn’t coming from me, or #StickChild, or someone with an axe to grind or political points to score – they’re the words of a cross-party committee of MPs.

Just saying.

Posted in Systems thinking | Tagged , , , | 12 Comments

The Weather Man

Sometimes when Stick Child and his friends are out playing, they see an old man walking his dog. They call this man ‘The Weather Man’, because he always asks the stick children to help him predict the weather. He asks the children to pick a pebble from the nearby stream, then he looks at it carefully, before declaring whether tomorrow’s weather will be sunny or rainy, hot or cold, windy or calm.

Stick children and weather man

The stick children like the old man, but think his weather forecasting antics are rather odd and amusing, because he’s not usually right. He thinks that if the pebble is larger or smaller, a different colour or a different shape than the one yesterday, this enables him to predict what tomorrow’s weather will be like. Often they see him walking his dog in his T-shirt and Bermuda shorts in torrential rain, or sweltering in a thick duffle coat on a scorching hot day because he got it wrong again.

One day, the stick children asked the old man why he thought he could predict the weather by comparing pebbles and he told them it was because he’d always done it that way. He also said that it must be a good indicator of forthcoming weather patterns because sometimes his predictions had actually been correct.

The stick children felt a bit sorry for him because he believed in what he was doing, but they all knew that his method was no good. Then one of them had an idea – her Mummy had just bought a new television and she didn’t know what to do with the old one. The stick children had a meeting (not like the ones grown-ups have, which go on for several hours) and after two minutes they decided to ask the old man if he wanted the old television, so he could watch real weather forecasts.

Next time they saw the old man, they told him about their plan and he was really excited, because he’d never owned a television before. That afternoon, the stick children (and a stick parent) took the television round to his house and set it up. They showed him how to find the weather forecast and explained how weather experts like Stick Lucy the Weather Lady use real science for predicting what the weather will be like.

Stick children and TV

The old man was very impressed and thanked the stick children for their thoughtfulness. Now he knows there is a better way to understand and forecast the weather than by trying to use pebbles from the stream, which means he no longer gets caught in downpours in his Bermuda shorts. After realising this, he couldn’t believe anyone would ever try and use pebbles at all.

Stick Child’s Thought of the Day

For anyone who knows Stick Child, you’ll probably have already figured out that the weather analogy is all about understanding measures properly. The stones represent binary comparisons (i.e. a completely useless ‘method’ prone to giving wrong signals) whilst the proper weather forecast reflects the use of control charts and associated scientific methods for understanding data properly. Better methods of understanding data lead to better choices – what’s not to like?

Bear in mind that neither control charts nor television weather reports are always 100% accurate, but they use real science, so they’re a whole lot better than pulling pebbles out of a stream.

If the old man was prepared to change his method to something that actually works, so can you.

Posted in Systems thinking | Tagged , , , , , , | 2 Comments

Stick Child’s Guessing Game

Stick Child dice game

Stick Child and his friends sometimes play a game where they take turns to throw dice and everyone tries to guess the number that will come up. It’s an enjoyable game, but the stick children know it’s just a bit of fun – even when one of them guesses the number correctly they know it’s luck, because when two dice are thrown, the total could be any number between two and twelve. None of the children really believe they had genuinely predicted the number, or that a particular number came up because of how the dice were thrown.

One day, Stick Child and his friends were playing the guessing game, when along walked Stick Cop. Now, the ‘Daily Stick’ newspaper may have you believe that most stick cops would treat what the children were doing as as ‘anti-social behaviour’, but really, most of us are not like that at all. Stick Cop just wanted to say hello to the children and make sure they were safe and enjoying themselves.

Stick Cop

After Stick Cop had played a couple of rounds of the guessing game with the stick children, Stick Child told him about his ‘straight lines’ theory. Stick Cop thought it was fascinating because he could see how it applied to his line of work. You see, Stick Cop works with some really clever people called ‘The Stick Analysts’, and although they sound like a 1960s rock band, they are actually a group of very bright folk who understand numbers and use their knowledge to try and help their bosses make good decisions.

One of the things The Stick Analysts do is try to work out what might happen with things like crime rates. They throw lots of numbers into a big machine and it produces charts like the ones Stick Child draws. Sometimes the machine tells them that crime might be going up or down over time, and this makes it possible to predict (to an extent) where crime rates might be heading.

Stick Analysts Machine

The Stick Analysts know, however, that accurate prediction is dependent upon on the overall crime rate trajectory remaining the same as it was at the point when they threw the numbers into the machine. Also, even if it continues to increase or decrease at exactly the same rate, even the cleverest of Stick Analysts could only say that the future crime rate could be anticipated to fall within a certain range. This is just like saying that the dice will produce a number somewhere between two and twelve on each throw.

Unfortunately, what sometimes happens is a Mystery Targets Monster gets hold of the good work done by The Stick Analysts and either swaps it for some indecipherable twoddle based on binary comparisons, or worse still, decides that it’s possible to choose one of those numbers somewhere within the predicted range and turns it into a target. This is because the Mystery Targets Monster refuses to listen to the experts and thinks it knows better.

Targets Monster

The Mystery Targets Monster earned its ‘mysterious’ qualification at college because often no one seems to know who put the targets in, or why. It’s a very cunning, elusive, but fundamentally confused (and grubby) creature that exists in a fantasy world where evidence about the dysfunctional effects of numerical targets is countered by simply being ignored.

Anyway, once the Mystery Targets Monster has ruined the report that The Stick Analysts have produced, it emits a shrill girlish giggle then runs back to its secret lair. The report then goes to the big police bosses who try to do their best with what they’ve been given.

Targets monster and chart

Meanwhile, the Mystery Targets Monster plays the dice guessing game by itself, becoming increasingly angry because its predicted number doesn’t come up every time, like it thinks it should. Whenever its predicted number fails to materialise, the Mystery Targets Monster goes out and roars unintelligibly at the first thing it sees, thinking that this will make the dice behave differently next time. It adopts the same approach with target setting, which of course is also a complete guessing game.

The Mystery Targets Monster just doesn’t get it at all.

Fortunately, Stick Child and his pals do. So do The Stick Analysts. And Stick Cop.

Hopefully you do too.

Posted in Systems thinking | Tagged , , , , , , , , | 3 Comments

Straight Lines

Stick kids and petThe other day, Stick Child was playing with his next-door neighbour, Stick Friend. At first, they were having a great time playing with her pet, ‘Stick Stick Insect’, but soon Stick Child noticed that Stick Friend seemed a bit quieter than usual, so he asked her if anything was wrong.

Stick Friend told Stick Child she felt a bit sad because her teacher had sent home a school report that graded her at level SP3c, when the national average for stick people of her age (8) is level SP3a. The teacher had told Stick Friend that the school would get in trouble if she and her classmates didn’t reach their targets by the end of the year.

Stick Child thought this didn’t make any sense – he supposed that the people who decided on these targets must have never heard of that thing called variation he learnt about in his red cars project. Knowing Stick Friend hadn’t done that project, Stick Child quickly thought of a way to explain the concept at a level that even an eight-year-old would understand.

He picked up a piece of paper and a pen and drew the following diagram:

Stick Childs education chart

Stick Child knew he only had about 15 minutes before dinner, so decided to not even talk about the dys-funct-ion-al behaviour that numerical targets cause, but just to concentrate on why it is totally mathematically silly to try and understand performance like this.

Using his drawing, Stick Child explained that some grown-ups believe things happen in straight lines, as in the diagram. In real life however, he pointed out that stick children learn and progress at different rates and in different ways, and showed Stick Friend that even when a child hits their targets by the time they finish ‘little stick person school’, this progress never ever happens in a perfectly straight line.

As you can see from the drawing, sometimes the child’s work level is a bit below average for the stage they’re at, and other times it’s a bit above average. This happens because of variation and a thing called regression to the mean, which is why the zig-zags tend to fluctuate around the average.

This means it’s pointless trying to compare isolated fixed points on a scale to… the average, a target, a peer, isolated points from the past (binary comparisons give kids nightmares!), or using daily totals or year-to-date figures to see whether performance is ‘on track’.

Stick Friend understood this concept straight away and felt much happier. The friends then played with Stick Stick Insect some more, before Stick Child went home for his dinner.

It took Stick Friend about five minutes to get her head around this stuff.

Even Stick Stick Insect understood most of it.

Stick stick insect

If hand-drawn stick insects can understand it, so can you.

Advanced Learning

At home that evening, Stick Child’s Daddy told him about some research that was done in the old days (1973). It involved two clever people called Kahneman &  Tversky, who discovered that some flight instructors believed praising trainee pilots after a smooth landing was a bad idea because they seemed to be a bit worse next time. The flight instructors also believed that telling the pilots off for a poor landing helped to make them better, because they often performed smoother landings after being shouted at.

The sad thing was that the pilots were actually getting better and better all the time – just not in a straight line – yet because the instructors didn’t understand variation or regression to the mean, they believed the best way to help them improve was to get angry with them when they were ‘failing’.

If you wished, you could draw a similar diagram to Stick Child’s, that shows gradual improvement of trainee pilots. (Or perhaps one that shows crime rates decreasing. Or profits going up. Or people living longer. Or unemployment going down. Or stock market shares increasing. Or anything else that improves or worsens over time).

Stick Child facepalm chart

As Stick Child points out – none of these things ever happen in a straight line. Hopefully, this illustrates why comparing a fixed point of performance (or anything else) against a straight line leads to muddled perceptions, impaired decision-making, and panicked reactions such as:

“We can only afford ‘X’ amount of ‘Y’ per day, otherwise we’ll miss the target!” 

Don’t do it guys! Have some self-respect.

Random Disclaimer

Note: The title of this blog is in no way inspired by the 2013 hit record ‘Blurred Lines’, by Robin #Sticke.

Posted in Systems thinking | Tagged , , , , | 4 Comments