The other day, Stick Child was playing with his next-door neighbour, Stick Friend. At first, they were having a great time playing with her pet, ‘Stick Stick Insect’, but soon Stick Child noticed that Stick Friend seemed a bit quieter than usual, so he asked her if anything was wrong.
Stick Friend told Stick Child she felt a bit sad because her teacher had sent home a school report that graded her at level SP3c, when the national average for stick people of her age (8) is level SP3a. The teacher had told Stick Friend that the school would get in trouble if she and her classmates didn’t reach their targets by the end of the year.
Stick Child thought this didn’t make any sense – he supposed that the people who decided on these targets must have never heard of that thing called variation he learnt about in his red cars project. Knowing Stick Friend hadn’t done that project, Stick Child quickly thought of a way to explain the concept at a level that even an eight-year-old would understand.
He picked up a piece of paper and a pen and drew the following diagram:
Stick Child knew he only had about 15 minutes before dinner, so decided to not even talk about the dys-funct-ion-al behaviour that numerical targets cause, but just to concentrate on why it is totally mathematically silly to try and understand performance like this.
Using his drawing, Stick Child explained that some grown-ups believe things happen in straight lines, as in the diagram. In real life however, he pointed out that stick children learn and progress at different rates and in different ways, and showed Stick Friend that even when a child hits their targets by the time they finish ‘little stick person school’, this progress never ever happens in a perfectly straight line.
As you can see from the drawing, sometimes the child’s work level is a bit below average for the stage they’re at, and other times it’s a bit above average. This happens because of variation and a thing called regression to the mean, which is why the zig-zags tend to fluctuate around the average.
This means it’s pointless trying to compare isolated fixed points on a scale to… the average, a target, a peer, isolated points from the past (binary comparisons give kids nightmares!), or using daily totals or year-to-date figures to see whether performance is ‘on track’.
Stick Friend understood this concept straight away and felt much happier. The friends then played with Stick Stick Insect some more, before Stick Child went home for his dinner.
It took Stick Friend about five minutes to get her head around this stuff.
Even Stick Stick Insect understood most of it.
If hand-drawn stick insects can understand it, so can you.
At home that evening, Stick Child’s Daddy told him about some research that was done in the old days (1973). It involved two clever people called Kahneman & Tversky, who discovered that some flight instructors believed praising trainee pilots after a smooth landing was a bad idea because they seemed to be a bit worse next time. The flight instructors also believed that telling the pilots off for a poor landing helped to make them better, because they often performed smoother landings after being shouted at.
The sad thing was that the pilots were actually getting better and better all the time – just not in a straight line – yet because the instructors didn’t understand variation or regression to the mean, they believed the best way to help them improve was to get angry with them when they were ‘failing’.
If you wished, you could draw a similar diagram to Stick Child’s, that shows gradual improvement of trainee pilots. (Or perhaps one that shows crime rates decreasing. Or profits going up. Or people living longer. Or unemployment going down. Or stock market shares increasing. Or anything else that improves or worsens over time).
As Stick Child points out – none of these things ever happen in a straight line. Hopefully, this illustrates why comparing a fixed point of performance (or anything else) against a straight line leads to muddled perceptions, impaired decision-making, and panicked reactions such as:
“We can only afford ‘X’ amount of ‘Y’ per day, otherwise we’ll miss the target!”
Don’t do it guys! Have some self-respect.
Note: The title of this blog is in no way inspired by the 2013 hit record ‘Blurred Lines’, by Robin #Sticke.
Brilliant, just brilliant.
I’ve posted this on a our Hospital yammer blog thingy, in the hope some of our senior managers read this, and they begin to think, think about how the NHS presents and analyses data.
Then maybe, just maybe, they might start asking that all important question “are measuring the right things, the right way?”
Another great blog post.
Had a great example at my place (not the police) recently of how people not understanding regression to the mean and variation causes extra work.
At our meeting to discuss our monthly (binary comparison) performance reports, one team had lots of downward red arrows that month. The managers there decided this was definitely a ‘signal’ that something was going wrong (it wasn’t – it was just ‘noise’).
The poor manager responsible for that team had to go away and investigate, write a report, and do a presentation about it. As well as work for themselves, they would have also involved team members in this.
Next month there were (surprise surprise) lots more green arrows on that team. Managers went away thinking their decisive action at the last meeting caused this improvement, or at least prevented things getting worse.
However, all that happened was that the team’s performance had a bit of variation (as all teams do everywhere), and mostly regressed in an upwards direction back towards the mean.
Does that count as a ‘face palm’?
Pingback: Straight Lines (and how they're silly) | Nice a...
Pingback: Stick Child’s Guessing Game | InspGuilfoyle
Reblogged this on anemone of promise.