Stick Child’s School Project

Over the last few weeks, our little friend Stick Child has been doing a really interesting school project about a thing called variation. Not everyone knows about variation. His project is called:

“Understanding Variation (For the Very Young or Pathologically Resistant)”.

Stick Child has learnt some pretty cool stuff during his project, which he thinks might be useful for grown-ups who struggle to use numbers properly when it comes to trying to understand performance information.

Here’s what he’s been up to…

Stick Child and red car

Armed with clip-boards and pencils, Stick Child and his friends have been standing outside their school and counting the number of red cars that drive past on different days. First of all, Stick Child ended up with a nice tally chart like this…

Stick Child tally chart

…then he used the daily totals to make an even nicer control chart like the one below:

Stick Child control chart

As you can see, there were different amounts of red cars each day. Stick Child’s teacher showed the class where the special dashed lines belong on the chart, and explained that in this case, every ‘X’ between them is completely normal because of this thing called ‘variation’. This means it’s a mistake to assume there’s any meaning behind different individual values, as well as a big waste of time trying to find out why one is different from another. (If you have difficulty sleeping and want to know how the lines are calculated, there is a step-by-step guide in Stick Child’s favourite bedtime book).

Next, Stick Child did some experiments with his chart. First of all, he randomly picked one of the numbers between the lines and called it a target. Then he tried to work out why sometimes the number of red cars hit the target and sometimes it didn’t.

Next, he tried to make the target influence the number of red cars by shouting at the chart and / or the cars as they drove past, but that didn’t work either.

Then, he said he would give 10p to one of his friends if she was able to make the target work. At first this didn’t make any difference, but later his friend said the target had been met. Stick Child looked closely at her chart and discovered that she had altered some of the numbers on it, so he told the teacher and kept his 10p.

Next, Stick Child picked random previous days’ totals on the chart and drew arrows between then to try and work out if the number of red cars was increasing or decreasing. Unfortunately, this just caused confusion because he got a different result every time; he quickly determined that making such binary comparisons was rubbish, so stopped doing it.

Stick Child comment with chart

Finally, Stick Child’s teacher timed how quickly the children had drawn their charts, then ranked them in a league table. Then she told the children that half of them were below average. None of the children could understand why she would do this, as they had worked very hard on their projects. It made them feel sad.

After a minute, the teacher told them this was actually just part of the lesson and that really she was very pleased they had done their best, because this is what really matters. Stick Child and his friends were glad that this silly way of assessing performance would never actually happen in real life.

What Stick Child Learnt

The project taught Stick Child and his friends lots of useful things. He learnt that there is no point worrying about why the total number of red cars was different on different days – this happens because of that thing called variation. He found out that unless systems conditions change (e.g. due to a road closure), those little ‘X’s will continue to appear anywhere between the dashed lines.

Stick Child report cardIn addition to this, Stick Child and his friends learnt that targets don’t make any difference to the amount of red cars that drive past his school. This is because variation doesn’t pay any attention to man-made follies, such as numerical targets. He remembered his Dad always says that numerical targets are arbitrary and likely to cause dysfunctional behaviour.

Stick Child also discovered that there is no point drawing arrows between two isolated numbers because it gives the impression of trends that simply do not exist. Finally, he learnt that league tables are a poor way of assessing performance. For his efforts, the teacher sent a nice letter to his parents.

Even though he is only nine, Stick Child knows that control charts do not just apply to red cars or school projects. They can also be used by grown-ups, for things like crime figures, response times, or almost any other set of numbers you might want to learn about. (That’s if the grown-ups really want to learn about these things).

If he understands this, so can you.

Posted in Systems thinking | Tagged , , , , , , , , , | 11 Comments

Stick Child and the Flat Tyre

Here’s the problem – you have a flat tyre!

flat tyre

So, let’s look at a couple of options for resolving this issue…

Option 1

One approach might be to inflate the tyre, then check the air pressure on a regular basis to see if it starts to go flat again. If it does, you could put more air in the tyre. Perhaps if the tyre still keeps going flat, you could increase the regularity of the air pressure checks and fill it with air more often. If, despite this, the tyre still keeps going flat, maybe you could shout at the driver a bit. That should do the trick! Problem solved.

Option 2

Another option might be to see what caused the tyre to go flat. In this case it was a nail, so you could just remove the nail, then either repair or replace the tyre.

nail in tyre2

Option 1 vs Option 2

I prefer Option 2, as it addresses the root cause of the tyre’s flatness, rather than introducing disproportionate retrospective checks and downstream reactions which focus on the symptoms. Apparently, there are scientific studies that prove if you put a nail through a tyre, it is likely to go flat. Rocket science, this is not.

What the Pro-Nail Brigade Might Say

Now, Option 2 might seem logical to you and I, but there are people out there who disagree. For example, whilst you’re trying to remove the nail, they might approach you and say things like:

“It’s not the nail that’s the problem because…

  • - it’s the way the nail was inserted into the tyre.
  • - it must have been the wrong type of nail.
  • - it’s the driver’s fault for the way the car was driven whilst the nail was in the tyre.
  • - the nail needs to be there, otherwise air would leak out of the hole.
  • - lots of nails would be bad, but there’s only one nail in this tyre and that’s okay.
  • - but I’ve always put nails in tyres / they look nice.
  • - I use things that look like nails, but they’re called something else, so that’s okay”.
  • - it has to be something else, anything else. Nails don’t cause flat tyres!”

(Delete or combine as applicable)

If this happens to you, it’s always fun to ask what evidence these statements are based on, as such people tend to skulk off and leave you alone after that. (Or break into an incoherent outburst, then run away). In either case, this means you can fix your tyre in peace and quiet.

The same people usually think the solution is to leave the nail where it is and blame individuals, introduce tougher sanctions, invoke more audit and inspection, or produce a lot of meaningless numbers which they think will tell them about the air pressure. Unfortunately, this totally misses the point, as we shall see…

Stick Child’s Perspective

Stick Boy helloFollowing his popularity in my last blog post, Stick Child is back to offer a view on the nail conundrum. You see, he has been learning lots of cool stuff from his Daddy, and is becoming a bit of a systems thinker. One of the books his Daddy read him at bedtime was by a man called Deming, who said, “Cease reliance on audit and inspection”.

Being very bright for a nine-year-old, Stick Child is able to apply Deming’s concepts to everyday life and came up with these diagrams, which demonstrate the futility of relying on audit and inspection as a means of redressing problems caused by adverse system conditions, like nails in tyres.

Method 1

Method 2

His art teacher will be impressed!

Anyway, the point is that the best you can hope for by using Method 1 is to identify a problem and maybe send it back upstream. This causes extra work (the inspection itself, and then the rework) and can be very costly. It focuses on the symptoms rather than the cause. It’s equivalent to repeatedly checking the tyre’s air pressure and putting more air in it as it slowly deflates, whilst leaving the nail embedded.

As long as destabilising system conditions are present, no amount of audit and inspection will prevent dysfunctional responses, such as misreporting of air pressure, imaginative re-definitions of what a ‘nail’ can be classified as, or worsening damage to the tyre.

Method 2 is better because it addresses the system conditions that cause the issue. This means that we have an opportunity to remove them and resolve the problem. Therefore, if you keep encountering unwanted reactions within your system, simply trace the root causes and eliminate them. And yes, I’m talking about numerical targets, league tables and binary comparisons!

As I said in the previous blog post – if our little friend Stick Child gets this, so can you!

Further Reading

For those of you interested in how this applies to crime data, see my written evidence to the House of Commons Public Administration Select Committee here. You can also follow the progress of the #CrimeStats inquiry on the PASC twitter account @CommonsPASC

 

Posted in Systems thinking | Tagged , , , , , , | 3 Comments

Understanding Targets (For the Under 10s)

Stick Child 1This is Stick Child.

Stick Child is just nine years old. Awww.

Stick Child wants to understand about targets, so he plays a game…

Stick Child likes paper planes, so he sets himself a target of making one in 60 seconds. How did he get this number? Well, he invented it in his head because it sounds nice. This is also how grown-ups decide targets.

Now, poor little Stick Child tried his hardest, but he just couldn’t make a paper plane in less than 85 seconds. This made him very sad.

Stick Child 2When his Grandma asked him if he had been able to make one in less than 60 seconds, he did a naughty thing and told her that he had managed to do it, even though this wasn’t true. Afterwards, Stick Child realised this wasn’t the right thing to do and felt bad.

Later on, Stick Child’s Daddy asked him why he looked so sad. Stick Child said he felt sad because he hadn’t told the truth to Grandma and he was very sorry. Daddy sat Stick Child on his knee and explained that numerical targets often cause this type of dysfunctional behaviour, but that it didn’t necessarily mean he was a bad kid. Then he gave him a cuddle.

Stick Child 3

Better still, Stick Child’s Daddy showed him how to fold paper quickly to make really good paper planes that fly well. He also showed him lots of designs in a book, which meant that Stick Child could learn how to make lots and lots of really good paper planes.

Stick Child practiced making paper planes and he became very good at it. He found that he was able to make some planes in as little as 25 seconds; others took longer, because they were a bit more complicated, but that didn’t matter because they were really good planes.

Stick Child 4This made Stick Child very happy. He was able to show his Grandma his planes and this made her happy too. Stick Child’s Daddy told him that although his 60 second target sounded nice, it didn’t actually help him make paper planes any better or quicker because targets do not provide a method.

Now Stick Child knows the way to become good at something is to learn about that thing, find a way to do the thing really well, and always do your best. He also learnt that targets make people do naughty things, even when they aren’t bad people.

If Stick Child can understand this, so can you.

Posted in Systems thinking | Tagged , , , , , | 9 Comments

Why ‘Year-To-Date’ is Rubbish

‘Year-to-date’ figures are often used in performance frameworks, both in the public and private sectors. In policing, ‘year-to-date’ figures are regularly used to track the number of reported crimes at any given point in the year, supposedly as an indicator of whether the police are doing a good job or not. Now, leaving aside the debate about whether crime rates should be used as a performance indicator at all (I might talk about that another time), let’s have a look at why the practice of ‘year-to-date’ itself is rubbish.

Stick Person Goes For a Run

Imagine a runner…

stickman running

Here we are – it’s our little friend, the stick person. Let’s say the stick person likes to run 12 miles at a time, and his personal best is 120 minutes and 1 second; an average of about 10 minutes per mile.

Next time the stick person sets out on a run, he aims to do his very best and see how quickly he can run the 12 miles. He knows that multiple factors will affect his performance, such as his fitness levels, his current weight, his choice of clothing, his diet, and so on.

He also knows that other, external factors will affect his performance; for example, the outside temperature, the wind, the terrain and so on. He might be held up for a few seconds waiting to cross a road. His shoelace might come undone. Something else might happen outside of his direct control that affects his final time.

So, our stickperson sets out to maximise his chances of a fast time by ensuring the systems conditions he can influence are favourable. He trains. He doesn’t run in a bulky duffle coat. He avoids drinking 10 pints of Guinness and eating a massive curry the night before his run.

stick person on beer

Anyway, once he sets off on his run, the stick person is smart enough to measure progress, because he knows measuring stuff is vital. He wears a heart rate monitor which helps him check if his heart rate is within a normal and safe range. He checks his stopwatch every so often to gauge progress. He processes this information as he runs along, taking into account the context around him, knowing that if his heart rate gets too high, he will have to slow down a little; likewise, he knows that when he’s running along an uphill section of the route, he is likely to cover ground a bit slower.

The stick person uses all this information to ensure he is doing his best at all times. He knows that some miles will be faster than others, but is not unduly concerned because he understands this is normal. It might be that he beats his personal best this time, or it might be that he’s just a few seconds too slow. Either way, his objective is to continually improve.

Now, imagine he adopted the ‘year-to-date’ method to pace himself - a strict 10 minutes per mile. Of course, this ignores all the factors that can influence his speed at any given time. So what happens? Well, he completes the first mile in 9 minutes 55 seconds, giving him 5 seconds ‘in the bank’. Unfortunately, the second mile is partly uphill and it takes him 10 minutes and 20 seconds, causing an overall ‘deficit’ of 15 seconds to that point.

stick person failing

Now the pressure is on, so he speeds up a bit, but realising he’s still a bit behind time, he decides to sprint the last couple of hundred yards of the next mile. This makes him feel tired, but at least he makes up some time. This process repeats itself as he focuses on each individual mile, until he collapses at the side of the road, exhausted. Poor stick person.

Clearly, no self-respecting runner would prefer that method over the stick person’s original approach. But wait! Bizarrely, the equivalent of worrying about individual mile timings (and sudden sprinting) is prevalent in many performance management situations, as we shall now see…

The Trouble With ‘Year-To-Date’

The problems with ‘year-to-date’ are many, especially when today’s figure is compared to:

  • The average.
  • The previous year’s figure (or an aggregation of previous years’ figures).
  • An arbitrary numerical target.

Have a look at the table below -

Year to date table

Here we can see two performance years that ended neck-and-neck. (It doesn’t matter what the numbers relate to). Firstly, imagine the reaction each month as management compare the ‘year-to-date’ figure with the monthly average required to finish the year ‘on track’ - Cue a mix of concern/anger/confusion (when it’s higher), and feelings of success and self-congratulation (when it’s lower). All of this, as you can see, is a big waste of time because looking at the whole year in retrospect, both rows come in at 480 anyway.

This occurs because of normal variation – the fluctuations amongst the numbers are caused by all those internal and external factors that affect how the system performs, as in the case of the stick person. As you can see, variation even applies to systems or processes that are stable. Therefore, there is no point whatsoever in getting excited about whether a number is a bit higher or lower than the average at any given point in time.

“About half of everything is below average”, as I like to say…Hahaha! ;-)

The misguided belief that some meaning can be ascribed to the types of fluctuations I’ve just talked about leads to exhortations such as, “We cannot afford to record more than 135 crimes per day”, or “Sales must exceed £150,000 per week”, and so on. It causes people to withhold surplus units of whatever’s being measured until the next period. It causes under-recording and other bizarre practices designed to keep the numbers under control. This is where our stick person disregards his knowledge about his surroundings and begins to run flat out.

And that’s just comparing the ‘year-to-date’ figure against averages…that’s bad enough, but check this out – what happens when you compare it against last year’s ‘year-to-date’ figures? I’ll tell you – it gets worse!

Stick person chart

This is because – guess what – last year’s figures were subject to variation too! The numbers went up and down. Crime, sales figures, unemployment rates, you name it – none of them happened in a nice flat line. We have ZIG ZAGS, people; ZIG ZAGS! However, do not be alarmed – this is just normal variation again. So, when we try and compare this year’s ‘year-to-date’ figure against last year’s this is even dafter than making a comparison with the average because we are comparing two moving variables. Cue wider fluctuations and more panic…

And it’s all so meaningless. As you can see from the table, both years came in at 480 anyway. Imagine how quickly our stick person would burn out if he adopted this method of measuring his performance as he runs along.

Finally, we consider the comparison between the ‘year-to-date’ figure and an arbitrary numerical target – in this case a nice 10% reduction. As you can see from the table, the target was only achieved during two months. This is because someone invented it in their head, without having any understanding of the systems conditions likely to influence performance. It’s just like our stickperson suddenly setting himself a target to run 9-minute miles, when he has never run faster than 10-minute miles. Targets do not provide a method for achieving stated aims.

stick person targets

Oh, and that would just make our poor stick person collapse at the side of the road even sooner. Poor stick person.

Summary

All of these ‘year-to-date’ methods are incapable of telling you anything about performance. FACT.

Furthermore, they are all quite capable of inducing dysfunctional behaviour, as people mistakenly assume there must be a meaning for the apparent differences between the numbers (caused by normal variation), then change tactics to try and get the ‘year-to-date’ figure on the preferred side of whatever number it is being compared against.

‘Year-to-date’ obscures genuine trends when they do exist, causes false signals and mistaken assumptions, makes people ask the wrong questions about the wrong things, causes unfair blame and arbitrary praise, leads to short-termism, knee-jerking and a fixation on today’s isolated number at the expense of understanding what the actual influencing factors are. Oh, and you may have noticed - the whole approach is based on making binary comparisons, which are known to be very rubbish indeed.

So, if you use ‘year-to-date’ in your performance framework, do yourself a favour and ditch it immediately, then go out and do something useful with your data instead.

Take some tips from the stick person!

stick person thumbs up

Posted in Systems thinking | Tagged , , , , , , , , , , | 13 Comments

The Tunnel

A teacher once angrily called me one of “the three most stupidest boys in the school”.

This is what happened…

At the age of 10 or 11 I went on a ‘school trip’ to a park about half a mile down the road from my junior school. Being 10 or 11 as I was, the attraction of the local park was quickly overtaken by the prospect of sneaking off with two of my pals and climbing down an adjoining grass embankment onto a disused railway line. This was good.

Better still, not far along this particular disused railway line there was a disused railway tunnel. Now, if you put a trio of errant 10 or 11 year old boys in the vicinity of a disused railway tunnel in close proximity to a mundane school outing only one thing is going to happen…

Tunnel 1

We went to explore the tunnel.

Tentatively at first, I remember the three of us standing at the black mouth of this great hole in the earth. Hesitating before the unknown. Listening for the slightest sound as we slowly edged forwards. Straining our eyes against the darkness. Imagining the trains that thundered through this chasm decades beforehand.

We advanced, literally shoulder-to-shoulder. We felt trepidation, fear, excitement. We crept into the tunnel not knowing what dangers it may hold for us. We knew we were taking a risk; at the very least we’d probably find ourselves in trouble if we were caught.

But as we marched onwards our eyes grew accustomed to the dark. We gained confidence, picked up speed and became surefooted. We stayed close together and the fruits of our joint enterprise soon materialised, in the form of the light at the end of this particular tunnel.

At that point we knew we’d made it, and that we’d only succeeded because we’d taken up the challenge together. I don’t think any one of us would have done it alone.

Tunnel - light at the end of

Heading back along the tunnel was easy. In fact, I can’t remember much about the return leg, except that as we emerged into the sunlight for the second time that day, our teacher was scrambling down the embankment, angrily gesticulating towards us and shouting, “It’s the three most stupidest boys in the school!”

I’m not sure the teacher’s outburst was technically grammatically correct, but the lesson here is about friendship, confronting fear, facing uncertainty, and being prepared to take risks, rather than accurate use of English grammar. Perhaps there’s something in there too about turning out okay when we grow up, I don’t know.

Final thoughts

I think this story could be an analogy for many things. If you can relate to it in any way, that’s brilliant and I’m glad it can have some meaning for you too.

From a personal perspective, I can’t help but think of the enormous progress there has been in UK police forces during the last year in respect of challenging some longstanding performance management practices (e.g. arbitrary numerical targets, binary comparisons, blah blah blah…) Many are now realising a much better future can be realised through ditching all that stuff and using our data/information more maturely.

More and more forces and individuals are gathering at the mouth of the tunnel; several are striding boldly into it. Progress is happening because no one is facing the challenge alone.

I hope 2014 will be the year where we collectively see the light.

Disclaimers

1. The pictures are not of the actual tunnel in this story – I found these great photos here, so all credit goes to the photographer, ‘Big Lew’.

2. Don’t try this at home kids. I was a professional 10/11 year old when I embarked upon this adventure.

Posted in Systems thinking | Tagged | Leave a comment

The Railway Children

Recently, I had the opportunity to have a look around a railway network control room. It looked a bit like this:

railway control roomPretty impressive eh? What struck me was what a fantastic example of a system the railway network is. Think about it – all those trains having to be coordinated so as to ensure the overall system functions effectively. Train crews need to be transported around the network in time to operate other train routes; certain trains need to be in certain places at certain times to facilitate the effective running of these routes. Every part of the system depends on every other part.

On top of that, different services sometimes need to have their carriages coupled and uncoupled at particular stages of their journey to enable trains to be split and sent in different directions. The arrival of services at railway station platforms must also be coordinated, taking into account the size of available platforms and many other factors. Signalling and other equipment has to operate effectively and trains must pass through points at the right moment to prevent congestion or delays.

If an incident occurs that causes a delay, then it’s almost inevitable that there will be a knock-on effect. The challenge then is to minimise the potentially cumulative impact of such an incident. This could involve re-routing particular services, or slowing others down so that they arrive at different points at different times and in a different order to the original plan. Essentially, the result is a dynamic reconfiguration of the system’s activity, intended to minimise disruption following an unforeseen event.

This is such a great example of a system because it highlights the importance of interdependencies, with the screens in the control room visually displaying real time performance information. If one small part of the system fails, there is likely to be an impact somewhere else in the system. If a state of balance is not restored quickly, then further consequences occur as other parts of the system begin to fall out of synch. The result is an accelerating snowball effect that can cause absolute chaos (as many travellers will be aware).

Having that ‘whole system’ view is necessary to maintain an optimal balance, i.e. homeostasis. However, in many traditional organisational structures, this crucial perspective can be obscured due to silo-based models and fragmented working practices (see this blog post for more), resulting in sub-optimisation and a breakdown in cohesion. Worse than that, where counterproductive performance management practices (such as the use of arbitrary numerical targets and league tables), and superficial use of data (e.g. binary comparisons) are present, this inadvertently encourages behaviour that damages the system and destabilises the natural balance required for it to function effectively.

As a result of not understanding systemic interdependencies and clumsy use of performance data, the risk of fundamentally misguided interference by managers is heightened, leading to knee-jerk reactions and an untold cycle of damage.

Staying with the railway theme, someone recently likened this behaviour to letting a bunch of children loose in an old-style signal box, pulling on the levers randomly without understanding what the levers do – perhaps even thinking they are making a positive difference. Meanwhile, the wrong points are being opened and closed, trains are being sent in the wrong direction, and maybe some are even crashing into each other, far away down the line where the consequences cannot be seen from the signal box.

railway signal levers

The question is this:

“Which approach to managing the system is dominant in your organisation?”

Posted in Systems thinking | Tagged , , , | 8 Comments

Angry Driver

Do you know anyone with this approach to performance management?

angry driver

Actually, it doesn’t work.

Posted in Systems thinking | Tagged , , , , | 3 Comments