Cast your mind back to Episode One of this series. No beer, no food, no chillies; just trains. The examples provided in that blog post highlighted the dire (but typical) methods used for presenting ‘performance’ information. One of my readers, (Ian Gilson) responded with an enlightening post of his own, which delved into railway performance measurement in more detail, demonstrating why the headline figures presented for timeliness can be misleading.
Anyway, I am currently visiting one of my favourite places in the world – Whitby, North Yorkshire – and whilst pottering about near the train station yesterday, I encountered another such performance board for train services. This is what I saw:
At first I noticed the same sort of pretty useless stuff as seen in my previous blog post, i.e.
- Comparisons between different work groups. This ignores local context and those system conditions existing in different parts of the overall rail network which affect performance. Also, cue individualist behaviour and sub-optimization (see my previous post) as work units strive to out-do each other at the expense of the overall system.
- The appearance of the inevitable asterix that relates to the definition of what ‘on time’ means in the eyes of the ‘Customer’s Charter’, along with the small print underneath. (Probably not what ‘on time’ means to the customer, but there you go).
- The declaration that, ‘All our service groups exceed the levels required’. In other words, arbitrary numerical targets have been achieved. How they have been achieved, or whether they correspond with what matters to the customer is unclear.
- That lost-looking pair of numbers on the second poster under ‘Customer contacts for the period’. Look – it’s this year vs last year again! A binary comparison or two is never too far away from this type of performance measurement. Anyway, what does it mean in this case? I’d have thought every person who buys a ticket would count as a ‘customer contact’. If it means something else, such as the number of complaints, letters of praise, or enquiries, why lump them all together like this? It makes no sense. Is this year’s lower figure supposed to be better or worse? No one knows.
None of this stuff actually tells us anything about performance. Furthermore, some of it actually risks generating the wrong type of behaviour. It’s not at all useful to the weary traveller who might either be sheltering from the rain and seeking inspirational reading material, or just trying to come to an informed conclusion about whether railway performance is any good. It’s data without context.
Now, on the other hand, I have to take my hat off to one aspect of what the railway people have done here. Look at the first poster. There’s something there that was completely absent from the performance document examined in Episode One – a bit of context. Some narrative. Some information about why performance data is what it is. Look closely – at the top right hand side of the first poster, there are reasons given why things happened:
- A flood.
- A vehicle hit a railway bridge.
- Signal failure.
There’s even dates! This is great to see, as it puts a bit of meat on the bones of the raw data, and highlights the fact that it is not always possible to achieve perfection; most importantly, it acknowledges that any shortfall is usually partly due to a range of external factors that affect performance. If a specific line’s performance for the month was adversely affected because of severe flooding on a particular day (and trains therefore failed to achieve their timeliness target), it would be wrong to castigate that work unit for ending up at the bottom of some league table. Conversely, if by looking beyond the raw data (in context of course) it becomes apparent that there is a repeated problem with the same set of signalling equipment, then the train company needs to take action to resolve that recurrent issue.
In my ideal world, the data would be presented using control charts (which give context within the data) alongside the narrative (which gives context around the data). Russ Ackoff would say that useful narrative accompanying badly-presented data (as with the overall presentation of these railway posters) is just doing ‘less of the wrong thing’, but hey, I’m trying to accentuate the positive.
The same applies to crime rates and other police data. There is little currency in presenting rows of numbers detailing total arrests or detected offences per team, often supplemented by worthless ‘this month vs last month’ binary comparisons. Measures should be used to help understand the performance and capabilities of the system, with a view to identifying opportunities for improvement. Simply presenting data in the traditional format does not achieve this, be it for train services or policing.
Conversely, if the right measures are presented in a useful format and with context, this enables managers to understand why things happen and what to do to improve performance. For example, if police managers were pondering an apparent change in crime rates or reported incidents, it would be useful to understand the narrative and context behind the numbers. This approach could indicate a description of factors such as:
- “That was when we did a proactive policing operation and caught loads of people committing crime, thereby increasing the volume of reported crime”.
- “Incidents were unusually high that week – as it happens we had fewer officers on duty available to police the town centre”.
- “Criminal Damage offences went off the scale for that area because one individual went on a tyre-slashing spree one night and damaged 30 cars”.
By understanding the context of the data (along with being statistically literate when it comes to interpreting the numbers in the first place), managers can begin to unearth the evidence base that either provides a mandate to change tactics, or not to knee-jerk. It also means more to the public, as opposed to “we had ‘X’ amount of things this year compared to ‘Y’ amount last year, and by the way, the target was achieved”. I’ve always found the public tend to be quite reasonable when they understand why something has happened – this applies as much to when a train is cancelled because a third party has crashed into a railway bridge, as it does when poorly-presented data invites either hysteria or complacency over crime rates.
At least some of the content of the board on the wall of Whitby train station suggests a step in the right direction. This is data with context. It makes a nice change.