Troubled Thinking

Well, the ridiculous examples of bad performance measurement keep flooding in. I’ve lost count of whether we should be on Episode 5 or 6 by this point, so I’ll just stick to new titles from now on. This time, I’m going to look at aspects of the Troubled Families agenda, after one of my readers sent me some very interesting documents…

The Department for Communities and Local Government (DCLG) defines troubled families as those who:

• Are involved in crime and anti-social behaviour.
• Have children not in school.
• Have an adult on out of work benefits.
• Cause high costs to the public purse.

Now, unlike some of my other posts, there is a very serious edge to this subject. If you’re just looking for a giggle about performance measurement in the toilet-cleaning world or on railway networks, then in the words of the character Morrell from Shane Meadows’ film A Room for Romeo Brass, “Why don’t you just leave, before things get really dark?” (It’s a great film by the way if you like that sort of thing).

We’ll start at the thin end of the wedge…

The Trouble with Targets

I’ve done this to death before, so rather than unpick why the following targets are daft, I’ll just show you them and let you work through the checklist that follows this excerpt from a document about troubled families.

“The education outcome is all children having fewer than three fixed-term exclusions and less than 15% unauthorised absences from school in the last three terms; the ASB outcomes are a 60% reduction in ASB and a 33% reduction in offending rates for under-16s in the last six months”.

(I particularly enjoy the 60% ASB reduction target. Apart from the insanely wide definitions of ASB that render accurate measurement impossible, this is another classic example of ‘our plan aims to ensure 40% of ASB remains’. Ha!)

Anyway, have a crack at the checklist.

Here’s what your completed table should look like:

Not too hard was it? Right, now onto the serious stuff…

The Trouble with Outcomes

First of all, let’s look at the significant limitations around the popular performance measurement term ‘outcomes’. There is a growing body of evidence that highlights the difficulties in defining and measuring outcomes. As true outcomes are usually difficult to quantify and measure, outcomes-based accountability programmes tend to rely on proxy indicators which are unsatisfactorily crude and primitive.

Furthermore, when attempting to tackle cross-cutting social issues it is notoriously difficult to establish a causal link between a unit of activity undertaken by a particular agency, and an eventual outcome. External environmental factors can also bear both positive and negative influences, thereby limiting the efficacy of claims that a single agency or specific intervention is the responsible factor in achieving the outcome.

Then there’s the question of who determines when the outcome has been achieved. Studies have shown that measuring the effectiveness of an intervention produces different results depending on when the evaluation takes place – i.e. efforts at determining whether the outcome is successful 6 months, 12 months or 2 years after an intervention will produce conflicting results. Which one (if any), is the correct point to stop the clock and declare success?

The Trouble with Payment by Results

Now here’s the rub. At the opposite end of the scale to the light-hearted jibes directed at silly performance indicators for dog poo, there exists some really dark thinking about how we should approach issues at the serious end of public sector business. In the case of the Troubled Families programme, the whole agenda is undermined by a particularly dangerous pathogen – Payment by Results (PbR).

The problem is that the whole ethos of PbR rests on the assumption that the considerations highlighted in the ‘Outcomes’ section of this post can be surmounted, along with spectacularly misguided confidence that the financial incentives offered under such schemes will not cause dysfunctional behaviour.

As you probably know, I argue that targets alone generate perverse incentives which lead to goal displacement, gaming and other horrible consequences. When bolstered by the prospect of hard cash for ‘results’, it is inevitable that many of the most unpalatable effects of target-driven performance management will be intensified. Extensive research across the social sectors, (in particular education, employment, health and policing) consistently indicates that adverse consequences are ubiquitously borne of public sector PbR programmes. To see examples of this for yourself, I highly recommend the following paper, which explores the phenomena in some detail, utilising a range of evidence:

Holding Accountability to Account (Rothstein, 2008)

Examples from within this particular article resonate with many of the very areas that the Troubled Families programme aims to address, such as educational achievement and work placements for the unemployed. It warns of ‘cream skimming’ (i.e. cherry picking those who are most likely to succeed against the PbR criteria; often at the expense of those most in need), as well as other behavioural risks that result from inappropriate performance incentives. It’s not pretty stuff.

The mind boggles at the prospects for the Troubled Families agenda. If previous experience is anything to go by, I fear for those families just below specified thresholds or slightly on the wrong side of definitions. I worry that the best efforts of caring individuals will be hamstrung by distorted priorities and wrong-headed cash-linked performance incentives. I am certain that the PbR-based approach will lend itself to overlooking purpose and marginalising many of those whom the programme is supposed to help.

One more thing – Please understand that this is not a criticism of the aims of the Troubled Families programme (and definitely not a slight on those committed people who are doing their very best to address the important and emotive issues it encapsulates), but rather an emphatic and considered health warning about the associated performance methodology which risks completely derailing those virtuous aims.

So there you have it. Not as pithy and light-hearted as usual, but as Morrell says in that film, “I can’t destroy you today lads – I’m on very serious business”.

See you next time.