One of the key tenets of systems thinking is understanding variation.
Deming said:
“Life is variation. Variation there will always be, between people, in output, in service, in product”.
This applies to weight loss as much as everything else. People who are trying to lose weight often become frustrated that despite their best efforts, the scales mock them, telling them they haven’t lost any weight since they weighed themselves yesterday, or worse still, they have gained a pound or two. This can also apply when comparing one’s weight to this time last week, or any other single point in time.
That’s the problem, you see – making a comparison to one other solitary value is meaningless. This is the case for a couple of reasons:
1. You are making the comparison against a moving variable that merely captures an arbitrary point in the past.
2. The comparison does not take into account external factors that affect the data. For example, in weight loss terms, several factors can influence what the scales tell you, such as the time of day you weigh yourself, your body’s degree of water retention at that moment in time, or how recently you have eaten.
This type of comparison is called a binary comparison and it tells us nothing. The method is flimsy, meaningless and basically rubbish. Here’s what one looks like on a chart:
Pants, isn’t it?
To establish whether there are any real trends in a set of data, the best way is to plot several data points on a control chart, such as in the one below:
Basically, if the data points don’t fall outside of the control limits (or form any of a handful of specific patterns that I won’t bore you with), then what you see is normal variation. Normal variation is normal!! Sometimes the numbers go up; sometimes they go down. There is always a degree of fluctuation in anything – the number of red cars that drive past per hour, the crime rate, your weight.
Here it is clear to see that the person to whom the data pertain has experienced a steady rate of weight loss over a three week period. But let’s go back to the traditional binary comparison method that people tend to use when trying to establish if they are losing weight. For the first couple of days our subject would be overjoyed to discover that their weight loss programme appears to be working. Unfortunately, this would be followed by shock and disappointment that he or she suddenly gained three pounds! Whaaat??!!
Worst of all, on the sixth day of the programme, the scales indicate the person is two pounds heavier than the previous day and a pound heavier than when they started!! How cruel. Imagine their sense of frustration and disappointment. “Stupid scales!” “Waste of time!” “The diet’s not working!” Recognise any of this?
Over the next few days, our subject experiences a loss of a few more pounds and would probably feel a bit happier. Then, on the second Friday of the programme, he or she records a ‘sudden’ and ‘massive’ weight gain of three pounds. (Stupid scales, stupid programme, not fair, etc etc). And so it continues. Some days the scales bring good news; other days frustration, based on whether the numbers are slightly higher or slightly lower than the previous day, or as compared to any other given single point in the past, such as ‘this time last week’.
This demonstrates why this method of comparing data is fundamentally flawed. A steady and identifiable trend of weight loss becomes obscured by honing in on just two readings. People become unhappy and wonder what they are doing wrong. The answer is probably nothing. It’s just normal variation, so stop worrying about it. In the same way, those trying to lose weight feel elated when the scales show an apparent decrease. Don’t fool yourself though – if you’re comparing your weight to yesterday the ‘decrease’ is just probably normal variation again. In the same way – should you experience a ‘sudden’ or ‘massive’ increase, remember that it’s neither sudden nor massive. Making a judgment on two data points is always unsound. Don’t do it!
Even if you don’t know how to construct a mathematically accurate control chart with control limits, just plot your weight loss data on any old chart over time. You will see ups and downs. That’s normal. Hopefully, over time, if you are doing the right things to lose weight, you will see a steady reduction. This is impossible to see when using the binary comparison approach.
Once you interpret data in this way you will see the bigger picture; the long term; the truth. Guess what – it goes beyond reading scales too. Listen to the news – “Teenage pregnancies are up 6% compared to last year”. “Unemployment fell by 15,000 compared to last month”. “Sales figures are up compared to last quarter”. “Crime is down compared to this time last year”. It’s all meaningless!
So, accept that all binary comparisons are naff and learn to understand variation instead.
InspGuilfoyle 
Hi, you’ve just said exactly what top S&C and nutritionist. John Beradi (Canadian) has been telling me for the last 18 months! He also says the scales are just a number and can only really mean anything when taking in conjunction with other measures such as skin folds, body measurements etc over a long period of time. 18 months in and 5stone and many many clothes size smaller maybe youve both got a point!!
Still get stressed though when the scales tell me the “wrong” thing!
Like your blogs, always look forward to new ones.
By the way I did realise that this wasn’t necessarily just
about weight loss. Just found it interesting in context of what I’ve been doing and how using one form of measurement isn’t necessarily correct either in many fields let alone using data in the way we do. Errr. I’ll stop rambling, just didn’t want you t think I was thick!!!!:-)
I absolutely agree with “plotting the dots”. However I’ve been taught that you can’t use an SPC chart for weight as the data is autocorrelated (the data points aren’t independent of each other). You can use a run chart instead. You could use a control chart if you are measuring the change in weight but then you end up with a chunky data problem (and perhaps a chunky thigh problem as well………..)
And policing has lurched from one set of media inflated politically manipulated data to another for decades. Is it any wonder we’re where we are now?
Sorry Mr G but you seem to have an obsession with Control charts, I bit my tongue last time but can’t resist any more.
Let’s look at your control chart, you have a nominal centre line that has no functional value since it is not a ‘target’. The lower and upper control lines therefore are also arbitrary, even if they are 3sigma from the centre. There are no warning limits (at 2sigma) which is a relief as the actions could only be ‘eat more pies’ or ‘amputate surplus limbs’.
Effectively the real problem is you are trying to measure a change in the value of W that is smaller than the random variations in the system (noise). If you were only to measure once a week the trend is still clearly visible. A simple line graph would do, adding complexity is a waste.
There is a false notion put about by ‘spreadsheet managers’ that if measurement is useful then even more measurement would be better. It’s not true you simply waste time and resources collecting meaningless data.
To dismiss all binary comparisons as meaningless is folly, as they are both the first and last step in most decision making processes. And to measure a trend, you are actually making a chain of binary comparisons. Though it is a very blunt tool and should be used with great care.
Any report that says “Crime” is up or down is nebulous anyway since it gives the same weight to different crimes. Murder is counted as one crime so is littering, dog fouling etc.
And as is traditional, Chilli tip.
Don’t just go for really hot ones, drop down the Scoville range and use more lower powered ones, you will get better flavours and the benefits of eating more fruit as part of the 5 a day.
Hmmm, I have to say I really like the flavour of Scotch Bonnetts despite their heat…
As for the charts, a line graph, a run chart (as Paul H pointed out), or ‘any old chart’ as I put it in the post, would be absolutely fine for this type of exercise. As most of my posts have some relevance to policing (in this case how crime figures are presented, i.e. usually as binary comparisons) this was the reason I focused on control charts in this post – they are the most useful way of presenting crime data. Perhaps I used a bit of artistic licence here for weight loss, technically speaking, but the principle is bang on.
You mention the centre line having no functional value as ‘it is not a target’. Targets have no functional value. The centre line and control limits come about as a result of calculations relating to the data used to construct the chart, as I’m sure you know. This means that their positioning is far from arbitrary. Personally I never bother with 2 Sigma warning limits – either there are signals in the data or there aren’t. In this case there are none, but that doesn’t mean that the values being plotted are worthless.
I agree that excessive amounts of measurement and data collection represent waste, and I am also with you on your points regarding crime data, but I don’t agree that it is folly to dismiss binary comparisons – to understand what the data are or aren’t telling you, it is essential to interpret them in context and this is impossible using just two values.
Dear MrG Thank you for a full and gracious reply. Though why you waste your precious spare time talking to random people on the internet, baffles me. But each to their own, I used to be a slave to Birds Eye’s until a friend clued me up to the thumb sized green ones. Heat, sweet and don’t wreak your ring quite so badly.
Oh and the phrase ‘spreadsheet manager’ was not aimed at you, I know you still get your boots dirty.
I’m going to respectfully disagree, I fail to see how using a Control chart which is designed for industrial process control and only monitors a single dimension is in any way relevant to crime. A quick count on the fingers says crime has 5 dimensions.
I get a vague feeling that you have collected the data first and then wrapped the chart round the data, rather than setting up the chart first and watching where the data falls. I might be wrong there.
Grab a brew, this may take a while.
Let’s go back and look at what the control chart was designed to do in the first place. In a production process there is an optimum condition as the outcome of the process, lets call it the target, for want of a better word. Now in each process there will be variation, so real world processes have tolerance levels +/- to the optimum.
The classic example is producing a cylindrical shaft with a diameter of 50.0 +/-1.0mm. So the control chart would have a central axis based at 50.0mm I.e. A target with a functional value.
Before each finished shaft is dismounted, a measurement is taken and recorded. When the operator set up the process, they would have carefully adjusted the tools to produce shafts of 50.0mm. As individually parts are made there will be small variations and looking at the chart, you should see the data points as a narrow scatter around the central axis.
It’s a bit late in the day to calculate the exact numbers, lets assume we are warning at +/- 0.8mm. And the upper limit is +/-1.0mm. I know rough as a badgers @¬$~ but let’s go with it.
In the classic example the process runs, and there is normal scatter, until tool wear starts and then the scatter band starts to drift up towards the warning level and once the number of results above that exceed the set limit the process either stops or signals and alarm. The other condition is tool recession where the cutting edge has been broken or jolted back into the fixture and the result jumps above the control line. That usually triggers a shut-down.
That’s how it is supposed to work, a real target, and warning and control lines based on acceptable deviations from that target.
Lets hammer the slimming example a little bit more. There is no target weight, eg. “I need to be 150kg” and so the central line just seems to be plonked at the median of the data set, and the UCL and LCL are calculated based on the basis that they are always 3sigma. Unless there is a reason to align the central line to the median, it is arbitrary, why not align it at 0.25 or at 0.75?
What happens if the notional slimmer hits the LCL? It should trigger an action. Do they get a big cake?
Lets leave that example crying in the corner because nobody loves it. And turn to crime, (not literally, though it does sound easier than working for a living).
We want a control chart for crime, and so we are not going to lump all crime together because that would be very silly. Let’s pick say, Serious Acquisitive Domestic, or whatever its called.
We need a target, now for crime that should be 0, (you know and I know that’s not realistic, but it would be political/career suicide to admit it. But let’s see what happens.)
Centre line =0
So can’t have negative occurrences of crime so no LCL. Where do you put the UCL? Given you have no direct control over the variables that contribute to crime, (they are many and the police are few and sadly getting fewer.)
Heavens, Let’s try and wrap this up, it was daylight when I started.
I don’t think I said the values were worthless exactly. What I said was they were not accurate, because they are the sum of both the true value and an unknown level of error. (I would describe it terms of signal to noise ratio, but we would be here all night.) Lets say a measurement is the true value summed with an unknown error caused by external factors between 0 and 10%, until the change in true value is >10% it will be indistinguishable from the error. And by reducing the sampling frequency it allows for the magnitude of the change to swamp the error.
Binary comparisons are unavoidable, and yes it is wrong to compare two single points in isolation. But most decisions resolve to yes/no, more/less, sooner/later, bigger/smaller etc.
If I still haven’t got it, please post a link to a decent training web page and I might eventually catch on.
Stay Dry.
Should it not be the case that unless the centre line, UCL and LCL represent specific targets and tolerances, that the centre line should be the mean average and the controls limits a chosen multiple of the standard deviation?
The centre line, UCL and LCL should never represent targets. The position of these lines is only ever derived from the data. As you say, the centre line is the mean average. The UCL and LCL are determined by establishing the median value amidst the data and using that figure to place them at 3 standard deviations either side of the centre line. This stps humans messing with it and treating any of the lines as targets. At least the data are objective…
Absolutely agree with you that targets have no functional value, and how could yo mo say that control limts are arbitary. What nonsence. Sorry trying hard not to flame!!
I didn’t mean to suggest all control limits are arbitary, just the ones in the example given.
Morning! You’re right- the data came before the chart (thats how it’s constructed). Whilst I take your point about crime being multidimensional, I find control charts very useful in preventing knee-jerk reactions and aiding evidence-based deployments. To go into that here would fill the page, but if you want to give me a call (you can get me via the WMP switchboard), or better still if you are in WMP yourself by any chance pop in for a coffee, I’d love to chat in person.
Evening All (sorry couldn’t resist)
No long reply today, been a pants week, and I need to run off and bathe in mountain air for a few days to get my head right. I’ve had further thoughts and your confirmation that you wrap the chart round the data, sort of confirms a notion that came to me earlier. But I can’t quite remember what it’s called, it might be a ??? moving p interval ??? but I’m not sure, and if it is then that raises a whole new set of questions.
Thanks for the coffee invite, but I’m a soft, southern, shandy drinker and it’s a bit far to go for a Max-Pax. 😉
Back Monday, until then have a short video from a chart genius.
Sorry matey, if you’re going down the lines of p-charts, np-charts, c-harts, u-charts, Binominal or Poisson models next, then I will bow down to your greater expertise, as that stuff goes way over my head. In my opinion, X-charts and XmR charts are absolutely fine for this type of simple everyday application. The phone call offer is still open if you want. Have a good weekend!
No I wasn’t going to go that far, it’s alright paddling near the shore, but it gets very deep, very quickly. So to sort of recap, ‘you utilise control charts as a tool to spot statistically significant deviations in a series of events.’
What a novel idea, my initial unvarnished reaction was “Hmmm, that’s either interesting or bananas”. Second reaction, on reading your reply is “need to know more about X and XmR charts”. If I go suddenly quiet, I’ll be reading up.
And if you want to baffle people leave a few of these lying about your desk.
http://en.wikipedia.org/wiki/File:Smith_chart_gen.svg
There might be a better way than ringing the nick, let me ponder it a bit.
Thanks for your considerable patience.
Great post. The principle could apply to all sorts of human endeavours. Take economic booms and busts. During the boom most people thought it would carry on forever. Now we are bust lots of people think it will go on forever unless we do something revolutionary. The boom ended, but we seem to fail to realise that the bust will too, eventually.
Take “how we think we are getting on with our life” as a more nebulous concept. If you’d asked me last Thursday I would have said “yeah, pretty good” and on Friday I would have said “oh Lord everything is terrible”. If I’d assumed that the trend was to continue I might have expected to be in the ground by Sunday. The trend turned and approaching this weekend I’m trying not to assume that I’ll be a millionaire by the middle of next week…
Interesting video. The factors that increased LE and wealth are the spread of western-style democracy and free trade. For that reason, I don’t completely share Rosling’s admirable faith in the future.
I have a print of Charles Joseph Minard’s classic ‘Napolean’s March’ on a wall in my house ( http://www.edwardtufte.com/tufte/posters ). And you thought you were a geek.
Good choice, hope Canada continues to treat you well.
For reasons of artistic choice rather than accuracy I have a
OK, the debate here seems to be all about the value of binary comparisons compared to longer-term trends. It’s great to know that you are 5% down this time compared to last time (whether that’s your weight or the crime levels in your division), but perhaps not so great if the time before that you were 10% up. So more data points give you a better picture.
But there are some other pieces of information that are important to a systems thinker, I suggest. The above chart is just telling me that this person’s weight fluctuates on a daily basis, but tends to be around the 140lbs mark. Now, is that good? Could or should your weight be different? How much different?
Your weight can be controlled by diet and exercise. But there are some basic factors that determine what your optimum weight could/should be.
– Some of these factors are not within your control, despite your best attempts (eg: age, height, gender), but are very relevant to your ability to manage your weight. Do you have any disabilities? Do you have any health issues?
– Some factors can be controlled, but it may be difficult to do so (eg, does your job involve manual labour, or are you desk bound? do you have to drive to work, or could you cycle/walk? Do you have time to exercise?
– Some factors are very much within your control. Diet. Exercise, either for weight loss or weight gain (muscle weights more than fat!).
– You can also do some extreme things to cut/gain weight – spend an hour or so in a sauna or sweat-suit, and you can easily lose 2 or 3lbs. Have a pint of blood taken out, and you are instantly 1lb lighter. Your weight also changes during the day, so you could easily have a chart like the one above just by getting on the scales at different times.
So, if you are to lose weight using systems thinking, I’d want to know a few more things first (ie, study the system). Your doctor wouldn’t tell you to go on a crash diet if you are well within your healthy weight range and then set you a target of weighing less than a group of people who are younger and shorter than you. But after checking other health indicators, he might wisely advise you to improve your fitness levels.
Relating it back to crime and policing, I’d want to know more about factors that influence crime levels, and the degree to which the police can control them. (One of the easiest ways is to not record it in the first place, but that would be cheating!) There’s not much point setting a target if it’s beyond your control and/or simply based on being lower than everyone else. But it would be sensible to ask you to be as good as you can.
Spot on once again. The data give you the evidence base to understand the system, and then take action on the factors that you can influence to improve it. With weight, it is crucial to understand the multitude of factors that can affect the variation, as you point out. Taking extreme measures may produce scales readings that are more pleasing to the eye, but this is likely to be short-term and may cause an adverse long term effect.
‘Being as good as you can be’ is an excellent mantra. It surpasses any numerical target for a start…
Mr Guilfoyle I’m no expert, but I think the above weight loss example is flawed. If this time last year I weighed 12 stones and now I weigh 10 stones, then who cares about the bit in the middle? The point is I’m 2 stones lighter! I’ve compared two data points and guess what it tells me something useful, that I’ve lost weight.
Bob – I kind of agree with you to an extent: If your aim was to lose weight so as to be fitter, healthier and feel good about yourself, then by being two stone lighter than when you started your weight loss programme you have achieved your purpose and that can only be a good thing. I’d still say that by understanding how you were going during the programme would be useful though – if you started plotting daily progress on a control chart after the first couple of months you would be able to see the gradual weight loss you were aiming for.
If you were able to correlate particular redcutions with particular exercise regimes or diets then that would help inform your decision making about what seems to be working. Conversely, you might notice a temporary trajectory of weight gain after Christmas, but understand what has caused it!
Being able to see gradual changes in this way helps plan your weight loss programme more effectively and should stop you feeling disheartened when inevitably, you appear to gain a couple of pounds here and there for no apparent reason along the way, due to normal variation.
Also, the binary comparison of today vs last year might be quite misleading if all that data in between is missing. It might be that a month ago you were actually 2.5 stones lighter and this was a healthier weight for you. Since then something has happened and you’re gaining weight consistently, whilst assuming victory because you’re two stone lighter than a year ago. Or, you might have lost two stone in the first three months then levelled out. Or you might be regularly yo-yo-ing between 10 stone and 12 stone every 3 or 4 months. This would be impossible to tell using the binary method.
Hope this makes sense!
Pingback: InspGuilfoyle | Police Inspector and Systems Thinker - Daylight Station
Pingback: The Curious Case Of The Chart That Didn’t Bark In The Night | thinkpurpose