PsyFi Search

Sunday 5 April 2009

Regression to the Mean: Of Nazis and Investment Analysis

Francis Galton, The Measuring Man

Francis Galton, cousin of Charles Darwin, was utterly uninterested in the both the workings of investment analysts and Hitler’s future plans for world domination. To be fair he died before both really got going. However, he has the dubious distinction of being at least partially to blame for some of the madder excesses of both.

What Galton was interested in was measuring things: mainly people, although he’d turn his hand to anything in the absence of a likely suspect or two. In so doing he uncovered a statistical phenomenon known as Regression to the Mean which lies behind many of the theories which still dominate stockmarket analysis and valuation techniques today and which many people misunderstand totally.

Galton’s Eugenics

To deal with Galton and Hitler first, however. Galton was the inventor of eugenics, a branch of science dedicated to analysing the fitness of the human species with, as its goal, the aim of improving the race. Like many a Victorian gentleman Galton was convinced that humanity was regressing into a less intelligent and less developed state. In fact what he was actually observing was the impact on people of moving them into unhygienic cities, working them in appalling conditions and giving them a lousy diet. Tricky stuff this nurture, nature thing.

Galton’s measurements were essentially designed to show that the rich and powerful were clever and the poor and weak were not, thus squaring the circle: the rich and powerful were naturally bound to ascend to the top of the tree and therefore had every right to be there. Nothing at all, then, to do with the fact that your great-great-great-grandmother had an illegitimate child with the King, was set up with an estate comprising half of Scotland in order to go away and stop bothering him and you inherited it by accident of birth. Clearly not.

Galton’s Statistics

Although Galton’s logic may have been a little faulty when it came to measuring such subjective things as intelligence, when he turned his attention to more objective stuff like height there wasn’t much wrong with his statistics. In a set of experiments on the size of seeds of successive generations of sweet peas he noted the odd fact that the larger seeds of one generation tended to result in smaller seeds in the next generation and vice versa. Applying a similar analysis to people he noted that although short people tended to have short children and tall people to have tall children the short children tended to be slightly taller than their parents and the tall children slightly shorter than theirs.

This led Galton to postulate the principle known as Regression to the Mean. What this says is that is an abnormally large value of a variable – let’s say it’s the return on the stockmarket in any given year – is likely to be succeeded by a lower value. You could say that it’s the statistical proof of the old saving “what goes up must come down”. However, what Galton also showed was that the results of such analyses tended to cluster around the mid-point – the so called mean. The average value if you like. Over time it’s this average value that all returns will converge on. Technically this convergence is called “regression”. So values converge on the average or “regress to the mean”.

Think about tossing a coin ten times. On average you’ll get five heads and five tails. If last time you got ten heads next time you’re more likely to approach the average value or regress to the mean. It’s just probability.

Stockmarket Returns and Sheep

Over the twentieth century the stockmarket returned, roughly, 12% per year. Most years saw a return a bit more or a bit less than this value. A few years saw a return much, much, much higher than this. A few saw returns much, much, much lower than this. What Galton’s findings suggest for stockmarket investors is that worrying over (or glorying in) exceptionally bad (or exceptionally good) years is pointless. Over time things will drift back to the mean. Over time, if you don’t let better judgement get in the way, you would have made about 12% a year.

Often, though, judgement does get in the way, not always for the better. There are certainly a few people who have done better than 12% over very long periods of time – too long to be explained by chance. There are lots and lots of people who’ve done worse. In both cases the explanation for the exceptional performance probably lies in psychology: the jitterings of markets mean that at both the high and low points there are opportunities for discerning investors to make cool judgements - sell at the highs and buy at the lows. Most investors, though, aren’t discerning. They’re just humans and they do what humans do. They flock like sheep. Baa.

So they buy when everyone else is buying and sell when everyone else is selling. Buy high, sell low is not textbook mantra but it’s done by far more people than do the opposite. In the sixties it was the Nifty Fifty, in the nineties it was dotcom – sooner or later it’ll be something else. The herd effect is well ingrained in our behaviours and if everyone else is doing something it stands to reason that we should be doing it to. Obviously. Baa.

Neither Students nor Stockmarkets are Normal

There are two things about mean regression that investors need to be careful of. Firstly the stockmarket doesn’t follow exactly the same distribution as naturally occurring phenomena (which technically is called a Normal or Gaussian distribution). So we get very bad years and very good years far more than the theory would predict. Over time returns will still tend to the mean, but we need to be aware that the markets may trend away from this for considerable periods of time. Simple psychology makes many investors extrapolate current conditions forever which is not sensible and is likely to lead to exactly the wrong sorts of behaviour. Clever academics can make the same mistake (see Alpha and Beta - Beware Gift Bearing Greeks).

Secondly, there’s a problem with mean regression which is that it’s a statistical effect. If you’re a below average student then you can assume you’ll get a below average mark in a test. You won’t get any closer to the mean on a second test just because of a statistical effect: it may be that you were unlucky to get an exceptionally low mark first time out and therefore you’ll do better next time but that’s not statistics, that’s you.

Stats Doesn’t Mean You Can Buck the Trend

The same applies to companies in the absence of any changes. Just as a low grade student will predictably score low grade marks so a low grade company will predictably get low grade results. However, very often underperforming companies will get shaken up by new management or shareholder discontent in much the same way as a student’s father may discipline them to get them to work harder. But this doesn’t happen by chance, which means the effort of stock analysis is not irrelevant and that simply picking a sample of underperforming companies won’t guarantee excess returns.

Although Galton observed mean regression in people he fundamentally misinterpreted it in the light of his own prejudices. Whereas we now know that regression is a statistical phenomenon where any lack of full correlation between parents and children will result in such an effect (i.e. children are not copies of their parents due to gene mixing and mutation) Galton attributed it to a combination of inheritance from parents and from prior ancestors. Hence he was able, or so he thought, to demonstrate that the inherited characteristics of the privileged flowed down from their distinguished lineages. This fundamental misunderstanding of the phenomena was the mainspring of the eugenics movement which ultimately led to Hitler and his death camps.

Statistics in the wrong hands is very, very dangerous. Losing money on the stockmarket because of it is possibly the least of our worries.

Breeding: The Human History of Heredity, Race, and SexThe Mismeasure of Man (Revised & Expanded)Chances Are . . .: Adventures in Probability

Related Posts: Risky Bankers Need Swiss Cheese Not VaR, Alpha and Beta: Beware Gift Bearing Greeks


  1. Statistics are such a powerful tool, and as you mention, so misused!
    The power of statistics are both in comparing similar sets, and in looking back at the progress of a series of values. It is distinctly not of any use in predicting where a series of values is likely to head, unless the series of values has been correctly profiled and characterized as being of a particular and known distribution.

    In particular, the returns on the stock market, do not follow any such known distribution. That the mean over a century has been 12% will be a result of a multitude of inputs such as world population, industrial output, wars, global taxation, and a myriad others... There's no guarantee that the 21st century will exhibit the same pattern, and in fact, given that massive shifts in population and global trade patterns occurring in the last decade or two, chances are that it will be radically different.

    As an aside, it is galling that finance professionals such as pension funds use backward looking statistics to project future returns, when a large percentage of their clients are not aware of the unsuitability of these statistic return expectations.

    Yours, and very grateful for your outstanding blog!

  2. Hi uchinadi

    Glad you enjoy it. All feedback is welcome, it's difficult to know at what level to pitch posts. Don't want to be too technical nor too simple.

    Your comments are all true. Perhaps shares are the least worst option a lot of the time. At least they ought to roughly track global growth (assuming we ever get any again!)

    Even mutual funds with their excessive fees would normally provide better returns than cash in a bank account so we perhaps shouldn't be too critical of their misuse of stats. The best possible bad option, maybe?


  3. I'm afraid that you've chosen an unfortunate (way to present the coin-tossing) example when explaining "regression to the mean":

    ``Think about tossing a coin ten times. On average you’ll get five heads and five tails. If last time you got ten heads next time you’re more likely to approach the average value or regress to the mean. It’s just probability.''

    Someone who is not familiar with the concept of independence (or, lack of correlation) of events may, very wrongly, conclude that if you get ten heads in a row then it is more likely to get tails in the next toss (or, e.g., to get more tails than heads in the next ten tosses). This of course cannot be the case if one assumes, as one typically does when evoking the coin-tossing example, that the coin is fair and the tosses are independent of each other. One non-technical way to explain it is that, when the eleventh coin toss happens, the coin ``doesn't know'' that ten heads in a row have just happened.

    What you say is correct: the expected number of heads-per-coin-toss is (always!) 0.5, and hence if the current average is 1-head-per-toss (ten heads in a row) then the expected average after twenty tosses (given that the first ten were all heads) is 0.75 = (10+10*0.5)/20: that is a ``reversion to mean'' from 1 to 0.75 (towards the mean of 0.5). And that, as you say, is ``just probability''.

    The way you formulated your (pithy) paragraph, however, might confuse some to believe that the ``magic of regression to the mean'' makes the coin more likely to yield tails after a long streak of heads. It doesn't, and it's not magic: indeed, it's ``just'' probability.

    Sorry for being picky about this, but it's worth appreciating the significant difference between the reversion to the mean in the coin tossing example and in the financial markets. While the coin doesn't know what happened to it in the previous ten tosses, the markets do know! In other words, the action of a market today is not independent of its action in the last ten days. It's not just probability that makes markets ``revert to the mean'': it's economics and psychology that do (too)! Another difference between investing and gambling in a casino.