Fooled by Randomness II - January 12, 2005

Crossroads Archive

In my previous column [go to previous column], I noted that human beings don't do well in interpreting the role of random fluctuations. Whether we are assessing the risk of dying from smoking, or trying to find a mutual fund in which to invest, we often fail to assign the appropriate probabilities, sometimes with disastrous consequences. Furthermore, when we look at outcomes, we typically fail to understand certain biases that affect the results significantly. The most important of these is the survivorship bias.

Suppose 1,000 graduates of a business school each start mutual funds and that each person has only a 50/50 chance of making money (i.e., posting a positive gain) in a single year. In other words, none of the 1,000 in the cohort is a good investor. After one year, there will be 500 who have made money, after two years, 250, and so on. After five years, there will be 32 who have a “successful” track record of making money in each of five consecutive years. These 32 will be shown on the top of the MorningStar mutual fund manager ratings, and most people will assume that their success is due to their skill as investors, and probably, that they all graduated from the same business school. In reality, of course, their performance is simply a statistical fluke—caused by random fluctuations.

Now, what about the other 900-plus investors—whose track record is mixed, to poor, to very poor? Most of them will have gone out of business—after two or three unsuccessful years, investors will pull their money out of their mutual fund, and these fund managers will be gone. They will not even make it to the bottom of the MorningStar list (where we can notice a lot of the business school graduates who don't do well at investing). Hence, we experience what is called the survivorship bias.

Let's take an oncologist who has a series of compounds, each having only a 50/50 chance of improving five-year survival in any given cancer patient. If enough compounds are tried, there will be some that have, simply by random chance, a positive effect on more patients than a negative effect. Of course, valid analysis will document the possible lack of statistical significance, but you get the point. Randomness plays a role in medical research.

Another way to look at the effect of randomness is as follows. If you are searching for a gene that causes Alzheimer's disease and you are one of 10 independent scientific groups conducting research, the chances of isolating the gene (assuming there is one) will be much less than if there are 100,000 groups doing the same type of research. The pace of medical discovery is partially random and therefore scales with the number of investigators. While each investigator may have skill equal to all the others, the one who happens to come upon the gene may get the Nobel Prize and be highly celebrated in our society, while the other 999,999 scientists will be also-rans. Randomness strikes again. And those who fail to make progress toward isolating the gene might not get their NIH grants renewed. Tenure may not be awarded. Only the survivors remain.

What is the worth of a Nobel Prize? If only one or two scientists are working in a field, the discovery might be more significant than if 1,000 or 10,000 scientists are laboring in the same area. Like the fabled monkeys on typewriters, randomness plays a role in medical discovery, yet our society doesn't recognize its importance.

But one area in which the issue of randomness needs to be much more carefully considered is in the pharmaceutical industry, as we shall see in my next column.

In my previous column [go to previous column], I noted that human beings don't do well in interpreting the role of random fluctuations. Whether we are assessing the risk of dying from smoking, or trying to find a mutual fund in which to invest, we often fail to assign the appropriate probabilities, sometimes with disastrous consequences. Furthermore, when we look at outcomes, we typically fail to understand certain biases that affect the results significantly. The most important of these is the survivorship bias.

Suppose 1,000 graduates of a business school each start mutual funds and that each person has only a 50/50 chance of making money (i.e., posting a positive gain) in a single year. In other words, none of the 1,000 in the cohort is a good investor. After one year, there will be 500 who have made money, after two years, 250, and so on. After five years, there will be 32 who have a “successful” track record of making money in each of five consecutive years. These 32 will be shown on the top of the MorningStar mutual fund manager ratings, and most people will assume that their success is due to their skill as investors, and probably, that they all graduated from the same business school. In reality, of course, their performance is simply a statistical fluke—caused by random fluctuations.

Now, what about the other 900-plus investors—whose track record is mixed, to poor, to very poor? Most of them will have gone out of business—after two or three unsuccessful years, investors will pull their money out of their mutual fund, and these fund managers will be gone. They will not even make it to the bottom of the MorningStar list (where we can notice a lot of the business school graduates who don't do well at investing). Hence, we experience what is called the survivorship bias.

Let's take an oncologist who has a series of compounds, each having only a 50/50 chance of improving five-year survival in any given cancer patient. If enough compounds are tried, there will be some that have, simply by random chance, a positive effect on more patients than a negative effect. Of course, valid analysis will document the possible lack of statistical significance, but you get the point. Randomness plays a role in medical research.

Another way to look at the effect of randomness is as follows. If you are searching for a gene that causes Alzheimer's disease and you are one of 10 independent scientific groups conducting research, the chances of isolating the gene (assuming there is one) will be much less than if there are 100,000 groups doing the same type of research. The pace of medical discovery is partially random and therefore scales with the number of investigators. While each investigator may have skill equal to all the others, the one who happens to come upon the gene may get the Nobel Prize and be highly celebrated in our society, while the other 999,999 scientists will be also-rans. Randomness strikes again. And those who fail to make progress toward isolating the gene might not get their NIH grants renewed. Tenure may not be awarded. Only the survivors remain.

What is the worth of a Nobel Prize? If only one or two scientists are working in a field, the discovery might be more significant than if 1,000 or 10,000 scientists are laboring in the same area. Like the fabled monkeys on typewriters, randomness plays a role in medical discovery, yet our society doesn't recognize its importance.

But one area in which the issue of randomness needs to be much more carefully considered is in the pharmaceutical industry, as we shall see in my next column.