Failure to understand how randomness affects our lives can produce spurious assumptions, such as the belief that chance correlations between two random variables are in fact the result of causality. My favorite example in this regard is a quote from former Lockheed Martin CEO Norman Augustine's book, Augustine's Laws: “Trailer parks cause hurricanes.” Think about any picture of destruction caused by a tornado or hurricane. Most likely, what comes to mind is a decimated trailer park. Does one therefore result from the other?
In fields in which randomness plays a role, it is particularly difficult to assess competence. Let me give you an example. Suppose you hear a knock on your door and open it to find a monkey dressed in a business suit, holding a laptop computer and a manuscript. You see the monkey has an armband identifying it as property of the Johns Hopkins neuroscience research lab. The monkey hands you the manuscript along with a note: “This is my manuscript that I wrote. I'd like to write more and want you to be my literary agent.” You read the manuscript and are startled to find it is better than John Grisham's latest novel.
Should you sign the monkey under contract to write novels, which you can then publish? What questions should you ask the monkey in order to ascertain its ability to create additional best-selling novels?
The answer to this question has to do with randomness. If you assume that the monkey types randomly at a keyboard, it is possible, theoretically, for it to type a novel that is as good as War and Peace. So the proper first question to ask the monkey (since you already know it did type this manuscript) is, How many other monkeys were there in the neuroscience laboratory typing manuscripts with you? If you learn that there were a billion monkeys randomly pecking away at keyboards, you might safely conclude that this manuscript was not the product of genius, but rather a random event among many other hundreds of millions of terrible manuscripts. On the other hand, if there were only a dozen or so monkeys typing away, then you would have to say that the probability this was due to a random event was very small. More likely, this is a monkey who was typing with purpose and knowledge, and is of superior literary intelligence.
So, in assessing performance, whether it is in discovering a new gene, developing a blockbuster drug, operating a hedge fund, writing novels, hitting home runs, or composing songs, one needs to ask two questions. First, what role does randomness or variability play in this particular endeavor? And second, how many people are in the cohort working in this particular field? In fields where outcomes are highly variable and lots of performers are competing, assessing true performance rather than chance success may be very difficult indeed.
The important idea here is that this is not an abstract issue pertaining only to monkeys typing manuscripts. Our much-vaunted system of empirical modern medicine can be fooled by randomness as well, as we shall see in future columns.
