Survivorship Bias's Fatal Flaw Love Your Work

There’s an important bias to avoid: Survivorship bias. Unfortunately, people who might otherwise do something with their lives hide behind survivorship bias. Just as important as knowing when survivorship bias matters is knowing when survivorship bias does not matter. Survivorship bias has a fatal flaw.


Example: Abraham Wald avoided survivorship bias to bring back more survivors
In WWII the US military was trying to improve their planes. Each time a plane came back from a mission, they made a record of the bullet holes. Since most bullet holes were on the wings and tails of the planes, the military concluded they needed to add more armor in the wings and tails.

But statistician Abraham Wald said, No – that’s not where you want to add more armor. You want more armor around the engine.

That seemed weird. Their map of bullet holes showed very little damage to the engine compartment.

 

What Wald noticed that the military hadn’t noticed is they were only seeing bullet holes on planes that returned from missions. The bullet holes they weren’t seeing were the bullet holes on planes that did not return. And the bullet holes on planes that did not return were the ones bringing the planes down.

Abraham Wald was cleverly taking into account what would become known as survivorship bias.

Example: How survivorship bias can be used by an investing con artist
In his book, Fooled by Randomness, Nassim Taleb tells a story of a con artist. He’d send out 10,000 letters. Half the letters predicted the stock market would go up in the next month. Half the letters, down.

The next month, the con artist would send not 10,000 letters, but only 5,000. The following month, 2,500. Then 1,250, and on and on.

Why did he keep sending fewer and fewer letters? Because he only sent follow-up letters to those who had received correct predictions. After enough letters, he had 150 or so victims hanging on his every word, eager to have this mystery genius invest money for them. Of course once the con artist received their money, they never heard from him again. They had been “fooled by randomness.” They had been fooled by survivorship bias.

Survivorship bias doesn’t account for ergodicity
Both these stories are useful examples of survivorship bias. In the first case, Abraham Wald used an awareness of survivorship bias to avoid getting a false signal from the data. In the second example, the recipients of the letters didn’t realize they could be getting a false signal from the letters.

Survivorship bias is an important phenomenon to understand, but survivorship bias has a fatal flaw: Survivorship bias doesn’t account for ergodicity.

What is ergodicity?
What is ergodicity? Imagine you enter a dimly-lit bar just as it opens. A table of patrons across the room light up cigarettes. You can see the cascading trails of smoke rising. When they’re done with their cigarettes, they don’t smoke anymore the rest of the night.

When you get home, you realize your clothes smell like smoke. How could this be? You were nowhere near the trails of smoke.

Well, after the trails of smoke rose from the cigarettes, they dissipated around the room, until a faint haze of smoke filled the entire room.

Randomness eventually touches everything
That’s ergodicity. The smoke was rising from the cigarettes in a random pattern. But when a random pattern continues for long enough, that random pattern eventually fills the entire space it could have filled. The smoke spread randomly, until it filled the whole room.

Ergodicity is why it’s not only 1% of Americans who are in the top 1% of income. As time passes, people enter and leave the top 1% of income. In a lifetime, 10% of Americans spend a year in the top 1%. More than half will spend a year in the top 10%.

Ergodicity is why – even though life expectancy is about 76 – a 76-year-old only has a 4% chance of dying. The small risks of dying each ye