A note on randomness: random data isn’t random. Data that has been generated in a truly random fashion can, by chance, contain patterns that look extremely ordered on small scales. Let’s look at some illustrative examples.
We have all heard of π (pi). Hopefully most of us will know it to 2 or 3 decimal places or that 22/7 is a decent approximation of it. π is what we call an irrational number, meaning that the sequence of its digits are random and that they go on forever – there is no ultimate number that can be said to be π, you can always calculate it to another decimal place. Indeed, being humans, some people can get a little competitive over such things. π has been calculated to at leat the ten trillionth digit and one chap from India decided to memorise as many as he could; he was able to recite more than two decimal places per second for over nine hours and got to the 70,000th decimal place. You get the picture, π is a popular number.
Contained within the random sequence of numbers that makes up π, though, is every type of order and every possible sequence you can conceive of. At some point in the infinite string of numbers you will see the integers 1 through 10, and 1 through 1,000, heck, even 1 to 1 billion. At some point you’ll see your birthday, your house number and your national insurance number all in a row. You’ll find every winning lottery ticket. You could say that the number 1 represents the letter a, 2 represents b and so on and you’ll find the entire works of Shakespeare, The Hunger Games and last year’s Christmas card to your mum all in there. These events are all unimaginably unlikely, but when you have an infinite list of numbers you also have an infinite number of chances and the probability of finding your sequence of choice somewhere goes up to one.
The reason any of this matters is that humans, as a species, are rubbish at noticing random data; or put another way, we’re really good at spotting patterns where there aren’t any. It’s not our fault, all the humans of times past that weren’t any good at spotting the pattern of a tiger amongst the random background of foliage got eaten. We are the offspring of the people who were good at spotting patterns. Many hundreds of thousands of years of evolution have made our brains the single greatest pattern recognition device in the known universe.
Whilst there were, and still are, advantages to this amazing ability there are also some distinct disadvantages. We get a lot of false positives. We see order in all sorts of places where there simply isn’t any. Astrology, complimentary and alternative medicine, the hot hands fallacy, the gamblers fallacy, pareidolia; these are all examples of apophenia: the human tendency to perceive patterns in random data.
So whilst we aren’t too likely to be eaten by wolves any more we are highly susceptible to finding meaning where there is none, and that kind of thinking can and does cause people real harm. At the benign end of the spectrum it is looking to your horoscope for your next steps in life instead of being proactive; at the more serious end it is gambling away your life’s savings on the spin of a wheel or pursuing a healthcare option that won’t do you any good and may endanger your life.
Part of being a good scientist is making sure that your data is showing a real effect and isn’t just noise, this can take years of training and sophisticated statistical analysis. Part of being a good sceptic, which everybody should be, is protecting yourself from those who would have you live your life according to random chance instead of what is real. Don’t waste your time running away from tigers in the long grass that aren’t really there.