Don't be "Fooled by Randomness"!

I am certain that many of you are already familiar with Nassim Nicholas Taleb 's remarkable book, "Fooled by Randomness." I remember reading it back in 2003 while pursuing my engineering studies at university. It is worth mentioning here that the notion of determinism (absence of randomness) and its application is a fundamental concept in engineering, so as student my mind was being shaped this way, that everything is deterministic and with the right measurement can be engineered. Therefore, Taleb's book had a profound impact on me, although I struggled to grasp some of its ideas and implications on investment and the stock market at the time (and perhaps even now). However, it convincingly argued, through practical examples, that randomness is an undeniable factor in the stock market (and life), and that most investors go wrong in their attempts to make sense of the market without adequately accounting for this element of randomness.

To be clear, this article is not about Taleb's book or the stock market, but rather about the concept of randomness itself. The perennial question arises: what is randomness, and where does it originate? Can we analyze it? And if we can, why can't we predict it? If we can predict it to some extent, at what degree does it become deterministic rather than random?

Throughout human history, the occurrence of events that appear to happen "randomly," without any discernible reasons or control, has played a pivotal role. Our survival and evolution, like that of other species, have been heavily influenced by our handling of randomness. We hunted and defended ourselves against wild animals as they appeared or attacked unpredictably. We migrated from one area to another based on the capricious changes in weather conditions. Our mating behaviors, guided by instincts and emotions, which seem random to us, were shaped by the same force of uncertainty since we lacked a comprehensive understanding of their origins or means of control. As time passed, our unique ability to recognize patterns made things progressively less random. We started identifying patterns in everything around us, from edible food to weather conditions and hazardous species. Consequently, we developed rational decision-making skills that substantially reduced the randomness in our lives. However, this relative order remained largely random to other species.

Rain Dancers

At its core, randomness reveals itself as the gaping hole in our comprehension when viewed from a specific perspective. Long ago, rain dances played a vital role in numerous tribes, while rain prayers reverberated through diverse religions, all aimed at invoking the favor of the rain gods. During those times, we overlooked the patterns of rain, which made these rituals (rain dances and prayers) have profound significance on our beliefs. However, fast forward to the present day! With the knowledge we have of weather forecasts and our ever-expanding grasp of meteorology, these practices have been stripped of their weighty seriousness and would be considered a joke now to get rain dancers to forecast the weather for us, as if it is a magic mystery (maybe with London as an exception, we certainly need magicians to forecast the weather there!). We now possess the ability to predict and anticipate weather patterns without leaning on such time-honored customs, rendering them as humorous tales from an era when randomness clung tightly to our understanding. This trend extends in almost every domain, I mean think about medicine and healthcare, what was the percentage of random diseases that we knew no cure or reason for 200 years ago versus now. The randomness factor is dramatically declining.

“One essential insight that has become evident over time is that more data and knowledge lead to less randomness. As we bridge the gap of knowledge exponentially, the randomness factor diminishes exponentially. “

Random number generators

Take, for instance, generating a random number in a digital system. To achieve true randomness, certain factors need to "confuse" the system's logic, such as incorporating noise or temperature sensors in the random number generation equation. Otherwise, the system would produce a "random" number based on a logic or algorithm it created, which, in reality, wouldn't be random. In cryptography, multiple methods ensure true randomness (or what experts call absolute randomness) for a given system. Yet, all these methods ultimately involve introducing a different level of ignorance into the system as part of the randomness equation; otherwise, it ceases to be truly random. Nevertheless, once this additional layer or level of ignorance is recognized, the randomness factor becomes eliminable, leading to determinism. For instance, placing a random number generator in a noiseless room with a constant temperature may still appear random from the system's perspective, but from the perspective of the person who placed it in that controlled environment, it is no longer random.

Thus, we arrive at the simple truth that randomness is merely the gap in our knowledge. Certain data or knowledge may be exceedingly complex to compute or comprehend, such as human psychology. For example, it has been demonstrated that humans are born with around 400 physiological traits resulting from genetic contributions from both parents and extended families, along with a unique genetic mosaic for each child. Add to this the environmental factors influencing the child's development, childhood memories or traumas, and the equation becomes incredibly intricate and challenging to solve. We may take the easy way out and label this complexity as a "psychological initial scheme," "emotions," or "behavior," effectively deeming it random, despite its evident deterministic nature.

Physicists' arguments about Randomness

When this idea crystallized in my mind, I went into some research and found that this debate has existed for some time, particularly among physicists. However, no strong argument against the claim of determinism and the absence of randomness was convincing. Quantum mechanics, with Heisenberg's uncertainty principle at its core, initially seemed to introduce fundamental unpredictability into reality, thereby suggesting true randomness. Nevertheless, certain interpretations of quantum mechanics propose that this perceived randomness may stem from our ignorance of "hidden variables," which ultimately supports my point. If the fact that "we can't know" renders something random, then the "ability to know" transforms it into a deterministic problem for which we lack a solution. However, throughout history, we have consistently managed to discover solutions and progress toward determinism rather than randomness.

Another interesting argument stated that even within deterministic systems, chaos theory explains how tiny differences in initial conditions can lead to drastically different outcomes, bringing Randomness to the system. While the system's nature is deterministic, its high sensitivity may make it seem random. This, however, does not negate the existence of randomness; rather, it emphasizes that deterministic systems can exhibit behavior that appears practically unpredictable.

This is indeed a valid point, but my argument does not assert that we can determine everything. Certain natural phenomena, quantum fluctuations in a vacuum, seem genuinely random to us. However, what I am proposing is not to label them as random but rather as unknown to us. There is currently no known cause or pattern for when an unstable atomic nucleus will decay, for instance, but that does not necessarily mean these processes are fundamentally random. It merely implies that we have yet to uncover the deterministic rules governing them. Given our innate pattern-seeking nature, capable of discerning order in perceived chaos, the idea that randomness is purely a product of our cognitive limitations or ignorance might be presumptuous. Consequently, I perceive randomness as a concept that emerges from the confines of our understanding and is not inherently woven into the fabric of the universe.

Now, the question arises: Why is this interesting? The fascination lies in the realization that randomness is essentially a limitation of our data and pattern recognition abilities. Guess who excels at pattern recognition far beyond humans? Artificial Intelligence. In other words, the only thing preventing AI from developing a higher level of knowledge perspective that would render things random to us (though not to AI) is data or, more precisely, the lack thereof. If we contemplate why we possess a higher level of consciousness or knowledge perspective compared to AI, despite its superior pattern recognition capabilities, the answer lies in the extensive data input we receive from our five senses. In contrast, AI, though equipped with a superior brain, lacks the multifaceted and generalized data input facilitated by our bodies.

As our understanding and models improve, we should be able to make better predictions, whether that's predicting the stock market, the weather, or even the outcome of quantum experiments. The existence of inherent randomness would set a limit to how accurate these predictions could be, but if randomness is just an illusion, these limitations could potentially be overcome.

Furthermore, if determinism holds true, then AI systems could also be capable of understanding and predicting human behavior better than ever before. The randomness we associate with human decisions could be better understood as complex patterns shaped by our biology (psychology and physiology) and experiences.

The next article will explore this very notion, but I cannot determine when it will be written, as my writing pattern is quite "random"!