The search for meaning in the Borgesian library of data

Evolution and revolution, artificial and biological, and the AI librarians that we need to bring order to chaos

You walk into The Library of Babel, an infinite repository of knowledge described in 1941 by Argentinian essayist, intellectual, philosopher and poet Jorges Luis Borges. The space is composed of interconnected hexagonal rooms; in each room four walls contain five shelves and each shelf holds 32 books of uniform size and format. Each book has 410 pages, each page has 40 lines of text and each line has 80 characters, including letters, punctuation marks and spaces. Within this vast honeycomb sit volumes containing every possible combination of those letters, punctuation marks and spaces. So shelved in this library are every play Shakespeare put to paper, every article I’ve ever written, there’s the collected wisdom output by humanity. But the wisdom is mixed in with an immeasurable number of volumes containing nonsense, volumes that have one recognizable word, some with two pages of coherence…but mostly gibberish. And despite this (seemingly) highly-structured physical environment, finding anything of use is an endless exercise of searching for an organizational scheme that makes the knowledge accessible. The scope of the problem is too much for any human librarian to solve.

In the author’s translated words: “When it was proclaimed that the library contained all books, the first impression was one of extravagant happiness. All men felt themselves to be the masters of an intact and secret treasure. There was no personal or world problem whose eloquent solution did not exist in some hexagon…As was natural, this inordinate hope was followed by an excessive depression. The certitude that some shelf in some hexagon held precious books and not that these precious books were inaccessible, seemed almost intolerable. A blasphemous sect suggested that the searches should cease and that all should juggle letters and symbols until they constructed, by an improbable gift of chance, these canonical books…Others, inversely, believed that it was fundamental to eliminate useless works. They invaded the hexagons…leafed through a volume with displeasure and condemned whole shelves…The library is so enormous that any reduction of human origin is infinitesimal. The other: every copy is unique, irreplaceable, but (since the library is total) there are always several hundred thousand imperfect facsimiles: works which differ only in a letter or a comma.”?

Borges’ multi-layered allegory uses the metaphor of an infinite library to highlight existential questions that speak to the nature of the human condition and the search for order in chaos. His librarians work on the edge of existence where the ability to acquire knowledge is, at best, limited and random, challenging our need for purpose. There’s a difficult tension between embracing curiosity in an ambiguous labyrinth—the idea is that an effectively unlimited amount of information could let you access a work of wisdom that’s inconceivable. But only if you can somehow impose order onto disorder—navigating to an island of meaning in an ocean of chaos. If that sounds familiar, it’s because the rise of commercial artificial intelligence (AI) has highlighted the philosophical and practical parallels between Borges’ vision of limitless possibility and how AI might shift the paradigm of extracting instruction, rather than irrelevance, from bigger, faster and more flows of data. Is AI the librarian who will finally be capable of usefully cataloging our digital world??

Effective AI must balance the sheer volume of information that exists with the comparatively tiny amount of information that has a practical use. It’s about using AI to recognize patterns, identify possible optimizations and potential problems, and recommend (or even agentically action) solutions. And when it comes to generative artificial intelligence (gen AI), it’s about establishing constraints that let gen AI models output a new multi-modal presentation of useful information drawn from the largely incomprehensible shelves in the Borgesian library.?

But like making sense of the library, AI (and how we use it) is a work in progress that regularly demonstrates the limits of information without understanding. How do we ensure that AI, like the hypothetical would-be librarians of Babel, helps us find truth and meaning in a universe of possibilities rather than crippling us with irrelevance and misinformation?

As enterprises figure out how they’re going to use AI, they’re essentially trying to bring to life librarians who can pull meaning out of noise. To ground that in telecoms, the use case could be upselling a consumer to a more expensive service plan or predicting equipment failure that would lead to a service outage before it happens. In either example it’s all a matter of finding useful information and doing something useful with it. 5G is the proving ground for telco AI as, for better or worse, the industry has already started to describe 6G as “AI-native.” If we want to deliver that future, it means taming a mind-boggling amount of complexity and solving the metaphorical problem of the Borgesian library within our particular domain.

In addition to that “AI-native” descriptor, 6G is also described as simultaneously an evolution and a revolution in capabilities. These are tricky terms that speak to the artificial and biological, the former of which is a manifestation of the biological. To get a handle on this we turn to Kevin Kelly’s 1994 work of brilliance Out of Control, which (I think) is a freewheeling exploration of system theory and the blurring line between the artificial and the biological.?

He writes that “‘Evolution’…is a common vernacular term meaning incremental change over time. But what in the world doesn’t alter gradually? Nearly all change around us is incremental. Catastrophic change is rare, and continual catastrophic change over long periods is almost unknown. Is all long-term change evolutionary? …Despite the confusion about the word ‘evolution,’ our strongest terms of change are rooted in the organic: grow, develop, evolve, mutate, learn, metamorphose, adapt. Nature is the realm of ordered change.?

“Disoredered change is what technology has been about until now. The strong term for disordered change is ‘revolution’—a type of drastic discontinuous change peculiar to human-made things. There are no revolutions within nature.?

“Technology introduced the concept of revolution as an ordinary mode of change. Beginning with the Industrial Revolution, and its spillovers the French and American Revolutions, we’ve seen an uninterrupted series of revolutions brought on by technological advances—the revolutions of electrical appliances, antibiotics and surgery, or plastic, of highways, of birth control, and so on. These days, revolutions, both social and technological, are announced weekly. Genetic engineering and nanotechnology—technologies which, by definition, mean we can make anything we desire—promise revolutions daily…The last revolution in technology will be to embrace evolutionary change. Science and commerce now seek to capture change—to instill it in a structured way—so that it works steadily, producing a constant tide of microrevolutions instead of dramatic and disruptive macrorevolutions. How can we implant change into the artificial so that it is both ordered and autonomous?”?

So constant evolution is the ultimate revolution, but revolution is a shock that our companies, our markets, our minds and our societies can’t control or productively leverage. Meaning if we could, somehow, impose order, predictability and structure to constant evolution, we could reap the rewards of that ultimate revolution; but further meaning that revolution is the result of constant evolution. However, the issue is that artificial evolution, as compared to biological evolution, operates at a very different time scale—a focus on constant evolution will perhaps deliver what feels like revolution more often than a laser-focus on affecting what you might call disruption. As an aside, there’s a soft schism in the study of biological evolution that’s applicable to artificial evolution. We’re all familiar with Darwinian evolution (Charles Darwin, On the Origin of Species from 1859) where random mutations are subject to natural selection of advantageous traits which are then passed on—“survival of the fittest.” But there’s also Lamarckian evolution (Jean-Baptiste Lamarck, Philsophe Zoologique from 1809) where environmental adaptation based on the use, or lack of use, of specific traits are acquired then passed on. That said, both hold truth, suggesting that things evolve, things adapt, on both immediate and long-term timescales. Whether those things are artificial or biological, I think, is an important area of inquiry, particularly in the sense that artificial things are necessarily starting to look a lot more like biological things.?

Back to the role of AI in all of this. We need it to balance infinite information and practical application. We need it to recognize patterns that can allow us to extract actionable meaning. We need it to generate creativity within constraints that make it functional in service of something of value. And we need it to do all of that in a way that somehow adheres to ethical, moral even, considerations that are ill-defined on the frontier of this new techno-humanistic construct.?

If nothing else, the problem has been defined, we have an idea of what the solution is, and now it’s a matter of getting from problem to solution. I don’t know how we do that, but as we collectively turn the crank, remember what Borges tells us: “Nothing is built on stone. All is built on sand, but we must build as if the sand were stone.”