Engineering-first AI Part II:
The development of AI has been remarkable, yet simply deploying "AI tools" for smarter analytics isn't enough to gain a competitive edge. The rapid advancement of hardware and compute power, often exemplified by Moore's law, contrasts with the declining returns in biotech, known as Eroom's law. The question arises: Can AI augment drug development and ensure a positive return on investment in R&D over time.
More concretely, given that 90% of drug candidates fail, it costs more than two Billion dollars to develop one and over 10 years we substantially impact the probability for success, finance and time to market?
Over a year ago, I discussed Immunai’s unique approach, the best-in-class platform in the clinical development of immunotherapies leveraging AI in Engineering-enabled AI. Our vertically integrated platform, and its analytical “brain”, the Immunodynamics Engine (IDE), aims to revolutionize drug development.?
The engineering aspect is for both the different components of the platform and, even more importantly, for the way we integrate, or “stitch”, these components into one entity. It is enabled through the curation and ingestion of unique data modalities into our growing proprietary AMICA? database via single-cell profiling of 100s of millions of immune cells and their interactions, and through our data engineering CellDB infrastructure to support AMICA integration and utilization, and last but not least, the application of a suite of AI frameworks enabling us to transform data into knowledge and recommendations. In this article, I will share my thoughts about our leveraging and enabling of artificial intelligence.?
This article is not aimed at serving as a comprehensive overview of Immunai’s AI capabilities, but rather to highlight some of the challenges that platform companies affecting drug development will have to deal with if they wish to leverage AI. Our approach to dealing with some of these challenges will create a moat and strong competitive advantage by ultimately increasing R&D productivity and the development of therapeutics with better clinical outcomes for patients.?
Two fundamental capacities constitute logical reasoning -- induction, the ability to derive a general rule by seeing a few instances of that rule, and deduction, the ability to deduce instances of a given general rule.
Imagine a simple pattern integer sequence doubling the previous number and adding one. Starting from zero, we can deduce the first few terms: 0, 1, 3, 7.?
Even without knowing this rule, by seeing these initial numbers, you might guess these numbers from logical thinking. However, there are other integer sequences defined by different rules that start with the same initial terms. If you don’t believe me, just type 0,1,3,7 here. This shows there can be several explanations for phenomena and also highlights how humans and computers solve problems differently. Humans are adept at creative solutions, while computers excel at applying rules precisely and quickly to get results.
In fields like science, engineering, and arts, “invention” goes beyond logical reasoning. Your favorite song or building, and perhaps, if you like math like me, your favorite mathematical proof. None of these was made possible by only following the principles of logical reasoning.?
Let’s delve into a concrete example from natural language processing (NLP), specifically within conversational AI. Over the past five years, a new generation of machine learning framework, on both the software and hardware sides, allowed to train significantly larger models capable of learning from vast amounts of data and solve various problems such as translation and text classification with a new level of quality. As a result, models such as ChatGPT4 are capable of furnishing conversational responses to textual questions. This ability to generate a novel conversational response, while not classified as “an invention”, undoubtedly represents a significant stride in this direction.
The revolution of generative artificial intelligence, encompassing Natural Language Processing (NLP) exemplified by ChatGPT and Computer Vision typified by DALL-E, seems to be unfolding like the unraveling of a well-guarded secret. However, is this perception truly accurate? What if I were to reveal that someone “fed” the algorithm with all the questions and their corresponding answers–wouldn’t that diminish the excitement considerably? The primary critique against recent breakthroughs in generative AI is that they haven’t attained genuine “understanding” but are instead trained on a sufficiently dense subset of the space, akin to mimicking a parrot, and cannot discriminate between true and false responses. This prompts the question: what exactly constitutes an understanding? Drawing from my perspective as a mathematician, it’s noteworthy that even proficient mathematicians occasionally prove complex theorems only to discover errors in their proofs later. Does this imply a lack of understanding or does it underscore the elusive and nuanced nature of comprehension itself?
With that perspective in mind, I aim to revisit the fundamental capacities of intelligent reasoning and delve into what they practically involve for the development and challenges of artificial intelligence. When discussing deduction in a context where numerous decisions must be made, deductive reasoning equates to comprehending the decision boundaries among the various labels–such as true and false, or colors like blue, green, yellow, and red.?
The capability to make inductions, extending beyond more validation, demands the proficiency to generate new rules and assess whether these rules can effectively explain the specific instances.??
Next, I will elucidate two significant classes of artificial intelligence frameworks: Discriminative AI and Generative AI.
Generative artificial intelligence: Generative models are machine learning models designed to generate new data samples resembling the training data they were trained on. These models grasp the underlying distribution of the data, allowing them to create novel instances. Generative models find applications in various domains, including image synthesis, data augmentation, and the generation of realistic content such as images, music, and text. They are recognized as a category of statistical models proficient in generating new data instances.?
I perceive Generative AI as related to “Induction”, as it involves learning to generate instances following the “general rule” underlying data instances it has observed. Furthermore, it can be viewed as an expansion, as Generative AI models improve their capabilities with more data, so it is a form of “Adaptive Induction”.
Discriminative AI: Discriminative models are utilized in Statistical Classification, primarily for supervised machine learning. These models concentrate on modeling the decision boundary between classes in a classification problem. The goal is to learn a function that maps inputs to binary outputs, indicating the class label of the input. Discriminative models (just as in the literal meaning) separate classes instead of modeling the conditional probability and don’t make any assumptions about the data points. However, these models lack the capability to generate new data points. Therefore, the ultimate objective of discriminative models is to separate one class from another.
I regard Discriminative AI as analogous to “Deduction”. It involves deducing whether a given instance is a “True” instance of the general rule. Furthermore, it can be viewed as an expansion, as Discriminative AI models improve the conditional probability when iterating with more data, so it is a form of “Adaptive Deduction”
2. On cats, dogs and Plato’s cave
A few months ago, over a discussion with a colleague and friend, who is a biopharma executive leader, I was asked the following: “AI can detect a cat (or a dog) in a picture, but it does not understand what a cat is– is it just a series of pixels that statistical probabilities indicate is most likely a cat rather than dogs or other objects?”?
Unfortunately, we don’t really know how to define human understanding. The concept of “consciousness” which underlies the mental faculty capable of understanding, is something that scientists and philosophers have been obsessing over for centuries and even millennia. One of the most famous quotes in philosophy “Cogito, Ergo Sum” (in Latin), and in translation to English means: I think, therefore, I am, by René Descartes, from 1637, is a philosophical discourse deducing “being” from “consciousness”, or the mental faculty of the “self” to “think”. But the question of how we understand what a cat is or what a chair is goes back even before Plato and Aristotle, two of the most important philosophers from the “Greek Philosophy school” who lived hundreds of years BC.?
Plato and Aristotle gave two different answers to this question. For Plato, there is the eternal “Realm of Forms,” where all the abstract, eternal, and unchanging forms exist. Forms are also translated as ideas. In this world, there is the “form or idea of a cat”, which stands as a perfect, eternal, and unchanging form of the cat; in that same world, there is also the “form of a chair.” The “Realm of forms” lives outside our physical world and is more real and true than the physical world. With our senses and consciousness, we are merely able to sense imperfect, changing instances of these forms or ideas in the physical world, and Plato described this in his “Allegory of the Cave,” where a group of people who have lived chained to a wall of a cave their whole lives. They watch shadows projected on the wall from objects passing in front of a fire behind them and give names to these shadows. Plato meant that our senses confound our perception of cats in this world, and we, mere humans, cannot “sense” the true form or idea of the cat in its eternal, unchanging form, except for glimpses. But this form does exist, and an understanding that a specific cat is indeed an instance of the idea of the cat in the physical world is equivalent to a divine glimpse in the “Realm of forms.”?
Aristotle's metaphysics was quite different; he believed that there is no other world except for our physical world, and the form of the cat is the universal property common to all cats in this physical world. In this approach, every object in our world contains two components: form and matter. For example, think about all the chairs you’ve sat on or seen during your lifetime. They are all made of atomic particles and exist in our physical world and are also distinct from each other. But they have something in common: their “chairness”, which probably stands for the fact that their function is to be sat on. They have different sizes, shapes, and colors. Some of them have four legs, but others don’t, but they’re all chairs. There are certain puzzles where you will see an object you identify as a chair, but then you’ll see a sign saying, “Please don’t sit on.” Is it still a chair? But without getting into these nuances here, what Plato and Aristotle would agree on is that the form of a chair or a cat exist, and sometimes the intellectual exercise of being able to distill it is non-trivial.?
After this digression on Western philosophy, I want to return to a toddler’s understanding of a cat. A toddler does not know about DNA, so when she thinks a specific object is a cat, it’s not because she took a DNA test and established it’s a cat, but because of specific sensing, she’s done during her life, through sight and hearing, and sometimes also through the feeling of the cat’s fur. But when she watches “Tom and Jerry”, she immediately knows that Tom is a cat even though it is a two-dimensional cartoon that doesn’t look like any real cat. For example, Tom stands on his two legs!?
In the exercise of training AI to detect a dog in a picture, the context that the AI model is shown is limited compared to the toddler, and in learning, one of the most nuanced and important aspects is context. Understanding depends on context.
领英推荐
3. Identifying challenges for the application of artificial intelligence in biology
Following this introductory exposition of logical reasoning, artificial intelligence frameworks, and? human versus machine “understanding,” this is an intermediate summary:
In the AI literature, “Transfer Learning” refers to leveraging similar yet different contexts to learn a related topic or function. For example, leveraging Natural Language Processing knowledge of Spanish, Italian, English, and Latin to study Portuguese instead of learning Portuguese from scratch. A similar and related notion of “Multi-task Learning” enables AI to learn multiple tasks together, such as learning multiple such languages together or identifying multiple animals in a picture. The basic concept behind both is that sometimes broadening the narrow concept of one class or function enables a more robust learning process that will enable learning faster and more effectively from fewer examples.
Similarly, but related to it, if we go back to the notion of “invention”, defining what constitutes invention remains elusive. It’s not about stringing together random words or letters; it necessitates internal cohesion, comprehensibility, and logic. This holds true even for compositions like "Für Elise" by Beethoven. Inventions typically involve taking something existing in one space and mapping or transferring it to another–for example, the invention of the wheel, which intricately maps from the realm of geometry and mechanics (the wheel and axle) to the realm of transportation. Another example is the invention of the radio: In April 1872, William Henry Ward received U.S. Patent 126,356 for a wireless telegraphy system, theorizing that convection currents in the atmosphere could carry signals like a telegraph wire.?
To empower artificial intelligence with inventive and discriminative capabilities that would allow for a deeper machine understanding, we must train models on datasets from diverse domains, along with a substantial set of potential mappings or transformations between these datasets and domains.?
However, the primary focus of this article is not a better articulation of the “machine understanding” concept. Instead, I aim to present that we have to build towards and integrate different artificial intelligence paradigms to address the main challenges ahead in biology.
By way of comparison, while almost every person can validate whether a text response to a question is valid or not, the challenge in biology lies in the ambiguity and sparseness of ground truth, for example, for a given patient at a given time, you will only be able to test only one treatment outcome, which leads to a sparse dataset, not to mention that figuring out the drug effects is not always straightforward, it takes a long time and can depend on multiple extraneous factors.
To emphasize the point of the importance of having a large, dense dataset to train your artificial intelligence models, a recent empirical study demonstrates that the in-context inference (ICL) of transformer-based GenAI produces excellent responses. This implies that when we ask the model to produce data points in areas where the context aligns or is close enough to what the model was trained on, the results are meaningful–representing “real” or close-to-real data points. However, moving farther from the true context, the generated data points may not accurately represent real-world scenarios. The concern arises when individuals unfamiliar with the space cannot effectively discriminate between “true” or “false”, compromising the purpose. This challenge is particularly pronounced in fields like biology or immunology, where the space is not easily equipped with “ground truth labels”.
And that is the fundamental challenge in biotech: will the drug work, and for which patient population? The fundamental challenge is the lack of an intrinsic way to test multiple drugs on a patient simultaneously. Moreover, the pharmaceutical and biotechnology industry is facing another major challenge: before advancing drugs to clinical trials, pharma and biotech companies have to rely on data from lab experiments with cell lines and in different animal models, not from humans– data that is known to read extremely poorly to human patients.?
Going back to Plato’s cave allegory, biopharma and biotech scientists can only look at the shadows on the wall of the cave, the preclinical wall, with data coming from cell lines and mice models only. How will they be able to deduce the true knowledge about how the drug is going to affect human patients? Is there a canonical way to reverse-engineer these shadows to deduce the actual effects of drugs on human patients??
I believe that with a careful and robust ingraining of broad contextualization schemes in artificial intelligence, coupled with wet-lab molecular profiling and functional validation, it is possible to create a more robust understanding of drug mechanisms of action (MoA) on a personalized level.?
4. Immunai’s analytical framework: introducing the Immunodynamics Engine (IDE)
As described above, over a year ago, I wrote this post, Engineering-enabled AI, to highlight how the leveraging of differentiated engineering infrastructure allowed us to create Immunai’s vertically integrated platform.?
Today, the unlocking of “Natural Language Understanding” (NLU) is only a matter of time, I would bet a decade. What led to the biggest breakthrough in NLU was a perfect storm combining: i. compute power (hardware), ii. advancement in computational models (Deep Learning & AI), and iii. The abundance of high-quality textual data is available online.?
In our case, our platform aims to unlock “Drug Development Understanding”, which is orders of magnitude more complex than natural languages. Unlike the abundance of high-quality textual data available online, having the right data available for drug development is a dual problem. First, it is important to know what data is most relevant and then make it readily available.
In drug development, one of the important areas is called pharmacodynamics, which is the study of the biochemical and physiological effects of drugs. The effects can include those manifested within animals, including humans, microorganisms, or combinations of organisms.
Immunai took a bet five and a half years ago that to measure drug effects and predict clinical outcomes, we should measure the effects drugs have on the immune system, both in the clinical setting, that is, how it affects patients in clinical trials or in the clinic, and also in preclinical setting, in the lab, studying the effects of the same drugs in-vitro and in-vivo.?
Adapting the notion of pharmacodynamics We introduced immunodynamics, the study of the immune effects of drugs, including those manifested within animals, including humans, and in different preclinical lab settings.
There are three issues here:?
But the most interesting and nuanced aspect is that measuring the immune system means that you need to measure the system, not merely elements in this system. Our approach is leveraging bio-technologies that allow for the measurements of cells with very high granularity and precision. Thesecalled single-cell sequencing technologies allow us to map immune cells from every sample we sequence, usually pre- and post-therapy, which means, before and after giving the drug. However, this only gives us a picture of the immune cells, and the harder question remains how to deduce the “Systems Immunology” map. This is our mission, and over the past five and a half years, we have been developing the Immunodynamics engine, IDE?, that systematically maps and studies the effects of different drugs on the immune system through our vertically integrated platform.
Soon, I will also publish “Engineering-enabled AI: part III”, where I will share a white paper with the technical details of our analytical framework and our artificial intelligence.
Summary
The next two decades will bring further vertical AI applications. What we have seen in 2023 is merely the beginning of a deep transformation the tech industry will go through. But for young companies that want to jump on the AI bandwagon, there is a clear warning - don’t start a company where you don’t have a clear differentiation. If you worry that with the advancement of artificial intelligence, your business will not have a “raison d’etre,” a reason to exist, you are probably right.?
However, medicine, biotechnology, aerospace, agriculture, and other industries that have not yet been disrupted by the computational transformation– they are all going to be transformed in the next two decades. For companies in these areas to create a differentiation, founders and executives must invest in building the right engineering infrastructure, on the basis of which, artificial intelligence will be deployed. Create new types of data modalities, own unique data that only they own, and experts that can label this data that others will not have access to. If you can’t think of how to create an engineering and data moat, my recommendation is that you shouldn’t do that.?
Our initial bet in 2018 was that the artificial intelligence transformation has already started, and we will have a way to create a differentiated company if we know how to measure the effects drugs have on the immune system better than others. And what we did over the first five years was just about building the engineering infrastructure and vertically integrating all the different components. In the past six months, we have been building sophisticated artificial intelligence models on AMICA? and augmenting our IDE engine with these.
I feel confident that the dedicated efforts we are concentrating on over the next few years to further our analytical capabilities and leverage the state-of-the-art artificial intelligence models will allow us to create a significant moat that will grow significantly over time and will allow us to unlock immune intelligence and drug development understanding.?