The Only Winning Move is Not to Play

In my many years in Cybersecurity a typical opening statement I'd offer to CISOs is that the objective of a cyber program is to play Tic-Tac-Toe with your potential hacker adversaries. If you are paying attention, there is no winner but also no loser.

The objective of the game is simply to ensure every match ends in a stalemate.

Many of you may recognize the source of the title I chose for this article

Ironically, not only was this movie one of the best teaching movies at the time about cybersecurity ("Sneakers" took first place when it came out), it also featured one of the most realistic outcomes of generative AI before GenAI was even a thing.

An Impossible Task

Why am I bringing up a movie made 41 years ago right now?

A few days ago, an article was published in Foreign Affairs Magazine titled "Why the Military Can't Trust AI" by Max Lamparth, Ph.D. and Jacquelyn Schneider .

It's a well written piece and while I am not agreeing nor disagreeing with the conclusions within, the crux of the piece was laid out in the middle of the paper

"LLMs may often correctly mimic human communication, but without the ability to internalize, and given the enormous size of the model, there is no guarantee that their choices will be safe or ethical. It is, therefore, not possible to reliably predict what an LLM will do when making high-stakes decisions."

I've written about this before. In fact I laid out the cause of the most concerning scenario in this paper (LLMs initiating nuclear war) - in my article about the recent eclipse. LLMs are built on logic, but war is human. Train an LLM about what to do in war, and it will apply logic. The LLMs naturally arrive at the conclusion that if the objective is to win, and we have big guns, use them.

The more fine-tuned the model, the more nuanced the strategy but the end result is virtually always the same per the testing.

This isn't surprising, nor unexpected.

If you have two nations run by AIs and their programmed goal is to ensure peace, prosperity, avoid poverty or famine, and permit a growth economy while fostering capitalism and individual wealth, I don't think the notion of going to war over a difference of beliefs, disagreements over land, or not sharing languages or customs would ever be within the AI's doctrine of rule.

Back to Wargames, ultimately the AI known as "WOPR" was designed to run simulations and learn from those simulations. But then, like the LLMs tested in Lamparth and Schneider's paper, it decided it may as well "just use the nukes" except it decided, since it had control over the real ones, that it would go ahead and launch. The only way humans could get it to not actually launch a full scale war in real life was to teach the AI that war was effectively an illogical game because it was unwinnable.

TicTacToe is unwinnable because the game is built to test your own perception of strategy. With two equal opponents, there is never a victor. The WOPR realized the same and eventually the world was safe from AI.

War itself being inherently illogical to an LLM is only half the problem though.

The Kobayashi Maru Test

Jumping moving references, The Kobayashi Maru was a test presented in the second Star Trek feature film. Commanding officers must face this simulation to see how people deal with no-win scenarios. It was only beaten once (or twice if you count the divided timelines) because a human chose to "cheat".

LLMs cheat also, but not by re-writing their code. They cheat through model drift and hallucination.

Military strategists rely on intelligence to make informed decisions. When LLMs are used for sensitive material, their ability to ingest, summarize, and output actionable information is the only thing that matters. If an LLM has incomplete information or is unclear on the objective, it could fabricate (hallucinate) critical missing pieces to achieve its programming.

If decisions are based on that LLM's output, the results could range from misleading to catastrophic.

Even if LLMs aren't used to make command decisions on how or when to start a war, a less directly malignant use case, intelligence gathering and summarizing with predictive modeling, could also be deeply impacted.

It puts military strategists in their own Kobayashi Maru. How can they trust their intel, how can the operate without it?

What's Being Done?

Quite a bit actually.

The Department of Defense announced the creation of an AI Task Force dubbed "Task Force Lima" which is focused on

...ensuring national security, minimizing risks, and responsibly integrating these technologies. The future of defense is not just about adopting cutting-edge technologies, but doing so with foresight, responsibility, and a deep understanding of the broader implications for our nation

The Department of the Navy issued a memorandum on the use of AI and LLMs, in which they specifically state

Commercial AI language models are not recommended for operational use cases until security control requirements have been fully investigated, identified and approved for use within controlled environments.

The U.S. Space Force Guardians also temporarily halted use of public LLMs while saying

LLMs offer great promise to assist DOD personnel in accomplishing a wide variety of tasks, however we must ensure that sensitive DOD data is safeguarded when using such platforms... Valid concerns have also been voiced about the traceability and validity of answers provided by LLMs

A Major in the Space Force indicated that this was a "temporary, strategic pause to evaluate the best path forward to align this capability into the USSF mission that protects the data of our service and Guardians"

In addition to these specific US Government initiatives, there are myriad movements in the public and private AI communities to manage and contain the risks of AI and LLMs.

Exercises including the ones outlined in the Lamparth/Schneider paper are important. They continue to expand the available knowledge and observable behaviors of LLMs and bring additional contexts to the discussions about the adoption and use of generative AI overall.

If you're interested in the field, you can create your own experiments inspired by your work, your hobbies or your interests then write about them and share. There's no more significant risk than not expanding your knowledge, as long as you follow local regulations and your company's business conduct guidelines.

Just remember once again, governance before deployment - always.

The thoughts contained in this piece are mine alone and do not reflect that of my organization.