Two Big Problems for AI: Alignment and Reification

AI is not a universe unto itself untethered from social realities. It’s a product, or conjuring up, of the social dynamics of the day. What could be better for these current times than word and image machines that transform our thoughts into our own delights? Put another way, even if all the technology was available in 1900, it is hard to see how the cultural mindset of that era could produce what we have now. At every point in time, culture and inventions intertwine, producing each other.

Two significant problems for AI and culture still loom. The first, the well-known alignment problem, refers to how an AI system stays true to its objective and is prevented from pulling away from the intended outcomes designers seek. The second, a type of reification , is when people take their thoughts and desires and by expressing them, often with deliberate manipulation of the system, turn the thoughts into a believed reality. The two interact in such a way that continual reification around disputed norms ensures perpetual misalignment of AI systems and balkanization of society and organizations.

Whew. Let’s unpack this.

Alignment is easier to understand. A well-cited example involves an autonomous vehicle and the death of a pedestrian where the autonomous vehicle failed to identify the pedestrian. The software error created a significant misalignment between the desired (safe driving) and the actual outcome (death). Another example is Microsoft’s experiment with a chatbot, Tay, where users quickly figured out how to make Tay say awful things . In this case, insufficient guardrails inside Tay failed to protect it from mischievous or malevolent users.

While these examples are easy to understand, they elide over a more complex question. To keep AI performing consistent with our objectives, we need to answer: “What are normative objectives?” and probably more importantly, “How does a pluralistic community of people agree on normative?” For another example that illustrates the difficulty in establishing alignment norms, we have no further to look than Elon Musk’s Grok. His chatbot is designed deliberately to be more “rebellious” and “spicy” in its dialogs. In this example, a billionaire decides what is normative and develops technology appropriate to his definition of norms. The Grok AI aligns with his norms.

This example, too, is somewhat simplistic. If we probe further, there are all sorts of cultural, micro-cultural, and personal variations of what we believe normative should be. How will AI and technology challenge this norm establishment problem?

One way is through personalization, and this is where generative AI excels. It can tune, in many cases well enough, the dialog it has with us in ways that align with our personal mental models and expectations. In time, I suspect we will be able fashion the generative AI dialog with ourselves unique to our personal view of things. Since generative AI can hyper-personalize, will this further balkanize our societies? What happens when normative becomes personalized? Are alternative facts a permanent feature? The alignment problem is exacerbated by the multiplicity of norms at play.

Reification is a bit harder to explain. This following scenario is one I that perhaps irreverently extend a great example by Daron Acemoglu regarding AI alignment. My extensions are in italics:

1. Human query: “Is Policy X effective?”
2. LLM: “No”
3. Future human communication (e.g., on social media, some paid by opponents of Policy X): “Policy X is not working”
4. LLM after new training on social metadata data: “Policy X is not working”
5. Lawmakers and politicians: “Policy X is not working. Let’s remove Policy X”
6. Policy X is removed

In this example, abstract or general concepts and severely shortened dialogs, turn from mere thoughts into a concrete reality with real-world repercussions. This is an extension of the concept of echo chambers . In our minds, echo chambers sound (pun intended) benign. In this example, this echo chamber of sorts is not benign. It can conjure up new realities quickly. The fight to control social media isn’t so much about the right to speak or the right to advertise. It is about the right to adversely reify, to create a factional and not communal reality out of the thin air of social media and AI.

From this perspective, the AI machinery is aiding in the creation of new realities while bypassing individual and community critical evaluation processes. Thought gets converted to reality too quickly without appropriate reflection about what is real and supported by evidence, much less what normative. It is as if the combination of social media and AI work as a replicator in the Star Trek universe. A replicator is a fictional device that takes your words and turns them into food items, spare parts, or nearly anything. Social media influencers are now conjurers.

Interaction with words leads to thoughts, and thoughts can lead to action, normative or not. In this regard, generative AI takes the promise of personalization of information to its furthest conclusion: development of people, one at a time, for better or worse. This technology can help us create our own thoughts. The community bonds between us help keep our individual actions, roughly speaking, normative.

So, what are we to do?

Setting aside larger global and national public policy questions and focusing inside the organizations, most people I talk to understand what needs to happen:

• Abide by applicable governing law
• Make sure monitoring the quality of human-technical solutions including AI are ongoing
• Make sure the monitoring has a holistic definition of quality that relies on evidence and also relies on the community to define what is normative. Typically, norms get codified in organizational principles and values, as they should
• Periodically pause to reflect ad perhaps restate the organization’s guiding principles and values
• Be able to detect when people and systems are bypassing the monitoring process, with direct attempts at reification

This is hauntingly like Chris’ Argyris’s concept of double-loop learning depicted here:

In double loop learning, an individual or community goes through the following steps:

1. They make a decision
2. Which affects real-world outcomes
3. Which generates feedback
4. Which then goes on to step 1) for another decision or moves on to step 5)
5. A second loop (loop B) in which the feedback interacts with individual or community mental models
6. Which then adjusts organizational decision-making rules, thereby impacting the next decision(s)

Adverse reification occurs when the information feedback loop that interacts with mental models becomes hyper-personalized and then similarly socialized without community feedback and reflection. The cycle goes like this:

1. The loop between information feedback <-> mental models (loop B) becomes hyper-personalized and gets detached from adjusting decision-making rules for organization (or individual)
2. Loop B gets multiplied across a faction, sort of like a meme that goes viral
3. As loop B spreads across a faction, often very small, it creates the illusion of a community belief system that then gets turned into a decision, bypassing the community-based decision-making rules processes

This adverse reification lops off step 6, crippling appropriate incorporation of evidence and adaptable norms and principles. This adverse reification can be initiated top-down or bottom-up and typically involves both directions. It can also be egged on by executive reality distortion fields . In short, the faction (and the AI) hijacks the whole loop by avoiding the decision-making rules process, preventing proper regulation of action. ?

Double-loop learning can be difficult for organizations and individuals due to resistance to change, fear of failure, or attempting too hard to control things for which people feel they are unilaterally responsible. I think the double-loop cycle is the antidote for misalignment between our human-technical systems and the outcomes required for our organizations to thrive.

This sounds like common sense, and it should be, I think. With the new media and AI systems available to us, we can now more easily turn individual, factional, and collective thoughts into reality. Within the political scene in the U.S., we are seeing spades of this happen in front of our sometimes-disbelieving eyes. When misalignment and short-circuited reification combine, bad stuff can happen very quickly.

If we can detect adverse reification attempts in people and in AI systems, we can better align the various human-technical systems we have with the outcomes we collectively desire. Vigilance and faith in the community are required.

?