Generative AI, Trust & Security: Fireside Chat at TrustCon'24

#TrustCon'24, held in San Francisco from 22-24 July 2024, was a sold-out conference of around 1200 Trust & Security professionals. While I have given >100 keynotes at various events, being interviewed in the Fireside Chat format in front of practitioners/industry professionals (as opposed to the usual researchers/academic audience) was a relatively new experience for me. I have WIPRO, who has funded our research at the AI Institute (sponsor: Anindito De ), and the fireside chat host, Karthikeyan Ravindren to thank.

Here, I reproduced part of the content we covered (the conversation had similar coverage but not the same, and to fit the time, we also skipped the domain).

Question 1: Prof. Sheth, how does the increase in the adoption of Generative AI change the content moderation landscape?

Answer: GenAI is used to create content - a lot of this content shows up as fake news and deep fake.? Following the text, GenAI-generated images and videos are going very rapidly (3.6 billion images are uploaded every day, and 12.6 billion images were created using just one of several platforms in 2023). Coupled with bots, these contents spread fast.? Often the content is made deliberately controversial since it is well known that such content spreads faster and wider. So, content moderators deal with all the big data problems: volume, velocity, and veracity. 30% of Indians say at least one in four videos they watch online are fake.? May 2024 Indian elections saw plenty of fake images and videos, and even Elon Musk has shared US election-related deepfake! All these make content moderators’ tasks harder. Costs for not being able to handle undesirable GenAI content can be high. 63% of organizations lost at least $50 million due to AI/ML governance failures, and GenAI had a role in many of these failures. Game Devs lose >$1.6B a year due to online toxicity.

(Here is how the first question/answer really went).

Question 2: Has the wider adoption of LLMs and GenAI tools increased the challenge for content moderators and consumers of the content? Are there specific technical reasons behind such challenges?

Answer: As GenAI technology has advanced, AI-generated content has become indistinguishable from human-generated content. In our recent paper, which received the outstanding paper award at the top NLP conference, we compared 15 LLMS and showed that recent LLMs create indistinguishable content from human-created content. Simultaneously, we show that detecting whether the content is human-generated or AI-generated has become extremely hard to impossible.?

The second technical reason is hallucination. The Cambridge dictionary calls hallucinate the word of the Year 2023! LLMs and foundation models learn billions and trillions of parameters by capturing complex relationships between the data they are trained on. However, when an LLM is used, it is no longer possible to tell or ensure that the response or answer is based on training data or facts contained in that data.?

In fact, you can consider hallucination to be a feature of GenAI models. You can put a positive spin on hallucination, which makes an LLM creative, but that comes at the cost of consistency and reliability. And for a content moderator or user, it becomes extremely hard to tell whether the AI model made up new information that is not factual.? Interestingly, if you ask an LLM for a reference for anything it returns, it could make up that reference, too. This happened in the well-known but unfortunate example of a lawyer who used ChatGPT in a court case. In our work, what was covered in the Washington Post, we develop a measure called the Hallucination Vulnerability Index to characterize a model’s vulnerability to produce hallucinations, identify types of hallucinations, and show which types of hallucination are more troublesome.?

Question 3: After success in developing textual models that address natural language processing, we now see rapid progress in vision models. How does this play for content moderation, and what new challenges do they pose?

Answer: In February of last year, an AI-generated image of the Pentagon under attack went viral on Twitter. As a result, many people believed and reshared the image, leading to an 800-point drop in the New York Stock Exchange. Sundar Pichai noted that multimodal AI generation is becoming a new challenge for search engines. He mentioned that in the near future, people will increasingly turn to Google to verify the factual accuracy of what they see online. Additionally, Sora recently announced that although many such models are publicly available, theirs is not yet. Deepfake audios are also a significant concern.

Question 4: If AI adds to the content management problem, does AI help counter the challenges? Is RAG a major component of the solution?

Several new research studies argue against this notion. One such example is the following: If we ask a large language model (LLM) who the current president of the USA is, and the LLM is only trained until 2019, it will likely answer Donald Trump, even if provided with a relevant document stating Joe Biden. This issue is known as knowledge conflict. LLMs have an availability bias, meaning they tend to rely more on their internal memory than on relevant documents that are provided. It says LLMs don't have the right ability to digest everything we provide in the context. We certainly need better techniques.?

Question 5: ? Can you elaborate on WIPRO and AI Institute's partnership and the research?

We are collaborating with Wipro to enhance Generative AI capabilities. Our joint effort includes a BioNLP project named Percuro, where we are developing a knowledge graph for biomedical domains and employing neuro-symbolic methods for tasks such as summarization and question-answering. Additionally, we are starting to explore the topics of safety and trust, including content moderation, deepfake detection, and mitigating hallucinations in AI.

WIPRO's announcement