August 2024 | This Month in Generative AI: Forensics Weaponized

by Hany Farid , UC Berkeley Professor, CAI Advisor

News and trends shaping our understanding of generative AI technology and its applications.

On July 17th, President Biden went into quarantine due to a COVID-19 infection. A few days later he announced his withdrawal from the 2024 presidential campaign. Soon after, a range of conspiracy theories began circulating online claiming that Biden was dead. His call a few days later into a press conference with Vice President Harris did nothing to mute these conspiracies and in fact provided even more fuel thanks to the faking of a forensic analysis of Biden's call.

Shortly after the press conference, a video with the caption "The White House Gets Caught Using Popular AI Voice Cloning Tool Eleven Labs to Fake Call To Vice President Kamala's HQ Event," circulated widely on X claiming to show evidence that Biden's voice was AI-generated.

The video shows a user purportedly uploading a recording of Biden's call to?ElevenLabs'?AI Speech Classifier, a freely accessible tool designed to detect if an audio was created using ElevenLabs' services. The tool returns a "very likely" of being AI-generated.

It wasn't until after the video reached millions of views that it was debunked.?ElevenLabs confirmed that the user did not upload the Biden audio. I also analyzed the audio using ElevenLabs' classifier and a model developed by our team at?GetReal Labs, and neither found evidence of AI generation or manipulation. There are also no obvious artifacts in the nearly four minute audio to suggest it was AI-generated.

While I applaud the development and deployment of the ElevenLabs' Speech Classifier, this type of abuse was fairly predictable. As commercial tools for classifying content as AI—or not—become more widespread, they can in some cases do more harm than good.?

And, most recently, an image of VP Harris and Governor Walz at a large rally was shared on X alongside a screenshot of its?misclassification as AI-generated by an online service. With over three million views, the overwhelming narrative in the comments was that Harris/Walz were doctoring photos to hide their unpopularity.

There has always been tension between the development and deployment of forensic techniques. For more than two decades, my academic research group has been developing and publishing techniques to detect manipulated media. Over these two decades the most common question I have received is "how do you ensure the adversary doesn't use your techniques to make better fakes?"

The answer is that we have approached disclosure of our techniques with a five-pronged policy:?

• We publish most of our forensic techniques.
• We don't publicly release code or data associated with our forensic techniques, but we do share with individual forensic researchers.
• When the work doesn't involve students, I will hold back certain techniques from disclosure.
• We strive to develop techniques that even when an adversary is made aware of our approach, a counter-attack is non-trivial.
• We develop a large suite of techniques so that even if an adversary can circumvent one technique, defeating all our defenses is non-trivial, time-intensive, and requires skill that we expect is out of the reach of the average person.

I am comforted to see the development and deployment of new forensics techniques ranging from ElevenLabs' AI Speech Classifier to Content Credentials. I am less excited to see error-prone services—with lofty claims of high accuracy in detecting manipulated media—being used and misused to further mislead the public.

We have to understand that ours is an inherently adversarial system where the adversary can and will weaponize our defenses against us. This will require us to be thoughtful on balancing accessibility, disclosure and security, and to deploy our technologies carefully and responsibly.

Beyond deploying forensic tools, the adoption of the C2PA standard to determine the provenance of digital files will become more important than ever as the use of AI becomes ubiquitous. The Content Authenticity Initiative, tasked with accelerating adoption of the C2PA standard, has now grown to more than 3,000 members and implementation is approaching “escape velocity.”

Consider joining the movement to restore trust and transparency online.