Our AI Friends: Confidants and Profiteers

In April 2022, Senator Menendez googled "How much is one kilo of gold worth." Chrome saved this search, and the FBI found it during a later raid. His search history became evidence for his conviction.

In 2024, Menendez might ask ChatGPT for advice on exchanging gold for dollars, discussing transaction details.

"I'm surprised how willing people are to share very personal details with an LLM," said Sam Altman in a July 2024 interview. He added that users feel more comfortable talking with AI than with a friend.

Altman recalls Reddit posts of people confessing uncomfortable things to LLMs. "They knew it wasn't a real person," he said, "and they were willing to have this hard conversation they couldn't have with a friend."

If you doubt humans share deep conversations with chatbots, look at the WildChat dataset. It contains one million user interactions with ChatGPT, collected by offering free access in exchange for chat histories. WildChat is built by Paul Allen's Institute for AI.

This past spring, a man in Washington feared his marriage was collapsing. "I am going a little crazy, still love her and want to win her back," he typed into ChatGPT. He sought help writing a letter protesting her decision to file for divorce. "Emphasize my shame and remorse for not nurturing and being a better husband, father, and provider," he wrote. He also asked ChatGPT for a poem "so epic that it could make her change her mind but not cheesy or over the top."

Others uploaded medical reports and screenshots of conversations with sexual partners.

Computers, unlike humans, don't judge. When people talk, we engage in "impression management," says Jonathan Gratch from USC — we regulate behavior to hide weaknesses.

People "don't see the machine as socially evaluating them like a person might," he said.

The AI industry lacks a solid strategy to monetize this ability to befriend people, but some early adopters have emerged: Snapchat announced it would use chatbot conversations to personalize ads.

Soon, we'll see product-related AI chatbots on corporate websites, from Herbalife to John Deere. These chatbots will assist with dietary help, potato farming, or naming your new dog.

There is no doubt: these chatbots will make users reveal desires, emotions, goals, and objectives. Tons of easily monetizable information.

This post is based on an article “Shh, ChatGPT. That’s a Secret” by Lila Shroff, an assistant editor at The Atlantic. The original text published in The Atlantic on October 2, 2024.