Trust and Safety
Simson Garfinkel
LinkedIn Influencer ? Chief Scientist @BasisTech, Lecturer @Harvard
If your website or service allows users to post comments or exchange messages with other users, then you will eventually confront the problem of abuse. You may not need a Trust and Safety (T&S) team, but you need to understand why T&S is important, and how the best practices have evolved in recent years.
If you are an entrepreneur, you may be surprised to know that Trust and Safety startups “raised $324 million globally in 2023, up from $48.5 million in 2019,” according to a May 29, 2024 article on pitchbook.com.
Trust and Safety Defined
The term “Trust and Safety” has evolved in recent years to describe the function that platforms perform to address user complaints and police their content and communities.
The term reflects an effort on the part of these professionals to present what they do with positive, platform-enhancing language. “Trust” because responding quickly to content that is abusive or toxic or simply false is thought to enhance the “trust” that users have in platform or online service. “Safety” because these efforts are thought to make the online platform a safer place for users.
And while the term “Trust and Safety” has been around for more than a decade, many people I encounter in the tech think that T&S is just for large organizations like Facebook, X and YouTube. While T&S is important for these organizations, the rise of disinformation, targeted harassment, and bots posing as humans made possible with generative AI means that any platform that accepts user-generated content will likely have the need for either T&S professionals, automated solutions, or both.
T&S faces two key problems today. The first is scale: using human to screen content and complaints just doesn’t scale, because much of problems come from automated attacks. This means that any modern T&S solution must use some degree of automation. So like systems that fight traditional spam and malware, T&S systems are in a perpetual arms race with the armies of people generating malicious, toxic content.
The second problem that T&S solutions face is avoiding bias and unwarranted censorship. In many cases there is no clear line between is acceptable content and what should be screened. Overly aggressive activities can also negatively impact a platform’s reputation.
Discord’s Trust and Safety group has published articles for its users on a wide variety of T&S topics, including what to do if your account is hacked, how to report abusive behavior on the platform, how to control content filters, what’s going on with the platform’s safety alerts and warnings, protecting discord servers from “raids” (this is a Discord thing), controlling how bots interact with user data, the company’s crisis text line, blocking and privacy settings, and other information.
The Trust & Safety page at Pinterest both tells what the T&S team does and is advertising two openings: “Manager; Vendor Management, Trust & Safety”, and “Lead Product Policy Manager, Spam.”
Incognia, a company that has developed software for identifying Internet users to reduce fraud and create “more secure digital experiences,” encourages its customers to create trust and safety teams. Such teams include a team lead (manager), people in courage of operations, policy writers, content moderators, and people who speicalize in fraud detection and prevention. “No one wants to establish an online community that people are going to think of as a lawless digital land to wreak havoc and commit fraud. Trust and safety teams are the professionals who ensure that the platform’s compliance and general atmosphere match the executive team’s vision for their service,” reads an article on the company’s website.
Trust & Safety Companies
The tech layoffs of 2023 hit Trust and Safety teams particularly hard. Like security and usability, it’s hard for C-level executives to understand T&S as anything other than a cost—one that presumably had to be cut. The problem is, T&S was a vital function to preserving platform integrity.
However, many big tech firms had organically grown their T&S function, resulting in services that were heavily labor intensive and built on top of tools design for customer support, not for stamping out AI-generated disinformation. The result: some of those teams laid off decided to build new T&S platforms from scratch, using their own AI solutions, and sell T&S services back to the tech giants. WIRED wrote about the “burgeoning trust and safety industry” back in November 2023.
Right now T&S companies seem to be clustering in two main areas, fraud detection and content moderation. There’s clearly overlap, however, since one way to detect fraud is by problematic content, and one way to find bad content is to link it back to a fraudulent account. The industry is also undergoing acquisition, with the 2022 acquisition of the risk intelligence firm Crisp by Kroll, and last year’s acquisition of SpectrumLabs by ActiveFence.
Some companies in the Trust and Safety space that are worth looking at include:
That’s a lot to track, and I won’t keep this list up to date. So you might also want to check:
Wired’s Victoria Elliott wrote about the T&S startup scene back in November 2023.
Annalee Newitz also wrote about T&S back in November 22 in NewScientist, but her article “Trust and safety – the most important tech job you’ve never heard of” is trapped behind a paywall.
Even The World Economic Forum wrote about T&S, “This is why you need to know more about digital trust and safety,” back in November 2022.
Getting Involved
Want to know more?
This July, the Trust and Safety Professional Association will host Trust Con 2024 in San Francisco on July 22, 2024. Searching on YouTube, I was able to find a playlist for Day One and Day Two of TrustCon 2023. I was also able to find the a few webinars, including the 58-minute “Wait, Who Decided I was In Charge?” and “That Escalated Quickly.” ?TSPA has a YouTube channel that’s worth reviewing.
In September, the third Trust and Safety Research Conference will be hosted at Stanford, September 26-27. The videos from the 2022 conference are available online.
For More Information
The Trust & Safety Foundation has created a public archive of “the History of Trust & Safety.”
The Digital Trust & Safety Partnership is a trade organization that includes many major tech players, including Apple, bitly, Discord, Google, LinkedIn, Match, Meta, Microsoft, Pinterest, reedit, TikTok, twicth and Zoom. Missing is X, which recently renamed its trust and safety group to be just “Safety,” according to a March 11, 2024 article in SocialMediaToday.
The Journal of Online Trust & Safety recently published eight issue (Volume 2, No. 3, May 28, 2024).
In March, Ben Goggin at NBC News wrote an article, “Big Tech companies reveal trust and safety cuts in disclosures to Senate Judiciary Committee”
?
Team Builder, Startup Cofounder and App Store Inventor
5 个月Trust and Safety are the most important topics, but Privacy is the key to the solution my old friend. https://www.dhirubhai.net/pulse/immediate-safety-how-transparency-flips-script-truanon-jakpe/
Geek who doesn't know how to present himself professionally on social media
5 个月Thank you for writing this. It also goes beyond systems that explicitly have messaging systems. Any user created text that can make it to other people becomes a T&S concern. Consider user or account names. Getting email from a service that says “User I-will-skin-you-alive would like to connect.” Many services see such things being abused for same, eg, “User cheap-viagra-5551234” is trying to reach you. My advice to such organizations is that if they see something being used for spam, it is also being used for harassment even if you aren’t (yet) aware of it.