Digital Amnesia

Now you can tear a building down But you can’t erase a memory These houses may look all run down But they have a value you can’t see - Living Colour, Open Letter (to a Landlord)

Something’s Rotten in the State of the Web

The year was 1998. The number of broken weblinks in some of his documents ticked off Tim Berners-Lee, the inventor of the web. As head of the World Wide Web Consortium (W3C), he decided to bring focus on the problem and penned “Cool URIs don’t change“. The piece, an optimistic artifact from a different era, explained why good behavior on the behalf of people administering websites guaranteed future-proofing the web. Among some of the gems from this page were:

Pretty much the only good reason for a document to disappear from the Web is that the company which owned the domain name went out of business or can no longer afford to keep the server running. Then why are there so many dangling links in the world??

In 2025, we all know how well this stuck.

In 2013, the New York Times highlighted research by Jonathan Zittrain and Kendra Albert, who showed linked citations in Supreme Court decisions were disappearing at an alarming rate: 49 percent of hyperlinks in Supreme Court decisions no worked at the time. (In a twist of fate, most of the links to Supreme Court pages in that 2013 articles no work, highlighting the point).

Almost a decade later, Zittrain found 72 percent of links the New York Times had linked to when Berners-Lee wrote this vanished. Estimates have ranged as high as 90 percent of all links created in the 1990s no being available today (a 2015 study showed that 98.4% of the links it had in its 20-year sample group were no available)

More recent content lookup are equally alarming:

• In 2023, 38% of all pages created in the previous decade – gone;
• 1 in 5 links on government pages – broken (of which 86% were internal to the sites);
• 32% of links from news site – dead;
• 11% of reference links on Wikipedia – unreachable (with 53% of pages having at least one broken link);
• On social media, 90% of tweets disappeared within 46 days.?

The issue is so widespread that it has a name: Link rot.

From legal citations to research citations, the problem has worsened over the years. While Tim Berners-Lee hoped for web pages lasting 200 years, the sad reality is that digital content has a hard time living even to a decade mark. Current estimates (and these numbers appear to be dropping every year):

• the average website will never celebrate its 5th birthday
• the average webpage may exist for only 3 years
• social web posts don’t make it to 2 years.

In a world of increasing digitalization, we are setting fire to Digital Alexandria on a regular basis.

The Emptiness of the Now

In 2011, Google gave up on dealing with outdated content. While its search crawlers improved, a deluge of new content (a 2016 estimate claimed more data would be created in 2017 than had been in the previous 5,000 years of humanity) threatened to make its goal of organizing the world’s information more complex. With less than 20% of searches being related to more obscure topics, Google, in a move well know in Silicon Valley, would pivot towards what was popular. The leading search engine in the western world, accounting for over 80 percent of all searches, decided to unveil “The Freshness Update,” using recency as a primary driver in its rankings.

The new QDF (“Query Deserves Freshness“) algorithm, would grant primacy to a recently written blog post (because it is classified as fresh) over a 200 year old manuscript. History lost its primacy to recency as driver for the search engine’s narratives.

The QDF algorithm would have several negative impacts on web diversity:

• Favoring large publishers who can generate frequent content updates over smaller, independent sites with potentially deeper but less frequently updated content
• Creating pressure for constant content refreshing, which can lead to superficial updates rather than substantial improvements
• Reducing the visibility of older but still valuable “evergreen” content that doesn’t require frequent updates
• Giving advantage to sites with more resources to produce new content regularly

This led to greater content homogenization, as publishers focused moved to recent events instead of more evergreen offering. It would also increase the primacy of the news cycle as a component of search, making pages related to event outside the news cycle less important than things related to what is happening. The less recent content would be pushed further and further back in search results, until it disappeared.

Mr. Clean

“Those who cannot remember the past are condemned to repeat it” – George Santayana

George Orwell’s 1984 reminds us history can be rewritten (the primary job of the Ministry of Truth in the book) by throwing outdated documents into a “memory hole” where they are destroyed forever. While Orwell had not envisioned our digital world, he looked to book burnings and other document destructions throughout history as examples of how reality could be reshaped based on access to a given record.

Today, bad actors looking to reshape the digital record use a variety of tools to clean up their image, ranging from content removal requests to strategic use of copyright claims and search engine manipulation.

A cottage industry of reputation management agencies handles digital content scrubbing as well as strategic content flooding (ie. creative recent positive content to overwhelm negative information) and astroturfing (creating fake positive reviews or commentary to improve perception) for anyone willing to spend a few thousands. These efforts are leading to a web where veracity is being undermined.

In the entertainment industry, many celebrities have hired reputation management firms to remove unflattering content before major career moves. When casting announcements happen, you’ll often notice old controversial tweets suddenly disappearing. In legal cases, they may have specialists work to have certain content removed from public view, while simultaneously pushing favorable narratives.

Similarly, politicians will use firms to clean up their social media profiles before announcing a campaign. Some may use DMCA takedowns (a legal process that allows copyright owners to request the removal of their intellectual property from websites without filing a lawsuit) to force unfavorable content online. When government change hands from one political party to another, government websites often undergo significant content removal or alteration, particularly on controversial policy areas.

All this is combined to scrubbing the digital reality to conform to a current narrative of what it should be (Winston Churchill said “History is written by the victors” and our current reality appears to follow the same process.)

Meanwhile, in a former Christian Science church building in San Francisco, a group of internet experts, toiling away from the mainstream, were working to ensure those redactions remain visible.