Is the internet’s memory at risk? The world's most-used archiving tool is under threat

For nearly 30 years, the Internet Archive, often described as the "Digital Library of Alexandria," has quietly built what many consider the most comprehensive record of the modern internet.

At the centre of this effort is the Wayback Machine, a platform that has preserved more than a trillion web pages and allowed users to revisit the digital past with remarkable precision.

From tracking deleted political statements to preserving vanishing news reports, the Wayback Machine has become indispensable to journalists, researchers, courts, and the public.

Yet now, this cornerstone of digital accountability is confronting a growing array of challenge, most recent being a growing resistance from major media organisations. The outcome of these pressures could determine not only the future of the Internet Archive but also the accessibility of the internet’s historical record.

More from Explainers

Time to choose ethical intelligence over artificial intelligence

FOBO (The fear of becoming obsolete): How anxiety triggered by AI is taking over workplaces

What is the Wayback Machine?

The Wayback Machine was established in 1996 by Brewster Kahle and Bruce Gilliat as a long-term project to document the rapidly expanding World Wide Web.

While conventional search engines such as Google focus on indexing current content, the Wayback Machine is designed to capture snapshots of websites at different points in time, enabling users to examine how content has evolved — or disappeared.

This process is powered by automated crawlers, including the widely used ia_archiverbot, which systematically navigate web pages, download their contents, and store them in a format that can later be reconstructed.

These archived versions include not just text, but images, scripts, and layouts, allowing users to view pages as they originally appeared.

A key feature of the platform is the “Save Page Now” tool, which enables individuals to manually archive a webpage, generating a timestamped record that can serve as verifiable evidence of what was published at a specific moment.

The scale of the archive is immense. By early 2026, the Wayback Machine had surpassed one trillion stored pages. Beyond websites, the broader Internet Archive hosts millions of digitised books, audio recordings, videos, and software.

This work addresses a fundamental issue of the internet era: impermanence. Web content is frequently altered or removed, often within a matter of months.

Estimates suggest that the average webpage remains unchanged for only about 100 days before being modified or deleted. This constant churn, commonly referred to as “link rot,” poses a significant challenge to preserving reliable records.

Why is the Wayback Machine essential?

The importance of the Wayback Machine extends well beyond nostalgia or casual browsing.

In journalism, the platform is widely used to verify claims, track editorial changes, and uncover discrepancies in reporting. It enables reporters to compare earlier versions of articles with updated ones, exposing modifications that might otherwise go unnoticed.

This function has played a role in high-profile cases, including scrutiny of editorial changes to a The New York Times article about Bernie Sanders during the 2016 US election cycle.

Legal systems have also come to rely on archived web pages. Courts in the United States and other jurisdictions accept Wayback Machine snapshots as evidence, using them to establish what information was publicly available at specific times.

In academic research, the archive helps preserve the integrity of citations. Scholars frequently reference online sources, but when those sources disappear, the validity of their work can be compromised.

By providing stable, permanent links, the Wayback Machine ensures that referenced materials remain accessible.

The platform also serves a role in human rights and accountability. In regions where governments censor online content or shut down websites, archived copies may be the only surviving record of suppressed information, including reports, blog posts, or activist materials.

Journalists themselves have emphasised the platform’s practical utility. Laura Flynn of The Intercept described the archive as an “essential tool” for fact-checking and retrieving historical material, while writer Micco Caporale highlighted its usefulness in accessing older cultural content and tracking changes in job listings over time, reported WIRED.

Caporale noted, “I've also been using the Wayback Machine a ton in my union organizing work to find old job listings so we know what the company claimed to hire people for vs. what duties they actually assigned or to see how different positions have been retooled at different points,” adding, “These posts also help us keep track of pay fluctuations across the organization over time.”

What is behind the growing trend of archival blocking?

Despite its broad utility, the Wayback Machine is increasingly being restricted by major online platforms and media organisations. An analysis by the artificial intelligence detection firm Originality AI found that at least 23 major news websites have blocked ia_archiverbot, preventing the Internet Archive from capturing their content.

Among those imposing restrictions are USA Today — which operates more than 200 outlets — as well as Reddit and The New York Times.

The impact of these decisions is substantial. When large networks such as USA Today Co. block archiving tools, hundreds of affiliated publications effectively disappear from the archival record.

Some organisations have adopted more nuanced approaches. The Guardian allows its content to be crawled but limits how it appears within the Wayback Machine, excluding it from the Internet Archive’s application programming interface (API) and filtering it from user-facing search results.

This also makes it significantly more difficult for the public to access historical versions of its reporting.

Publishers have offered various explanations for these restrictions. A spokesperson for USA Today Co. stated that “this effort is not about specifically blocking the Internet Archive” but instead part of broader measures to prevent automated scraping.

Similarly, Robert Hahn, the Guardian’s director of business affairs and licensing, said the organisation has been engaged in discussions with the Internet Archive over “concerns over potential misuse by AI companies of content sets crawled for preservation purposes.”

At the centre of the issue is the rise of generative artificial intelligence. Media companies are increasingly concerned that archived content could be used to train AI models without compensation, potentially enabling competitors to replicate or summarise their work.

A spokesperson for The New York Times, Graham James, said that “the issue is that Times content on the Internet Archive is being used by AI companies in violation of copyright law to directly compete with us.” The organisation did not clarify whether this concern is based on documented instances or precautionary assumptions.

Reddit has similarly cited AI-related concerns in its decision to restrict access to its data.

Why is this move contradictory?

The move by publishers to block the Wayback Machine has drawn criticism from within the journalism community, particularly because many of these organisations rely on the archive for their own reporting.

Mark Graham, director of the Wayback Machine, highlighted this inconsistency, stating, “They're able to pull together their story research because the Wayback Machine exists. At the same time, they're blocking access.”

There are documented instances where media organisations have used the archive to support investigative work while simultaneously preventing their own content from being preserved.

The stakes are particularly high when it comes to tracking changes in published content. Without independent archives, news organisations could effectively control access to their own historical records, limiting the ability of outsiders to verify edits or examine how stories have evolved.

How does AI play into this?

As AI companies seek vast datasets to train models, archived web content has become a valuable resource. The Wayback Machine’s extensive collection makes it particularly attractive for this purpose.

However, media companies increasingly view their content as proprietary data that can be monetised through licensing agreements with AI developers. By blocking the Internet Archive, they aim to prevent unauthorised use of their material.

What was once freely accessible information is now being treated as a strategic asset, with companies seeking to regulate its distribution and usage.

The unintended consequence of this approach is the potential fragmentation of the historical record. If large portions of the web are excluded from public archives, future researchers may encounter gaps in documentation, limiting their ability to reconstruct events or analyse trends.

The cumulative effect of these challenges poses a significant threat to the Wayback Machine’s mission. Without consistent access to major sources of information, the archive’s ability to provide a comprehensive record of the web will be diminished.

This could lead to a scenario where only partial histories are preserved, with key content either inaccessible or entirely lost.

Mark Graham warned, stating that “there's no question that the general locking-down of more and more of the public web is impacting society’s ability to understand what's going on in our world.”

Is there a pushback?

In response to the growing restrictions, dozens of journalists and advocacy groups have mobilised in support of the Internet Archive. More than 100 journalists have signed a coalition letter defending the Wayback Machine’s role in preserving the public record.

The initiative, supported by organisations such as the Electronic Frontier Foundation and Fight for the Future, points out the archive’s importance in an era where traditional physical archives are declining.

The letter states, “In previous generations, journalists would turn to the physical archives of a local newspaper or of a local public library to access historical reporting and follow the threads of the present back into history,” and adds, “With many newspapers closed, and no clear path for local public libraries to preserve digital-only reporting, the work of safeguarding journalism’s record increasingly falls to the Internet Archive.”

Signatories include prominent figures such as Rachel Maddow as well as independent reporters and media workers across different sectors. For many journalists, the issue is not merely about access to information but about preserving the integrity of the historical record.

As newsrooms shrink and local publications disappear, the responsibility for maintaining archives has increasingly shifted to digital platforms like the Internet Archive.

Is there a replacement for Wayback Machine?

The Internet Archive has indicated that it remains engaged in discussions with publishers that have restricted access, expressing hope that some may reconsider their positions.

However, the broader conflict — encompassing copyright law, AI development, and data ownership — shows little sign of resolution.

At present, there is no widely available alternative that matches the scale or accessibility of the Wayback Machine.

Its potential decline would leave a significant gap in the digital ecosystem.

Why does this come at a bad time for Internet Archive?

In addition to publisher restrictions, the Internet Archive is already contending with other legal challenges that have strained its resources.

One major case involved book publishers who sued the organisation over its “Open Library” initiative, which allowed users to borrow digitised versions of books.

After losing an appeal in 2024, the Internet Archive was required to remove more than 500,000 titles from its collection. The court ruled that its system of Controlled Digital Lending constituted copyright infringement rather than fair use.

Another legal dispute centred on the “Great 78 Project,” an effort to preserve vintage audio recordings. Major music labels, including Sony and Universal, sought damages that could have reached hundreds of millions of dollars.

The case was ultimately settled in 2025 for a confidential amount, though it is widely understood to have been substantial.

The organisation has also faced serious cybersecurity threats. In October 2024, the Internet Archive experienced a major data breach that exposed information from approximately 31 million users. This incident was followed by sustained distributed denial-of-service (DDoS) attacks, which disrupted access to the platform for an extended period.

Operating with limited resources, the Internet Archive has had to allocate significant funds toward strengthening its security infrastructure.

These battles have had broader implications for the organisation’s financial stability. As a nonprofit entity operating under a 501(c)(3) designation, the Internet Archive relies heavily on donations.

Prolonged litigation, settlements and cyberattacks have placed additional pressure on its funding, raising concerns about its long-term sustainability.

With inputs from agencies