The internet is forever, except when it isn't. "Link rot"—where once-valid links to websites become broken over time as pages move or sites go offline—is a real problem for people who try to do research online. The Internet Archive helps solve this problem by making submitted content available in the "Wayback Machine."
The difficulty, of course, is getting people to remember to submit links for archival.
This is where Parker Higgins's new Twitter bot comes in. The @LinkArchiver account automatically submits links from accounts it follows to the Internet Archive. If a Twitter user follows @LinkArchiver, it will follow back and continue to add links even if the user unfollows the bot. This means links shared on Twitter can be preserved in perpetuity with no effort on the user's part.
The low-effort aspect is very appealing to Higgins. "I'm most excited by how passive the whole set-up is," he told Opensource.com. "If you rely on people to pick and choose what is important to archive, you miss a lot of the most important stuff. By just grabbing a copy of every link as it's posted, this bot should help ensure we don't end up missing that context."
After initially developing the bot, Higgins contacted the Internet Archive. His concerns that the automation would cause problems were quickly dispelled. "It's effectively a rounding error in terms of the traffic they handle," he says, though he did give API requests a custom user-agent string at their request. The scalability concerns are on the Twitter side: The service limits an account's number of followers and the rate of new followers. This constrains the abilities on a single instance of LinkArchiver.
Fortunately, LinkArchiver is available on GitHub under the AGPLv3 license.
With a small server and a Twitter account, anyone can run this bot. Higgins envisions people running LinkArchiver instances that focus on a particular interest or social circle. "One thing that occurs to me is that you could turn off the follow-back behavior and curate the following list to a specific group or interest. For example, one bot could follow a group of friends or classmates, or major media outlets, or every U.S. Senator and Representative and archive the links they tweet."
This is not Higgins's first foray into Twitter bots: @securethenews, @pomological, and the ever-popular @choochoobot are among his previous work. These bots are all write-only. LinkArchiver is the first interactive bot he has developed, which required learning several new skills. This effort was done as part of Higgins' participation in Recurse Center, a 12-week retreat of sorts for programmers.
Higgins encourages pull requests and additional instances of LinkArchiver bots.
Comments are closed.