Internet Archive Blogs

The Internet Archive will be part of a team that is working to address a key challenge for students with disabilities: getting books in accessible formats. This participation aligns with an existing Internet Archive program to make materials available and accessible to readers with disabilities.

The number of students with disabilities at colleges and universities has grown over the past few decades. Many of those students have print disabilities, including the largest subgroup, those with learning differences.  Students with print disabilities require text to be reformatted for screen readers, text-to-speech software, or other forms of audio delivery, often with human intervention. Universities are required to perform this reformatting on request but are rarely staffed to do that work at scale and this type of reformatting and remediation can cost hundreds or even thousands of dollars. Once the work has been done for a student at one university, the reformatted book is almost never made available for use by students with disabilities at other universities.  Without collaboration and coordination across campuses efforts are wasted and students with disabilities often wait weeks to get texts in a form they can access and use.

A newly-funded pilot project, “Federated Repositories of Accessible Materials for Higher Education,” aims to address this problem. This is a two-year pilot program that has recently been funded by a $1,000,000 grant from The Andrew W. Mellon Foundation to the University of Virginia (as principal investigator) with a primary goal of reducing the duplication of remediation activity across the seve (7) universities participating in the pilot. It will also support the cumulative improvement of accessible texts and decrease the turnaround time for delivering those texts to students and faculty.

Within this program, the Internet Archive will participate as one of several repositories of digitized books, both to provide initial digital copies (for remediation) and to receive and hold remediated book files. Those improved books can then be shared with other schools and organizations that provide services to people with disabilities. They may also be used as a starting point for further conversion into additional formats (such as Braille) that may be needed to support specific reader needs.

The Internet Archive’s role in this pilot project dovetails with our existing program to make materials available and accessible to readers with disabilities. Our current program allows any organization that is already working with people with disabilities, known as Qualifying Authorities, to access the digital files of over 1.8 million books (about 900,000 of which are otherwise unavailable). Those Qualifying Authorities, especially Disability Student Service teams at colleges and universities, are then able to streamline their preparation and remediation of these digital books for people with print disabilities. Because they work directly with individual readers, Qualifying Authorities are also able to enable existing (and qualified) Internet Archive users for an account with disability access. With that access, these users can enjoy expanded and immediate access to the Internet Archive’s full collection of books (through archive.org or OpenLibrary).

We are excited to participate in and support the wider community of teams working to make books accessible for all.

When you visit a public library, you get to meet the librarians and others who build and care for those collections. You know there are people who empty the garbage cans, who put back the borrowed books, who maintain the computers, and who determine what ends up on the shelf.

A digital library, on the other hand, is “just” a web site.  You don’t really see the people who build it – we are often anonymous. But the Internet Archive wasn’t built by computers and algorithms.

From its inception, the Internet Archive has been built by thousands of people who understand that we have an opportunity to use the Internet to give everyone access to knowledge. Every person on the planet should have the opportunity to learn and to make a contribution.

This goal – Universal Access to All Knowledge – inspires the people who have built the Internet Archive over the past 23 years.  

People clean and repair the buildings that we occupy. People do payroll, choose our health plans, answer the phones, plan our events, reply to user emails, clean up spam, and pay our bills. People design and build the computers that hold the collections. People construct the network that carries data to every corner of the world. People write software that processes, backs up, and delivers files. People design and test and build interfaces. People digitize analog media and type in metadata. People curate collections, establish collaborations, and manage projects.

There’s no way I can mention all of these people by name. Even if I listed every employee from the past 23 years, I would still be missing the volunteers, the people from other organizations who worked on joint projects with us, the pro bono lawyers, the delightfully compulsive collectors, the funding organizations, the idea generators, our sounding boards for crazy ideas, the individuals who have donated money or materials, and the hundreds of thousands of people who have uploaded media into the archive.

Libraries are built by people, for people.  Thank you so much to all of the people who have contributed to building the Internet Archive, whether they were employees or our huge group friends and family.  We would not be here without you, and we hope you will continue to help bring universal access to all knowledge in the future.

Happy Valentine’s Day!

With many thousands of software items up at the archive, we’re both very useful and also very intimidating, depending on how exactly you know what you’re looking for. While it’s great when your search query gives you exactly what you need (like, say, a manual for the greatest elevator simulator of all time or a lovely flip-album of floppy disk sleeves), it’s not so great when it doesn’t.

Our rather expansive approach to acquisition of items means that if you have a long-hazy memory of something you want to see again or want to do a query in a generalized “show me all the shooters that came out for this platform”, you’ve got a lot of digging ahead of you. I’ve had many lovely conversations with people who are looking for something specific software or game-wise, that have ended with being able to point them to an emulated version of it. Other times, I have to hand them a way to look inside a CD-ROM image from nearly 20 years ago, like this URL inside a GIF CD-ROM from 1992, which was a lovely rendered image of the Apple Logo and semi-transparent balls.

Here’s the image, which is just nice to look at:

Beyond the findability problem, there’s also the deeper problem that computer history has a lot of buried bodies. There were conflicts and issues related to interoperability, who ran what standards, and which programs actually did what they were supposed to. These problems persist in the modern world, but they have rapidly become the province of several abstract layers away: “my Playstation 4 doesn’t play every Playstation 3 game”, or “I can’t paste this image into my twitter post with a simple copy-paste, I have to put it in a paint program and copy-paste that.”

It used to be a lot, lot worse.

Which brings us to .ZIP.

A SHORT (COMPRESSED) HISTORY TO COMPRESSION

Since computers have come onto the scene, connections between them (and to the user) have always suffered for lack of bandwidth. Sending text, data, images and sounds between different locations has always been some level of slow or undependable. There have been lots of innovations across the decades to deal with it; one of them is compression techniques.

This is where the computer takes a file or sets of files, combines them, finds similar parts, and replaces those similar parts with one-off references to them. The algorithms to do these have become more complicated over time and require more computing power on the compressing end, and in some cases the decompressing end.

And here’s the thing: There have been a lot of file compression formats.

So many of them, in fact, that there’s some legitimate concern that there are compressed files out there for which no decompression program exists anymore. That’s certainly the case for a lot of proprietary file storage formats that were meant to run with one specific program (think a game data file, or a word processing program), but we’re sticking to generalized “File Compression Utility” formats in this essay.

Just in the IBM/DOS world, here are some file compression format extensions that have been created for a variety of reasons and which have been considered as in use:

ARJ, LZH, PAK, ARC, ZOO, SQZ, HYP, ARCE, ARC128, ARC286, .PAK, UC2, LHA, LBR, SFX, HAP, HA, DWC, LAR, SQZ, PIT, SIT, ICE

Some of these were made for other machines, but were made available via utility to the DOS world. They’ve got great names, reflected in the filename but just barely; names like Hamnersoft HAP/ Knowledge Dynamics, Voof, Zoo, Novosielski, ShrinkIt, and ReeveSoft Freeze. Pretty much all have fallen to the wayside in various usage (as has DOS itself) so we don’t generally see new versions of these show up.

Except .ZIP. ZIP won the battle, and is the dominant compression scheme for “files” (as opposed to video/audio compression).

But what is .ZIP?

ZIP is ZIP, except Not ZIP

Co-created by Phil Katz and Gary Conway in 1989, .ZIP was a reaction to a lawsuit. In the growing realm of file compression utilities, one format, .ARC, created by System Enhancement Associates, had started to rise, and PKWARE (Katz’ company) made a competing product, PKARC, that used original .ARC source code but rewrote it in faster routines, making it speedier. System Enhancement Associates sued PKWARE and won in a settlement, resulting in abandoning .ARC and a new format being created. The bad blood and publicity from the lawsuit helped drive adoption/conversion to the replacement format, .ZIP.

(I actually made a documentary about this part of the story.)

ZIP’s wide adoption and easy, clear documentation of the format meant support for it started expanding over time. Besides compressing the files themselves, a format like .ZIP preserves timestamps, has integrity checks, and maintains directory structure. (Many others do this as well.). If you uncompress a .ZIP file from 1992, you’ll be able to see when it was created and compressed, and other important data from a historical perspective. Also, if the file is from the early 1990s, chances of unpacking these .ZIP files successfully with any of a large range of current methods are really, really high. Drag it to your Windows, OSX or *nix environment, and chances are you’ll do fine.

The closer you get to now, though, and problems arise.

The most damning issue is that different operating system versions approach .ZIP slightly differently, which mostly works, and lets you even treat a .ZIP file like a little disk drive or folder, adding and removing files within it while preserving the compression. Why unpack 800 megabytes of files when you only need this single 5 megabyte one? Similarly, you can construct a new .ZIP file on your desktop, adjust a bunch of parameters within it, and poof, a .ZIP file you can attach to e-mail or pass along via other ways.

But between 1989 to now, with ZIP being 30 years old, there have been expansions to the format, small changes that make it backwards compatible, but with nothing to easily tell a user that they’re using an out of date or different uncompression program.

The current cross-platform king is Info-ZIP, which has a homepage that credits the many people who have worked on it and access to the versions from over the years. It has been continually maintained to handle new issues, and is generally excellent at backwards compatibility. It’s probably your best bet to getting the information back out of a .ZIP file.

But that’s not what everyone uses.

“It Doesn’t Work”

On dozens of software items at the Internet Archive are reviews where a strange phenomenon happens:

• Some reviews indicate the contents were just what they were looking for.
• Some declare it broken, and terrible and truncated.

They’re both right.

One of the most problematic technical issues on a day to day basis with computers are the bit limits. When you hear discussions of “8-bit”, “16-bit”, “32-bit” and “64-bit”, it usually reflects some resource within the system (graphics, filesystem, pipeline) being limited to a certain amount of addressing. If your daily job is computer development, this is probably old news to you; but not everyone’s daily job is computer development.

In general, a modern system will be some amount of 64-bit, with some 32-bit addressing thrown in a few corners simply because it’s not thought there’ll be a use for more. 32-bit is, very roughly, about 3 gigabytes of information.

This means that when someone on the Archive uploads a .ZIP file that is larger than 3 gigabytes, there’s a somewhat good chance that a patron who downloads that file will not have the ability to uncompress/unpack that file using the tools on their specific desktop. If they use the internal tools (or a downloaded tool) to go through that .ZIP, the program (or even the operating system itself) won’t know what to do with this very large file, and begin throwing out errors.

However, since the nature of .zip files is to be somewhat resilient, some files will make it out. It’ll start to unpack them, then declare a corruption or a bug and stop working. So it looks like some of it’s there, but not what the user was expecting or needed.

What Is The Lesson Here?

As the Internet Archive continues growing in acquiring software and files, our propensity for easily searchable and accessible programs means that people will rush in, encounter a file like a .ZIP file, and not know about this 30 year+ history with that format and issues that could arise. How could they be expected to?

In earlier eras of computer history, the user was expected to be able to build and pilot the ship as comfortably as ride in it as a passenger. Thankfully, those days are mostly behind us and picking up a piece of technology and using it runs into issues like placement of buttons or lacking a headphone jack, instead of concerns of header information or data formats.

But under this surface of ease and frictionless experience is the occasional roiling current of decisions, movements and changes. It reflects how truly unsettled our computer world is, and how, every once in a while, we get a glimpse into it in ways that are not obvious.

It’s a privilege to be able to hold and present these vintage programs and documents from technology and time long past. But these items lived in an environment and support structure now truly gone, and it is sometimes a period of rediscovery for researchers professional, academic and hobbyist to re-learn what we’ve forgotten.

Hopefully the archive can help remember that too.

Further Reading

• Episode of Computer Chronicles about File Compression, including Phil Katz
• Zip File Format
• Phil Katz, ZIP’s co-creator
• Lempel-Ziv-Welch, a very widely used compression scheme
• The “3 Gigabyte Limit“, a clear example of 32-bit addressing limits

Guest blog post by professor Tom Gally

As international travel becomes cheaper and easier, many of the tourists who now swamp Venice, Barcelona, San Francisco, and Hong Kong are visiting a foreign country for the first time. Surprised, fascinated, and sometimes repulsed by what they see, they eagerly post to social media their photos and impressions. Such reports are the source of much of what we believe, consciously or unconsciously, about places we haven’t visited yet.

Centuries ago, too, travelers were eager to tell their stories to people back home, and those stories helped to create the images and stereotypes that were formed about other lands and people. Many of those stories can be found in the thousands of travel books that are available in the text collections of the Internet Archive.

Here is a description, from a book published in London in 1701, of an Englishman’s first impressions of Paris:

Having enter’d this famous City, we were set down near the Louvre, and drop’d in first at a paltry House where the Fellow call’d himself in his Sign Le grand Voyageru, (or great Traveller) and pretended to Speak all Languages, but could scarce speak his own. Finding here but indifferent Accommodation, our Man provided us a Lodging in a House, where liv’d no less than two and twenty Families; thither we were carried in Sedans with Wheels, drag’d along by one Man, no Hackney-Coaches being then to be had. This was on a Sunday, and I was not a little surpriz’d to see Violins about the Streets, and People singing and dancing every where, as if they had been mad.

Though the language is archaic, the sentiments—bragging about visiting a famous city, complaining about accommodation and transportation, frowning at the local customs—would not be out of place in a tourist’s Facebook post today.

In the early 1790s, King George III sent an envoy to the Emperor of China. Though the diplomatic mission was unsuccessful in its main purpose—to obtain trade concessions for Britain similar to those granted to the Portuguese and Dutch—it yielded a three-volume official report, by George Staunton, that contains a fascinating account of the long voyage halfway around the world (volume 1) and of the Chinese empire as seen through British eyes (volume 2). The report also includes many carefully engraved illustrations of sights in China—the Instagram posts of the era (volume 3).

Other travelers’ accounts I’ve dipped into include Travels from St. Petersburg, in Russia, to Diverse Parts of Asia by John Bell (1763) (volume 1, volume 2), Travels in America by George Howard (1851) (here), and a large compendium titled Cyclopædia of Modern Travel by Bayard Taylor (1856) (here).

Lately, I’ve also been exploring the Internet Archive’s rich collection of books written by British and American visitors to Japan in the 19th and early 20th centuries. Until the 1850s, Japan had been shut off nearly completely from the rest of the world for more than two hundred years, and people elsewhere were eager to learn about the mysterious country. Many sailors, traders, diplomats, missionaries, journalists, and individual travelers who were able to visit Japan wrote later about their experiences, and I’ve compiled a list of more than 240 of their books.

I myself moved to Japan in 1983 and have lived here ever since. As I read now the accounts of Westerners who arrived at Nagasaki or Yokohama in 1858 or 1869 or 1880 or 1905, I recall my own vivid first impressions of the country 36 years ago. While there are many differences—they rode rickshas, I took commuter trains; those Victorians were shocked by the casual nudity, this Californian was surprised by how formally people dressed—our experiences were also similar in many ways. And those who, as I did, stayed for more than a year or two and learned the language gradually came to see how their initial assessments had also been incomplete and sometimes biased.

Several times a week, I pass through the bustling Shibuya crossing in Tokyo, and in recent years I’ve noticed more and more foreign tourists taking pictures of that famous location. After reading travelers’ accounts from more than a century ago, I increasingly wonder how tourists today are perceiving this country that is now my home, and I speculate how people elsewhere, seeing those photos posted to Instagram and Twitter and Weibo, will come to view that intersection and this country. I never would have thought deeply about this, and I certainly wouldn’t be contrasting our experiences with those of 19th-century visitors, if it weren’t for the great collections of books that the Internet Archive makes available for anyone in the world to read.

Tom Gally was born in Pasadena, California, in 1957. Since moving to Japan, he has worked as a translator, teacher, lexicographer, and writer. He is now a professor in the Graduate School of Arts and Sciences at the University of Tokyo and is compiling a book of excerpts from travelers’ accounts to be titled Japan As They Saw It. Samples can be read at the book’s website.

By Paul Soulellis

We usually think about archives as places of abundance. Deep, rich sites that house a multitude of perspectives. This can certainly be true, but archives are also sites of erasure, allowing some voices or perspectives to be minimized and excluded when they don’t fit into normative narratives.

Traditionally, stories involving people of color, queer people, and other historically-marginalized voices have been left out of archives, or diminished, because of ignorance, homophobia, and racism. Histories aren’t “discovered” in archives; rather, we use archives to actively construct versions of history, stories that accommodate our own subjective positions and ideologies. All too frequently, these stories favor the familiar structures of oppressive power—whiteness, patriarchy, and capitalism.

Likewise, the public domain is a remarkable construction that allows us to define who is or isn’t included in normative narratives. The public domain proclaims certain material as property owned by no one; cultural material in the public domain, theoretically, belongs to everyone. As copyright law enables new content to enter the public domain each year, it’s important to look closely at which voices are amplified in the celebration of open culture. There is no actual public domain. There is no site or territory or designation that reflects an authentic condition of “making public.” 

Rather, it’s a complex, evolving structure defined by the institutions that serve as portals to cultural material—museums, libraries, courts, and archives like this one. They carry a responsibility to give (or deny) access to materials that traverse in and out of the public domain. But as an institutional construct, the public domain can easily fail to reflect any true nature of “the public;” without careful consideration, access to the public domain ends up repeating and perpetuating, in a highly predictable way, the same oppressive structures that govern society and culture.     

What can be done? It’s crucial that we carefully examine our archives and search for lost voices, stories of failure, non-linear trajectories, and other non-conventional perspectives. We must refuse to accept traditional timelines at face value, and work to amplify marginalized material that has otherwise gone unnoticed, or erased. When confronting an archive or any presentation of historic cultural material, it’s irresponsible not to ask urgent questions like: What forces shaped this? Who was excluded? Who else should be included here in order to better understand the material at hand? Once engaged, we can actively work to change the shape of history, giving it dimension and depth and greater representation for all who were involved. This is what I’ve been calling queer archive work.  

I’m really grateful to the Internet Archive for inviting me to help shape their effort to present newly available material in the public domain. During my residency here, for the last 3 weeks, I’ve been searching archive.org for forgotten material — in particular, evidence of African-American culture, Native American culture, early LGBTQ voices, and other artifacts from 1923 that in the past would have been forgotten or actively left out of celebrations of open access culture. If something seemed to be missing, I tried to find it elsewhere and upload it to archive.org. Remarkably, I found the first openly lesbian book of poetry ever published in North America, On A Grey Thread, by the Bay-area poet Elsa Gidlow, from 1923. It had never been digitized, but a PDF from the author’s estate was sent to me for this project and is now online, as of a few days ago.

The result is QUEER.ARCHIVE.WORK 2, 1923 INTERNET ARCHIVE EDITION. It’s an edition of 100 copies that I edited, designed, and printed myself at a small press in Berkeley, and it features 15 lesser-known historical artifacts. All of it is now available on archive.org. I’m very proud that the Internet Archive enabled me to create this project. By bringing these items together in a loose assemblage, in the form of a publication, my hope is to create a place for forgotten voices to co-mingle. I think by doing more of this work, we can challenge what we think or assume we know about the early years of the 20th century, and imagine other kinds of histories.

For more see:
http://soulellis.com
http://queer.archive.work
https://queer.archive.work/2/index.html
https://archive.org/details/soulellis

The Internet Archive would appreciate some help from a volunteer programmer to create software that would help determine if a book cover is useful to our users as a thumbnail or if we should use the title page instead. For many of our older books, they have cloth covers that are not useful, for instance:

But others are useful:

Just telling by age is not enough, because even 1923 cloth covers are sometimes good indicators of what the book is about (and are nice looking):

We would like a piece of code that can help us determine if the cover is useful or not to display as the thumbnail of a book. It does not have to be exact, but it would be useful if it knew when it didn’t have a good determination so we could run it by a person.

To help any potential programmer volunteers, we have created folders of hundreds of examples in 3 catatories: year 1923 books with not-very-useful covers, year 1923 books with useful covers, and year 2000 books with useful covers. The filenames of the images are the Internet Archive item identifier that can be used to find the full item:  1922forniaminera00bradrich.jpg would come from https://archive.org/details/1922forniaminera00bradrich.   We would like a program (hopefully fast, small, and free/open source) that would say useful or not-useful and a confidence. 

Interested in helping? Brenton at archive.org is a good point of contact on this project.   Thank you for considering this. We can use the help. You can also use the comments on this post for any questions.

FYI: To create these datasets, I ran these command lines, and then by hand pulled some of the 1923 covers into the “useful” folder.

bash-3.2$ ia search "date:1923 AND mediatype:texts AND NOT collection:opensource AND NOT collection:universallibrary AND scanningcenter:*" --itemlist --sort=downloads\ desc | he\
ad -1000 | parallel --will-cite -j10 "curl -Ls https://archive.org/download/{}/page/cover_.jpg?fail=soon.jpg\&cnt=0 >> ~/tmp/cloth/{}.jpg"

bash-3.2$ ia search "date:2000 AND mediatype:texts AND scanningcenter:cebu" --itemlist --sort=downloads\ desc | head -1000 | parallel --will-cite -j10 "curl -Ls https://archive.\
org/download/{}/page/cover_.jpg?fail=soon.jpg\&cnt=0 >> ~/tmp/picture/{}.jpg"