Centralization can lead to price controls, censorship without due process, lack of reader privacy, and resistance to innovators. We need lots of publishers, booksellers, authors, and readers—and lots of libraries. If many actors work together, we can have a robust, distributed publishing and library system, possibly resembling the World Wide Web.

The courts struck down as monopolistic an attempt by top libraries and Google to build a massive e-book collection. They proposed a collective licensing system, the Book Rights Registry, that would have the right to license exclusively to Google any book not claimed by an author or publisher. It would have limited options for readers. Now some proponents of the nonprofit Digital Public Library of America (DPLA) are encouraging legislative action that we fear might lead to a similar collective licensing approach (see “The Library of Utopia.”)

We could be helping nurture the seeds of a distributed library and publishing system—seeds that have already been planted.

All libraries could lend e-books, just as they lend physical books, avoiding a dependency on centralized databases. Libraries are already buying as many e-books as they can, and even small libraries can offer large collections: a single hard drive can hold over 150,000 books (as searchable color PDFs) and their catalogue data. It is not difficult to lend books digitally, as technologies used by Netflix and Amazon show.

Patrons of thousands of libraries can already borrow over 200,000 purchased and scanned e-books free from the Internet Archive. Most large publishers have recently banned e-book sales to libraries, but we hope this restriction is temporary. Even as we acquire current e-books, we need to scan existing ones, but again, this work is already under way. We scan 1,000 books a day at 31 libraries in seven countries with funding from libraries and foundations.

Scanning centers such as those at the Boston Public Library and the Library of Congress digitize hundreds of books a day. Libraries working with the Internet Archive have already put over two million public-domain books online for free downloading and lending, and for use by people unable to read printed books.

Now is our chance to build an online library accessible to all. To equal the Boston Public Library or university libraries like those at Yale or Princeton, we need 10 million books. These could be acquired in four years for approximately $160 million. The DPLA, with its broad support, can help build this library system, or it could end up building an overly centralized library by using collective licensing systems like the Book Rights Registry. If we work together, we can achieve universal access to knowledge by building on the positive lessons of the Internet and World Wide Web.

Brewster Kahle is the founder of the Internet Archive. Rick Prelinger is an archivist, writer, and filmmaker.

http://www.technologyreview.com/article/40261/

We want to buy more digital books from you.

We currently buy, lend, and preserve eBooks from publishers and booksellers, but we have not found many eBooks for sale at any price.  The Internet Archive is running standard protection systems to lend eBooks from our servers through our websites, openlibrary.org and archive.org.   In this way, we strive to provide a seamless experience for our library patrons that replicates a traditional library check-out model, but now with eReaders and searching.

By buying eBooks from you, we hope to continue the productive relationship between libraries and publishers. By respecting the rights and responsibilities that have evolved in the physical era, we believe we will all know how to act: one patron at a time, restrictions on copying, re-format for enduring access, and long term preservation.

We understand these are early days, and prices will evolve.  What we would like to do, however, is not lose the relationship libraries have built up with publishers just because we are now buying and lending electronic books rather than physical ones.

Our checkbook is open.   Please sell to us.

Sincerely,

Brewster Kahle

Digital Librarian, Internet Archive

(inspired by one from Douglas County’s in Colorado)

• minor bump up to “thusnelda” release
• “upgrade” from 1-pass video encoding to 2-pass video encoding
• direct ffmpeg creation of the video (you’ll need to re/compile ffmpeg minimally with “–enable-libtheora –enable-libvorbis” configure flags)

some notes:

• You’d want to adjust the “scale=WIDTH:HEIGHT” accordingly, as well as the “-r FRAMES-PER-SECOND” related args, to your source video.
• I made a small patch to allow *both* bitrate target *and* quality level for theora in ffmpeg, after comparing the other popular tool “ffmpeg2theora” code with the libtheoraenc.c inside ffmpeg.  It may not be necessary, but I believe I saw *slightly* better quality coming out of theora/thusnelda ogg video.  For what it’s worth, my minor patch is here:  http://archive.org/~tracey/downloads/patches/ffmpeg-theora.patch
• The way we compile ffmpeg (ubuntu/linux) is here.  (Alt MacOS version here )
• (Edited post above after I removed this step) It’s *quite* odd, I realize to have ffmpeg transcode both the audio/video together, only to split/demux them back out temporarily.  However, for some videos, the “oggz-comment” step would wipe out the first video keyframe and cause unplayability in chrome (and the expected visual artifacts for things that could play it).   So, we split, comment the audio track, then re-stitch it back together.

BEFORE YOU BEGIN
You will need to have wget installed (it’s free!), and it helps to have some understanding of basic unix commands.  It may also help to read How Archive.org Items Are Structured so that you understand terminology used here.

The basic method for using wget to download files is:

1. Generate a list of item identifiers from which you wish to grab files
2. Create a directory to hold the downloaded files
3. Construct your wget command to retrieve the appropriate files
4. Run the command and wait for it to finish

We strongly recommend trying this process with ONE identifier first to make sure you get what you want as output before you try to download files from many items.

Step 1: Generate list of identifiers

The list of identifiers should be in plain text and should contain one identifier per line with no extraneous characters or spaces.  You can name it anything you’d like, but I will use the name itemlist.txt in all examples here.

How do you get a list of identifiers?  Of course you can just paste or type them into a file individually if you’d like, but if you are working with large numbers of items this will be impractical.  The easiest way to get a list is to use advanced search to return a csv file.

Determine your search query using the search engine.  In this example, I am looking for items in the Prelinger collection with the subject “Health and Hygiene.”  There are currently 41 items that match this query.  Once you’ve figured out your query:

1. Go to the advanced search page and paste your query into the query box of the section titled “Advanced Search returning JSON, XML, and more.”
2. Choose “identifier” from the list of fields to return.
3. Optionally sort the results (sorting by identifier is handy)
4. Enter a number into the “Number of results” box that matches (or is higher than) the number of results your query returns
5. Choose the “CSV format” radio button.
6. Click the search button (may take a while depending on how many results you have)

An alert box will ask if you want your results – click “OK” to proceed.  A file called search.csv will be downloaded to your default download location (often your Desktop or your Downloads folder).  The contents of the CSV will look like this:

“identifier”
“AboutFac1941″
“Attitude1949″
“BodyCare1948″
“Cancer_2″
“Careofth1949″
“Careofth1951″
“CityWate1941″

You’ll need to remove the first heading line “identifier” and remove the double quotes from each line.  The easiest way to do this is to open the file in Excel or a similar spreadsheet program, copy the column of identifiers (minus the heading), and paste it into a text program like TextWrangler or Notepad.  Save your new text file as itemlist.txt (or whatever name you prefer).  The contents of the file should now look like this:

AboutFac1941
Attitude1949
BodyCare1948
Cancer_2
Careofth1949
Careofth1951
CityWate1941

You can use this advanced search method to create lists of thousands of identifiers, although we don’t recommend using it to retrieve more than 10,000 or so items at once (it will time out at a certain point).

Step 2: Create a directory for downloaded files

Open a terminal emulator, such as Terminal (on Mac) or Cygwin (on Windows), and navigate to the location where you’d like to store your downloaded files.  Make sure there is sufficient storage space here.  You may want to store files on an external drive, in a folder on your Desktop, etc.  This example assumes you want to create a folder on your Desktop.

1. Change directories (cd) to your Desktop
2. Make sure itemlist.txt is on your Desktop
3. Create a new folder (directory) for your downloads called archivedownloads (mkdir archivedownloads) – this folder can be called anything you like
4. cd archivedownloads

Step 3: Create wget command

Your wget command will look something like this:

wget -r -H -nc -np -nH –cut-dirs=2 -e robots=off -i ../itemlist.txt -B ‘http://www.archive.org/download/’

Explanation of each option in the wget command:

-r
recursive download; required in order to move from the item identifier down into its individual files

-H
enable spanning across hosts when doing recursive retrieving (the initial URL for the directory will be on www.archive.org, and the individual file locations will be on a specific datanode)

-nc
no clobber; if a local copy already exists of a file, don’t download it again (useful if you have to restart the wget at some point, as it avoids re-downloading all the files that were already done during the first pass)

-np
no parent; ensures that the recursion doesn’t climb back up the directory tree to other items (by, for instance, following the “../” link in the directory listing)

-nH
no host directories; when using -r, wget will create a directory tree to stick the local copies in, starting with the hostname ({datanode}.us.archive.org/), unless -nH is provided

–cut-dirs=2
completes what -nH started by skipping the hostname; when saving files on the local disk (from a URL likehttp://{datanode}.us.archive.org/{drive}/items/{identifier}/{identifier}.pdf), skip the /{drive}/items/ portion of the URL, too, so that all {identifier} directories appear together in the current directory, instead of being buried several levels down in multiple {drive}/items/ directories

-e robots=off
archive.org datanodes contain robots.txt files telling robotic crawlers not to traverse the directory structure; in order to recurse from the directory to the individual files, we need to tell wget to ignore the robots.txt directive

-i ../itemlist.txt
location of input file listing all the URLs to use; “../itemlist” means the list of items should appear one level up in the directory structure, in a file called “itemlist.txt” (you can call the file anything you want, so long as you specify its actual name after -i)

-B ‘http://www.archive.org/download/’
base URL; gets prepended to the text read from the -i file (this is what allows us to have just the identifiers in the itemlist file, rather than the full URL on each line)

 Additional options that may be needed sometimes:

-A
-R
accept-list and reject-list, either limiting the download to certain kinds of file, or excluding certain kinds of file; for instance, -R _orig_jp2.tar,_jpg.pdf would download all files except those whose names end with _orig_jp2.tar or _jpg.pdf, and -A “*zelazny*” -R .ps would download all files containing zelazny in their names, except those ending with .ps. See http://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html for a fuller explanation.

 Step 4: Run the command

Run the command from within the directory or folder you created to hold the downloaded files.  You will see your progress on the screen.  If you have sorted your itemlist.txt alphabetically, you can estimate how far through the list you are based on the screen output.

Depending on how many files you are downloading and their size, it may take quite some time for this command to finish running.

Tips:

• You can terminate the command by pressing “control” and “c” on your keyboard simultaneously while in the terminal window.
• If your command will take a while to complete, make sure your computer is set to never sleep and turn off automatic updates.
• If you think you missed some items (e.g. due to machines being down), you can simply rerun the command after it finishes.  The “no clobber” option in the command will prevent already retrieved files from being overwritten, so only missed files will be retrieved.

Brewster Kahle is honored to be inducted today into the newly formed Internet Hall of Fame administrated by the Internet Society.  As the Society that administers much of the Internet turns 20 years old, they have introduced this award at their annual meeting in Geneva.

“Brewster Kahle is the founder and Digital Librarian of the Internet Archive in 1996. An entrepreneur and Internet pioneer, Brewster invented the first Internet publishing system and helped put newspapers and publishers online in the 1990′s.” –Official Biography of the Internet Society

The Internet Archive’s work, to help build a robust library system reinforcing the publishing systems of the Internet including the World Wide Web, is likely to have been a reason for this award.   As well, this is for Brewster’s earlier work on the first Internet publishing system and search engine, Wide Area Information Servers, in 1989.

The Mellon grant will go towards rebuilding LibriVox’s technical infrastructure, and improving accessibility of the LibriVox website.

“It’s fantastic to get this support from the Mellon Foundation,” said LibriVox founder Hugh McGuire. “It will be put to good use, helping our hard-working volunteers create many more free audiobooks.”

LibriVox, a volunteer project of the Internet Archive, gets volunteers from around the world to make audio recordings of public domain texts, and gives those recordings away for free. All LibriVox audiobooks are hosted at the Internet Archive.

Founded in 2005, LirbiVox has to date produced 5,371 free audiobooks, in 31 languages. Popular audiobooks include “The Adventures of Sherlock Holmes,” by Arthur Conan Doyle, “Adventures of Huckleberry Finn” by Mark Twain, and “Jane Eyre,” by Charlotte Brontë. In addition to novels, the LibriVox collection includes numerous texts of importance from philosophers such as Kant, Descartes, and Hume, political documents such as the “Universal Declaration of Human Rights,” and scientific texts including Einstein’s “Relativity,” and Darwin’s “Origin of the Species.”

“The LibriVox collection is one of the most popular on the Internet Archive,” said Brewster Kahle, Founder and Director of the Internet Archive. “100 million downloads is awesome. LibriVox is an integral part of our commitment to making important texts available to the world in the best format for people, and we are thrilled at the support from the Mellon Foundation.”

Cori Samuel, a long-time LibriVox volunteer, who has recorded some of the project’s more popular books, was in shock at the numbers. “It’s hard to believe that what started out as a small project among some passionate people on the web has turned into something so big. It’s incredible to imagine that we could have touched the lives of 100 million listeners.”

For more information, please contact Hugh McGuire, LibriVox founder: hughmcguire@gmail.com.   Job posting.

Last Thursday we pushed out changes to drop the “www.” prefix from our urls
so that we have the newer/shorter style urls start like:

http://archive.org

We intend to keep this change permanently.

We know there will be a few minor breaks here and there especially from some third-party applications that might not handle “301 Moved Temporarily” redirects (if you have something flash-based that needs http://www.archive.org/crossdomain.xml we caught that breakage and that url still works now (that is, it can be either requested either with or without the lead “www.” as an exception now). We’re happy to work with anyone having issues — feel free to reply to this post and let us know.

Best wishes, and now go spend those four characters saved on something fun

————————-

Sorry for the interruption, but we are moving most of the petabytes of the Wayback Machine from their happy home in a shipping container on the Oracle/Sun campus to our new datacenter.

We are now 1/2 way through the move, and 1/2 of the machines are down.    We are not certain if they will be up tomorrow.    We are looking to move the rest of the machines next Tuesday, so we will have some interruptions then as well.

We will update this post as things move along.

Again, we apologize.

-brewster

A fun trip through the history:

The Grateful Dead started a tradition in the 60′s to allow taping of their concerts and sharing those tapes as long as no one made any money. Key: no one made any money.

A generation of deadheads became fantastic archivists. The songs were annotated, the loss in quality through copying was minimized. New tapers were brought into the fold.

Other bands, and their fans, started to join in: taping became one of the ways to participate in a community around a band’s music.

When music went digital, they wanted to put an end to generational loss through copying by passing on uncompressed files. They even developed a way to compress these files losslessly saving file size without losing quality: shn then flac formats, I believe, were developed by the tapers.

The Internet made digital music trading possible, but the files were big. The ETree community started a database of shows to help facilitate accurate transfer. John Gilmore and John Perry Barlow translated the Dead’s taper permission to the Internet generation.

The Internet Archive offered to the ETree community to provide “unlimited storage, unlimited bandwidth, forever, for free” in an email in 2002. The reply came back:  “We don’t believe you. But if you could, that would be our dream.” We decided to give it a try.    But posting concerts on a website was different from tape trading so we decided to ask for some sort of permission from the taper friendly bands. ETree volunteers started to ask for email confirmation and these were then put in each band’s collection page.

About 1-3 bands sign up and 30 concerts are uploaded every day ever since! Volunteer administrator Tyler Huff has been keeping a milestone post going.

These concerts have been downloaded or listened to almost 800,000,000 times. The Grateful Dead collection alone is over 65,000,000 for the over 8,000 concert recordings on the site.

The total data space for the live music archive is now 112,000,000,000,000 Bytes (112TB), and the Grateful Dead collection is 12TBytes.

This is working for the Internet Archive because our mission is to make the cultural works of humankind available to all. It is also showing that we can share and be happy about it.

Thank you to all the 5,200 bands that have participated. Thank you to the over 5,800 people that have uploaded shows. Thank you to the fabulous admins Tyler Huff, John Dailey, Dave Mallick, Diana Hamilton, Peter Hedeman, Brad LeBlanc, Marc Pujol, Tom Horton, Mark Goldey and Tom Anderson. Thank you to the Grateful Dead and those that created and supported this tradition such as John Gilmore, John Perry Barlow, Geoff Sears, and Matt Vernon. Thank you to the Archive staff such as Jon Aizen, Parker Thompson, Tracey Jaquith, Alexis Rossi, and Jeff Kaplan.

Support Live Music!  Please go out to see a band or join one.