Celebrating 100 million tasks (uploading and modifying archive.org content)

Just over 8-1/2 years ago, I wrote a multi-process daemon in PHP that we refer to as “catalogd”.  It runs 24 hours a day, 7 days a week, no rest!

It is in charge of uploading all content to our archive.org servers, and all changes to uploaded files.

We recently passed the 100 millionth “task” (upload or edit to an archive “item”).

After starting with a modest 100 or so tasks/day, we currently run nearly 100,000 tasks/day.  We’ve done some minor scaling, but of the most part, the little daemon has become our little daemon that could!

Here’s to the next 100 million tasks at archive.org!

-tracey

This entry was posted in News, Technical and tagged , , , , . Bookmark the permalink.

One Response to Celebrating 100 million tasks (uploading and modifying archive.org content)

  1. niku says:

    Assuming on an average 3 tasks/item (a conservative estimate, I think), 100,000 tasks/day means some 30,000 items are now added or edited per day. That is not a bad number! Congratulations!

Comments are closed.