Category Archives: Software Archive

Saving 500 Apple II Programs from Oblivion

Among the tens of thousands of computer programs now emulated in the browser at the Internet Archive, a long-growing special collection has hit a milestone: the 4am Collection is now past 500 available Apple II programs preserved for the first time.

playable_screenshot

To understand this achievement, it’s best to explain what 4am (an anonymous person or persons) has described as their motivations: to track down Apple II programs, especially ones that have never been duplicated or widely distributed, and remove the copy protection that prevents them from being digitized. After this, the now playable floppy disk is uploaded to the Internet Archive along with extensive documentation about what was done to the original program to make it bootable. Finally, the Internet Archive’s play-in-a-browser emulator, called JSMESS (a Javascript port of the MAME/MESS emulator) allows users to click on the screenshot and begin experiencing the Apple II programs immediately, without requiring installation of emulators or the original software.

In fact, all the screenshots in this entry link to playable programs!

playable_screenshot (1)

If you’re not familiar with the Apple II software library that has existed over the past few decades, a very common situation of the most groundbreaking and famous programs produced by this early home computer is that only the “cracked” versions persist. Off the shelf, the programs would include copy protection routines that went so far as to modify the performance of the floppy drive, or force the Apple II’s operating system to rewrite itself to behave in strange ways.

Because hackers (in the “hyper-talented computer programmers” sense) would take the time to walk through the acquired floppy disks and remove copy protection, those programs are still available to use and transfer, play and learn from.

One side effect, however, was that these hackers, young or proud of the work they’d done, would modify the graphics of the programs to announce the effort they’d put behind it, or remove/cleave away particularly troublesome or thorny routines that they couldn’t easily decode, meaning the modern access to these programs were to incomplete or modified versions. For examples of the many ways these “crack screens” might appear, I created an extensive gallery of them a number of years ago. (Note that there are both monochrome and color versions of the same screen, and these are just screen captures, not playable versions.) They would also focus almost exclusively on games, especially arcade games, meaning any programs that didn’t fall into the “arcade entertainment” section of the spectrum of Apple II programs was left by the wayside entirely.

With an agnostic approach to the disks being preserved, 4am has brought to light many programs that fall almost into the realm of lore and legend, only existing as advertisements in old computer magazines or in catalog listings of computer stores long past.

playable_screenshot (2)

It gets better.

Easily missed if you’re not looking for it are the brilliant and humorous write-ups done by 4am to explain, completely, the process of removing the copy protection routines. The techniques used by software companies to prevent an Apple II floppy drive from making a duplicate while also allowing the program to boot itself were extensive, challenging, and intense. Some examples of these write-ups include this one for “Cause and Effect”, a 1988 education program, as well as this excellent one for “The Quarter Mile”, another educational program. (To find the write-up for a given 4am item in the collection click on the “TEXT” link on the right side of the item’s web page.)

These extensive write-ups shine a light on one of the core situations about these restored computer programs.

As 4am has wryly said over the years, “Copy Protection Works!” – if the copy protection of a floppy disk-based Apple II program was strong and the program did not have the attention of obsessed fans or fall into the hands of collectors, its disappearance and loss was almost guaranteed.  Because many educational and productivity software programs were specialized and not as intensely pursued/wanted as “games” in all their forms, those less-popular genres suffer from huge gaps in recovered history. Sold in small numbers, these floppy disks are subject to bit rot, neglect, and being tossed out with the inevitably turning of the wheels of time.

This collection upends that situation: by focusing on acquiring as many different unduplicated Apple II programs as possible, 4am are using their skills to ensure an extended life and documented reference materials for what would otherwise disappear.

Classifying Animals with Backbones title screen

Already, the collection has garnered some attention – the “Classifying Animals With Backbones” educational program linked above has a guest review from one of the creators describing the process of the application coming to life. And a particularly thorny copy protection scheme on a 1982 game of Burger Time went viral (in a good way) and was read 25,000 times when it was uploaded to the Archive.

In a few cases, the amount of effort behind the copy protection schemes and the concerned engineering involved in removing the copy protection are epics in themselves.

Speed Reader II 091286 screen 3 - main menu

As an example, this educational program Speed Reader II contains extensive copy protection routines, using tricks and traps to resist any attempts to understand its inner workings and misleading any potential parties who are duplicating it. 4am do their best to walk the user through what’s going on, and even if you might not understand the exact code and engineering involved, it leaves the reader smarter for having browsed through it.

This project has been underway for years and is now at the 500 newly-preserved program mark – that’s 500 different obscure programs preserved for the first time, which you can play and experience on the archive.

Get cracking!

Algernon title screen

(The usual notes: The “Play in Browser” technology used at the Internet Archive is still relatively new, and works best on modern machines running newest versions of browsers, especially Firefox, Chrome and Brave. Javascript (not Java) needs to be enabled on the machine to work. (By default on all browsers, it is.) The manuals for many of the programs are not directly available in many cases, so some experimentation is required, although educational programs often worked to be understood without any manuals for the use of their audiences. Thanks to 4am for housing their collection at the Internet Archive and the many individuals on the MAME and JSMESS teams who have made this emulation possible.)

Internet Archive Does Windows: Hundreds of Windows 3.1 Programs Join the Collection

Microsoft Windows was, to some people, too little, too late.

Released as Version 1.0 in 1985, the graphic revolution was already happening elsewhere, with other computer operating systems – but Microsoft was determined to catch up, no matter what it cost or took. Version 1.0 of their new multi-tasking navigation program (it was not quite an “Operating System”) appeared and immediately got marks for being a step in the right direction, but not quite a leap. Later versions, including versions 2.0 and 2.1, finished out the late 1980s with a set of graphics-oriented programs that could be run from DOS and allow the use of a mouse/keyboard combination (still new at the time) and a chance for Microsoft to be one of the dominant players in graphical interfaces. It also got them a lawsuit from Apple, which ultimately resulted in a many-years court case and a settlement in 1997 that possibly saved Apple.

Meanwhile, the Windows shell started to become more an more like an operating system, and the introduction of Windows 3.0 and 3.1 brought stability, flexibility, and ease-of-programming to a very wide audience, and cemented the still-dominant desktop paradigms in use today.

In 2015, the Internet Archive started the year with the arrival of the DOS Collection, where thousands of games, applications and utilities for DOS became playable in the browser with a single click. The result has been many hundreds of thousands of visitors to the programs, and many hours of research and entertainment.

This year, it’s time to upgrade to Windows.

win31logo

We’ve now added over 1,000 programs that run, in your browser, in a Windows 3.1 environment. This includes many games, lots of utilities and business software, and what would best be called “Apps” of the 1990s – programs that did something simple, like provide a calculator or a looping animation, that could be done by an individual or small company to great success.

windows

Indeed, the colorful and unique look of Windows 3/3.1 is a 16-bit window into what programs used to be like, and depending on the graphical whims of the programmers, could look futuristic or incredibly basic. For many who might remember working in that environment, the view of the screenshots of some of the hosted programs will bring back long-forgotten memories. And clicking on these screenshots will make them come alive in your browser.

screenshot_00 (2)screenshot_00 (3)screenshot_00 (4)When they focused on it, a developer could produce something truly unique and beautiful within the Windows 3.x environment. Observe this Role-Playing Game “Merlin”:

screenshot_01

But on the whole, the simple libraries for generating clickable boxes and rendering fonts, and an intent to “get the job done” meant that a lot of the programs would look like this instead:

payoff

(Then again, how complicated and arty does a program to calculate amortization amounts have to be?)

Windows 3.1 continues to be in use in a few corners of the world – those easily-written buttons-and-boxes programs drive companies, restaurants, and individual businesses with a dogged determination and extremely low hardware requirements (a recent news story revealed at least one French airport that depended on one).

Many people, though, moved on to Microsoft’s later operating systems, like Windows 95, ME, Vista, 7, and so on. Microsoft itself stopped officially supporting Windows 3.1 in 2001, 15 years ago.

But Windows 3.1 still holds a special place in computer history, and we’re pleased to give you a bridge back to this lost trove of software.

If you need a place to start without being overwhelmed, come visit the Windows Showcase, where we have curated out a sample set of particularly interesting software programs from 20 years ago.

As is often the case with projects like this, volunteers contributed significant time to help bring this new library of software online. Justin Kerk did the critical scripting and engineering work to require only 2 megabytes to run the programs, as well as ensure that the maximum number of Windows 3.1 applications work in the browser-based emulator. (Justin thanks Eric Phelps, who in 1994 wrote the SETINI.EXE configuration program). db48x did loader programming to ensure we could save lots of space. James Baicoianu did critical metadata and technical support. As always, the emulation for Windows and DOS-based programs comes via EM-DOSBOX, which is a project by Boris Gjenero to port DOSBOX into Javascript; his optimization work has been world-class. And, of course, a huge thanks to the many contributing parties of the original DOSBOX project.

Making Your DOS Programs Live Again at the Internet Archive

MSDOSSince the beginning of the year, the Internet Archive has been making a large amount of DOS-based games and programs run in the browser, much like our Console Living Room and Internet Arcade collections. Many thousands of people have stopped by and tried out these programs, enjoying such classics as Llamatron 2112 or Dangerous Dave. With countless examples of DOS programs going back spanning 30 years, there’s lots of great software to try out and experiment with. Here’s a great place to start.

If you want to just try out the software, we’re done here. Go into our stacks and have a great time!

However, some people have asked about adding DOS software they created or which they have which isn’t part of our collections, and especially how to make these programs boot in a window like our currently available programs do.

This is a quick guide to getting your DOS programs up and emulating in the browser. If any of these instructions are unclear to you, please contact the Software Curator at jscott@archive.org.

Please note: these instructions are for DOS programs, not Windows programs.

First, you should register for your Internet Archive library card if you haven’t already.

getcardNext, you should upload your DOS software as a .ZIP file. It is important that your program and any support files be inside a single .ZIP file and not uploaded separately.

uploadWhen you upload, you’ll be asked to fill out all sorts of information about your program. Be sure to be as complete as possible, including the description, date of creation, who the author or authors were, and so on. You’re the curator of this software – help the world understand why they should look at it!

Set the “Collection” to Community Software.

Finally, at the bottom of this upload screen, there is an add additional metadata option.

metadataAdd these two metadata pairs:

  • Set “emulator” to “dosbox”.
  • Set “emulator_ext” to “zip”.

Finally, and this is very important … inside the .ZIP file you uploaded is the program that starts the program running. It might be an .EXE, .BAT or .COM file.  For example, if your ZIP file has a single file in it, called LEMON.EXE, then that’s the program that “starts” your program.

  • Set “emulator_start” to this program.

After double-checking your work, click on “Upload and Create your Item” and the system will upload your program to the Archive, and if all goes well, your program will be emulated in our pages after a few minutes.

Again, if you have any questions or experience any issues, contact Jason Scott, the software curator at the Archive, at jscott@archive.org.

Let’s bring the DOS prompt back! And let a thousand programs bloom!

 

archive.org download counts of collections of items updates and fixes

Every month, we look over the total download counts for all public items at archive.org.  We sum item counts into their collections.  At year end 2014, we found various source reliability issues, as well as overcounting for “top collections” and many other issues.

archive.org public items tracked over time

archive.org public items tracked over time

To address the problems we did:

  • Rebuilt a new system to use our database (DB) for item download counts, instead of our less reliable (and more prone to “drift”) SOLR search engine (SE).
  • Changed monthly saved data from JSON and PHP serialized flatfiles to new DB table — much easier to use now!
  • Fixed overcounting issues for collections: texts, audio, etree, movies
  • Fixed various overcounting issues related to not unique-ing <collection> and <contributor> tags (more below)
  • Fixes to character encoding issues on <contributor> tags

Bonus points!

  • We now track *all collections*.  Previously, we only tracked items tagged:
    • <mediatype> texts
    • <mediatype> etree
    • <mediatype> audio
    • <mediatype> movies
  • For items we are tracking <contributor> tags (texts items), we now have a “Contributor page” that shows a table of historical data.
  • Graphs are now “responsive” (scale in width based on browser/mobile width)

 

The Overcount Issue for top collection/mediatypes

  • In the below graph, mediatypes and collections are shown horizontally, with a sample “collection hierarchy” today.
  • For each collection/mediatype, we show 1 example item, A B C and D, with a downloads/streams/views count next to it parenthetically.   So these are four items, spanning four collections, that happen to be in a collection hierarchy (a single item can belong to multiple collections at archive.org)
  • The Old Way had a critical flaw — it summed all sub-collection counts — when really it should have just summed all *direct child* sub-collection counts (or gone with our New Way instead)

overcount

So we now treat <mediatype> tags like <collection> tags, in terms of counting, and unique all <collection> tags to avoid items w/ minor nonideal data tags and another kind of overcounting.

 

… and one more update from Feb/1:

We graph the “difference” between absolute downloads counts for the current month minus the prior month, for each month we have data for.  This gives us graphs that show downloads/month over time.  However, values can easily go *negative* with various scenarios (which is *wickedly* confusing to our poor users!)

Here’s that situation:

A collection has a really *hot* item one month, racking up downloads in a given collection.  The next month, a DMCA takedown or otherwise removes the item from being available (and thus counted in the future).  The downloads for that collection can plummet the next month’s run when the counts are summed over public items for that collection again.  So that collection would have a negative (net) downloads count change for this next month!

Here’s our fix:

Use the current month’s collection “item membership” list for current month *and* prior month.  Sum counts for all those items for both months, and make the graphed difference be that difference.  In just about every situation that remains, graphed monthly download counts will be monotonic (nonnegative and increasing or zero).

 

 

Mirroring the Stone Oakvalley Music Collection

soamc_logo

The Internet Archive has begun mirroring a fantastic collection of music called the “Stone Oakvalley Music Collection”. When you visit one of their websites, the archive.org mirror is one of the choices for download. Going forward, the Archive will offer a full backup of the entire site (over a terabyte) for permanent storage.

Why the Stone Oakvalley Collection is important

Manufactured from the early 1980s to the mid 1990s, the Commodore 64 computer was a revolutionary piece of hardware and a critical introduction to programming for generations. It also had, within its design, a very well-regarded sound chip: the 6581/8580 SID (Sound Interface Device), whose unique properties in wave generation and effects gave a special sound in the hands of the right developers and musicians.

MOS_Technologies_6581

 

This successful piece of hardware was manufactured in the millions across the life of the C64, and in the late 1980s, the introduction of the Commodore Amiga computer brought to life an improved chipset for generating sound; the 8364, or PAULA. With a range of improvements to what sounds and music could come out of this chip, the Amiga soared with capabilities that took years to match in other machines.

paula8364The Archive hosts many examples of music generated by these chips: our C64 Games Archive has videos in the hundreds of games played on a Commodore 64, and searching for terms like “Amiga Music”, “Chiptunes” and “C64 Music” will yield a good amount of sound to enjoy.

But nothing comes close to the Stone Oakvalley Collection in terms of breadth, dedication, and craft in ensuring the unique sound of these chips can be enjoyed in the future.

setup01

The process, which is documented here, involved setting up a large amount of Commodore hardware connected to servers which would reboot the machines, over and over, playing thousands of pieces of music in different configurations, and automatically cataloging and saving the resulting waveforms. Considerations for modifications of the chipset over the years, of stereo versus mono recordings, and verification of the resulting 400,000 files have provided the highest quality of snapshots of this period.

Browsing the Collection

Currently, there are two websites for Stone Oakvalley’s collection – one based around the C64, and the other based around the Amiga.  Impeccable work has been done to catalog the music, so if there are songs or games you remember, they are likely to be saved on the site (and powered from Archive.org’s servers). Otherwise, browse the stacks of the sites and enjoy a soundscape of computer history.

The Internet Archive strives to provide universal access to the world’s knowledge. Through mirroring, hosting and gathering of data, our mission allows millions to gain ad-free, fast access to information and materials. Be sure to check our many collections on our main site.

Inviting the Internet Over to Play

B062ql8CMAErlkJ

At our Annual Event last week, the Archive announced a variety of new projects and plans, including our new beta interface, our compact book scanner, and our progress in tracking political ads on television. The event (full video is here) went very well, with lots of activities and social gathering before and afterwards, and included the first public unveiling of our newest project, the Internet Arcade.

Photo by Kyle Way

Photo by Kyle Way

It was obvious we were on to something – the smallish room with the two stations set up to play emulated arcade games from the collection was constantly packed. Players young and old tried out classic video games, including parents showing their children games they’d played in their own teenage years. All of it was running off the Archive’s own web pages through standard web browsers, with no special plug-ins – and it held up well. We even tracked high scores.

B1F2VfECYAAHKr3

The party, of course, was just the beginning – over the weekend, we quietly announced that the Internet Arcade was available through the main site. With over 900 arcade machines in the collection, most every major machine released between 1976 and 1988 was included. (The emulation system we use, JSMESS, is a Javascript port of a long-running emulation project called MESS/MAME, which has had hundreds of contributors over the years – we salute them.)

After an initial tweet or two, the Arcade’s existence went from a mention by Waxy and Laughing Squid, to sites like Hacker News and Mashable, and from there it hit larger and larger audiences. Within a few hours news had spread to a whole range of sites, including Joystiq, The Verge, Engadget, CNN, PC World, Gizmodo, Ars Technica… and, well, let’s just say a very large amount of sites were reporting on this story.

And that’s when the world showed up.

We’re still counting, but we know hundreds of thousands of people came, many of them all at once, to play.

And as these thousands of curious visitors and first-time callers came to the Archive to try out our collection, minor inefficiencies became showstoppers and the site was temporarily crushed. Our brave administration team persevered, repairs were made, and the site settled in for the new reality:

That's a lot of new visitors!

Everything’s fine and normal… then we crash and fix things… and WOW that’s a lot of new visitors!

This crush of new visitors are coming to the Internet Archive, possibly for the first time ever, and we welcome them with open arms. After all, that’s what we were founded for –  our stated purpose is to function as the Internet’s Library, with stored websites, digitized texts, music, movies and software.  It’s our mission as a non-profit library: make as much of culture and information available to as many people as possible. You can lose a workday or a whole winter in our virtual stacks, and our users often do.

Meanwhile, the story continues to have legs, appearing in newspapers, on radio shows, video podcasts, and message boards around the world.

And then we made it to TV news:

B1hYwH-CQAAb5_f

So now that we have (apparently) the world’s attention… ahem ahem..

Even we don’t know where this story is going to lead. But one thing is sure – video games and software are as important a part of history and culture as books, movies and music have been in the past.  And we’re dedicated to bringing all of this to you, the Internet. Sure, it can be a bit surprising when the entire internet comes over to play, but we wouldn’t have put out the welcome mat if we didn’t want you to visit.

As a non-profit, we depend heavily on user donations to stay afloat – we even take Bitcoin and subscriptions. Keeping 20 petabytes of information flowing, fast and free, is what we’re working on day and night and the positive messages and feedback we’ve gotten this past week (and over the years) tell us we’re doing the right thing.

The JSMESS emulation project is one of many open-source projects the Internet Archive is involved with, and while a lot of it is fun and games we’ve got a serious side too, gathering up disappearing web resources and important historical events into our archives to preserve for next generations. We hope that after you relive your childhood or live out a second new one, you’ll stick around and see what else we have here. It’s quite a place.

Game on!

 

 

 

 

 

Free the Screenshots!

As the Archive moves more widely into the archiving of software, it quickly becomes apparent that there’s going to be an awful lot of programs online without much indication of what they are. With many thousands of programs or program collections to choose from, determining what might be inside becomes a pretty involved task.

In the case of movies, images and texts, there are previews that help show what is contained in the files in a given item. These are extremely helpful, as they not only show the quality or style of the works, but give all sorts of information that might not be reflected in the metadata.

Starting now, the same will be true for many types of software.

screenshot_01

The Atari 800 graphical masterpiece Astro Chase.

Using a combination of the JSMESS emulator and screen capturing software, the Archive has begun automatic “playing out” of sets of programs, snagging shots of what the software does, and then providing it as a guidepost of what is to come with that program.

For example, work has just been completed on the playable Sega Genesis Library,  where the directory view of the items in the collection show helpful screenshots, and individual games show animated playthroughs of the beginning of the cartridge.

00_coverscreenshotThe process is still evolving – currently it requires real-time capture (that is, capturing the first five minutes of a program takes an actual five minutes), but with multiple machines moving through collections, screenshots will be available for huge amounts of programs in coming weeks and months.

Along with the obvious graphical prettiness comes an even greater cultural benefit: the freeing of screenshots.

As these shots have often been done manually or have been gathered by hand, there has risen a tendency to put watermarks or credits with the images to indicate who did the work. While it’s an understandable urge to want some kudos for the effort, it meant that the very work being lauded (the graphics of the program) was being vandalized to ensure credit where credit was due.

None of the screenshots we are generating will have watermarks, and can be used freely for other purposes as you see fit.

To celebrate this, we’ve created a compilation of all the Sega Genesis screenshots generated by the project so far. The compilation is here. Be warned – it’s 4.3 gigabytes of 16,900 screenshots of 573 cartridges! (There’s a way to browse it at this link.)

Many screenshots are simply informative, but many more are truly works of art, as artists and programmers strained the edges of these underpowered machines to create the most evocative images possible. With this screenshotting effort underway, that work will hopefully get a new life and respect on the web.

Free the Screenshots!

screenshot_36

 

 

The Internet Archive Declares Spacewar!

spacewarLike everything else in history, debate rages about when the “first” video game came into being.  Games and demonstrations such as “Tennis for Two” (1958), “NIM” (1951) and “Mouse in the Maze” (1959), played on million dollar equipment for the amusement and experimentation of limited audiences.

One contender in this group is “Space War!”, a 1962 collaboration of multiple students at the Massachusetts Institute of Technology. Playing off the cathode-ray tube of a Digital Equipment PDP-1 (of which less than 60 were sold), this two-player space-battle game has been lauded as a major advancement in computer gaming for over 50 years.

Now, it’s possible to play it at the Internet Archive.

As part of our larger Historical Software collection, there is now an entry for Space War!

sw1This entry covers the historical context of Space War!, and instructions for working with our in-browser emulator. The system doesn’t require installed plugins (although a more powerful machine and recent browser version is suggested).

The JSMESS emulator (a conversion of the larger MESS project) also contains a real-time portrayal of the lights and switches of a Digital PDP-1, as well as links to documentation and manuals for this $800,000 (2014 dollars) minicomputer.

sw2You’re going to need a friend to play – the game requires two human players on the same keyboard. And don’t worry, everyone gets sucked into the star in the center the first few times. You’ve got to have your orbital dynamics down before you’re truly ready to be a space warrior.

With over a half-century of history behind it, Space War! still holds up as a great example of what would become a dominant form of media in the decades since – the space video game.

The Internet Archive continues to add more historical software frequently – bringing the computing past to the computer future. Stay tuned!

Magazine in movie “WarGames” is discovered using an Internet Archive Collection

 

 

01_Title

An intrepid researcher wanted to figure out what magazine was used in movie WarGames and using the Internet Archive collection found it was Creative Computing.  (which was a key magazine for me in the 70’s when I sold personal computers during the pre-Apple ][, kit days).

Reading the gory details of this hunt is fun.  http://mw.rat.bz/wgmag/