Category Archives: Television Archive

Mapping 400,000 Hours of U.S. TV News

Posted on by

TVnewMap2
We are excited to unveil a couple experimental data-driven visualizations that literally map 400,000 hours of U.S. television news. One of our collaborating scholars, Kalev Leetaru, applied “fulltext geocoding” software to our entire television news research service collection. These algorithms scan the closed captioning of each broadcast looking for any mention of a location anywhere in the world, disambiguate them using the surrounding discussion (Springfield, Illinois vs Springfield, Massachusetts), and ultimately map each location. The resulting CartoDB visualizations provide what we believe is one of the first large-scale glimpses of the geography of American television news, beginning to reveal which areas receive outsized attention and which are neglected.

Watch4-year

 

Watch TV news mentions of places throughout the world for each day.

 

Compare-Contrast

 

Select a TV station and time window to view their representations of places.

 

Keep in mind that as you explore, zoom-in and click the locations in these pilot maps, you are going to find a lot of errors. Those range from errors in the underlying closed captioning (“two Paris of shoes”) to locations that are paired with onscreen information (a mention of “Springfield” while displaying a map of Massachusetts on the screen). Thus, as you click around, you’re going to find that some locations work great, while others have a lot more error, especially small towns with common names.

What you see here represents our very first experiment with revealing the geography of television news and required bringing together a bunch of cutting-edge technologies that are still very much active areas of research. While there is still lots of work to be done, we think this represents a tremendously exciting prototype for new ways of interacting with the world’s information by organizing it geographically and putting it on a map where it belongs!

Virtual Machines: Unlocking Media for Research

In addition to our public web-based research service, we are facilitating scholars, like Kalev, and other researchers in applying advanced data treatments to our entire collection, at a speed and scale beyond any individual’s capacity. As responsible custodians of an enormous collection of television news content created by others, we endeavor to secure their work within the context of our library. Therefore, rather than lending out copies of large portions of the collection for study, researchers instead work in our “virtual reading room” where they may run their computer algorithms on our servers within the physical confines of the Archive. We hope our evolving demonstrations of this data queries in — results out — process may help forge a new model for how exceptional public interest value can be derived from media without challenging their value and integrity to their creators.

The Knight Foundation and other insightful donors are providing critical support in our ongoing efforts to open television news and join with others in re-visioning how digital libraries can respectfully address the educational potential of other diverse media. We hope you will consider lending your support.

The Atlantic

NSA TV Clip Library

Posted on by


When the American people find out how their government has secretly interpreted the Patriot Act, they are going to be stunned and they are going to be angry.  Senator Ron Wyden May 26, 2011

Recent revelations of the extent of National Security Agency surveillance and weakening of our digital infrastructure give substance to the warnings of Senator Wyden and others. To assist journalists and other concerned citizens in reflecting on these issues, the Internet Archive has created a curated library of short television news clips presenting key statements and other representations.

NSA-issues TV News Quote Library

The experimental, Chrome and Safari only, library launches today with more than 700 chronologically ordered television citations drawn from the Archive’s television news research service. The TV quotes can be browsed by rolling over clip thumbnails, queried via transcripts and sorted for specific speakers. Citation links, context, links to source broadcasters and options to borrow can be explored by following the More/Borrow links on each thumbnail.

new mp4 (h.264) derivative technique — simpler and easy!

Posted on by

Greetings video geeks!  😎

We’ve updated the process and way we create our .mp4 files that are shown on video pages on archive.org

It’s a much cleaner/clearer process, namely:

  • We opted to ditch ffpreset files in favor of command-line argument 100% equivalents.  It seems a bit easier for someone reading the task log of their item, trying to see what we did.
  • We no longer need qt-faststart step and dropped it.  we use the cmd-line modern ffmpeg “-movflags faststart”
  • Entire processing is now done 100% with ffmpeg, in the standard “2-pass” mode
  • As before, this derivative plays in modern html5 video tag compatible browsers, plays in flash plugin within browsers, and works on all iOS devices.   it also makes sure the “moov atom” is at the front of the file, so browsers can playback before downloading the entire file, etc.)
Here is an example (you would tailor especially the “scale=640:480” depending on source aspect ratio and desired output size;  change or drop altogether the “-r 20” option (the source was 20 fps, so we make the dest 20 fps);  tailor the bitrate args to taste):
  • ffmpeg -y -i stairs.avi -vcodec libx264 -pix_fmt yuv420p -vf yadif,scale=640:480 -profile:v baseline -x264opts cabac=0:bframes=0:ref=1:weightp=0:level=30:bitrate=700:vbv_maxrate=768:vbv_bufsize=1400 -movflags faststart -ac 2 -b:a 128k -ar 44100 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 1 -map 0:0 -map 0:1 -acodec aac -strict experimental stairs.mp4;
  • ffmpeg -y -i stairs.avi -vcodec libx264 -pix_fmt yuv420p -vf yadif,scale=640:480 -profile:v baseline -x264opts cabac=0:bframes=0:ref=1:weightp=0:level=30:bitrate=700:vbv_maxrate=768:vbv_bufsize=1400 -movflags faststart -ac 2 -b:a 128k -ar 44100 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 2 -map 0:0 -map 0:1 -acodec aac -strict experimental -metadata title='”Stairs where i work” – lame test item, bear with us – http://archive.org/details/stairs’ -metadata year=’2004′ -metadata comment=license:’http://creativecommons.org/licenses/publicdomain/’ stairs.mp4;

Happy hacking and creating!

PS: here is the way we compile ffmpeg (we use ubuntu linux, but works on macosx, too).

new video and audio player — video multiple qualities, related videos, and more!

Posted on by

Many of you have already noticed that since the New Year, we have migrated our new “beta” player to be the primary/default player, then to be the only player.

We are excited about this new player!
It features the very latest release of jwplayer from longtailvideo.com.

Here’s some new features/improvements worth mentioning:

  • html5 is now the default — flash is a fallback option.  a final fallback option for most items is a “file download” link from the “click to play” image
  • videos have a nice new “Related Videos” pane that shows at the end of playback
  • should be much more reliable — I had previously hacked up a lot of the JS and flash from the jwplayer release version to accommodate our various wants and looks — now we use mostly the stock player with minimal JS alterations/customizations around the player.
  • better HD video and other quality options — uploaders can now offer multiple video size and bitrate qualities.  If you know how to code web playable (see my next post!) h.264 mp4 videos especially, you can upload different qualities of our source video and the viewer will have to option to pick any of them (see more on that below).
  • more consistent UI and look and feel.  The longtailvideo team *really* cleaned up and improved their UI, giving everything a clean, consistent, and aesthetically pleasing look.  Their default “skin” is also greatly improved, so we can use that now directly too
  • lots of cleaned up performance and more likely to play in more mobile, browsers, and and OS combinations under the hood.

Please give it a try!

-tracey

 

For those of you interested in trying multiple qualities, here’s a sample video showing it:

http://archive.org/details/kittehs

To make that work, I made sure that my original/source file was:

  • h.264 video
  • AAC audio
  • had the “moov atom” at the front (to allow instant playback without waiting to download entire file first) (search web for “qt-faststart” or ffmpeg’s “-movflags faststart” option, or see my next post for how we make our .mp4 here at archive.org)
  • has a > 480P style HD width/height
  • has filename ending with one of:   .HD.mov   .HD.mp4   .HD.mpeg4    .HD.m4v

When all of those are true, our system will automatically take:

  • filename.HD.mov

and create:

  • filename.mp4

that is our normal ~1000 kb/sec “derivative” video, as well as “filename.ogv”

The /details/ page will then see two playable mpeg-4 h.264 videos, and offer them both with the [HD] toggle button (seen once video is playing) allowing users to pick between the two quality levels.

If you wanted to offer a *third* quality, you could do that with another ending like above but with otherwise the same requirements.  So you could upload:

  • filename.HD.mp4       (as, say, a 960 x 540 resolution video)
  • filename.HD.mpeg4   (as, say, a 1920 x 1080 resolution video)

and the toggle would show the three options:   1080P, 540P, 480P

You can update existing items if you like, and re-derive your items, to get multiple qualities present.

Happy hacking!

 

 

 

files.xml and metadata API now showing video file’s width, height, and duration

Posted on by

For those video oriented of you who inspect our _files.xml directly, or use a metadata API (JSON) for items, you’ll likely be pleased to know that for all new items, as well as items that update (eg: review posted, metadata changes, rederives), that:

we now detect and stamp into each video file fields for
width
height
length (duration in seconds)

here is an example file listing (in JSON results) demonstrating:
{“name”:”anamorphic.mov”,
“source”:”original”,
“format”:”h.264″,
“length”:”3.03″,
“height”:”720″,
“width”:”1280″,
“mtime”:”1355278126″,
“size”:”375646″,
“md5″:”94a0b479869ab7ee997b36b45d9edb7a”,
“crc32″:”0b8e4162”,
“sha1″:”c21b65ee8e9153f0812d7397fe89224d6f3f80fd”}

we will even attempt to “correct” the width in the case of “anamorphic” videos (eg: the pixels are “rectangular” and not “square”) to the scaled width in square pixel terms (like the one above)

cheers!

Happy News Year! An Exhibit of International News

Posted on by

In celebration of the new year, we’d like to take you on a tour of news broadcasts from around January 1st from more than 60 stations in 30 countries. We hope the Happy News Year exhibit will highlight the amazing breadth of culture and opinion available through daily television news.

This exhibit includes content from Internet Archive’s television collections, Mosaic: World News from the Middle East and Scola.

Happy News Year screen shot

You may also be interested in the Understanding 9/11 Television News Archive.

Understanding 9/11: A Television News Archive

Posted on by

We are proud to announce the launch of Understanding 9/11: A Television News Archive, a library of news coverage of the events of 9/11/2001 and their aftermath as presented by U.S. and international broadcasters. A resource for scholars, journalists and the public, the library presents one week (3,000 hours from 20 channels over 7 days) of news broadcasts for study, research and analysis, with select analysis by scholars.

911 collection pageTelevision is our preeminent medium of information, entertainment and persuasion, but until now it has not been a medium of record. Scholars face great challenges in identifying, locating and adequately citing television news broadcasts in their research. This archive attempts to address this gap by making TV news coverage of this critical week in September 2001 available to those studying these events and their treatment in the media.

Background on the Television Archive

Internet Archive is a non-profit library founded in 1996 that started by attempting to collect every webpage from all websites. This is a major task but it is doable even by a non-profit.

Another medium, television, struck us as historically under-appreciated, despite its tremendous importance. Television is pervasive and persuasive, but it is difficult to access programs for research and analysis.  We felt that TV should be a medium of record, a moniker generally reserved for newspaper publishing. As we learned in high school, to effectively understand we need to be able to ‘compare and contrast’. We need to be able to quote.

Talking with the Library of Congress in 2000 we found that they were not systematically recording TV. Talking with the Federal Broadcast Information Service which was collecting TV for the US intelligence community, we found it would probably be difficult to get the recordings from them for library use. The notable Vanderbilt TV News archive at that time was struggling financially and only captured several hours of television news each night. As a result, we decided to create the Television Archive to help preserve this culturally important medium.

Starting in late 2000, we began collecting Russian, Chinese, Japanese, Iraqi, French, Mexican, British, American, and other stations… 20 channels of TV in DVD quality.

When the events of September 11, 2001 occurred, we, like most Americans, urgently wanted international perspectives on the United States. Stunned by the attacks, we tried to figure out what we could do to help.  Seventy-one people and organizations worked together to get one week of TV News up on the Internet to be launched on October 11, 2001. (Bear in mind this is 3 years before YouTube started.) Launched at the Newseum in Washington DC, we made a website that allowed anyone to research the collection of 20 channels for the week of September 11th.

Today, we are relaunching this collection with an updated interface with a conference at NYU.

LEARNING FROM RECORDED MEMORY: 9/11 TV News Archive Conference

Posted on by

LEARNING FROM RECORDED MEMORY: 9/11 TV News Archive Conference

Co-sponsored by Internet Archive and New York University’s Moving Image Archiving and Preservation Program, Tisch School of the Arts

Wednesday, August 24, 4:00-6:00 pm; reception follows

New York University, Tisch School of the Arts, 721 Broadway, 6th Floor, Michelson Theater, New York, NY 10003

This conference highlights work by scholars using television news materials to help us understand how TV news presented the events of 9/11/2001 and the international response. Our collective recollection of 9/11 and the following days has become inseparable from the televised images we have all seen. But while TV news is inarguably the most vivid and pervasive information medium of our time, it has not been a medium of record. As the number of news outlets increases, research and scholarly access to the thousands of hours of TV news aired each day grows increasingly difficult. Scholars face great challenges in identifying, locating and adequately citing television news broadcasts in their research.

The 9/11 Television News Archive (http://archive.org/details/911) contains 3,000 hours of national and international news coverage from 20 channels over the seven days beginning September 11, plus select analysis by scholars. It is designed to assist scholars and journalists researching relationships between news events and coverage, engaging in comparative and longitudinal studies, and investigating “who said what when.” What kinds of research and scholarship will be enabled by access to an online database of TV news broadcasts? How will emerging TV news studies make use of this service? This conference offers contemporary insights and predictions on new directions in television news studies.

SCHEDULE

4:00:  Welcome: Richard Allen, Chair, Department of Cinema Studies, Tisch School of the Arts, NYU
4:05:  Brewster Kahle, Founder and Digital Librarian at the Internet Archive
4:15:  Brian A. Monahan, Iowa State University
4:25:  Deborah Jaramillo, Boston University
4:35:  Marshall Breeding, Vanderbilt Television News Archive
4:45:  Mark J Williams, Department of Film and Media Studies, Dartmouth College
4:55:  Carolyn Brown, American University
5:05:  Michael Lesk, Rutgers University
5:15:  Beatrice Choi, New York University
5:25:  Scott Blake, Artist
5:35:  Discussion
6:00:  Reception (Remarks by Dennis Swanson, President of Station Operations, Fox Television)

SPEAKERS

Welcome: Richard Allen, Chair, Department of Cinema Studies, Tisch School of the Arts, New York University

 

Brewster Kahle, Internet Archive

“Introducing the 9/11 TV News Archive”

Brewster Kahle is the founder and Digital Librarian of the Internet Archive in 1996.   An entrepreneur and Internet pioneer, Brewster invented the first Internet publishing system and helped put newspapers and publishers online in the 1990’s.  

 

Brian A. Monahan, Iowa State University

“Mediated Meanings and Symbolic Politics: Exploring the Continued Significance of 9/11 News Coverage”

In-depth analysis of television news coverage of the September 11 attacks and their aftermath reveals how these events were fashioned into “9/11,” the politically and morally charged signifier that has profoundly shaped public perception, policy and practice in the last decade.  The central argument is that patterned representations of 9/11 in news media and other arenas fueled the transformation of September 11 into a morality tale centered on patriotism, victimization and heroes.  The resulting narrow and oversimplified public understanding of 9/11 has dominated public discourse, obscured other interpretations and marginalized debate about the contextual complexities of these events. Understanding how and why the coverage took shape as it did yields new insights into the social, cultural and political consequences of the attacks, while also highlighting the role of news media in the creation, affirmation and dissemination of meanings in modern life.

Brian Monahan has extensively researched news coverage of 9/11, resulting in a number of scholarly presentations and a book, The Shock of the News: Media Coverage and the Making of 9/11 (2010, NYU Press).

 

 

Deborah Jaramillo, Boston University

“Fighting Ephemerality: Seeing TV News through the Lens of the Archive”

The experience of watching the news on TV as events unfold is often complicated by the space of exhibition — typically, the domestic space. When hour upon hour of news is catalogued and archived — placed in a space of focused study — the news and the experience become altogether different. What was meant to be ephemeral acquires permanence, and what is usually a short-term viewing experience becomes a rigorous, frame-by-frame examination. In this presentation I will discuss how the archive challenges researchers to adopt new ways of seeing and explaining TV news.

Deborah L. Jaramillo is Assistant Professor in the Department of Film and Television, Boston University.

 

Marshall Breeding, Vanderbilt Television News Archive

“An Overview of the Vanderbilt Television News Archive”

Marshall Breeding will give a brief overview of the Vanderbilt Television News Archive and how it carries out its mission to preserve and provide access to US national television news.   He will relate the incredibly diverse kinds of use that the archive receives, including: academic scholarly research; individuals seeking coverage of themselves or family members that may have appeared on the news in life-changing events; those needing historic footage for current journalism, documentaries or other creative works; or corporations or non-profits researching news coverage of their vested topics.  Breeding will also outline some of the constraints it faces in how it provides access to its collection.

Marshall Breeding is the Executive Director of the Vanderbilt Television News Archive and the Director for Innovative Technology and Research for the Vanderbilt University Library.

 

Mark J. Williams, Department of Film and Media Studies, Dartmouth College

“Media Ecology and Online News Archives”

Online TV news archives are a crucial digital resource to facilitate the awareness
of and critical study of Media Ecology.  The 9/11 TV News Archive will fundamentally
enhance our capacity for the study of historical TV newscasts. Two significant
research and teaching outcomes for this area of study are A) to better understand
the role of television news regarding the mediation of society and its popular
memory, and B) to underscore the significance of television news to the goal of
an informed citizenry.  The 9/11 TV News Archive will enhance and ensure the continued
study of the indelible tragic events and aftermath of 9/11, and make possible
new interventions within journalism history and media history, via online capacities
for access and collaboration.

Mark J. Williams is Associate Professor in the Department of Film and Media Studies, Dartmouth College.

 

Carolyn Brown, American University

“Documentation and Access: A Latino/a Studies Perspective on Using Video Archives”

This talk will explore the possibilities and potential of using accessible video news archives in two areas: immigration research in the field of communication and documentary journalism. I will speak of the significance of video news archives in my current film, The Salinas Project, and discuss my continuing research on Latino/as and immigration in the news.

Carolyn Brown is Assistant Professor in the School of Communication and Journalism at American University. She produced daily news shows for MSNBC News and Fox News Channel, and has worked as a producer and senior producer in local news in San Francisco, Washington, D.C., and Phoenix.

 

Michael Lesk, Rutgers University

“Image Analysis for Media Study”

Focusing on television news coverage of the 9/11 attacks, this talk will outline strategies for automatic quantitative analysis of television news imagery.

After receiving a PhD degree in Chemical Physics in 1969, Michael Lesk joined the computer science research group at Bell Laboratories, where he worked until 1984. From 1984 to 1995 he managed the computer science research group at Bellcore, then joined the National Science Foundation as head of the Division of Information and Intelligent Systems, and since 2003 has been Professor of Library and Information Science at Rutgers University, and chair of that department 2005-2008. He is best known for work in electronic libraries, and his book “Practical Digital Libraries” was published in 1997 by Morgan Kaufmann and the revision “Understanding Digital Libraries” appeared in 2004.  He is a Fellow of the Association for Computing Machinery, received the Flame award from the Usenix association, and in 2005 was elected to the National Academy of Engineering. He chairs the NRC Board on Research Data and Information.

 

Beatrice Choi, New York University

“Live Dispatch: The Ethics of Audio Vision Media Coverage in Trauma and the Legacy of Sound from Shell Shock to 9/11”

What experiential narratives—sensory, aesthetic and political—are invisible to those exposed to traumatic events? Considering September 11, 2001, the media coverage of the event is predominantly visual. People drift in and out of news footage, covered in dust and ash as they exclaim that witnessing the attacks was like watching a movie . In contrast, the wailing of sirens, the staccato thud of feet running from the stricken towers, and the chaotic overlap of voices break through—sometimes even swallow—the visual narratives spun for 9/11. For contemporary American traumatic events, this inquires into how porous the sensory modalities are in experiencing and remembering shock. How, after all, do sensory representations of traumatic events leave in/visible marks on documentation? I address these questions by exploring sound as an alternate modality, evoking a different level of traumatic indexicality. First, I draw attention to the sensory discrepancy between audio and visual content dispersed for American traumatic events, taking 9/11 as the focal event. By investigating the most highly represented media vehicles in the event—television and radio—I delve into a critical visual-acoustic analysis, looking specifically at FDNY radio transmissions and NY1 Aircheck news footage. Finally, I examine the discursive legacy sound imparts in moments of American crisis from shell shock accounts in the late 19th – 20th century to post-9/11 narratives of post-traumatic symptoms. In delineating this legacy, I hope to reveal the ways in which these documented discourses evolve past preconceived sensory boundaries in the experience of trauma.

Beatrice Choi is an NYU MA Graduate from the Media Culture Communication program. She has worked with the 9/11 archives for a year as a Moving Imagery Exhibitions Intern at the National September 11 Memorial & Museum, and recently completed a thesis on Post-Traumatic Landscapes, focusing primarily on post-Katrina New Orleans.

 

Scott Blake, artist

“9/11 Flipbook and Quantitative Media Study”

Scott Blake has created a flipbook consisting of images of United Airlines Flight 175 crashing into the south tower of the World Trade Center. Accompanying the images are essays written by a wide range of participants, each expressing their personal experience of the September 11th attacks. In addition, the authors of the essays were asked to reflect on, and respond to, the flipbook itself. Not surprisingly, the majority of the essayists experienced the events through news network footage. Blake is distributing his 9/11 Flipbooks to encourage a constructive dialog regarding the media’s participation in sensationalizing the tragedy. To further illustrate his point, Blake conducted a media study using the 9/11 TV News Archive to count the number of times major news networks showed the plane crashes, building collapses and people falling from the towers on September 11, 2001.

While best known for his Barcode Art, Scott Blake has created new works that are scandalous, witty, fun, pornographic, humorous and about a thousand other adjectives viewers might use when seeing them for the first time. A self-described “frivolous artist,” he mows over conceptual and visual boundaries to make work that is as thought provoking as it is entertainingly tongue-in-cheek.

RECEPTION

Remarks by Dennis Swanson, President of Station Operations, Fox Television

THANKS TO

We thank the many people at New York University and Internet Archive who have helped to make this conference possible.