TV News Lab: Hyperaudio improving TV news video captioning and sharing

In a new blog series, TV News Lab, we’ll demonstrate how the Internet Archive is partnering with technology, journalism, and academic organizations to experiment with and improve the TV News Archive, our free, public, online library of TV news shows. Here we interview Mark Boas, founder of the Hyperaudio project, an organization that works to make audio and video more accessible and shareable on the web, by providing an easy-to-use interface for copying and pasting bits of transcripts to create mash ups of shareable video. You can find the open source code powering Hyperaudio on GitHub.

Mark Boas talks to the Internet Archive about Hyperaudio.

NW: What is the problem you’re trying to solve by applying Hyperaudio technology to the TV News Archive?

MB: People find TV news credible. It’s very hard to fake TV news. I’d love to see people using TV news to back up any sort of political or other expression a public official is trying to make, by showing the source material and also the arguments about those statements. I think this also has implications for improving media literacy.

(An example mix made at Chattanooga Public Library.)

NW: What stands in the way of people sharing TV news video right now?

MB:  One of the problems is that audio and video on the web has been a black box in a way. It has not been very well integrated into the web because it’s difficult to do that. If you see a big block of text, it’s easy to highlight, copy, paste and send it off. But if you have an interesting piece of audio to share, how do you do that? There are ways to do it, but it’s not intuitive.

Coupled with that is it’s also hard to find audio on the internet. If you’re searching for search terms, you may or may not find what you want, but only if someone has added sufficient metadata so it’s discoverable. Transcripts allow you to search, but also provide a way to share. And the key to that is that you need not just the transcript, but also you need to match the words in the transcript to the proper times in the audio.

NW: Why is it hard to match the transcript to the audio in a video?

The first step is getting a good quality transcript. It’s great that the TV News Archive uses open captions, but it’s not perfect. (Note: the TV News Archive is searchable via closed captioning, but there’s often a several-second lag between the captions and the video, as well as other quality issues.) The transcript usually needs to be cleaned up. The better the transcript, the better the match. Closed captions are done in real time by humans who make mistakes.

The next challenge is to try and minimize the time it takes to match the words in the transcript to the audio. If we want to automate the process, we need to figure out how to do that more quickly. It’s very intensive on the computing side. I’m experimenting with chunking up the video to speed up the process. I think we’ll see that the matching is an exponential task: a one hour transcript might take 30 minutes and a three hour transcript might take more than three times that. But if we split it up into smaller chunks, the processing might become more efficient

How do people try out Hyperaudio?

Hyperaudio is not a commercial software as a service. It’s more of a demo of the underlying technology. We work with groups like the Studs Terkel Radio Archive (WFMT Chicago), to help them make the most of their content and data; whatever we make flows back into our open source code on GitHub. What we do is very experimental, but it will give you an idea what’s possible. If you want to experiment with TV News Archive, you can do that at http://newsarchive.hyperaud.io/. More info on our experiments and collaborations can be found on our blog.