Tag Archives: ffmpeg

new mp4 (h.264) derivative technique — simpler and easy!

Posted on by

Greetings video geeks!  😎

We’ve updated the process and way we create our .mp4 files that are shown on video pages on archive.org

It’s a much cleaner/clearer process, namely:

  • We opted to ditch ffpreset files in favor of command-line argument 100% equivalents.  It seems a bit easier for someone reading the task log of their item, trying to see what we did.
  • We no longer need qt-faststart step and dropped it.  we use the cmd-line modern ffmpeg “-movflags faststart”
  • Entire processing is now done 100% with ffmpeg, in the standard “2-pass” mode
  • As before, this derivative plays in modern html5 video tag compatible browsers, plays in flash plugin within browsers, and works on all iOS devices.   it also makes sure the “moov atom” is at the front of the file, so browsers can playback before downloading the entire file, etc.)
Here is an example (you would tailor especially the “scale=640:480” depending on source aspect ratio and desired output size;  change or drop altogether the “-r 20” option (the source was 20 fps, so we make the dest 20 fps);  tailor the bitrate args to taste):
  • ffmpeg -y -i stairs.avi -vcodec libx264 -pix_fmt yuv420p -vf yadif,scale=640:480 -profile:v baseline -x264opts cabac=0:bframes=0:ref=1:weightp=0:level=30:bitrate=700:vbv_maxrate=768:vbv_bufsize=1400 -movflags faststart -ac 2 -b:a 128k -ar 44100 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 1 -map 0:0 -map 0:1 -acodec aac -strict experimental stairs.mp4;
  • ffmpeg -y -i stairs.avi -vcodec libx264 -pix_fmt yuv420p -vf yadif,scale=640:480 -profile:v baseline -x264opts cabac=0:bframes=0:ref=1:weightp=0:level=30:bitrate=700:vbv_maxrate=768:vbv_bufsize=1400 -movflags faststart -ac 2 -b:a 128k -ar 44100 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 2 -map 0:0 -map 0:1 -acodec aac -strict experimental -metadata title='”Stairs where i work” – lame test item, bear with us – http://archive.org/details/stairs’ -metadata year=’2004′ -metadata comment=license:’http://creativecommons.org/licenses/publicdomain/’ stairs.mp4;

Happy hacking and creating!

PS: here is the way we compile ffmpeg (we use ubuntu linux, but works on macosx, too).

files.xml and metadata API now showing video file’s width, height, and duration

Posted on by

For those video oriented of you who inspect our _files.xml directly, or use a metadata API (JSON) for items, you’ll likely be pleased to know that for all new items, as well as items that update (eg: review posted, metadata changes, rederives), that:

we now detect and stamp into each video file fields for
width
height
length (duration in seconds)

here is an example file listing (in JSON results) demonstrating:
{“name”:”anamorphic.mov”,
“source”:”original”,
“format”:”h.264″,
“length”:”3.03″,
“height”:”720″,
“width”:”1280″,
“mtime”:”1355278126″,
“size”:”375646″,
“md5″:”94a0b479869ab7ee997b36b45d9edb7a”,
“crc32″:”0b8e4162”,
“sha1″:”c21b65ee8e9153f0812d7397fe89224d6f3f80fd”}

we will even attempt to “correct” the width in the case of “anamorphic” videos (eg: the pixels are “rectangular” and not “square”) to the scaled width in square pixel terms (like the one above)

cheers!

Improved theora/ogg video derivatives!

Posted on by

We’ve made our ogg video derivatives slightly better via:

  • minor bump up to “thusnelda” release
  • “upgrade” from 1-pass video encoding to 2-pass video encoding
  • direct ffmpeg creation of the video (you’ll need to re/compile ffmpeg minimally with “–enable-libtheora –enable-libvorbis” configure flags)

ffmpeg -y -i ‘camels.avi’ -q:vscale 3 -b:v 512k -vcodec libtheora -pix_fmt yuv420p -vf yadif,scale=400:300 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 1 -an -f null /dev/null;

ffmpeg -y -i ‘camels.avi’ -q:vscale 3 -b:v 512k -vcodec libtheora -pix_fmt yuv420p -vf yadif,scale=400:300 -r 20 -threads 2 -map_metadata -1,g:0,g -pass 2 -map 0:0 -map 0:1 -acodec libvorbis -ac 2 -ab 128k -ar 44100 -metadata TITLE=’Camels at a Zoo’ -metadata LICENSE=’http://creativecommons.org/licenses/by-nc/3.0/’ -metadata DATE=’2004′ -metadata ORGANIZATION=’Dumb Bunny Productions’ -metadata LOCATION=http://archive.org/details/camels camels.ogv

some notes:

  • You’d want to adjust the “scale=WIDTH:HEIGHT” accordingly, as well as the “-r FRAMES-PER-SECOND” related args, to your source video.
  • I made a small patch to allow *both* bitrate target *and* quality level for theora in ffmpeg, after comparing the other popular tool “ffmpeg2theora” code with the libtheoraenc.c inside ffmpeg.  It may not be necessary, but I believe I saw *slightly* better quality coming out of theora/thusnelda ogg video.  For what it’s worth, my minor patch is here:  http://archive.org/~tracey/downloads/patches/ffmpeg-theora.patch
  • The way we compile ffmpeg (ubuntu/linux) is here.  (Alt MacOS version here )
  • (Edited post above after I removed this step) It’s *quite* odd, I realize to have ffmpeg transcode both the audio/video together, only to split/demux them back out temporarily.  However, for some videos, the “oggz-comment” step would wipe out the first video keyframe and cause unplayability in chrome (and the expected visual artifacts for things that could play it).   So, we split, comment the audio track, then re-stitch it back together.

improved h.264 derivatives!

Posted on by

We have thoroughly tested a newer and simpler way to create h.264 derivatives!

Changes you’ll notice:

  • More pixels!  previously 320 x 240    goes to 640 x 480 pixels
  • Slightly higher video bitrate — from about 512kb/s   to   about  700kb/s bitrate
  • Switching from mp4creator container maker to ffmpeg container + qt-faststart
  • Less back-end commands to make high-quality derivative

Nice things about this derivative (similar to prior derivative):

  • Plays in adobe flash plugin
  • Plays on all versions of iphone and ipad
  • Starts quickly, nearly instant seeking even to unbuffered areas of the video

Here’s a sample of how we do it with just 3 simple commands.  (We do/you should adjust “-r” argument appropriately to your video’s frames-per-second.  We also adjust the “640” in the “-vf scale” argument to be appropriate for the video’s *actual* aspect ratio, etc.  So for example, the 640 might become 852 for 16:9 widescreen video.  Although for our .mp4 specific derivative and playback ability on iPhone (1st gen and thus all versions), we would actually downrez that to 640×360).

ffmpeg -deinterlace -y -i 'camels.avi' -vcodec libx264 -fpre libx264-IA.ffpreset -vf scale=640:480 -r 20 -threads 2 -map_meta_data -1:0 -pass 1 -an tmp.mp4


ffmpeg -deinterlace -y -i 'camels.avi' -vcodec libx264 -fpre libx264-IA.ffpreset -vf scale=640:480 -r 20 -threads 2 -map_meta_data -1:0 -pass 2 -acodec aac -strict experimental -ab 128k -ac 2 -ar 44100 -metadata title='Camels at a Zoo - http://www.archive.org/details/camels' -metadata year='2004' -metadata comment=license:'http://creativecommons.org/licenses/by-nc/3.0/' tmp.mp4

qt-faststart tmp.mp4 'camels.mp4'

our preset file:
http://www.archive.org/~tracey/downloads/libx264-IA.ffpreset

 

For the adventurous out there, you can create this same setup by building ffmpeg on mac, linux, or windows.  Linux is easy, but personally, I’m a mac gal.  So here’s some ffmpeg build tips on the mac.

Happy viewing!

 

better mp4 (h.264) derivatives at archive.org!

Posted on by

Late last week, we pushed live a new video deriving technique, as well as in the process updated our audio/video file reader, ffmpeg.

New items will benefit from this newer method, and prior items can be re-derived by users if they desire (probably by the end of the year, we will rederive all our movies automatically).

The video will have significantly less “noise”, a higher PSNR (Peak Signal-to-Noise Ratio), and less”blocking” — all at similar or faster deriving speed to build and the same bitrate and filesize!

example new derivative frame

example new derivative frame

example old derivative frame

example old derivative frame

We now open the source video file up with ffmpeg, resize and convert it to raw video, and pipe it to the most recent build of “x264” tool (opting for baseline profile for iPhone, etc. compatibility).
For the very curious (and the very geeky 😉 here is a how we make our h.264 MPEG4 video files now:

• ffmpeg -i camels.avi -vn -acodec libfaac -ab 64k -ac 2 temp.aac
• ffmpeg -an -deinterlace -i camels.avi -s 320x240 -r 20 -vcodec rawvideo -pix_fmt yuv420p -f rawvideo - 2>/dev/null | ffmpeg -an -f rawvideo -s 320x240 -r 20 -i - -f yuv4mpegpipe - 2>/dev/null | x264 --bitrate 512 --vbv-maxrate 768 --vbv-bufsize 1024 --profile baseline --pass 1 /dev/stdin --demuxer y4m -o temp.h264
• ffmpeg -an -deinterlace -i camels.avi -s 320x240 -r 20 -vcodec rawvideo -pix_fmt yuv420p -f rawvideo - 2>/dev/null | ffmpeg -an -f rawvideo -s 320x240 -r 20 -i - -f yuv4mpegpipe - 2>/dev/null | x264 --bitrate 512 --vbv-maxrate 768 --vbv-bufsize 1024 --profile baseline --pass 2 /dev/stdin --demuxer y4m -o temp.h264
• mp4creator -c temp.h264 -r 20 t2.mp4
• mp4creator -c temp.aac -interleave t2.mp4
• ffmpeg -i t2.mp4 -acodec copy -vcodec copy -metadata title="Camels at a Zoo - http://www.archive.org/details/camels" -metadata year="2004" -metadata comment="license:http://creativecommons.org/licenses/by-nc/3.0/" camels_512kb.mp4
• mp4creator -optimize camels_512kb.mp4

–Tracey Jaquith

Audio and Video improvements

Posted on by

Hi Patrons,

Yesterday we made live a large update to the way we create the audio and video displayed on our site.  Most folks might not notice the changes, so here’s a rundown:

For Audio:

  • We can now read and make mp3s from 24-bit Flac files. This has been requested for many years and we are thrilled to get it working.
  • The Ogg audio files that we create from audio files will now be using an updated “libvorbis” library.  (The library we were using before today was from 2001! 😎 )
  • We are no longer making 64kb MP3s (or zips or m3u playlists of those files).  This was a judgement call — given how poor the sound quality is for these files and the fact that most people are getting more and more bandwidth to their devices and computers.
  • Simplified back-end system, relying more and more on “ffmpeg” for format conversion.
  • We will now (try to) make derivatives from “.aac” (Advanced Audio Coding) files and “.ra”/”.rm” (Real Audio) files.
  • General ability to read more kinds of audio files more reliably.

For Movies:

  • The Ogg Video files that we create from movies files will now be using an updated “libvorbis” library for their audio.  (Previously we were using the “non reference” library ogg encoder.  Now we are using the much asked for and newer “libvorbis” library).
  • Updated ffmpeg to v0.5. This allows for a much wider range of source audio/video containers and codecs.  We will be able to derive HD-quality video formats like DV-50 and DV-100.   (For those interested in ffmpeg, changelog).
  • Better detection of widescreen movies (so less of our movies on our site will incorrectly appear “squooshed”).
  • General ability to read more kinds of video files more reliably.
  • Noting the prior point, we were able to get streaming videos for about 170 TV archive items that we could not process previously.

Enjoy!

–Tracey Jaquith

Bookmark and Share

Fast and reliable way to encode Theora Ogg videos using ffmpeg, libtheora, and liboggz

Posted on by


archive.org has started to make theora derivatives for movie files, where we create an Ogg Theora video format output for each movie file. after trying a bunch of tools over a good corpus of wide-ranging videos, i found a neat way to make the Archive derivatives.

High Level:

  • use ffmpeg to turn any video to “rawvideo”.
  • pipe its output to *another* ffmpeg to turn the video to “yuv4mpegpipe”.
  • pipe its output to the libtheora tool.
  • for videos with audio, ffmpeg create a vorbis audio .ogg file.
  • add tasty metadata (with liboggz utils).
  • combine the video and audio ogg files to an .ogv output!

Detailed example:

  • ffmpeg -an -deinterlace -s 400×300 -r 20.00 -i CapeCodMarsh.avi -vcodec rawvideo -pix_fmt yuv420p -f rawvideo – |  ffmpeg -an -f rawvideo -s 400×300 -r 20.00 -i – -f yuv4mpegpipe – |  libtheora-1.0/lt-encoder_example –video-rate-target 512k – -o tmp.ogv
  • ffmpeg -y -i CapeCodMarsh.avi -vn -acodec libvorbis -ac 2 -ab 128k -ar 44100 audio.ogg
  • oggz-comment audio.ogg -o audio2.ogg TITLE=”Cape Cod Marsh” ARTIST=”Tracey Jaquith” LICENSE=”http://creativecommons.org/licenses/publicdomain/” DATE=”2004″ ORGANIZATION=”Dumb Bunny Productions” LOCATION=http://www.archive.org/details/CapeCodMarsh
  • oggzmerge tmp.ogv audio2.ogg -o CapeCodMarsh.ogv

WTFs:

  • Why the double pipe above? Some videos could not go directly to yuv4mpegpipe format such that libtheora (or ffmpeg2theora) would work all the time.
  • We do the vorbis audio outside of libtheora (or ffmpeg2theora) to avoid any issues with Audio/Video sync.
  • We convert to yuv420p in the rawvideo step because ffmpeg2theora has (i think) some known issues of not handling all yuv422 video inputs (i found at least a few videos that did this).
  • We add the metadata to the audio vorbis ogg because adding it to the video ogv file wound up making the first video frame not a keyframe (!)

So this will end up working in Firefox 3.1 and greater — the new HTML “video” tag:

<video controls=”true” autoplay=”true” src=”http://www.archive.org/download/commute/commute.ogv”> for firefox betans </video>

This technique above worked nicely across a wide range of source and “trashy” 46 videos that I use for QA before making live a new way to derive our videos at archive.org.

-tracey jaquith