November 5, 2004

The MJPEG tools

Author: Preston St. Pierre

When it comes to video processing, many free software users don't look beyond the well-known mainstream utilities, like transcode and MPlayer. When these utilities don't produce the desired output, the MJPEG Tools suite might be able to help. Mature, feature-rich, and of extremely high quality, the MJPEG Tools provide video manipulation facilities that you can use either as a complete end-to-end solution or as a toolbox from which you can select the exact tool you need for a given job. Whether you need assistance with capturing, improving, or encoding video, the MJPEG Tools can help.

The original intent of the MJPEG Tools was to provide a package which would enable Linux users to capture and play back video through a PCI card based around the Zoran ZR36067 MJPEG chip. Wrapped in the standard Audio/Video Interleaved (AVI) container format, MJPEG -- properly known as Motion JPEG -- is essentially a sequence of JPEG still images which, when played back fast enough, show as a movie. The main reason most people don't come across MJPEG as a day-to-day video format is that it produces enormous video files -- up to several megabytes a second -- so it's a terrible format for sharing video. The flip side of the coin, however, is that MJPEG is an ideal format for video editing. There are no key frames, and no part of the video is built from adjacent frames, as is done in most compressed formats. You can cut, copy, splice, and generally edit sections of an MJPEG stream quickly and accurately on an individual frame basis.

Zoran-based capture cards, like the Pinnacle DC10+ and the Iomega Buz, capture and compress the JPEG frames in realtime, sending MJPEG video straight into the computer. The MJPEG Tools project started out with a couple of useful utilities to allow you to use these cards under Linux. Once PCs became powerful enough to do the capture and compression in real time, the tools were expanded to allow software encoding from less capable hardware, such as simple BT8x8-based TV capture cards. The developers then enhanced the post-processing capabilities of the tool suite, and it's this part of the package which is of most interest to many people today. The figure below shows where the MJPEG Tools can fit into a video processing workflow.

Capturing video

Zoran-based capture cards fell into obsolescence some time ago; today videophiles often choose digital video (DV) hardware. However, for those who do have Zoran-based cards or another analogue capture device, the capturing facilities of the MJPEG Tools are still extremely usable.

Although the capture itself happens in MJPEG format, the majority of the tools actually use a more raw format called YUV4MPEG2. The MJPEG video stream is typically converted (on the fly) into a YUV4MPEG2 video stream, because that's the actual format most of the MJPEG Tools use.

The capture tool in the MJPEG Tools is called lavrec, and it can be used to capture directly from a Zoran-based capture card, or from a regular video capture card using software encoding. Either way, the result on the disk is an AVI file containing an MJPEG video stream. You can edit this video stream using another tool in the MJPEG Suite called glav, which is a simple GUI-based video editor. glav allows you to play the MJPEG video forwards and backwards at various speeds, and to select sections and cut, copy, and paste them. It ultimately creates what is known as an edit list, which is a small text file that describes the edited video sequence. An edit list file can be used in place of an MJPEG AVI file as input to many of the tools in the MJPEG suite.

The final step in this part of the process is to convert the MJPEG stream (or edit list) into YUV4MPEG2 format using a tool called lav2yuv, which sends the YUV4MPEG2 stream to its standard output.

As indicated in Figure 1, for those with DV equipment a similar tool chain exists. This article won't go into details, but the chain basically consists of a DV frame grabber such as dvgrab, and the kino video editor for removal of unwanted sections and the splicing in of new parts. The DV equivalent of the edit list file is an SMIL file, and it's this file that you give to one of the tools in the smilutils package (specifically smil2yuv) which emits it as a YUV4MPEG2 stream. From there the MJPEG Tools utilities can be used on the video.

Filters

What happens once lav2yuv (or smil2yuv) has converted the video from your VCR or camcorder into a YUV4MPEG2 stream depends on what you want to end up with. The end goal is often a file you can burn onto disc for playing on your DVD player, or maybe to send to friends to play on a computer. Before you do that you might consider employing a few filters to tidy the video up a bit, and this is the area where the MJPEG Tools really shine.

The design of the MJPEG suite follows the Unix philosophy of "one tool to do one job." Each tool uses standard Unix input and standard output streams, which means you can build up a filter chain using a traditional Unix pipeline. Let's look at some of the options.

The first thing you might want to do is denoise the video. If the video came from a nice clear source this may not be necessary, but if the video came from an old VHS tape or other noisy source, this step removes, or at least reduces, flickers and specks. The MJPEG yuvdenoise utility can take a few parameters for fine-tuning thresholds and such, but for the most part it adapts dynamically to the content of the video it finds. It is typically run without parameters, like this:

lav2yuv myvideo.eli | yuvdenoise | ...

with the lav2yuv part being replaced by smil2yuv if the source came via DV.

The next step is quite often the software scaler, which changes the size of the video frame. The frame size is normally whatever the native size of the capture card is, and that varies between devices. You will likely need to rescale the frame size to match the requirements of your destination medium. The MJPEG Tools's yuvscaler can scale up or down (i.e. bigger or smaller), and can use any of a number of algorithms to do the work. You can choose the exact size you want, but most people probably pick one of the standard modes that are preconfigured for VCD, SVCD, and DVD.

If our example pipeline were heading towards a DVD output, a yuvscaler command would be:

lav2yuv myvideo.eli | yuvdenoise | yuvscaler -O DVD | ...

For a particularly noisy video source yuvmedianfilter, which improves the image by removing low frequency noise, is sometimes useful. The excellent MJPEG Tools HOWTO has this to say on the subject:

Using yuvmedianfilter's capability to only filter the chroma (-T) is moderately effective at reducing noise in dark scenes without softening the image during normal (brighter) scenes. Median filtering of the luma (-t) will produce a lower bitrate but can cause loss of detail (softening). Chroma only medianfiltering is less aggressive and is a good choice to use in combination with yuvdenoise.

This kind of language is typical of most video manipulation documentation, which is no doubt a great thing if you understand what the heck it's on about, but a bit intimidating if you don't. But if you keep track of the goal rather than trying to understand the process too deeply, it's not too hard to get improved results. The HOWTO information above, together with a read of the yuvmedianfilter man page, suggests that if our video has excessive noise in the dark scenes, adding this filter to the chain might help:

lav2yuv myvideo.eli | yuvdenoise | yuvscaler -O DVD | yuvmedianfilter -T 3 | ...

Then again, it might not. Or maybe it would help if we put it in before the scaling step. The thing about these filters is that until you have experience with them the best way to find out if they help is just to try them on a five-second clip and fiddle with the parameters a bit. You can see how things look by putting yuvplay on the end of the chain, which sends the video to the screen:

lav2yuv myvideo.eli | yuvdenoise | yuvscaler -O DVD | yuvmedianfilter -T 3 | yuvplay

There are other filters that are useful in certain situations. For example, if you have a video source with fuzzy edges (video captured from a VCR often has this) you should use the yuvscaler filter to crops it off or black it out.

There are lots of other filters you can use to generate video from still images, change the audio track, and so on. Sometimes it's hard to know when to stop.

Encoding

Once you finally have a chain of filters that produces a video stream as you want it, you have to encode it to the format required by your output medium. The MJPEG Tools specialise in producing MPEG output destined for VCD or DVD players.

A script in the MJPEG Tools package called lav2mpeg will encode your MJPEG file to one of a set selection of output formats. This script might occasionally do what you want, but best quality output is always achieved using the individual tools, so as to keep control of the steps.

Step one of the encoding process is normally to extract the sound from the original video file. For DVDs or VCDs the target is the MPEG-1 Layer 2 sound format. The MJPEG Suite provides lav2wav to extract the video's audio track to a WAV file, and mp2enc to encode that WAV into MP2 format. These tools are normally used with a command like:

lav2wav myvideo.eli | mp2enc -o myvideo.mp2

There are various switches to control the format of the output file, or you can use an alternative MPEG audio encoder like toolame if you're more familiar with that. Note that the command above doesn't apply any filters -- there's no point in denoising a video stream if you're going to extract only the audio from it!

Step two is to encode the video. The video encoder in the MJPEG Tools is called mpeg2enc, and here lies a slight problem. Because of some legal uncertainties with software that uses the MPEG-2 codec, some Linux distributors, including SUSE, play safe and choose not to include the mpeg2enc binary in their MJPEG Tools package. SUSE users can grab the MJPEG Tools package from packman, which is complete; other distributions might have a similar alternative package, or there's always the source.

mpeg2enc encoding is where having a good, clean video stream from your selected filters makes a big difference. The better the video you put in, the better the encoded output will be. The basic command for mpeg2enc looks something like this:

lav2yuv myvideo.eli | | mpeg2enc -f 8 -o myvideo.m2v

The MPEG encoder has a number of predefined formats, of which number 8 is standard-format DVD. There are other formats for VCD, SVCD, and so on, and lots of other options for special requirements -- see the man page.

The output from mpeg2enc is either MPEG-1 video or MPEG-2 video, without sound. The final step is to merge the sound file and the video file using a tool called a multiplexer. The multiplexer in the MJPEG Tools suite is called mplex, and it might be used like this:

mplex -f 8 myvideo.mp2 myvideo.m2v -o myvideo.mpg

Again, I use the preset format number 8, which produces a file suitable for burning onto DVD and watching with a regular DVD player. There are other presets and lots of options to fine-tune the results.

A popular alternative to all this is to encode to DivX (MPEG4) format. Although DivX is still almost completely incompatible with hardware DVD players, it is very popular for files destined to be played on computers. MJPEG Tools defers the creation of DivX video to other applications which are more suited to the task, such as mencoder, transcode, and avifile. There is a demonstration script in the MJPEG Tools package called lav2avi that does a basic job of DivX encoding using mencoder, which can be used as a basis for a more fine-tuned version if required.

Conclusion

Video processing is one of those endeavours where having a large, well-equipped toolkit is better than having a single program that tries to do everything. The subject is just so wide-ranging, and can present so many diverse problems, that it's hard to predict what difficulty your project is likely to present you with next.

This is, therefore, an area where the Unix philosophy of one tool for one job really shines, and the MJPEG Tools suite is an excellent implementation of that philosophy. Its video capture features are useful to people with the right hardware, its filters are useful to people with videos that need cleaning up or reformatting, and its encoding tools are useful to people who need to get their videos into the required destination formats. It plays nicely with the other major tools in the area, and lends itself to the idea of just picking one tool from the suite if that is all that is required to improve an established workflow.

Video processing tasks are rarely easy, but most things are possible given the right tools, which is why the MJPEG Suite should definitely be part of your toolbox.


My thanks to Steven M. Schultz from the MJPEG users mailing list, who provided technical assistance with this article.

Derek Fountain is a freelance writer and software developer specialising in Linux and open source scripting languages. He lives in Perth, Western Australia.

Click Here!