Playing Movies in a Java 3D World, Part 1
Pages: 1, 2, 3

5. Managing the Movie

JMF offers a high-level way of accessing specific movie frames. The code fragment below illustrates the main elements. I've left out error checking and exception handling.

// create a movie player, in a 'realized' state
URL url = new URL("file:" + movieFnm);
Player p = Manager.createRealizedPlayer(url);

// create a frame positioner
FramePositioningControl fpc = 
   (FramePositioningControl) 
    p.getControl("javax.media.control.
                   FramePositioningControl");

// create a frame grabber
FrameGrabbingControl fg = 
    (FrameGrabbingControl) 
    p.getControl("javax.media.control.
                  FrameGrabbingControl");

// request that the player changes to a 'prefetched' state
p.prefetch();

// wait until the player is in that state...

// move to a particular frame, e.g. frame 100
fpc.seek(100);

 // take a snap of the current frame
Buffer buf = fg.grabFrame();    

// get its video format details
VideoFormat vf = (VideoFormat) buf.getFormat();

// initialize BufferToImage with video format
BufferToImage bufferToImage = 
                      new BufferToImage(vf);

// convert the buffer to an image
Image im = bufferToImage.createImage(buf);

// specify the format of desired BufferedImage
BufferedImage formatImg = 
          new BufferedImage(
                  FORMAT_SIZE, FORMAT_SIZE, 
                  BufferedImage.TYPE_3BYTE_BGR);

// convert the image to a BufferedImage
Graphics g = formatImg.getGraphics();
g.drawImage(im, 0, 0, 
                FORMAT_SIZE, FORMAT_SIZE, null);
g.dispose();

A media player passes through six states between being created and started. A player in the realized state knows how to render its data, so can provide visual components and controls when asked. I require two controls: FramePositioningControl and FrameGrabbingControl. FramePositioningControl offers methods like seek() and skip() for moving about inside of a movie to examine a particular frame. FrameGrabbingControl supplies grabFrame(), which pulls the current frame from the video track of the movie.

For these controls to work, the player must be moved from its realized state into a prefetched state. This prepares the player for playing the media, and the media data is loaded.

The call to prefetch() is asynchronous, which means that my code must include a waiting period until the state transition is finished. The standard JMF coding solution is to implement a waitForState() method, which causes execution to pause until a state change event wakes it up.

The desired frame can be located in the track with seek(), and then grabbed with grabFrame(). The code must go through several translation steps to convert the grabbed Buffer object into the BufferedImage object required by JMFMovieScreen. Note that the BufferedImage object uses the TYPE_3BYTE_BGR format, which is necessary for the Java 3D parts of the program to employ texturing by reference.

Sun's JMF website contains a useful collection of small examples, one of which, Seek.java, shows how to use FramePositioningControl to step through a movie.

Hacking in Three Steps

Unfortunately, the code outlined above fails, at least in the JMF Performance Pack for Windows v.2.1.1e. I went through several rewrites to get to a working version of JMFSnapper.

Hack 1. The two controls, FramePositioningControl and FrameGrabbingControl, are unavailable in the default player module used in JMF. (The Solaris and Win32 performance packs each support two different MPEG players.) The "native modular" player is required, which is selected by calling:

Manager.setHint(Manager.PLUGIN_PLAYER, new Boolean(true));

This player is a heavyweight component, which interacts poorly with lightweight Swing GUIs such as JFrame and JPanel. However, I don't need to display the player. A more serious consequence of using the native modular player is a much longer loading time for the media, and erratic playing (e.g., varying play rates and dropped frames).

Hack 2. After pondering for a while, I decided the best way to speed up the player was to give it less work to do. I stripped the audio tracks out of the MPEG files, and made sure the files were saved in the (relatively) simple MPEG-1 format. Any number of video editing tools are available to do these tasks. I used two freeware utilities: MPEG Properties and FlasKMPEG. The former is a simple utility that supplies movie format information, while the latter is a decent editor.

The stripped-down movies play promptly, their frame rates are constant, and no frames are lost.

Nevertheless, the FramePositioningControl class is unreliable. On my WinXP machine, seek() almost always failed, and skip() worked correctly perhaps four times out of five.

Hack 3. I bid a tearful farewell to FramePositioningControl. My frame-grabbing algorithm relies on calling FrameGrabbingControl's grabFrame() method at regular intervals while the player is running the movie.

I now have code that reliably catches frames from video-only MPEG-1 files. It also works fairly well with files that have video and audio tracks, but the player is slow to start. Also, the erratic playing causes frames to be grabbed erratically.

I added some "waiting" code at the start of JMFSnapper to deal with video-and-audio movies. The JMFSnapper object waits for a player to start (that is, to enter its started state), and also waits for the first movie frame to become available.

Waiting for the First Frame

The JMFSnapper constructor calls a waitForBufferToImage() method that repeatedly calls hasBufferToImage() until it detects the first video frame.

hasBufferToImage() calls FrameGrabbingControl's grabFrame(), and checks if the returned Buffer object contains video format data. It uses this data to initialize a BufferToImage object, which is employed subsequently to translate each grabbed frame into an image.

// globals
private FrameGrabbingControl fg; // frame grabber
private BufferToImage bufferToImage = null;
private int width, height;    // frame dimensions


private boolean hasBufferToImage()
{
  Buffer buf = fg.grabFrame();   // take a snap
  if (buf == null) {
    System.out.println("No grabbed frame");
    return false;
  }
    
  // there is a buffer, but check if it's empty
  VideoFormat vf = (VideoFormat) buf.getFormat();
  if (vf == null) {
    System.out.println("No video format");
    return false;
  }

  System.out.println("Video format: " + vf);
  // extract the image's dimensions
  width = vf.getSize().width;     
  height = vf.getSize().height;

  // initialize bufferToImage with video format
  bufferToImage = new BufferToImage(vf);
  return true;
}

A minor drawback of this coding approach is that the first video frame (which causes hasBufferToImage() to return true) is discarded after the BufferToImage object is initialized. The frame isn't made available as a BufferedImage to JMFMovieScreen.

Taking a Snap

The most important public method of JMFSnapper is getFrame(), which is called periodically to get the current frame in the running movie.

// global
private BufferedImage formatImg;  // frame image

synchronized public BufferedImage getFrame()
{
  // grab the current frame as a buffer object
  Buffer buf = fg.grabFrame();
  if (buf == null) {
    System.out.println("No grabbed buffer");
    return null;
  }
    
  // convert buffer to image
  Image im = bufferToImage.createImage(buf);
  if (im == null) {
    System.out.println("No grabbed image");
    return null;
  }

  // convert the image to a BufferedImage
  Graphics g = formatImg.getGraphics();
  g.drawImage(im, 0, 0, 
                FORMAT_SIZE, FORMAT_SIZE, null); 

  // Overlay current time on top of the image
  g.setColor(Color.RED);
  g.setFont(new Font("Helvetica",Font.BOLD,12));
  g.drawString(timeNow(), 5, 14);

  g.dispose();
 
  return formatImg;
}  // end of getFrame()

The methods getFrame() and closeMovie() are both synchronized in JMFSnapper. closeMovie() terminates the player, and may be called at any time. The synchronized keywords ensure that the player can't be closed while a frame is being extracted from it.

The formatImg BufferedImage object is initialized in JMFSnapper's constructor:

formatImg = new BufferedImage(
              FORMAT_SIZE, FORMAT_SIZE, 
              BufferedImage.TYPE_3BYTE_BGR);

6. Other Approaches to Frame Grabbing

Sun's JMF examples website offers two other ways of grabbing frames from a movie.

The VideoRenderer

The DemoJMFJ3D example is a combined Java 3D and JMF application, which shows how to wrap a video around a cylinder.

The Java 3D part is virtually identical to what I've discussed--a BufferedImage using the BufferedImage.TYPE_3BYTE_BGR format is passed to an ImageComponent2D object, and then becomes the cylinder's texture. The image can also use the BufferedImage.TYPE_4BYTE_ABGR format, which is required by Solaris in order to support texturing by reference.

The JMF side of the program is quite different from mine. An implementation of JMF's VideoRenderer interface is attached to the TrackControl object for the video track of the movie. Once the TrackControl object is started, the process() method of VideoRenderer is automatically called for each frame encountered in the video. process()'s input argument is the Buffer object (that is, the grabbed frame). Rather than use the Buffer-to-BufferedImage translation steps I've outlined, DemoJMFJ3D builds the BufferedImage by carrying out a low-level, byte array copy between the Buffer's raw data and a pixel map for the BufferedImage.

A lot of the code in DemoJMFJ3D is used in a 3D chat-room example in the book Java Media APIs: Cross-Platform Imaging, Media and Visualization, by A. Terrazas, J. Ostuni, and M. Barlow. I recommend this book as a good introduction to JMF, and it also has several very interesting chapters on Java 3D.

A Processor Codec Plugin

The FrameAccess example utilizes more advanced elements of JMF, centered around a Processor codec plugin.

The Processor class is an extended version of Player, which offers more capabilities for processing media data. A codec plugin (an implementation of the JMF interface Codec) is capable of reading frames from a track, processing them in arbitrary ways, and then writing them back to the track. In particular, Codec's process() method is called each time a frame in encountered in the track. It's supplied with a Buffer object holding the input frame, and an empty Buffer object for the output.

FrameAccess attaches a Codec plugin to the movie's video track, and uses the input frame Buffer object passed to process() to generate some basic statistics about the video. This example could easily be modified to convert the Buffer object into a BufferedImage, either using my approach or the byte array technique of DemoJMFJ3D.

Unfortunately, the Processor class isn't required to support plugins; as a consequence, plugins don't work in JMF 1.0, and in some 2.0-based versions.

It's a good idea to search the jmf-interest mailing list before utilizing Sun's JMF examples, since many of the programs have problems in different versions of JMF.

Check back here next week for the conclusion to this two-part article, where Andrew will discuss another version of the movie screen, using Quicktime for Java.

---

In May 2005, O'Reilly Media, Inc., released Killer Game Programming in Java.

• Chapter 22, "Flocking Boids" (PDF format), is available free online.

• You can also look at the Table of Contents, the Index, and the full description of the book.

• For more information, or to order the book, click here.

Andrew Davison has had a varied and interesting career as an educator, a researcher, and an author. Formerly with the Computer Science Department at Melbourne University, he now lives in Thailand and teaches at the Prince of Songkla University.

---

Return to ONJava.com.