Playing Movies in a Java 3D World, Part 2:
Pages: 1, 2, 3

3. A Frame-By-Frame Movie

To understand the inner workings of QTSnapper, it helps to have a general idea of the structure of a QuickTime movie. Each movie may be composed of multiple audio and video tracks, overlapping in time. The general idea is illustrated by Figure 4.

Figure 4. The internals of a QuickTime movie

Each track manages its own data, such as the type of media it contains, and the media itself. The media container has its own data structures, including its duration and playing rate (i.e., how many samples should be shown per second). The media is a series of samples (or frames), the first one starting at time 0 (with respect to media time). The samples are indexed, with the first sample at position 1 (not 0).

Simplified views of QuickTime's track and media structures are given in Figure 5.

Figure 5. The internals of a QuickTime track and media

For more comprehensive information, have a look at the movie section of the QuickTime tutorial.

Accessing the Movie's Video Media

The QTSnapper constructor opens the movie:

// globals
private boolean isSessionOpen = false;
private OpenMovieFile movieFile;
private Movie movie;


// in the constructor,
// start a QuickTime session
QTSession.open();
isSessionOpen = true;

// open the movie
movieFile = 
   OpenMovieFile.asRead( new QTFile(fnm) );
movie = Movie.fromFile(movieFile);

The call to QTSession.open() initializes QuickTime prior to its use. There should be a corresponding call to QTSession.close() at termination time.

The video track is located (if one exists) and its media accessed:

// more globals
private Track videoTrack;
private Media vidMedia;


// in the constructor, 
// extract the video track from the movie
videoTrack = 
    movie.getIndTrackType(1, 
            StdQTConstants.videoMediaType,
            StdQTConstants.movieTrackMediaType);
if (videoTrack == null) {
  System.out.println("Sorry, not a video");
  System.exit(0);
}

// get the media used by the video track
vidMedia = videoTrack.getMedia();

Once the media is exposed, various information is extracted from it:

// more globals
private MediaSample mediaSample;
private int numSamples;  // number of samples
private int sampIdx;     // current sample index
private int width;       // frame width
private int height;      // frame height


// in the constructor
numSamples = vidMedia.getSampleCount();

sampIdx = 1;   // get first sample in the track
mediaSample = vidMedia.getSample(0, 
  vidMedia.sampleNumToMediaTime(sampIdx).time,1);

// store width and height of image in the sample
ImageDescription imgDesc = 
      ImageDescription) mediaSample.description;
width = imgDesc.getWidth();
height = imgDesc.getHeight();

sampIdx is a counter that will iterate through the samples (the first sample starts at position 1).

The movie image's width and height are obtained by examining the first sample, under the assumption that all of the samples use the same dimensions.

Measuring FPS

The number of frames per second returned by QTSnapper will be used later to compare different implementation strategies for the class. The necessary elements are initialized in the constructor:

// frame rate globals
private long startTime;
private long numFramesMade;

// initialize them in the constructor
startTime = System.currentTimeMillis(); 
numFramesMade = 0;

Wrapping Up

As the application is about to terminate, stopMovie() is called in QTSnapper. It reports the FPS, and shuts down QuickTime.

// globals
private DecimalFormat frameDf = 
    new DecimalFormat("0.#"); // 1 dp


synchronized public void stopMovie()
{
  if (isSessionOpen) {
    // report frame rate
    long duration = 
        System.currentTimeMillis() - startTime;
    double frameRate = 
       ((double) numFramesMade*1000.0)/duration;
    System.out.println("FPS: " + 
                  frameDf.format(frameRate));

    QTSession.close();  // close down QuickTime
    isSessionOpen = false;
  }
}

stopMovie() and getFrame() are synchronized so that it's impossible to terminate the QuickTime session while a frame is being copied from the movie.

Catching a Frame

getFrame() returns a single sample (a frame) from the movie as a BufferedImage object. The frame is selected using the index number stored in sampIdx (which goes from 1 to numSamples, and then repeats).

// globals
private BufferedImage img, formatImg;


synchronized public BufferedImage getFrame()
{
  if (!isSessionOpen)
    return null;
  if (sampIdx > numSamples)   
    // start back with the first sample
    sampIdx = 1;

  try {
    /* Get the sample starting at the 
       specified index time */
    TimeInfo ti = 
       vidMedia.sampleNumToMediaTime(sampIdx);
    mediaSample=vidMedia.getSample(0,ti.time,1);
    sampIdx++;

    writeToBufferedImage(mediaSample, img);

    // resize img, writing it to formatImg
    Graphics g = formatImg.getGraphics();
    g.drawImage(img, 0, 0, 
         FORMAT_SIZE, FORMAT_SIZE, null); 

    // Overlay current time on image
    g.setColor(Color.RED);
    g.setFont(
      new Font("Helvetica", Font.BOLD, 12));
    g.drawString(timeNow(), 5, 14);
    g.dispose();

    numFramesMade++;  // count frame
  }
  catch (Exception e) {
    System.out.println(e);
    formatImg = null;
  }

  return formatImg;
} // end of getFrame()

The sample is readily obtained by calling the getSample() method from QTJ's Media class. Unfortunately, there's still the tricky question of converting the sample into a BufferedImage, which I've hidden away inside of my writeToBufferedImage() method.

The method performs some fancy translation dance steps, which I lifted from Chris W. Johnson's MovieFrameExtractor.java example, available from the quicktime-java mailing list.

The gory details, copiously commented, can be studied in the code. A "raw" image is extracted from the sample, and then decompressed as it's written into a QuickTime version of a Graphics object. The uncompressed data in the Graphics object is copied into another "raw" image, and then into a pixel array (a PixMap). Finally, this array is written into the DataBuffer part of an empty BufferedImage.

Does the Application Work? Does it Work Well?

Yes, Movie3D displays a movie, but large movies play slowly. This is due to getFrame()'s slowness in supplying frames, which can be quantified by looking at the FPS numbers.

For the movie in Figure 1, the reported FPS on a slow Windows 98 machine is in the range of 15 to 17 frames per second. However, the TimeBehavior object is requesting an update every 40 milliseconds, which should translate into frames appearing at nearer to 25 FPS.

getFrame() is slow because of the time-consuming conversion of the sample to a BufferedImage. As the current call to getFrame() gets bogged down converting a frame, further requests are delayed until the current one is finished.

I'll look at two ways of attacking this problem: permitting getFrame() to skip frames when it finally gets to process a request, and trying a different conversion strategy in getFrame(). I'll look at each of these in turn, starting with frame skipping.

Pages: 1, 2, 3

Next Page