Playing Movies in a Java 3D World, Part 2:
Pages: 1, 2, 3

4. A Movie that Skips Frames

The new Snapper class, QTSnapper1, still returns a frame when its getFrame() method is called. It differs from QTSnapper in that it supplies a frame corresponding to the current running time of the movie.

For example, getFrame() may retrieve frames 1, 2, 5, 8, 12, etc., depending on the when the method is called. Consequently, the movie progresses at a good speed, but the lack of frames may cause the movie to jitter.

In comparison, QTSnapper will return all of the frames (1, 2, 3, 4, etc.), but the delay between the getFrame() calls will make the movie run slowly. However, there won't be any jitter, since no frames are dropped.

The crucial element in QTSnapper1 is the notion of "current running time" for the movie. My approach is to calculate the current running time for the QTSnapper object when getFrame() is called, and convert it to a movie running time and then finally to a sample index value.

QTSnapper1 has the same public methods as QTSnapper, so it can be utilized in QTMovieScreen with minimal changes. The difference only becomes apparent when a movie is played--the movie rattles along at a good speed. Measurements, detailed later, put the "apparent" frame rate for the example in Figure 1 at about 31 FPS, compared to 16 FPS for QTSnapper.

Accessing the Movie's Video Media

QTSnapper1 follows the same steps as QTSnapper to access the movie's video. Once the video is available, several media values are stored as globals, to be used later by getFrame():

// globals
private Media vidMedia;
private int numSamples;
private int timeScale;   // media's time scale
private int duration;    // duration of the media


// in the constructor,
// get the media used by the video track
vidMedia = videoTrack.getMedia();

// store media details for later
numSamples = vidMedia.getSampleCount();
timeScale = vidMedia.getTimeScale();
duration = vidMedia.getDuration();

Getting a Frame Now

The new element of getFrame() is how it calculates the index value used to access a particular sample. The rest of the method, the call to writeToBufferedImage() and the code for writing the current time on the image, is the same as in QTSnapper.

// globals
private MediaSample mediaSample;
private BufferedImage img, formatImg;

private int prevSampNum;
private int sampNum = 0;
private int numCycles = 0;
private int numSkips = 0;


// inside getFrame(),
// get the time in secs since start of QTSnapper1
double currTime = 
  ((double)(System.currentTimeMillis() - 
            startTime))/1000.0;

// use the video's time scale
int videoCurrTime = 
     ((int)(currTime*timeScale)) % duration;

try {
  // backup the previous sample number
  prevSampNum = sampNum;

  // calculate the new sample number
  sampNum = vidMedia.timeToSampleNum(
                     videoCurrTime).sampleNum;

  // if no sample change, then don't generate
  // a new image
  if (sampNum == prevSampNum) 
    return formatImg; 

  if (sampNum < prevSampNum)    
    numCycles++;   // movie has just started over

  // record the number of frames skipped
  int skipSize = sampNum - (prevSampNum+1);
  if (skipSize > 0)  // skipped frame(s)
    numSkips += skipSize;

  // get a single sample starting at the 
  // sample number's time
  TimeInfo ti = 
      vidMedia.sampleNumToMediaTime(sampNum);
  mediaSample = vidMedia.getSample(0,ti.time,1);

getFrame() calculates the current time in seconds, measured from when QTSnapper1 started:

double currTime = 
  ((double)(System.currentTimeMillis() - 
            startTime))/1000.0;

Each piece of QuickTime media has its own time scale, ts, such that one unit of time is 1/ts seconds. The time scale constant must be multiplied to currTime to get the current time in movie time units:

int videoCurrTime = 
    ((int)(currTime*timeScale)) % duration;

The time is corrected modulo the media duration, to allow the movie to repeat when the current time passes the end of the movie.

The time is mapped to a sample index number by calling Media's timeToSampleNum() method:

sampNum = vidMedia.timeToSampleNum(
                     videoCurrTime).sampleNum;

The previously used sample number is stored in prevSampNum, to allow a number of tests and calculations to be carried out.

If the "new" sample number is the same as the previous one, then there's no need to go through the lengthy process of converting the sample to a BufferedImage; getFrame() can return the existing formatImg reference.

If the new sample number is less than the previous one, this means that the movie has looped, and a frame from the start of the movie is about to be shown. This looping is registered by incrementing numCycles.

If the new sample number is greater than the previous number plus one, then the number of skipped samples is recorded.

Wrapping Up

stopMovie() prints the FPS and closes the QuickTime session, in a similar way to the stopMovie() method in QTSnapper. It also reports additional information:

long totalFrames = 
   (numCycles * numSamples) + sampNum;

// report percentage of skipped frames
double skipPerCent = 
    (double)(numSkips * 100) / totalFrames;
System.out.println("Percentage frames skipped: "+ 
              frameDf.format(skipPerCent) + "%");

// 'apparent' FPS (AFPS)
double appFrameRate = 
      ((double) totalFrames * 1000.0) / duration;
System.out.println("AFPS: " + 
      frameDf.format(appFrameRate));  // 1 dp

appFrameRate represents the "apparent" frame rate, which is the total number of samples that have been iterated over since QTSnapper1 began. It's "apparent" in the sense that not all of those samples will have necessarily been rendered to the screen.

Does the Application Work? Does it Work Well?

With QTSnapper1 instead of QTSnapper, slow movies (such as the one in Figure 1) play much faster. The numbers reported at termination time put the apparent frame rate at about 31 FPS, the actual frame rate at around 16 FPS, and the percentage of skipped frames near 50 percent. Surprisingly, this high amount of missed frames is not that apparent on screen.

For other, smaller, movies, the speed-up is less noticeable; the percentage of skipped frames is around five percent to 10 percent.

Unfortunately, there are two problems: scrambled pixels when frames are skipped, and a lack of control over the number of frames that may be skipped.

Scrambled Pixels

Whenever QTSnapper1 skips a movie frame, the next retrieved frame will contain some scrambled pixels. This effect can be seen in Figure 6. The incorrect pixels use values from an earlier stage in the video.

Figure 6. Partly scrambled image

I figured out a solution with the help of people from the quicktime-java mailing list (my special thanks to George Birbilis and Dean Perry).

The problem is that all of my movie examples use temporal compression, a compression scheme that takes advantage of similarities between successive video frames. If two successive frames have the same background, then there's no need to store the background again. Instead, only the differences between the two frames are stored.

This technique, which is employed by almost all the popular video formats, means that the extraction of an image from a frame will depend on that frame and potentially several previous ones, as well.

Temporal decompression is dealt with by a QuickTime DSequence object, which is employed by my writeToBufferedImage() method. The DSequence constructor specifies that QuickTime should use an offscreen image buffer during the decompression phase.

The frame's image is written to the buffer, where it's combined with earlier frame data. The resulting image is passed to the next stage of the conversion.

This works well when QTSnapper1 is decompressing samples in sequence, with no skipped frames (e.g., 1, 2, 3, 4), but skipping spells trouble. For example, what happens if QTSnapper1 skips frames 5 and 6, and then decompresses frame 7? The frame is written to a QuickTime image buffer, and combined with earlier frame data. Unfortunately, data from frames 5 and 6 is missing, and so the resulting image will be incorrect.

In brief, the scrambled pixels in an image are due to the use of temporal compression in the movie. One alternative is to use spatial compression, which compresses each frame individually, in a style similar to JPEG or GIF compression for a single image. This means that all the information for decompressing a frame is present in the frame itself; there's no need to examine earlier frames.

The QuickTime MOV format supports a spatial compression scheme called Motion-JPEG (M-JPEG). I used the export tool in QuickTime 6 Pro to save the example in Figure 1 as a MOV file using the M-JPEG A codec. When this movie was played by the Movie3D application, no scrambling occurred.

Limiting Frame Skipping

Another issue with QTSnapper1 is that getFrame() doesn't place any limit on the number of frames that can be skipped. In my tests, the upper limit was a skip size of three frames, which wasn't noticeable. However, if getFrame() is given a larger sample (e.g., larger dimensions or larger resolution) to convert, then it's increased slowness will be compensated for by more skipped frames. The movie quality could deteriorate very significantly.

5. Trying to Generate the Image Faster

The sample-to-BufferedImage conversion method (writeToBufferedImage()) used in QTSnapper and QTSnapper1 is taken from an example by Chris W. Johnson. Is there another, hopefully faster, way of extracting an image from a sample?

The standard book on QTJ is QuickTime for Java: A Developer's Notebook, by Chris Adamson, O'Reilly, January 2005. Chapter 5, covering QuickDraw, contains a ConvertToJavaImageBetter.java example, which shows how to grab a sample as a PICT image and convert it to a Java Image object. This example may also be found on the quicktime-java mailing list.

The conversion isn't straightforward, relying on the addition of a dummy 512-byte header to the PICT object so it can be treated as a PICT file by the QuickTime version of an ImageProducer.

I used Adamson's code as the basis of yet another Snapper class, called QTSnapper2. It renders a frame sequence without skipping, in the same way as QTSnapper, but employs PICT-to-Image translation.

On small movies, QTSnapper2's performance is indistinguishable from QTSnapper, but for the slightly larger movie of Figure 1, its average frame rate descends to about 9 FPS, compared to QTSnapper's 16 FPS. In other words, PICT-based translation is slower than Johnson's technique.

Click here to return to part 1 of this two-part article.

---

In May 2005, O'Reilly Media, Inc., released Killer Game Programming in Java.

• Chapter 22, "Flocking Boids" (PDF format), is available free online.

• You can also look at the Table of Contents, the Index, and the full description of the book.

• For more information, or to order the book, click here.

Andrew Davison has had a varied and interesting career as an educator, a researcher, and an author. Formerly with the Computer Science Department at Melbourne University, he now lives in Thailand and teaches at the Prince of Songkla University.

---

Return to ONJava.com.