
Playing Movies in a Java 3D World, Part 2:
Pages: 1, 2, 3
4. A Movie that Skips Frames
The new Snapper
class, QTSnapper
1, still returns a frame when its
getFrame()
method is called. It differs from QTSnapper
in that it
supplies a frame corresponding to the current running time of the
movie.
For example, getFrame()
may retrieve frames 1, 2, 5, 8, 12, etc.,
depending on the when the method is called. Consequently, the movie
progresses at a good speed, but the lack of frames may cause the
movie to jitter.
In comparison, QTSnapper
will return all of the frames (1, 2, 3, 4, etc.),
but the delay between the getFrame()
calls will make the movie run
slowly. However, there won't be any jitter, since no frames are dropped.
The crucial element in QTSnapper
1 is the notion of "current running
time" for the movie. My approach is to calculate the current running
time for the QTSnapper
object when getFrame()
is called, and convert
it to a movie running time and then finally to a sample index value.
QTSnapper
1 has the same public methods as QTSnapper
, so it can be
utilized in QTMovieScreen
with minimal changes. The difference only
becomes apparent when a movie is played--the movie rattles along at
a good speed. Measurements, detailed later, put the "apparent" frame
rate for the example in Figure 1 at about 31 FPS, compared to 16 FPS
for QTSnapper
.
Accessing the Movie's Video Media
QTSnapper
1 follows the same steps as QTSnapper
to access the movie's
video. Once the video is available, several media values are stored
as globals, to be used later by getFrame()
:
// globals
private Media vidMedia;
private int numSamples;
private int timeScale; // media's time scale
private int duration; // duration of the media
// in the constructor,
// get the media used by the video track
vidMedia = videoTrack.getMedia();
// store media details for later
numSamples = vidMedia.getSampleCount();
timeScale = vidMedia.getTimeScale();
duration = vidMedia.getDuration();
Getting a Frame Now
The new element of getFrame()
is how it calculates the index value
used to access a particular sample. The rest of the method, the call
to writeToBufferedImage()
and the code for writing the current time
on the image, is the same as in QTSnapper
.
// globals
private MediaSample mediaSample;
private BufferedImage img, formatImg;
private int prevSampNum;
private int sampNum = 0;
private int numCycles = 0;
private int numSkips = 0;
// inside getFrame()
,
// get the time in secs since start of QTSnapper1
double currTime =
((double)(System.currentTimeMillis() -
startTime))/1000.0;
// use the video's time scale
int videoCurrTime =
((int)(currTime*timeScale)) % duration;
try {
// backup the previous sample number
prevSampNum = sampNum;
// calculate the new sample number
sampNum = vidMedia.timeToSampleNum(
videoCurrTime).sampleNum;
// if no sample change, then don't generate
// a new image
if (sampNum == prevSampNum)
return formatImg;
if (sampNum < prevSampNum)
numCycles++; // movie has just started over
// record the number of frames skipped
int skipSize = sampNum - (prevSampNum+1);
if (skipSize > 0) // skipped frame(s)
numSkips += skipSize;
// get a single sample starting at the
// sample number's time
TimeInfo ti =
vidMedia.sampleNumToMediaTime(sampNum);
mediaSample = vidMedia.getSample(0,ti.time,1);
getFrame()
calculates the current time in seconds, measured from
when QTSnapper
1 started:
double currTime =
((double)(System.currentTimeMillis() -
startTime))/1000.0;
Each piece of QuickTime media has its own time scale, ts
, such that one unit
of time is 1/ts
seconds. The time scale constant must be multiplied
to currTime
to get the current time in movie time units:
int videoCurrTime =
((int)(currTime*timeScale)) % duration;
The time is corrected modulo the media duration, to allow the movie to repeat when the current time passes the end of the movie.
The time is mapped to a sample index number by calling Media
's
timeToSampleNum()
method:
sampNum = vidMedia.timeToSampleNum(
videoCurrTime).sampleNum;
The previously used sample number is stored in prevSampNum
, to allow
a number of tests and calculations to be carried out.
If the "new" sample number is the same as the previous one, then
there's no need to go through the lengthy process of converting
the sample to a BufferedImage
; getFrame()
can return the existing
formatImg
reference.
If the new sample number is less than the previous one, this means
that the movie has looped, and a frame from the start of the movie
is about to be shown. This looping is registered by incrementing
numCycles
.
If the new sample number is greater than the previous number plus one, then the number of skipped samples is recorded.
Wrapping Up
stopMovie()
prints the FPS and closes the QuickTime session, in a
similar way to the stopMovie()
method in QTSnapper
. It also reports
additional information:
long totalFrames =
(numCycles * numSamples) + sampNum;
// report percentage of skipped frames
double skipPerCent =
(double)(numSkips * 100) / totalFrames;
System.out.println("Percentage frames skipped: "+
frameDf.format(skipPerCent) + "%");
// 'apparent' FPS (AFPS)
double appFrameRate =
((double) totalFrames * 1000.0) / duration;
System.out.println("AFPS: " +
frameDf.format(appFrameRate)); // 1 dp
appFrameRate
represents the "apparent" frame rate, which is the total
number of samples that have been iterated over since QTSnapper
1 began.
It's "apparent" in the sense that not all of those samples will have
necessarily been rendered to the screen.
Does the Application Work? Does it Work Well?
With QTSnapper
1 instead of QTSnapper
, slow movies (such as the one
in Figure 1) play much faster. The numbers reported at termination
time put the apparent frame rate at about 31 FPS, the actual frame
rate at around 16 FPS, and the percentage of skipped frames near
50 percent. Surprisingly, this high amount of missed frames is not that
apparent on screen.
For other, smaller, movies, the speed-up is less noticeable; the percentage of skipped frames is around five percent to 10 percent.
Unfortunately, there are two problems: scrambled pixels when frames are skipped, and a lack of control over the number of frames that may be skipped.
Scrambled Pixels
Whenever QTSnapper
1 skips a movie frame, the next retrieved frame
will contain some scrambled pixels. This effect can be seen in Figure 6.
The incorrect pixels use values from an earlier stage in the video.
Figure 6. Partly scrambled image
I figured out a solution with the help of people from the quicktime-java mailing list (my special thanks to George Birbilis and Dean Perry).
The problem is that all of my movie examples use temporal compression, a compression scheme that takes advantage of similarities between successive video frames. If two successive frames have the same background, then there's no need to store the background again. Instead, only the differences between the two frames are stored.
This technique, which is employed by almost all the popular video formats, means that the extraction of an image from a frame will depend on that frame and potentially several previous ones, as well.
Temporal decompression is dealt with by a QuickTime DSequence
object, which
is employed by my writeToBufferedImage()
method. The DSequence
constructor
specifies that QuickTime should use an offscreen image buffer during the
decompression phase.
The frame's image is written to the buffer, where it's combined with earlier frame data. The resulting image is passed to the next stage of the conversion.
This works well when QTSnapper
1 is decompressing samples in sequence, with
no skipped frames (e.g., 1, 2, 3, 4), but skipping spells trouble. For
example, what happens if QTSnapper
1 skips frames 5 and 6, and then
decompresses frame 7? The frame is written to a QuickTime image buffer,
and combined with earlier frame data. Unfortunately, data from frames 5
and 6 is missing, and so the resulting image will be incorrect.
In brief, the scrambled pixels in an image are due to the use of temporal compression in the movie. One alternative is to use spatial compression, which compresses each frame individually, in a style similar to JPEG or GIF compression for a single image. This means that all the information for decompressing a frame is present in the frame itself; there's no need to examine earlier frames.
The QuickTime MOV format supports a spatial compression scheme called Motion-JPEG (M-JPEG). I used the export tool in QuickTime 6 Pro to save the example in Figure 1 as a MOV file using the M-JPEG A codec. When this movie was played by the Movie3D application, no scrambling occurred.
Limiting Frame Skipping
Another issue with QTSnapper
1 is that getFrame()
doesn't place any limit
on the number of frames that can be skipped. In my tests, the upper limit
was a skip size of three frames, which wasn't noticeable. However, if getFrame()
is given a larger sample (e.g., larger dimensions or larger resolution) to
convert, then it's increased slowness will be compensated for by more
skipped frames. The movie quality could deteriorate very significantly.
5. Trying to Generate the Image Faster
The sample-to-BufferedImage
conversion method (writeToBufferedImage()
)
used in QTSnapper
and QTSnapper
1 is taken from an example by Chris W.
Johnson. Is there another, hopefully faster, way of extracting an
image from a sample?
The standard book on QTJ is
QuickTime for Java: A Developer's Notebook, by Chris Adamson, O'Reilly, January 2005. Chapter 5, covering QuickDraw, contains a ConvertToJavaImageBetter.java
example,
which shows how to grab a sample as a PICT image and convert it to a
Java Image
object. This example may also be found on the quicktime-java
mailing list.
The conversion isn't straightforward, relying on the addition of a
dummy 512-byte header to the PICT object so it can be treated as a
PICT file by the QuickTime version of an ImageProducer
.
I used Adamson's code as the basis of yet another Snapper
class,
called QTSnapper
2. It renders a frame sequence without skipping, in
the same way as QTSnapper
, but employs PICT-to-Image
translation.
On small movies, QTSnapper
2's performance is indistinguishable from
QTSnapper
, but for the slightly larger movie of Figure 1, its average
frame rate descends to about 9 FPS, compared to QTSnapper
's 16 FPS.
In other words, PICT-based translation is slower than Johnson's
technique.
Click here to return to part 1 of this two-part article.
In May 2005, O'Reilly Media, Inc., released Killer Game Programming in Java.
Chapter 22, "Flocking Boids" (PDF format), is available free online.
You can also look at the Table of Contents, the Index, and the full description of the book.
For more information, or to order the book, click here.
Andrew Davison has had a varied and interesting career as an educator, a researcher, and an author. Formerly with the Computer Science Department at Melbourne University, he now lives in Thailand and teaches at the Prince of Songkla University.
Return to ONJava.com.
