Making Media from Scratch, Part 2
Pages: 1, 2, 3

The next thing we do is to create a CSequence. This object provides us the ability to compress frames. We have to call this with each frame to compress, in order, and there's an interesting reason for this. If we were using a compression scheme meant for single images, such as JPEG, we could do the images in any order, since each frame would have all of the information it needed to be decompressed and rendered. This is generally not true of video compression schemes, which often use "temporal compression": techniques to compress data by eliminating redundant information between frames, such as an unchanging background. Because of this approach, decoding a given frame might depend on information from one or more previous frames, which is why we have to do our compression through an object that understands that we're working with a series of images.

The CSequence constructor looks like this:

CSequence seq = new CSequence (gw,
                               gRect, 
                               gw.getPixMap().getPixelSize(),
                               CODEC_TYPE,
                               CodecComponent.bestFidelityCodec,
                               StdQTConstants.codecNormalQuality, 
                               StdQTConstants.codecNormalQuality, 
                               KEY_FRAME_RATE,
                               null,
                               0);

These arguments are, in order:

• QDGraphics src: the QDGraphics from which to get image data. In our case, the offscreen GWorld into which we draw.
• QDRect srcRect: the portion of the src to use. In our case, the whole thing.
• int colorDepth: an int indicating the likely depth (4-bit color, 32-bit color, etc.) at which the frames are likely to be viewed. Pass 0 to let the Image Compression Manager choose for you. More info lives in the docs for the native function.
• int cType: the codec type, as described above.
• CodecComponent codec: often used to request a specific behavior of the given codec, such as the CodecComponent constants bestSpeedCodec, bestFidelityCodec, or bestCompressionCodec.
• int spatialQuality: a quality setting for the images, from codecMinQuality, through low, normal, and high, up to codecMaxQuality and, in for codecs that allow it, codecLosslessQuality (all in StdQTConstants).
• int temporalQuality: the quality setting for inter-frame compression, with values as above.
• int keyFrameRate: the maximum number of frames allowed between "key frames," which are the frames that have all of the information they need, and that may be needed for multiple subsequent frames to decompress.
• ColorTable clut: a custom color lookup table, often set to null to let QuickTime use the table from the source image.
• int flags: one or more behavior flags, logically OR'd together. One interesting option is codecFlagWasCompressed, which hints that the source image was previously compressed and gives the codec a chance to compensate for the artifacting and other image degradation that occurs when an image has been compressed with a lossy codec (like JPEG).

Once we've created the CSequence, we get an ImageDescription object, which we'll need later when adding samples to the Media.

Now we can start the loop to draw, compress, and add frames. We calculate a rectangle, fromRect, inside of the original image. This will be the source of this frame. Next, we create a Matrix that maps and scales from its original location and size to the offscreen buffer's location and size; in other words, a rectangle at (0,0) with dimensions VIDEO_TRACK_WIDTH by VIDEO_TRACK_HEIGHT. Calling GraphicsImporter.draw() performs the scaled drawing of the region into the offscreen QDGraphics.

Matrix drawMatrix = new Matrix();
drawMatrix.rect (fromRect, gRect);
importer.setMatrix (drawMatrix);
importer.draw();

Next, we compress the image that was drawn into the offscreen QDGraphics:

CompressedFrameInfo cfInfo =
        seq.compressFrame (gw, 
                           gRect, 
                           StdQTConstants.codecFlagUpdatePrevious, 
                           compressedImage);

The arguments to this call are:

• QDGraphics src: the source image to compress.
• QDRect rect : what portion of that image to use.
• int flags: behavior flags. Among the most useful is codecFlagUpdatePrevious, which is used for codecs that use temporal compression. Another interesting option not needed here is codecFlagLiveGrab, which you'd use if you were generating images from a live source, possibly image capture, and needed to compress frames as quickly as possible. In the typical QuickTime style, the desired flags are mathematically OR'd together.
• RawEncodedImage: a RawEncodedImage into which the compressed frame will be written. This is the object we made sure was big enough with that getMaxCompressionSize() call earlier.

The compressFrame call returns a CompressedImageInfo object, which has an important method called getSimilarity(). This value represents how similar the compressed image is to the one compressed just before it. A value of 255 means the images are identical. 0 means the compressed frame is to be a "key frame," meaning it has all the image data it needs, it does not depend on other frames, and other frames may depend on it. Other values simply represent image difference, where low values mean low similarity.

With the frame now compressed into the RawEncodedImage, we can add a sample to the VideoMedia, with the addSample method inherited from the Media superclass:

videoMedia.addSample (imageHandle, 
                      0, 
                      cfInfo.getDataSize(),
                      20,
                      imgDesc,
                      1,
                      (syncSample ?
                       0:
                       StdQTConstants.mediaSampleNotSync)
                      );

The arguments to this method are:

• QTHandleRef data: a reference to the sample data; in this case, to the RawEncodedImage.
• int dataOffset: an offset into the data. This is 0 in our case, since we're using all of the RawEncodedImage that was populated by compressFrame().
• int dataSize: the number of bytes of data, starting at dataOffset, to use. Again, we're using the whole RawEncodedImage.
• int durationPerSample: how long this sample lasts, expressed in units of the media's timescale. Since our timescale is 600, a duration of 20 equals 1/30th of a second.
• SampleDescription sampleDesc: an object that tells the media what to do with the sample data being passed in. This is why we got an ImageDescription from the CSequence earlier.
• int numberOfSamples: the number of samples provided by this call. For video, this is typically one frame. For other kinds of media, there are some performance considerations described in the native docs.
• int sampleFlags: behavior flags. The interesting value here is whether or not this is a "key frame," also known in QuickTime as a "sync sample." We set the mediaSampleNotSync flag if our earlier call to CompressedFrameInfo.getSimilarity() returned non-zero. Note that failing to set this flag correctly is a popular cause of movies that "blur" when scrubbed or played from any point other than the first frame, as explained in an Apple tech note.

Once the loop finishes, we do the same clean-up tasks as with the text-track samples in Part 1 -- declare that we're done editing and insert the media into the video track:

videoMedia.endEdits();
videoTrack.insertMedia (0, // trackStart
                        0, // mediaTime
                        videoMedia.getDuration(), // mediaDuration
                        1); // mediaRate

Finally, we save the movie to disk, exactly as before.

The Result!

Here, for those with QuickTime 5 or 6, is a videotrack.mov movie produced by the sample code. If you recompile and re-run the code with different codecs and different sizes, you'll see some fairly dramatic differences in file size and image quality. I've used 160x120 to keep the file size small, in order to avoid abusing O'Reilly's bandwidth, and the compression artifacts here are more visible than in the 320x240 version.

Also remember that while we just copied a scaled section of an image into the offscreen buffer, you can do any kind of imaging with this buffer before compressing it into a frame. For example, you could do the drawing commands in the QDGraphics class, or use the QTImageDrawer to use Java 2D Graphics methods to draw into the QuickTime world. With some bit-munging, you might even find a way to render 3D graphics from JOGL into QuickTime ... anyone up for rendering Finding Nemo directly into a QuickTime movie?

This completes our tour of QuickTime media structures, in which we've gone from the high-level view of what makes up a movie to the low-level mucking around with individual samples. This is a little "closer to the metal" than QTJ usually requires, but if you believe in keeping simple tasks easy and complex tasks possible, this has been an example of the latter.

Chris Adamson is an author, editor, and developer specializing in iPhone and Mac.

---

Return to ONJava.com.