ONJava.com    
 Published on ONJava.com (http://www.onjava.com/)
 See this if you're having trouble printing code examples


Streaming QuickTime with Java Streaming QuickTime with Java

by Chris Adamson, author of QuickTime for Java: A Developer's Notebook
01/12/2005

This isn't an excerpt from my soon-to-be-released QuickTime for Java book, though now I wish it was.

You see, the thing is, a lot of us in the QTJ world had long assumed that the streaming APIs in QTJ were broken, and I didn't want to commit to covering something that was broken. OK, I went ahead and did so by covering capture, which various people got working through different techniques, but I didn't want to do it twice. Besides, the case against streaming seemed particularly bad. No one could get its demo code working--this post to the quicktime-java list is typical of many users' desperation with getting Apple's AudioBroadcaster and DrawableBroadcaster demos working. To make things worse, one of the demos depended on a GUI preview component class that was taken out of QTJ in version 6.1 as QTJ radically scaled back its GUI offerings, offering components only for Movies, MovieControllers and GraphicsImporters, not streaming Presentations, video capture, or certain graphic niceties like "composited" content created from multiple sources. So, given official demos that didn't seem to work in the first place and that now had key classes deprecated (and throwing RuntimeExceptions if they were actually used in Java 1.4), the prognosis for actually streaming content with QTJ 6.1 seemed very bad.

Which is why I'm delighted, and more than a little surprised, to report that streaming actually does work in QTJ 6.1. In this article, I'll introduce the basics of simple webcasting with QTJ.

Requirements

QuickTime's streaming API, represented in Java by the package quicktime.streaming works only on Mac OS (Classic and OS X). The classes are present in QTJ for Windows, but they won't do anything. However, you can use QuickTime for Windows as a client for a stream, and if running in Java isn't critical, you can get the Darwin Streaming Server, an open source project that's runnable on Windows 2000 Server and 2003 Server, as well as Solaris 9 and Red Hat Linux 9.

Related Reading

QuickTime for Java: A Developer's Notebook
By Chris Adamson

The simplest thing to stream with QuickTime is live content, which is what I cover here, so you'll at least need an audio input device, such as a built-in mic or a headset. Having a QuickTime-supported camera, like an iSight, is far more impressive, of course.

What Streaming Is, What It Isn't

Given the overloading of the term "stream," it's not easy to nail down what the term "streaming" even means. For example, QuickTime has long supported a "fast-start" feature--a movie could start playing if QuickTime determined that it had enough data that it could start playing and would not run out of data at the current download rate--that some people mistook for a form of streaming. And to be sure, that has its advantages: it's easier to set up and it ensures that all packets get to the client. But real streaming, that is to say, streaming that corresponds to Internet Engineering Task Force (IETF) standards, is a different matter entirely, not supported by QuickTime until QuickTime 5, and not exposed to Java until QTJ 6.

This form of streaming allows the server to control the transmission, and is heavily optimized to keep things running in real-time. Clients don't have to download potentially huge files, and this approach is particularly well-suited to live broadcasts. In fact, QuickTime's streaming uses two types of "real-time" streaming protocols: Real-time Transport Protocol (RTP) to send packets of media data, and Real-Time Streaming Protocol (RTSP) for control information. RTP uses the potentially lossy UDP, so all parties are designed to tolerate the dropping of packets during a transmission. This means that clients have to gracefully handle not getting all of the data for a video frame or an audio sample. This is preferable to a TCP/IP-based approach, which could make an indeterminate of retries (and thus take an indeterminate amount of time) to get a missed packet.

Presentations and SDP Files

In QuickTime, streaming is based around the idea of a Presentation, represented in QTJ as the class quicktime.streaming.Presentation. In many ways, it is the streaming equivalent of a Movie--where a movie might have audio and video tracks, and a collection of metadata to tie it all together, the presentation will combine some metadata with multiple audio and video streams. It's also worth noting that audio and video are the kinds of media you're most like to stream, since certain other media types supported by QuickTime (sprites, Flash content) do not handle lost packets well, and are thus not well-suited to streaming.

You might have guessed that to set up a stream, you'll need to create a Presentation and kick it off. But how? The most common way to do so is to create a Session Description Protocol (SDP) file, and toss it over to the static factory method Presentation.fromFile(). The SDP file is in a reasonably straightforward text format, defined by RFC 2327 and several updates. I find this a lot easier in theory than in practice, but let's worry about implementation details later. Here's an SDP used by some of Apple's streaming examples, and in Tim Monroe's QuickTime Toolkit Volume Two:


v=0
c=IN IP4 224.2.1.2/20/1
m=audio 1000 RTP/AVP 12
m=video 2000 RTP/AVP 101
a=rtpmap:101 H263-1998

To explain this file, line by line:

This is all well and good, but when I used it with my sample program, I got a video stream but no sound. So I used a very different SDP, one that was originally provided with the QTJ DrawableBroadcaster demo. Yes, the one they deprecated:


m=audio 2656 RTP/AVP 96
c=IN IP4 239.60.60.60
a=rtpmap:96 x-qt
m=video 2700 RTP/AVP 96
a=rtpmap:96 x-qt

The big difference here is that both audio and video use the same dynamic payload mapping, and it's not to a real codec, but the genericized x-qt. The win here is that you can pick any of QuickTime's audio and video codecs at runtime, instead of forcing the issue in the SDP file. The downside is that this may not be parsable by non-QuickTime clients, whereas using sufficiently standardized and/or old codecs and specifying them in the SDP makes it more likely that other clients (Real, JMF, etc.) will be able to handle your stream.

So there's your SDP file. Now to make a Presentation out of it.

Creating the Presentation

My streaming server program is called LittleBroadcast, and it's really not a lot of code, just over 140 lines. I'll step through it in this article, commenting general sections, but providing its full listing. It's also available in a .tar.gz file, along with the SDP file and an Ant build file, in the Resources section below.


package com.mac.invalidname.qtjstreaming;

import quicktime.*;
import quicktime.std.*;
import quicktime.util.*;
import quicktime.qd.*;
import quicktime.io.*;
import quicktime.streaming.*;
import quicktime.app.time.*;

import java.io.*;
import java.awt.*;
import java.awt.event.*;

public class LittleBroadcast extends Tasking implements ActionListener {

This is the typically long list of QuickTime imports, including qd in order to use its QDGraphics to provide an offscreen drawing surface for the camera, io to read the SDP file, streaming for the streaming API, and time to pick up some tasking utilities to give the Presentation time to run. On that last point, notice that the class extends Tasking--that provides a task() that's called back periodically. For this application, it's used to continually call the Presentation's idle() method and give it some cycles to work with. You'll learn in the book is that the same thing is needed for Movies, but the tasking is almost always handled for your automatically. No such luck with Presentations. (Or capture, for that matter, but I digress.)


    boolean broadcasting = false;

    public static final int BROADCAST_WIDTH = 176;
    public static final int BROADCAST_HEIGHT = 144;

    Button startStopButton;
    Button configButton;

    Presentation pres;
    int presenterTimeScale = 600;

These are the server's instance variables. broadcasting is a flag used to determine what to do when the start/stop button is pressed. Next is a pair of constants for the size of the broadcast video, followed by the buttons for the server GUI. Finally, there's a Presentation object, and its timescale. (The time-keeping system for the media, where a timescale of 600 means there are 600 units in a second; 600 is a common default in QuickTime.)


    public static void main (String[] args) {
        System.out.println ("main");
        try {
            QTSession.open();
            new LittleBroadcast();
        } catch (QTException qte) {
            qte.printStackTrace();
        }
    }

There's nothing particularly special about this main. I put all of the smarts in the constructor in case I needed to create an inner class, for which I'd need a instance. Maybe you'll find that useful if you extend this code.


    public LittleBroadcast() throws QTException {
        System.out.println ("LittleBroadcast constructor");
        QTFile file = new QTFile (new File ("little.sdp"));
		try {
			MediaParams mediaParams = new MediaParams();
            mediaParams.setWidth (BROADCAST_WIDTH);
            mediaParams.setHeight (BROADCAST_HEIGHT);

            QDGraphics myGWorld =
                new QDGraphics (new QDRect (
                    BROADCAST_WIDTH, BROADCAST_HEIGHT));
            mediaParams.setGWorld (myGWorld);

			PresParams presParams =
                new PresParams( presenterTimeScale,
                            QTSConstants.kQTSSendMediaFlag | 
                            QTSConstants.kQTSAutoModeFlag |
                            QTSConstants.kQTSDontShowStatusFlag,
                            mediaParams );
			pres = Presentation.fromFile(file, presParams );

The first thing the constructor does is to load an SDP file called little.sdp. But this isn't all it needs to create the Presentation--a number of parameters need to be set by the server application for use in the Presentation.fromFile() call. First, you create a MediaParams object, on which you can set the video height and width. Another important thing you have to do is to provide the camera a drawing surface, which you do by creating a QDGraphics and setting it on the MediaParams. Yes, the names are weird, because the QTJ designers wanted to stress the similarity to an AWT Graphics object, but every method that gets or sets with such an object uses its native-API name, GWorld. Finally, you create PresParams to set parameters for the whole Presentation. This takes a somewhat arbitrary timescale, some behavior flags mathematically OR'ed together, and the MediaParams. The possible behavior flags, all defined in QTSConstants, include:

With the SDP file reference, parameters, and GWorld set up, you create the Presentation with Presentation.fromFile().


            // find audio stream
            Stream audioStream = null;
            for (int i=1; i<=pres.getNumStreams(); i++) {
                System.out.println ("stream: " + i + ": " + 
                                    pres.getIndStream(i));
                Stream aStream = pres.getIndStream (i);
                if (pres.hasCharacteristic(aStream, 
                    StdQTConstants.audioMediaCharacteristic)) {
                    audioStream = aStream;
                    break;
                }
            }
            System.out.println ("audioStream = " + audioStream);
            pres.setVolumes (audioStream, 100, 100);
            System.out.println ("created presentation, gworld == " +
                                pres.getGWorld() + ", size == "+
                                mediaParams.getWidth() + "x" + 
                                mediaParams.getHeight() + ", streams == " +
                                pres.getNumStreams());  

This isn't genuinely necessary, but it shows you how to tour through the Presentation to pick out individual streams. Presentation.getIndStream will return a Stream by index (note that QuickTime indexes are, as always, 1-based). It iterates over these asking for the audioMediaCharacteristic to find the audio stream (for video, you'd ask for the visualMediaCharacteristic). This example sets the volume on the audioStream, for both the left and right channels, to its maximum value of 100.

Finally, the println dumps a bunch of interesting metadata from the Presentation and the MediaParams.

Configuring the Presentation


            SettingsDialog sd = new SettingsDialog (pres);
            System.out.println ("Did settings");
            pres.preroll();
            broadcasting = false;

These are the final steps to set up the presentation. The SettingsDialog presents the user with a choice of input devices for audio and video (the two streams the SDP file specified as being in the Presentation). Each stream can also be customized with a compressor (MPEG-4, Sorenson Video 3, H.263, etc.) and a packetizer (which is sometimes dictated by the compressor; watch to see if it changes automatically when you change compressors). An example of this GUI is shown in Figure 1.


Figure 1. SettingsDialog for a Presentation

In the pictured case, the audio has defaulted to the computer's line in. To change it to the iSight, you'd click the Source button, which brings up a device selection list, as shown in Figure 2.


Figure 2. Source selection dialog

The last thing to do is to call Presentation.preroll(), which, like Movie.preroll(), gives the Presentation an opportunity to pre-allocate resources and get ready to begin streaming the Presentation.

Providing a Control GUI


            // Make monitor window
            startStopButton = new Button ("Start");
            configButton = new Button ("Configure");
            startStopButton.addActionListener (this);
            configButton.addActionListener (this);
            Frame monitorFrame = new Frame ("QTJ Streaming");
            monitorFrame.setLayout (new BorderLayout());
            Panel buttonPanel = new Panel();
            buttonPanel.add (startStopButton);
            buttonPanel.add (configButton);
            monitorFrame.add (buttonPanel, BorderLayout.SOUTH);
            monitorFrame.pack();
            monitorFrame.setVisible(true);

This sets up a trivial GUI for controlling and configuring the Presentation, basically just offering a start/stop button and a configure button. The buttons are referred to this as an ActionListener, meaning this class will need to provide an actionPerformed method later to handle button clicks. A screenshot of the control GUI is shown in Figure 3.


Figure 3. Monitor/control window

One interesting question you might ask at this point is: "Since when are we worrying about providing a server with a GUI?" Presumably, this is a holdover from the Classic Mac OS, which didn't have a command line to start applications and send arguments to them. But moreover, one would usually like to provide a preview of the streaming data, and if you're going to have a preview window, why not have a configuration GUI, too?

Anyways, this is probably moot, because QTJ 6.1 does not provide an AWT Component that you could use to provide the preview. Hopefully at some point, the QTFactory will get a new overload for makeQTComponent that takes a Presentation and returns a Component that shows the streaming video. It's probably possible to hack one up entirely in Java with some QuickDraw voodoo. If, in each task() callback (see below) you take the GWorld created earlier, convert it to a Pict, and write that as a uniquely named file, you'll see each one is different, meaning the GWorld is getting fresh data each time. So if you instead took the GWorld and could write its pixels to an AWT Component on each pass, you'd have an onscreen preview. Anyone game to try this? See you on the quicktime-java list.

Details


            // add shutdown handler to make sure presentation 
            // gets stopped
            Thread presentationStopper = new Thread() {
                    public void run() {
                        try {
                            pres.stop();
                        } catch (QTException qte) {}
                    }
                };
            Runtime.getRuntime().addShutdownHook (presentationStopper);

This shutdown hook makes sure the Presentation is stopped before the program exits. This is important because, like the SequenceGrabber, the Presentation will be happy to keep running after your application quits, tying up a port, burning cycles, keeping other applications from using your capture device, etc.


		} catch ( QTException e ) {
			e.printStackTrace();
			System.exit (-1);
		}
    }

Finally, the constructor catches any QTExceptions that may have been thrown along the way.


    public void actionPerformed (ActionEvent ae) {
        System.out.println ("actionPerformed");
        try {
            if (ae.getSource() == startStopButton) {
                if (broadcasting) {
                    pres.stop();
                    stopTasking();
                    broadcasting = false;
                    startStopButton.setLabel ("Start");
                    System.out.println ("Stopped");
                } else {
                    pres.start();
                    startTasking();
                    broadcasting = true;
                    startStopButton.setLabel ("Stop");
                    System.out.println ("Started");
                }
            } else if (ae.getSource() == configButton) {
                new SettingsDialog (pres);
            }
        } catch (QTException qte) {
            qte.printStackTrace();
        }
    }

This is pretty straightforward handling of the start/stop and config buttons. If the clicked button is start/stop, the config GUI has to call start() or stop() on the Presentation, start or stop tasking (the periodic callbacks to this class' task() method), set the broadcasting flag for the benefit of the next button click, and change the label on the button. If the user clicked on configure, it launches a new SettingsDialog for the Presentation.


	public synchronized final void task() throws QTException {
		pres.idle(null);
	}

}

This final method implements the task() method inherited from Tasking and periodically called after startTasking() is called by the start button handler. By simply calling Presentation.idle(), it gives the presentation time to acquire current data from the capture devices, encode it, and stream it out.

Running the Streaming Client

The simplest way to get a client to see the broadcast is to use QuickTime Player to open the same SDP file that the server used to create the Presentation. This will invoke the SDP Importer to connect to the streams and start parsing their contents. Note that the client cannot be on the same machine as the server, apparently since the server will have occupied the ports used for the presentation, denying the client the ability to bind to those ports. Figure 4 shows what streaming from my desktop looks like (when I'm playing with my Macross and Escaflowne toys, that is).


Figure 4. QuickTime streaming client

If you're using QuickTime Player, you can use its Get Info command to reveal the two streams and their formats. In Figure 5, you can see there are two streams: an uncompressed 44.1kHz audio stream, and an H.263 video stream.


Figure 5. Client info window

Conclusion

As I said, kicking off a stream with QuickTime for Java is a lot easier than it has been assumed to be. The simplest case, broadcasting from capture devices, takes less than 150 lines of code. By far, the hardest part is understanding the SDP file, which proved extremely finicky and whose documentation presupposes a lot of knowledge that application-level programmers may not have. It's also unfortunate that QTJ doesn't provide a preview component anymore, but it may do so in the future, and a little bit of GWorld/QuickDraw hacking may produce such a component in the future.

This article only covered how to set up a broadcast for live-capture data. Other available Sourcers, such as those to broadcast QuickTime files from disk or arbitrary content, will be discussed in a future installment of this series.

Thanks to various members of quicktime-api for help deciphering SDP files for this article.

Resources

Chris Adamson is an author, editor, and developer specializing in iPhone and Mac.


Return to ONJava.com

Copyright © 2009 O'Reilly Media, Inc.