ONJava.com    
 Published on ONJava.com (http://www.onjava.com/)
 See this if you're having trouble printing code examples


Enabling Peer-to-Peer BitTorrent Downloads with Azureus

by Jacobus Steenkamp
06/22/2007

The type of traffic distribution on the Internet today is quite different from the type you might have encountered only a few years ago. In the past, the vast majority of internet bandwidth was used to transfer character streams (in most cases HTML) over either HTTP or HTTPS. This trend has changed over the past few years, with a great deal of bandwidth (33 to 50 percent by some estimates) now being used to distribute large files over peer-to-peer connections. BitTorrent is one of the more popular protocols being used for peer-to-peer file transfers, and enabling your Java applications to use this protocol has never been easier.

Peer-to-peer networks rely primarily on the bandwidth and hardware of the participants in the network rather than on a relatively small set of centralized servers. It is, therefore, much cheaper in terms of bandwidth and energy costs for the content provider to distribute large files using a peer-to-peer network rather than through the traditional client-server approach. There are quite a few examples in the industry where peer-to-peer networking has already been taken advantage of:

The BitTorrent Protocol

The BitTorrent Protocol, which was designed and first implemented by Bram Cohen in 2001, is arguably the most popular and efficient peer-to-peer protocol currently in use.

To start sharing a file (or set of files) using BitTorrent the first peer, or initial seeder, creates a torrent file that contains all the metadata information required by clients to start downloading the shared file. This typically includes the name of the shared file (or files), the number of pieces the file has been broken down into, the checksum of each of the pieces, and the location of the tracker server which serves as a central point that coordinates all the connected peers. Unlike the rest of the traffic in a BitTorrent peer group (or swarm), communication to the tracker server is usually performed over HTTP.

Given a torrent file, a BitTorrent client would typically start off by connecting to the tracker server and getting the details of all other peers on the network. It would then start requesting pieces of the shared file from the rest of the swarm and use the checksum values in the torrent file to validate the received data. This BitTorrent process is very nicely illustrated on Wikipedia.

Azureus

Due to the openness of the BitTorrent protocol, numerous compatible BitTorrent clients have been implemented in a variety of programming languages and computing platforms. Out of all the options out there Azureus, which is implemented using Java and SWT, has proven itself to be one of the more popular and feature rich clients available. In fact, Azureus is the second most downloaded application on the Alltime Top Downloads list on SourceForge. One can argue that Azureus's popularity probably makes it one of the most successfulconsumer targetted Java desktop applications in the world.

In addition to being a great BitTorrent client, Azureus also contains functionality to create torrent files, set up a tracker server and an initial seeder. In the rest of the article we will be looking at how you can leverage these features for use in your own applications and take advantage of the cost benefits that peer-to-peer file distribution offers.

Getting Started with the Azureus API: A Simple Torrent File Downloader

In this section, we are going to implement a simple command-line application based on the Azureus API (or engine) to download a data file using the BitTorrent protocol. The URL of the torrent file will be passed in at the command line.

public class SimpleStandaloneDownloader {
  ...
  private static AzureusCore core;
  ...
  public static void main(String[] args) throws Exception{
    
    //Set the default root directory for the azureus engine.
    //If not set, it defaults to the user's home directory.
    System.setProperty("azureus.config.path", "run-environment/az-config");
    ...
    String url = null;
        ...      
        url = args[0];
    ...
    core = AzureusCoreFactory.create();
    core.start();
    ...
    System.out.println("Attempting to download torrent at : " + url);
    
    File downloadedTorrentFile = downloadTorrentFile(new URL(url));
    
    System.out.println("Completed download of : " + url);
    System.out.println("File stored as : " + downloadedTorrentFile.getAbsolutePath());
    
    File downloadDirectory = new File("downloads"); //Destination directory
    if(downloadDirectory.exists() == false) downloadDirectory.mkdir();
    
    //Start the download of the torrent 
    GlobalManager globalManager = core.getGlobalManager();
    DownloadManager manager = globalManager.addDownloadManager(downloadedTorrentFile.getAbsolutePath(),
                                                               downloadDirectory.getAbsolutePath());

    DownloadManagerListener listener = new DownloadStateListener();
    manager.addListener(listener);    
    globalManager.startAllDownloads();
    
  }
}

The singleton AzureusCore instance is the central axis on which the whole Azureus API revolves. After creating it (using the AzureusCoreFactory) and starting it, you are ready to start using its functionality. It should be noted that AzureusCore spawns its own Threads internally and generally runs asynchronously to the rest of the application.

After having downloaded the torrent file from the passed in URL using the downloadTorrentFile() method, the torrent is submitted to Azureus's GlobalManager instance, which is responsible for managing downloads. The DownloadManager that gets returned by the addDownloadManager() method can be used to retrieve a wealth of statistics on the download, including the data send rate and number of connected peers. In this example we have registered a DownloadManagerListener instance (implemented by the DownloadStateListener class) to track when the torrent data file has started downloading and to print out the completed percentage to the command line.

private static class DownloadStateListener implements DownloadManagerListener{
  ...
    public void stateChanged(DownloadManager manager, int state) {
      switch(state){
        ...
        case DownloadManager.STATE_DOWNLOADING :
          System.out.println("Downloading....");
          //Start a new daemon thread periodically check 
          //the progress of the upload and print it out 
          //to the command line
          Runnable checkAndPrintProgress = new Runnable(){
           
            public void run(){
              try{
                boolean downloadCompleted = false;
                while(!downloadCompleted){
                  AzureusCore core = AzureusCoreFactory.getSingleton();
                  List<DownloadManager> managers = core.getGlobalManager().getDownloadManagers();

                  //There is only one in the queue.
                  DownloadManager man = managers.get(0);
                  System.out.println("Download is " + 
                             (man.getStats().getCompleted() / 10.0) + 
                             " % complete");
                  downloadCompleted = man.isDownloadComplete(true);
                  //Check every 10 seconds on the progress
                  Thread.sleep(10000);
                }
              }catch(Exception e){
                throw new RuntimeException(e);
              }

            }
          };
          
          Thread progressChecker = new Thread(checkAndPrintProgress);
          progressChecker.setDaemon(true);
          progressChecker.start();
        break;
        ...
      }
    }
    
    public void downloadComplete(DownloadManager manager) {
      System.out.println("Download Completed - Exiting.....");
      AzureusCore core = AzureusCoreFactory.getSingleton();
      try{
        core.requestStop();
      }catch(AzureusCoreException aze){
        System.out.println("Could not end Azureus session gracefully - " +
                           "forcing exit.....");
        core.stop();
      }
    }
    ..
  }
}

The DownloadStateListener instance will be notified by the Azureus engine for events relating to the torrent download. In the implementation above we start off a daemon thread that prints out the completed percentage of the file to the command line every 10 seconds once downloading starts. We then issue the shutdown of the Azureus engine and exit the application once the download is complete (method downloadComplete()). As mentioned earlier, the Azureus engine generally runs asynchronously to the rest of the application, so the only way to gather the completed percentage of a download is to continually check (in this case with a daemon thread) the statistics provided by the DownloadManager.

When we build and execute the above application we should get something like the following:

C:\SimpleDownloader>java -cp Azureus2.jar;torrent-client.jar 
client.SimpleStandaloneDownloader 
http://localhost:8080/torrent-server/download.torrent

Attempting to download torrent at : http://localhost:8080/torrent-server/download.torrent
Completed download of : http://localhost:8080/torrent-server/download.torrent
File stored as : C:\DOCUME~1\jacks\LOCALS~1\Temp\torrentDownload55518torrent
Checking....
Downloading....
Download is 61.9 % complete
Download is 61.9 % complete
Download is 65.9 % complete
Download is 75.8 % complete
Download is 78.6 % complete
Download is 78.7 % complete
Download is 82.2 % complete
Download is 93.8 % complete
Download is 99.8 % complete
Finishing Download....
Download is 100.0 % complete
Download Complete - Seeding for other users....
Download Completed - Exiting.....
Download Stopped.

A More Advanced Example: Regular Distribution of Data Files to a Large Audience

While the SimpleStandaloneDownloader class is useful for illustrating the basic usage of the Azureus API, it does not add anything to what you can do with any out-of-the-box BitTorrent client. In this section we shall look at implementing a solution for a more real world scenario--the distribution of large data files to numerous client at regular intervals.

The regular distribution of large files has become a core part of many software platforms. Although the actual applications themselves are quite diverse and range from World of Warcraft to Windows XP and the Java Runtime Environment, they all have one thing in common--Automatic Updates.

Following the traditional client-server approach and allowing clients to download these files from a relatively small number of servers can be quite costly in terms of bandwidth and energy requirements. This is especially true if there are a large number of clients that require updates. The traditional client-server approach is also very prone to falling victim to flash crowds (or slashdotting as it is also known) once a new update is made available.

Peer-to-peer networks, by contrast, share the load more evenly amongst the participants and do not suffer from the same problems outlined above. In fact, the more people in the swarm, the better the file will be distributed. In this section we will look at how it is possible to build a solution that distributes regular updates over BitTorrent (as is being done by Blizzard for World of Warcraft) using the Azureus engine.

Server/Tracker Implementation

The server implementation is a combination of the tracker server and the initial seeder that will make the file available to the swarm. It provides administrators with a web interface through that they can upload a new file to be distributed to swarm as well as some useful statistics for monitoring the tracker and initial seeder.

False Triggers
Figure 2. The Tracker/Initial Seeder Management console (Click to enlarge.)

The TorrentSeederTrackerServer Class

The TorrentSeederTrackerServer singleton class forms the core part of the server implementation. Once initialized by the ServerInitialiser ServletContextListener it is placed as an attribute in the web container's ServletContext where it can be accessed by all the various components in the application.

The single most important function of the TorrentSeederTrackerServer instance is to create a torrent file for a new upload, start the tracker server and create an initial seeder that will distribute the file to the swarm. All of the above is accomplished in the seedFile(File upload) method.

public class TorrentSeederTrackerServer {
  ...
  public static final String DATA_FILE_NAME = "distributed-data.dat";
  ...
  public static final int TRACKER_PORT_NUMBER = 6671; 
  ...
  private AzureusCore core;
  /**
   * The root directory which this server will use to store all the files
   * and settings required by the Azureus engine.
   */
  private File serverRoot;
  /**
   * The file to distribute over BitTorrent
   */
  private File sourceFile;
  /**
   * The current torrent being hosted.
   */
  private TRHostTorrent currentTorrent;
  ...
  public void seedFile(File upload) throws TRHostTorrentRemovalVetoException, TOTorrentException, TRHostException{
    
    //Move the source file to a location within our torrent directory.
    sourceFile = new File(serverRoot,DATA_FILE_NAME); 
    ...
    copy(upload,sourceFile);
    ...
    TRHost trHost = core.getTrackerHost();
    ...
    TOTorrentCreator torrentCreator = TOTorrentFactory.createFromFileOrDirWithComputedPieceLength(sourceFile, getTrackerURL());
    TOTorrent torrent = torrentCreator.create();
    currentTorrent = trHost.hostTorrent(torrent, true, false);
    
    //However, we also need to implement a seeder that 
    //will be responsible for uploading the initial 
    //file to the network.
    File bEncodedFile = new File(serverRoot + File.separator + "data.torrent");
    TorrentUtils.writeToFile(torrent,bEncodedFile);
   
    GlobalManager glManager = core.getGlobalManager();
    DownloadManager seederManager = glManager.addDownloadManager(bEncodedFile.getAbsolutePath(), serverRoot.getAbsolutePath());
    seederManager.setForceStart(true);
    seederManager.addListener(new DownloadManagerListener(){
      ...
      public void stateChanged(DownloadManager manager, int state) {
        
        switch(state){
                  ...
          case DownloadManager.STATE_CHECKING:
            lastStatus = "Torrent Checking...";
          break;
          case DownloadManager.STATE_DOWNLOADING:
            lastStatus = "Torrent Downloading...";
          break;
          ..
        }
      }
    });
    glManager.startAllDownloads();
    running = true;
  }  
  ...

Once the tracker server and the initial seeder are up and running, clients can call any of the methods below to gather some statistics on the application. In the example application this is done by the various JSP pages.

public long getTotalDownloaded(){}
public int getLeecherCount(){}
public int getPeerCount(){...}
public long getDataSendRate(){...}
public long getSeedFileSize(){...}
public int getNumberOfConnectedPeers(){...}
public int getNumberOfConnectedSeeds(){...}
public int getNumberOfPiecesForFile(){...}
public Date getCreationTime(){..}

The other classes in the server implementation add in the necessary glue code that provides users with the means to manage the TorrentSeederTrackerServer instance. The NewTorrentFileUploadServlet servlet for instance, handles file uploads with the help of the Jakarta Commons File Upload library while the TorrentDownloadServlet servlet is used make the produced torrent file (bEncodedFile) downloadable to clients. The full source code for this article is provided in the Resources section.

The Client Implementation

The client implementation is a Swing application that runs on the machines of all the peers that wish to receive the regular updates.

Figure 2
Figure 2. The client implementation Swing application downloading a torrent file

The client implementation does not only serve to download a file to this user, but also helps to seed the file throughout the rest of the swarm. In fact, it is recommended that the client implementation is run for as long as possible. This will ensure that there is always a great number of peers in the network, which, in turn, ensures that the swarm remains healthy.

Figure 3
Figure 3. The client implementation application seeding a file

Another important feature of the client implementation is the ability to pick up when the torrent file on the tracker server is updated by a new administrator upload. Once this is detected, the client implementation will reset itself and start downloading the latest data file.

The ClientTorrentManager class

The ClientTorrentManager singleton class in the client implementation is what the TorrentSeederTrackerServer class is to the server implementation--the core component where all the logic is situated. The ClientFrame class, which is the caller in this case, starts off by setting the name of the tracker server (setServer(String server) method) and the download destination directory (setDestination(File destination) method). It then invokes the startTorrentProcess() method, which downloads the torrent file from the server and starts the torrent data file download process.

public class ClientTorrentManager implements DownloadManagerTrackerListener{
  ...
  /**
   * The relative path on the server where the torrent of the latest data file is kept.
   */
  public static final String TORRENT_PATH = "/torrent-server/download.torrent";
  
  /**
   * Singleton instance of this class
   */
  private static ClientTorrentManager trClient;
  ...
  private AzureusCore core;
  private DownloadManager fileDownloadManager;
  ...
  private File torrentFile;
  private File destination;
  
  private String server;
  private URL torrentFileURL;
  ...  
  private Stack<String> messages; //Might be used by calling clients.
  ...
  public boolean startTorrentProcess() throws IllegalStateException{
    ...
    if(torrentFile == null){
      //We need to download the torrentFile 
      boolean success = downloadTorrentFile();
      ...
    }
    
    //Now that we have the torrent file, 
    //we can fire up the manager and start downloading...
    GlobalManager globalManager = core.getGlobalManager();
    fileDownloadManager = globalManager.addDownloadManager(torrentFile.getAbsolutePath(), 
                                                           destination.getAbsolutePath());
    fileDownloadManager.setForceStart(true);
    File saveLocation = fileDownloadManager.getSaveLocation();
    if(saveLocation.exists() == true){
      //This file already exists on disk
      //attempt to delete it if the size does not match the torrent's
      long torrentFileSize = fileDownloadManager.getSize();
      long actualFileSize = saveLocation.length();
      if(torrentFileSize != actualFileSize){
        addMessage("Length of torrent file (" + torrentFileSize + ") " +
                   "and file already on disk (" + actualFileSize + ") " + 
                   "does not match. " +  
                   "It will be overwritten.");
                   
        saveLocation.delete();
      }
    }
    
    fileDownloadManager.addTrackerListener(this);
    fileDownloadManager.addListener(new DownloadManagerListener(){
      ...
      public void stateChanged(DownloadManager manager, int state) {
        
        switch(state){
          ..
          case DownloadManager.STATE_ERROR:
            addMessage("Torrent Error Encountered... - " + manager.getErrorDetails());
          break;
          ...
        } 
      } 
    });

    globalManager.startAllDownloads();
    addMessage("Started downloading torrent file....");

    running = true;
    return true;
  }
  ...
  public void setDestination(File destination) {
    this.destination = destination;
  }
  ...
  public void setServer(String server) {
    this.server = server;
  }
  ... 
}

With the torrent download process up and running, the ClientFrame gathers statistics for display to the user every few seconds using the following methods:

  public boolean isDownloading(){...}
  public boolean isSeeding(){...}
  public double getPercentDone(){...}
  public double getTotalBytesUploaded(){...}
  public double getTotalBytesDownloaded(){...}
  public double getDataSendRate(){...}
  public double getDataReceiveRate(){...}
  public Collection<String> getMessages(){..}
  public boolean isRunning() {...}

Detecting When a New Data File Is Uploaded to the Server

As stated earlier, the client implementation also needs to be able to detect when a new data file has been uploaded to the server. Fortunately, this is made possible by the Azureus API through the registration of a DownloadManagerTrackerListener instance (in this case the ClientTorrentManager itself) as a listener on the DownloadManager instance.

        ...
        fileDownloadManager.addTrackerListener(this);
        ...

The scrapeResult() method is then subsequently invoked for all tracker related events. If it then found that the original torrent file is no longer available, then all registered listeners are notified (in the application the ClientFrame) and the appropriate action is taken.

public class ClientTorrentManager implements DownloadManagerTrackerListener{
...
private List<WeakReference<TorrentFileListener>> torrentChangeListeners;
...
public void scrapeResult(TRTrackerScraperResponse response) {
    addMessage("Tracker Scrape Result : " + response.getStatusString());
    if(response.getStatus() == TRTrackerScraperResponse.ST_ERROR){
      if(response.getStatusString().indexOf("File Not Found") != -1){
        //We need to notify all the listeners
        for(WeakReference<TorrentFileListener> ref : torrentChangeListeners){
          ref.get().trackerTorrentFileChanged(new TorrentFileEvent(this));
        }
      }
    }
  }
...
}

Conclusion

The size and amount of files being distributed over the internet is increasing daily. If we as content providers are to carry on distributing these files using the traditional client-server paradigm, so too will our costs. Hopefully this article will serve as an introduction to peer-to-peer file distribution, the cost benefits it offers, and how you can leverage the freely available Azureus engine to implement BitTorrent file sharing in your own applications.

Resources

Jacobus Steenkamp has been developing in Java for the last 5 years and has worked in various sectors including the financial, pharmaceutical and energy industries.


Return to ONJava.com.

Copyright © 2009 O'Reilly Media, Inc.