ONJava.com -- The Independent Source for Enterprise Java
oreilly.comSafari Books Online.Conferences.

advertisement

AddThis Social Bookmark Button

Building Highly Scalable Servers with Java NIO
Pages: 1, 2, 3

The Complex Solution

Our initial approach was based on the M-N architecture. This proved to be a bad option. The main problem was keeping the whole system thread-safe. There were many interaction points between dispatcher and worker threads, all of them requiring careful synchronization. An even worse problem is that the group formed by a selector and its associated selection keys is not safe for multithreaded access. If a selection key is changed in any way by a thread while another thread is calling its selector select() method, the typical result is the select() call aborting with an ugly exception. This happened often when worker threads closed channels and indirectly cancelled the corresponding selection keys. If select() is being called at that time, it will find a selection key that was unexpectedly cancelled and abort with a CancelledKeyException. This is perhaps the most important lesson we learned about I/O multiplexing and Java NIO: A selector, its selection keys, and registered channels should never be accessed by more than one thread.



We tried to enforce this rule by delegating to the selector thread all tasks related to selector structures. But after a while, the architecture was getting very complex, inefficient, and error-prone. We decided it was time to admit defeat and start over using a simpler architecture.

The Working Solution

The simplest way of avoiding the concurrency problems between dispatcher and worker threads was not to use worker threads at all and let the event-dispatch threads to do all of the work (the M dispatchers/no workers architecture option described above). What we lost in flexibility, we gained in simplicity. There was no more need to ensure thread safety on the handler or on the selector, and there was no need to delegate tasks between threads. The life of a selector thread is as simple as:

  • Block on the select call. Return only when there are operations ready to be performed.
  • Execute all operations that are ready. This includes accepting or establishing connections, reading, writing, registering new sockets with the selector, changing the set of operations monitored, and so on.
  • Go back to the select call.

In the M-N architecture, steps 1 and 2 were happening simultaneously, creating many concurrency problems. Doing them in the same thread put an end to the concurrency bugs that plagued our first attempt.

However, this architecture has some issues that must be carefully considered.

It is necessary to ensure a good distribution of work between threads. We used a simple round-robin algorithm to assign incoming connections to event-dispatch threads. So far, this has been working fine. But in other situations, it may be necessary to take in consideration other factors such as the activity going on in each thread.

Another problem is preventing the event-dispatch threads from blocking for too long. This can happen if processing a request requires reading or writing to a disk or to a database. To minimize this problem, we increased the ratio of event-dispatch threads to CPUs (to four or five). This does not prevent a thread from blocking, but when that happens, it is more likely that another one will be ready to take over the idle processor time.

One final problem is that sometimes it is necessary for external threads to interact with objects managed by the event-dispatch threads. For instance, there may be a timer to close idle connections. But the timer's thread should not close the connection directly, for the reasons mentioned previously. The solution is to provide a way for external threads to schedule tasks for execution on the event-dispatch thread of a selector. This is done by calling either invokeLater() or invokeAndWait() of the SelectorThread class. If these names seem familiar to you, it's because they exist in the java.awt.EventQueue class to serve a similar function in Swing.

The Main I/O Cycle

Now that all of the pieces are in place, we can go on to the main event-dispatch loop:

public void run() {
  while (true) {
    // Execute all the pending tasks.
    doInvocations();
    
    // Time to terminate? 
    if (closeRequested) {
      return;
    }
    
    int selectedKeys = selector.select();
    if (selectedKeys == 0) {
      // Go back to the beginning of the loop 
      continue;
    }
     
    // Dispatch all ready I/O events
    Iterator it = selector.selectedKeys().
      iterator();
    while (it.hasNext()) {
      SelectionKey sk = (SelectionKey)it.next();
      it.remove();      
      // Obtain the interest of the key
      int readyOps = sk.readyOps();
      // Disable the interest for the operation
      // that is ready. This prevents the same 
      // event from being raised multiple times.
      sk.interestOps(
        sk.interestOps() & ~readyOps);
          
      // Retrieve the handler associated with 
      // this key
      SelectorHandler handler = 
         (SelectorHandler) sk.attachment();          
        
      // Check what are the interests that are 
      // active and dispatch the event to the 
      // appropriate method.
      if (sk.isAcceptable()) {
        // A connection is ready to be completed
        ((AcceptSelectorHandler)handler).
          handleAccept();
        
      } else if (sk.isConnectable()) {
        // A connection is ready to be accepted            
        ((ConnectorSelectorHandler)handler).
          handleConnect();            
          
      } else {
        ReadWriteSelectorHandler rwHandler = 
          (ReadWriteSelectorHandler)handler; 
        // Readable or writable              
        if (sk.isReadable()) {
          // It is possible to read
          rwHandler.handleRead();              
        }
            
        // Check if the key is still valid, 
        // since it might have been invalidated 
        // in the read handler (for instance, 
        // the socket might have been closed)
        if (sk.isValid() && sk.isWritable()) {
          // It is read to write
          rwHandler.handleWrite();                              
        }
      }
    }
  }
}

This is what the loop does:

  1. Execute scheduled tasks
    These are the tasks that were scheduled for execution on the event-dispatch thread by external threads (by calling invokeLater() or invokeAndWait()).

  2. Check if close was requested
    Checks if it is time to close the selector. For graceful shutdowns.

  3. Call select()
    Wait for I/O events.

  4. Dispatch I/O events
    Obtains the set of selection keys with operations ready, iterates over them and dispatches the operations to the appropriate handlers.

When all ready selection keys are dispatched, it's time to go back to the beginning of the loop, repeating the process once more.

Conclusion

Developing a fully functional router based on I/O multiplexing was not simple. It took us three months to get the architecture right. Part of this time was spent learning an unfamiliar I/O model. The other part was spent dealing with its inherent complexity. In the end, we were quite happy with the results. In internal benchmarks, the router was able to handle up to 10,000 clients with no significant drop in throughput. For comparison, we implemented a version based on the thread-per-client model, which was only able to reach 3,000 clients, with a significant drop in throughput as the number of clients increased.

Should you use I/O multiplexing in your next server? It depends. If the number of simultaneous clients will never exceed more than one or two hundred, then go for a simpler threaded model. But if you plan to support hundreds or even thousands of simultaneous clients, you should definitely consider using I/O multiplexing. If you do use it, we hope this article and the companion source code will help you avoid some of pitfalls of using I/O multiplexing with Java NIO.

Resources

Nuno Santos is a software engineer at WIT-Software, a software company in the mobile telecommunication networking field.


Return to ONJava.com.