Building Highly Scalable Servers with Java NIOby Nuno Santos
About one year ago, a client of the company where I work asked us to develop a router for telephony protocols (i.e., protocols used for communication between a SMS center and external applications). Among the requirements was that a single router should be able to support at least 3,000 simultaneous connections.
It was clear that we could not use the traditional thread-pooling approach. Most thread libraries do not scale well, because the time required for context switching increases significantly with the number of active threads. With a few hundred active threads, most CPU time is wasted in context switching, with very little time remaining for doing real work. As an alternative to thread pooling, we decided to use I/O multiplexing. In this approach, a single thread is used to handle an arbitrary number of sockets. This allows servers to keep their thread count low, even when operating on thousands of sockets, thereby improving scalability and performance significantly. Unfortunately, there is a price to pay: an architecture based on I/O multiplexing is significantly harder to understand and to implement correctly than one based on thread pooling.
The support for I/O multiplexing is a new feature of Java 1.4. It builds on two features of the Java NIO (New I/O) API: selectors and non-blocking I/O. The article "Introducing Nonblocking Sockets" provides a good introduction to these two features.
In this article, we describe the lessons we learned while designing and implementing our router, focusing on architectural issues such as I/O event dispatching, threading, management of client data, and protocol state. This is not an introductory article; the intended audience is developers that already have a basic knowledge of I/O multiplexing and Java NIO, but haven't yet used those technologies to develop a full-scale server.
The article includes the source code for a echo server and client based on the architecture described. Both the server and the client are functional and can be complied and executed without any modification. The source code can also be used as a starting point to develop a full server.
I/O Event Handling
The I/O architecture of our router was strongly inspired by the Swing
event-dispatch model. In Swing, events generated by the user interface are
received by the JVM and stored in an event queue. Inside of the JVM, an event
dispatch thread (implemented in the class
monitors this queue and dispatches incoming events to interested listeners. This
is a typical example of the Observer pattern.
In our router, there is also an event dispatch thread, implemented in the
SelectorThread. As the name suggests, this class encapsulates
a selector and a thread. The thread monitors the selector, waiting for incoming
I/O events and dispatching them to the appropriate handlers.
SelectorThread class generates four types of events,
corresponding to the operations defined on
java.nio.channels.SelectionKey: connect, accept, read, and write.
Handlers register with the
class to receive events. Depending on the type of events they are
interested in, they must implement one of the following interfaces:
ConnectSelectorHandler: For establishing outgoing connections.
AcceptSelectorHandler: For receiving incoming connections.
ReadWriteSelectorHandler: For reading or writing data to a connection.
Figure 1 describes the class hierarchy of these interfaces. We chose not to define a single interface for all of the possible events because a single handler will likely be only interested in some of the events. For instance, a handler that accepts connections will most likely not need to establish connections. This separation allows handlers to implement only the operations they really need.
Figure 1. Class hierarchy for I/O event handlers
The Life of a Handler
One important difference from the thread-per-client model is that all read and write operations are non-blocking, forcing the programmer to deal with partial reads and writes. When the handler receives a read event, it means only that there are some bytes available in the socket's read buffer. This data may contain either part of a packet, a full packet, or more than one packet. All cases have to be considered while reading. A similar situation occurs when writing. It is only possible to write as much as the space available on the socket's write buffer. A call to write will return as soon as the buffer space is exhausted, regardless of whether the data has been fully written or not. This has a direct impact on the lifecycle of a handler, which needs to deal with all of these situations.
A handler is basically a state machine reacting to I/O events. Its typical lifecycle is the following:
Waiting for data
The handler is interested in reading but not in writing, since there is nothing to send to the client. Therefore, it activates read interest and waits for the read event.
After receiving the read event, the handler retrieves the available data from the socket and starts reassembling the request. During this state, it is not interested in receiving any type of event. If a packet is fully reassembled, it starts processing it (state 3). Otherwise, it saves the partial packet, reactivates read interest, and continues waiting for data (state 1).
The handler enters this state whenever a request is fully reassembled. While here, the handler is not interested in either reading or writing (assuming that it only processes a request at a time). All interest in I/O events is disabled.
When the reply is ready, the handler tries to send it immediately using a non-blocking write. If there is not enough space on the socket's write buffer to hold the entire packet, it will be necessary to send the rest later (step 5). Otherwise, the packet is sent and the handler can reactivate read interest, waiting for the next packet.
Waiting to write
When a non-blocking write returns without having written all of the data, the handler activates interest in the write event. Later, when there is space available in the write buffer, the write event will be raised and the handler will continue writing the packet.
Figure 2 shows the state transition diagram of a handler.
Figure 2. State transition diagram of an handler
Dispatching I/O Events
class is responsible for supporting the lifecycle of the handlers. For
that, it manages the following information for each handler:
SelectableChannel: The channel to be monitored for events.
- The handler itself: The object to be notified of events.
- An interest set: The set of operations to be monitored.
Handlers must provide these elements when they register
will then register the channel with the internal selector, using
the interest set to activate monitoring of the corresponding I/O operations. The
handler is stored as an attachment, which is a convenient way of associating
application data with a registered channel. Internally, the following method
call is performed to register a handler:
channel.register(selector, interestSet, handler);