ONJava.com -- The Independent Source for Enterprise Java
oreilly.comSafari Books Online.Conferences.

advertisement

AddThis Social Bookmark Button

Monitoring Session Replication in J2EE Clusters
Pages: 1, 2

Running the Replication Latency Tester

You can download the complete tester program.



To use the Replication Latency Tester, simply execute java -jar ReplicationLatencyTester.jar in one of your nodes. The server thread and the client thread of the program will start to work with default values for the IP multicast address and IP multicast port. The server side of the program will start sending serialized objects, and the client side will start obtaining metrics based on the latency of the multicast network and the size of the object being sent.

An even more realistic scenario will be simulated if you start the tester on several nodes of your network. This will reproduce an architecture in which you have various application server instances running in different nodes and sending information to each other, using IP multicast. One Replication Latency Tester alone is not very significant; nor does it reflect a real-case scenario. (Replication shouldn't be activated when running a single instance of an application server; that would imply overhead without providing any higher availability.)

For each object received, these are the calculated values:

  • Delay in receiving the object (msecs): This is the elapsed time between the creation of the object in the server and its deserialization and invocation in the client thread.
  • Size of object received (bytes): This is the size, in bytes, of the object received.
  • Ratio delay/size (msecs/Kb): This is the ratio between the delay experienced in reconstructing the object and the object's size.

For each battery of tests:

  • Average ratio (msecs/Kb): This is the average ratio for the group of objects.
  • Average delay (msecs): This is the average delay for the group of objects.

You can persist these average metrics to a file simply by specifying the corresponding configuration parameter. The information will be stored in a file called lateststatistics.trc. Persisting this information will affect the average figures in a multi-node test, because this operation is done in the same thread as the serialization and deserialization and hence can introduce some time lag.

Both the server thread and the client thread can be configured with different parameters:

  • The server-side configuration can be overridden by modification of the server.properties file.
  • The client-side configuration can be overridden by modification of the client.properties file.

These files need to be placed in the directory where the tester is run. If these files are not found, the default values are used. These default values are specified in the class Constants.class.

These are the configurable parameters for the server thread:

Property Meaning Default Values
ipaddress Multicast address to be used to send information 230.230.230.1
port Port to be used to send information 2608
message Text message to be included in the object that will be serialized Hi there!
delay

Delay applied between each send operation and the next one (msecs)

1000
iterations Number of operations to be performed 100

And these are the configurable parameters for the client thread:

Property Meaning Default Values
ipaddress Multicast address to be used to send information 230.230.230.1
port Port to be used to send information 2608
verbose Print on standard error the detailed information for each object received false
dumptofile Write to the lateststatistics.trc file the statistics for each series of objects false

Without verbose output activation, the result should look something like the display in Figure 3:

Figure 3
Figure 3. Non-verbose output

With verbose output active, the standard output shows more detailed information as objects are being received, as seen in Figure 4:

Figure 4
Figure 4. Verbose output

Analyzing Replication Latency Tester Results

Several things need to be considered when you are analyzing the results obtained with the tester:

  1. Durations are measured using System.currentTimeMillis() and this approach may include much more in the metrics than just the code's execution time. Time taken by other processes on the system or time spent waiting for I/O can result in inaccurately high timing numbers. However, since this is only a two-thread program, this shouldn't affect our statistics much.

  2. The measurements of the latency between the client and the server assume that the clocks are set to the same values in every node in the test.

  3. In order to make the metrics as accurate as possible, the default behavior of the program is to show as little information as possible on the standard output (console). It is well known how System.out.println("") can affect the performance of a Java application.

  4. The size of the object is calculated by first serializing the object and then obtaining the size of the byte array created. Different approaches for obtaining the size of an object use the memory state by invoking Runtime.getRuntime().freeMemory(). I am not in favor of this approach, because different threads and the behavior of the garbage collector may affect this type of measurement.

The sizes of the objects being serialized in the tester are a good approximation of real session objects (sizes between 3Kb and 5Kb). Any objects that are much larger than that will imply considerable overhead.

As a rule of thumb, any average ratios that exceed 30 to 40 msecs/kb indicate that the nodes and the multicast network might be overloaded. You can extrapolate from these metrics and apply them to your application: If your session objects are 10Kb in size, you can expect that your application server will take more than 300 msecs (30 msecs/Kb*10Kb) to make the object available in other nodes (application servers do many more things simultaneously than what my application does, such as identifying the application, servlet and session for the object, of course). This implies that whenever your servlet/JSP engine is using multicast for replicating objects across application server instances, the objects will not appear on other nodes before 300 msecs have passed. Consequently, if a failover happens and the load balancing takes place in less than 300 msecs (which you can expect from most application servers), your client request will find an older version of the object in the new node and the application will become inconsistent.

As mentioned at the beginning of the article, most application servers do not replicate the entire session in every request and they send only deltas (they replicate changes to the session instead of the entire session). That saves time and resources for the subsequent modifications to become available in other application server instances. However, you always have to consider the worst-case scenario: a crash after the first serialization.

Problem-Solving Hints

You can do various things to solve a performance problem detected with the Replication Latency Tester tool. Each applies to a specific deployment, depending on the application involved and the resources available. Covering each one of them and when it is applicable is beyond the scope of this article. However as brief introduction to future analysis, here are the main possible solutions:

  • Improve the infrastructure for your application server (mainly network behavior and RAM).
  • Reduce the number of replicating nodes in your cluster (some application servers allow you to isolate the nodes that work together in small groups).
  • Reduce the HTTP session object to the minimum amount of relevant information.
  • Save your session information to a database or file regularly, freeing the HTTP session object of your application.

Obviously, all of these will affect your J2EE system (requiring either changes in code/configuration or additional resources) and need to be analyzed in detail before you decide which one applies to your case.

Conclusion

There are many reasons why the environment of a cluster may change through the lifecycle of an application. Thus, it is necessary to regularly monitor the cluster's proper functioning and prevent performance and stability issues. This article showed how to use a simple Java tool to obtain various metrics related to correct cluster behavior. It also discussed why it is important that serialization and IP multicast take place rapidly and covered how to monitor both. The next step: apply the necessary corrections once the problem is detected. But that's a subject that's sufficiently complex to merit its own article.

References

Fermin Castro is a product manager at Oracle Corporation.


Return to ONJava.com.