ONJava.com    
 Published on ONJava.com (http://www.onjava.com/)
 See this if you're having trouble printing code examples


J2EE Clustering with JBoss

by Ivelin Ivanov
08/20/2003

In a recent article, Bill Burke and Sacha Labourey explained the key components of JBoss 3 clustering. We will now present several new clustered services recently introduced in JBoss 3.2.2, which was not yet released at the time of the writing of that article. Basic knowledge of JBoss 3 clustering and farming is a prerequisite for this article.

Several design patterns commonly occur in clustered enterprise applications. Though they are not covered by the current J2EE specification, JBoss provides several built-in services to help the development of components that follow these patterns.

Singleton Service

A clustered singleton service is deployed on multiple nodes in a cluster but runs on only one of the nodes. The node running the singleton service is typically called the master node. When the master fails, another master is selected from the remaining nodes and the service is restarted on the new master.

clustered singleton diagram
Figure 1. Clustered singleton service

Many times, an application needs to ensure that a certain task is run exactly once. Thus, only one of the nodes in a cluster should execute the task. The other nodes should knowingly remain passive. Examples of singleton tasks include:

Related Reading

Java Enterprise Best Practices
By The O'Reilly Java Authors

While it is fairly easy to implement such singleton tasks in a single VM, the solution will usually not work immediately in a clustered environment. Even in the simple case of a task activated upon startup on one of the nodes in a two-node cluster, several problems must be addressed:

The logic to solve these problems is unlikely to be included in the design of a single-VM solution. However, a solution can be found to address the case at hand and it can be patched on the startup task. This is an acceptable approach for a few startup tasks and two-node clusters.

As the application grows and becomes more successful, more startup tasks may be necessary. The application may also need to scale to more than two nodes. The clustered singleton problem can quickly become mind-boggling for larger clusters, where the different node startup scenarios are far more difficult to enumerate than in the two-node case. Another complicating factor is communication efficiency. While two nodes can directly connect to each other and negotiate, 10 nodes will have to establish 45 total connections to use the same technique.

This is where JBoss comes in handy. It eliminates most of the complexity and allows application developers to focus on building singleton services regardless of the cluster topology.

We will illustrate how the JBoss clustered singleton facility works with an example. First, we will need a service archive descriptor. Let's use the one that ships with JBoss under server/all/farm/cluster-examples-service.xml. The following is an excerpt:

<!--    | This MBean is an example of a cluster Singleton    -->
<mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
          name="jboss.examples:service=HASingletonMBeanExample">
</mbean>
<!- - -->
   <!--    | This is a singleton controller which works similarly to the
           | SchedulerProvider (when a MBean target is used)    -->
   <mbean code="org.jboss.ha.singleton.HASingletonController"
          name="jboss.examples:service=HASingletonMBeanExample-HASingletonController">
      <depends>jboss:service=DefaultPartition</depends>
      <depends>jboss.examples:service=HASingletonMBeanExample</depends>
      <attribute name="TargetName">jboss:service=HASingletonMBeanExample</attribute>
      <attribute name="TargetStartMethod">startSingleton</attribute>
      <attribute name="TargetStopMethod">stopSingleton</attribute>
   </mbean>
<!- - -->

This file declares two MBeans, HASingletonMBeanExample and HASingletonController. The first one is a singleton service that contains the custom code. It is a simple JavaBean with the following source code:

public class HASingletonMBeanExample
  implements HASingletonMBeanExampleMBean {

    private boolean isMasterNode = false;

    public void startSingleton() {
        isMasterNode = true;
    }

    public boolean isMasterNode() {
        return isMasterNode;  
    }

    public void stopSingleton() {
        isMasterNode = false;  
    }
}

All of the custom logic for this particular singleton service is contained within this class. Our example is not too useful; it simply indicates, via the isMasterNode member variable, whether the master node is running the singleton. This value will be true only on the one node in the cluster where it is deployed.

HASingletonMBeanExampleMBean exposes this variable as an MBean attribute. It also exposes startSingleton() and stopSingleton() as managed MBean operations. These methods control the lifecycle of the singleton service. JBoss invokes them automatically when a new master node is elected.

How does JBoss control the singleton lifecycle throughout the cluster? The answer to this question is in the MBean declarations. Notice that the HASingletonMBeanExample-HASingletonController MBean also takes the name of the sample singleton MBean and its start and stop methods.

On each node in the cluster where these MBeans are deployed, the controller will work with all of the other controllers with the same MBean name deployed in the same cluster partition to oversee the lifecycle of the singleton. The controllers are responsible for tracking the cluster topology. Their job is to elect the master node of the singleton upon startup, as well as to elect a new master should the current one fail or shut down. In the latter case, when the master node shuts down gracefully, the controllers will wait for the singleton to stop before starting another instance on the new master node.

A singleton service is scoped in a certain cluster partition via its controller. Notice that, in the declaration above, the controller MBean depends on the MBean service DefaultPartition. If the partition where the singleton should run is different than the default, its name can be provided to the controller via the MBean attribute PartitionName.

Clustered singletons are usually deployed via the JBoss farming service. To test this example, just drop the service file above in the server/all/farm directory. You should be able to see the following in the JBoss JMX web console:

JMX Console, HASingletonController
Figure 2. Controller MBean view. The MasterNode attribute will have value True on only one of the nodes.

JMX Console, HASingletonMBeanExample
Figure 3. Sample singleton MBean view. The MasterNode attribute will have the same value as the MasterNode attribute on the controller MBean.

Scheduler Service

The JBoss 3 Scheduler service is covered in detail by the JBoss 3: Administration and Development book. It has three interdependent components:

Often, applications need to schedule tasks that have to be executed once in the scope of the application, whether it is running standalone or in a cluster of multiple nodes. Examples of such tasks include regular database cleanup, email notifications, and scheduled reports.

All JBoss ScheduleProvider services accept an MBean attribute that enables them to schedule tasks on only one node in the cluster (of one or more nodes). The attribute name is HASingleton and its value is a Boolean type. When set to True, all ScheduleProvider MBeans registered with the same name and deployed in the same cluster partition will coordinate and make sure that the schedule is only provided to the schedule manager on one of the nodes. When set to False, each of the schedule providers will act independently and as a result, they will all schedule tasks with their local scheduler managers. The default value of the attribute is True, which allows transparent transition of standalone schedule providers to a clustered environment.

The name of the partition where a singleton schedule provider service will be deployed can be set via the PartitionName attribute. By default, the value assumes the default JBoss partition name.

Here is an excerpt from the example service archive descriptor cluster-examples-service.xml, located in the server/all/farm directory in the JBoss installation.

<!--
 | This MBean is an example of an HA Schedule Target
 | which is identical to a regular Schedule Target
 | (the example class is the same, just the MBean has different names)
 - ->
<mbean code="org.jboss.varia.scheduler.example.SchedulableMBeanExample" 
       name="jboss.examples:service=HASchedulableMBeanExample">
</mbean>
<!- - -->

<!--
 | The Schedule Manager has to be started whenever
 | schedules are needed.
 |
 | Uncomment only if not started by 
 | another service (e.g. schedule-manager-service.xml)
 - ->
<mbean code="org.jboss.varia.scheduler.ScheduleManager"
       name="jboss:service=ScheduleManager">
   <attribute name="StartAtStartup">true</attribute>
</mbean>
<!- - -->

<!--
 | This is a single schedule Provider which works like the
 | one in schedule-manager-service.xml
 |
 | The key difference is the explicit use of the HASingleton MBean attribute
 | to make the provider a clustered singleton.
 | When HASingleton is set to true the MBean will usually declare dependency 
 | on a cluster partition. In this case it is the DefaultPartition.
 | When not explicitly set the attribute defaults to true. 
 |
 | The same attribute can also be used for the other schedule providers as well:
 | DBScheduleProvider and XMLScheduleProvider
 | 
 |
 - ->
<mbean code="org.jboss.varia.scheduler.SingleScheduleProvider"
       name="jboss:service=HASingleScheduleProvider">
   <depends>jboss:service=DefaultPartition</depends>   
   <depends>jboss:service=ScheduleManager</depends>
   <depends>jboss.examples:service=HASchedulableMBeanExample</depends>
   <attribute name="HASingleton">true</attribute>
   <attribute name="ScheduleManagerName">jboss:service=ScheduleManager</attribute>
   <attribute name="TargetName">jboss:service=HASchedulableMBeanExample</attribute>
   <attribute name="TargetMethod">
        ;hit( NOTIFICATION, DATE, REPETITIONS, SCHEDULER_NAME, java.lang.String )
   </attribute>
   <attribute name="DateFormat"></attribute>
   <attribute name="StartDate">NOW</attribute>
   <attribute name="Period">10000</attribute>
   <attribute name="Repetitions">10</attribute>
</mbean>
<!- - -->

The deployment descriptor above is almost identical to the one provided by schedule-manager-service.xml but is intended to be deployed via farming. Although not necessary, since the default value is True, the descriptor specifies the HASingleton attribute of the schedule provider for illustration of its usage.

Notification Service

When it comes to reliable, mission-critical, and sophisticated J2EE messaging, one API stands out: JMS. The JBossMQ service is a robust JMS implementation that meets the high standards commanded by the API for distributed, transactional, and secure messaging. Then why another notification service?

Most typical clustered applications are deployed in a symmetric, homogeneous environment where components need to notify each other about various lifecycle or domain-change events, much like AWT and Swing widgets exchange events. This is where the JBoss cluster notifications come in. They fill the need for reliable, quick, and lightweight events. Cluster notifications do not require new skills from developers familiar with JMX, because they comply with the JMX notifications API.

clustered notification diagram
Figure 4. Clustered notification service

If you have used JMX notifications before, the clustering extension should be a breeze. There are two common ways to employ clustered notifications: by extending HAServiceMBeanSupport or by delegating the work to instances of this class.

Let's look at an example that comes with the JBoss distribution. Here is an excerpt from the familiar descriptor cluster-examples-service.xml, located in server/all/farm.

<!--
 | This MBean is an example showing how to extend a cluster notification broadcaster 
 | Use the sendNotiication() operation to trigger new clustered notifications.
 | Observe the status of each instance of this mbean in the participating cluster partition nodes.
 - ->
<mbean code="org.jboss.ha.jmx.examples.HANotificationBroadcasterExample" 
	name="jboss.examples:service=HANotificationBroadcasterExample">
	<depends>jboss:service=DefaultPartition</depends>           
</mbean>
<!- - -->

<!--
 | This MBean is an example that shows how to delegate notification services 
 |    to a HANotificationBroadcaster. 
 | Use the sendNotiication() operation to trigger new clustered notifications.
 | Observe the status of each instance of this mbean in the participating cluster partition nodes.
 - ->
<mbean code="org.jboss.ha.jmx.examples.HANotificationBroadcasterClientExample" 
	name="jboss.examples:service=HANotificationBroadcasterClientExample">
	<depends>jboss.examples:service=HANotificationBroadcasterExample</depends>
	<attribute name="HANotificationBroadcasterName">
              jboss.examples:service=HANotificationBroadcasterExample
    </attribute>
</mbean>
<!- - -->

The MBean implemented by the HANotificationBroadcasterExample class provides common clustering services like shared distributed state, replication management, and notifications. It is a convenience class that can be extended and customized or used as is. Its most important attribute is PartitionName, which fixates the partition of nodes where all MBeans with the same name will work together.

The other MBean in this example is implemented by the HANotificationBroadcasterClientExample class. It uses the former MBean's services to broadcast notifications to the local listeners, as well as remote cluster listeners that have subscribed to the HANotificationBroadcasterExample MBean. At the same time, the client MBean will also receive all notifications sent through any instance of the broadcaster. Clear as mud?

Maybe a picture will help. Here is a screenshot of the client MBean attributes:

HANotification Client Attributes
Figure 5. HANotification Client Attributes

The first attribute points to the name of the cluster notification broadcaster to which we will subscribe. The next attribute of interest is ReceivedNotifications. When a message is sent to any deployed instance of the broadcaster MBean on any of the partition nodes, it will be received and listed in the Value column for this attribute. In the screenshot above, there are two received notifications, iivanov2 and mau. Even though it is not obvious from the picture, one of these messages was received from a remote host.

Now, a screenshot of the MBean operation of interest:

HANotification client operation
Figure 6. HANotification client operation

The name is not very friendly, but it is distinct and will do for this example. When this operation is invoked with a text message as its argument, the value will appear in the ReceivedNotification attribute on each of the client MBeans deployed in the same partition.

Let's look at the code of the client MBean to see that it is actually straightforward. I made an attempt to annotate the code and keep it simple, so that there is no need for additional analysis.

public class HANotificationBroadcasterClientExample
  extends ServiceMBeanSupport
  implements HANotificationBroadcasterClientExampleMBean, NotificationListener
{

  /**
   * 
   * On service start, subscribes to notification sent by this broadcaster or
   * its remote peers.
   * 
   */
  protected void startService() throws Exception
  {
    super.startService();
    addHANotificationListener(this);
  }

  /**
   * 
   * On service stop, unsubscribes to notification sent by this broadcaster or
   * its remote peers.
   * 
   */
  protected void stopService() throws Exception
  {
    removeHANotificationListener(this);
    super.stopService();
  }

  /**
   * Broadcasts a notification to the cluster partition.
   * 
   * This example does not ensure that a notification sequence number 
   * is unique throughout the partition.
   * 
   */
  public void sendTextMessageViaHANBExample(String message) 
    throws InstanceNotFoundException, MBeanException, ReflectionException
  {
    long now = System.currentTimeMillis();
    Notification notification =
      new Notification(
        "hanotification.example.counter",
        super.getServiceName(),
        now,
        now,
        message);
    server.invoke(
        broadcasterName_,
        "sendNotification",
        new Object[] { notification },
        new String[] { Notification.class.getName() }
        );
  }

  /**
   * Lists the notifications received on the cluster partition
   */
  public Collection getReceivedNotifications()
  {
    return messages_;
  }

  /**
   * @return the name of the broadcaster MBean
   */
  public String getHANotificationBroadcasterName()
  {
    return broadcasterName_ == null ? null : broadcasterName_.toString();
  }

  /**
   * 
   * Sets the name of the broadcaster MBean.
   * 
   * @param 
   */
  public void setHANotificationBroadcasterName(String newBroadcasterName)
    throws InvalidParameterException
  {
    if (newBroadcasterName == null)
    {
      throw new InvalidParameterException("Broadcaster MBean must be specified");
    }
    try
    {
      broadcasterName_ = new ObjectName(newBroadcasterName);
    }
    catch (MalformedObjectNameException mone)
    {
      log.error("Broadcaster MBean Object Name is malformed", mone);
      throw new InvalidParameterException("Broadcaster MBean is not correctly formatted");
    }
  }

  protected void addHANotificationListener(NotificationListener listener) 
                                         throws InstanceNotFoundException
  {
    server.addNotificationListener(broadcasterName_, listener, 
      /* no need for filter */ null, 
      /* no handback object */ null);
  }

  protected void removeHANotificationListener(NotificationListener listener) 
                 throws InstanceNotFoundException, ListenerNotFoundException
  {
    server.removeNotificationListener(broadcasterName_, listener);
  }

  public void handleNotification(
    Notification notification,
    java.lang.Object handback)
  {
    messages_.add(notification.getMessage());
  }

  // Attributes ----------------------------------------------

  Collection messages_ = new LinkedList();

  /**
   * The broadcaster MBean that this class listens to and delegates HA notifications to
   */
  ObjectName broadcasterName_ = null;

}

The key points of the code are highlighted. Notice how subscription to the cluster broadcaster is accomplished via the MBean server API.

It is also important to note that the subscription is local. Both the client and the MBean server reside in the same VM. The sendNotification() invocation is also local to the VM. The cluster broadcaster hides the implementation details of working together with its remote peers to deliver notifications throughout all nodes.

If you aren't too concerned with dependencies on JBoss-specific classes, you can directly extend the cluster broadcaster class, in which case all method calls will be direct instead of proxied through the MBean Server. Here is the code for an extended broadcaster:

public class HANotificationBroadcasterExample
  extends HAServiceMBeanSupport
  implements HANotificationBroadcasterExampleMBean
{
  
  /**
   * 
   * On service start, subscribes to notification sent by this broadcaster or
   * its remote peers.
   * 
   */
  protected void startService() throws Exception
  {
    super.startService();
    addNotificationListener(listener_, /* no need for filter */ null, /* no handback object */ null);
  }
  
  /**
   * 
   * On service stop, unsubscribes to notification sent by this broadcaster or
   * its remote peers.
   * 
   */  
  protected void stopService() throws Exception
  {
    removeNotificationListener(listener_);
    super.stopService();
  }
  
  /**
   * Broadcasts a notification to the cluster partition.
   * 
   * This example does not ensure that a notification sequence number 
   * is unique throughout the partition.
   * 
   */
  public void sendTextMessage(String message)
  {
    long now = System.currentTimeMillis();
    Notification notification =  
      new Notification("hanotification.example.counter", super.getServiceName(), now, now, message);
    sendNotification(notification);
  }

  /**
   * Lists the notifications received on the cluster partition
   */
  public Collection getReceivedNotifications()
  {
    return messages_;
  }

  Collection messages_ = new LinkedList();
 
  NotificationListener listener_ = new NotificationListener()
  {
    public void handleNotification(Notification notification,
                                   java.lang.Object handback)
     {
       messages_.add( notification.getMessage() );
     }
  };
}

This class behaves similarly to the previous one. If you examine its MBean View in the JMX console, you will notice that it has the same ReceivedNotifications attribute. The messaging operation is called sendTextMessage().

Conclusion

This text shows more examples where clustering is both powerful and complex. Fortunately, JBoss eliminates much of the complexity, allowing developers to enjoy the benefits of high availability. Writing singleton services in a distributed environment with JBoss is almost as easy as writing single VM singletons. If you previously invested in the JBoss Scheduler for a standalone server environment, you can keep your code and make it automatically valid in a clustered environment. Finally, lightweight notifications are no longer the prerogative of standalone applications. JBoss cluster notifications are yet another step towards practical horizontal scalability.

Ivelin Ivanov is a frequent contributor to the open source community and has recently participated in projects that include Apache Cocoon, JBoss, GNU XQuery and Jakarta Commons.


Return to ONJava.com.

Copyright © 2009 O'Reilly Media, Inc.