Taking JUnit Out of the Box

by Amir Shevat

"Nobody likes bugs." Most articles about testing utilities start with this sentence. And it's true--we would all like our code to act exactly as we planned it to work. But like a rebellious child, when we release our code into the world it has a tendency to act as if it has a will of its own. Fortunately, unlike in parenthood, there are things we can do to make our code behave exactly as we would like.

There are many tools designed to help up test, analyze, and debug programs. One of the most well-known tools is JUnit, a framework that helps software and QA engineers test units of code. Almost everyone that encounters JUnit has a strong feeling about it: either they like it or they don't. One of the main complaints about JUnit is that it lacks the ability to test complex scenarios.

This complaint can be addressed by doing some "out of the box" thinking. This article will describe how JUnit can perform complex test scenarios by introducing Pisces, an open source JUnit extension that lets you write test suites composed of several JUnit tests, each running on a remote machine serially or in parallel. The Pisces extension will allow you to compose and run complex scenarios while coordinating all of them in a single location.

Related Reading

JUnit Pocket Guide
By Kent Beck

JUnit Basics

Two basic objects in JUnit are TestCase and TestSuite. The TestCase provides a set of utility methods that help run a series of tests. Methods such as setup and teardown create the test background at the beginning of every test and take it down at the end, respectively. Other utility methods do a variety of tasks, such as performing checkups while the test is running, asserting that variables are not null, comparing variables, and handling exceptions.

Developers that want to create a test case need to subclass the TestCase object, override the setup and teardown methods, and then add their own test methods while conforming to the naming convention of testTestName.

Here is what a simple TestCase subclass might look like:

public class MyTestCase extends TestCase {

    * call super constructor with a test name 
    * string argument.
    * @param testName the name of the method that 
    * should be run.
    public MyTestCase(String testName){
    * called before every test
    protected void setUp() throws Exception {
    * called after every test
    protected void tearDown() throws Exception {
    * this method tests that ...
    public void testSomeTest1(){
    * this method tests that ...
    public void testSomeTest2 (){

TestSuite is composed of several TestCases or other TestSuite objects. You can easily compose a tree of tests made out of several TestSuites that hold other tests. Tests that are added to a TestSuite run serially; a single thread executes one test after another.

ActiveTestSuite is a subclass of TestSuite. Tests that are added to an ActiveTestSuite run in parallel, with every test run on a separate thread. One way to create a test suite is to inherit TestSuite and override the static method suite().

Here is what a simple TestSuite subclass might look like:

public class MyTestSuite extends TestSuite {
    public static Test suite() {

        TestSuite suite = 
            new TestSuite("Test suite for ...");
        // add the first test
        MyTestCase mtc = 
            new MyTestCase("testSomeTest1");
        // add the second test
        MyTestCase mtc2 = 
            new MyTestCase("testSomeTest2");
        return suite;

Running a test or a test suite is easy, as there are several GUIs you can use, starting with the GUI provided by JUnit and going all the way to IDEs such as Eclipse.

Figure 1 shows how the Eclipse IDE presents a TestSuite.

JUnit integration with Eclipse
Figure 1. JUnit integration with Eclipse

Because it is not the main subject of this article and because there are many articles about JUnit, this article will only provide a brief overview of the basic concepts of JUnit. See the Resources section for articles that cover JUnit in greater depth.

JUnit Strengths and Weaknesses

JUnit is an easy-to-use and flexible open source testing framework. As with every other project, it has many strengths but also some weaknesses. By using the automated JUnit testing framework, which does not need human intervention, one can easily accumulate a large number of JUnit tests and ensure that old bugs are not reproduced. In addition, JUnit helpfully integrates with build utilities, such as Ant, and with IDE utilities, such as Eclipse.

JUnit's weaknesses are also well known. It supports only synchronous testing and offers no support for call backs and other asynchronous utilities. JUnit is a black-box testing framework, so testing problems that do not directly affect functionality (such as memory leaks) is very hard. Additionally, it does not provide an easy-to-use scripting language, so you must know Java to use JUnit.

Another big limitation is that JUnit tests are limited to only one JVM. This limitation becomes a big issue when one is trying to test complex and distributed scenarios. The rest of this article covers this problem and ways to solve it.

Testing Complex Scenarios

Why do we need to test complex distributed scenarios?

  1. Tests that verify the integrity of a small unit are useful but limited. Experience has taught us that most bugs are found in the integration phase. These problems range from two modules that fail to work together, all the way to two separate applications that are acting up. Whether it is two application servers, a client/server environment, or even a peer-to-peer scenario, it is highly important to test these complex scenarios because of the tricky bugs that can be found in them. But it is nearly impossible to do so with JUnit.
  2. Although Java is platform-independent, it is wise to check an application on more than one operating system. Your application may work very well on all operating systems, but it may not function exactly the way you expect it to work on some of them. Running the same set of tests over and over again for every OS is time-consuming, and with JUnit you cannot perform distributed tests, so you don't have a way to run the same series of tests simultaneously on several JVMs each running on different OSes.
  3. Some units of code can only be tested in a multiple-JVM scenario. For example, testing the opening of a connection (TCP socket or HTTP connection) and the integrity of the information retrieved from it is not possible (or very hard) to test with only one JVM, which is the only option that exists with JUnit.

Coordinating JUnit Tests Using Ant

It is possible to coordinate several tests running on different JVMs using only JUnit and Ant. Ant can run tasks both serially and in parallel (using the <parallel> tag), and can be configured to stop if any of these tasks fail.

There are several limitations to this approach. First, using JUnit together with Ant still limits you to one box and one operating system. Second, as you accumulate many tests, the Ant XML file grows ever bigger and more unmanageable. Third, we found that in some cases the forked JVMs remained up and running even after the Ant task had terminated.

The Pisces project came into the world because of these limitations and in order to fully equip JUnit with the ability to test complex and distributed scenarios. The next sections of this article will introduce this open source project.

Taking JUnit Out of the Box with Pisces

Pisces Basics

Project Pisces is an innovative open source project that extends the JUnit framework. Like many other extensions, Pisces adds functionality to JUnit while keeping the way you use it exactly the same.

At the heart of Pisces lies the ability to run JUnit tests on a remote JVM on the same computer, or on a separate one. These remote tests are wrapped in JUnit tests that are run locally so that the developer or QA person performs regular (local) tests using regular JUnit GUI tools. The wrapper object for the remote tests is called RemoteTestCase and it extends the TestCase object.

Figure 2 shows the remote test and its wrapper.

The remote test and its wrapper
Figure 2. The remote test and its wrapper

On every remote location we run a Pisces Agent application with a unique agent name. That agent performs the actual JUnit test and returns the result to the local test. Now, once we can run a remote test wrapped in a local test, we can create a more complex scenario that is composed of several remote tests. Figure 3 shows a test suite composed of several remote tests.

A test suite composed of several remote tests
Figure 3. A test suite composed of several remote tests

Default Out Rerouting

Every agent runs a set of JUnit tests and each test may write information to the default out. Because it is important that the tester or developer gets this information while the test is running, the default out is copied to the main local test suite's console. This process eliminates the need to go and look at each of the agent consoles in the test suite.

Pisces Pluggable Communication Layer

Communication between the agents and the main local test is a complex matter, and Pisces must be able to work in a range of different environments and network topologies. In order to accommodate this issue, Pisces has a pluggable communication layer. Each of these implementations solves the problems that occur in their specific network environments.

Two basic communication layers are provided by Pisces by default--a multicast implementation, which is easy to configure but works only in a LAN environment, and a JMS implementation, which requires the installation of Message-Oriented Middleware (MOM) but works in most environments.

Setting up and Running Pisces Tests

  1. Configuring and Running Pisces Agents

    As I mentioned before, the Pisces test suite is composed of several JUnit tests running on remote agents. Each agent is a Java application that, according to instructions it receives from the main test runner, runs the JUnit test and reports its result and the default out printouts back to the main test runner.

    The easiest way to run an agent application is to configure and run the relevant executable script in the scripts folder provided in the Pisces build. Alternatively, you can also build your own script based on those already provided. The script allows users to configure general parameters for the agent, such as a unique identifier, and for the communication layer, a multicast IP and port.

    For example, a script for running an agent with a multicast communication layer on Linux OS might look like this:

    # the folder were the agent can find junit.jar
    export JUNIT_HOME=/opt/junit3.8.1
    # the folders were the agent can find
    # the junit tests 
    export JUNIT_TEST_CLASSPATH=../examples/
    # the multicast port that the communication
    # layer uses 
    export PISCES_MCP=6767
    # the multicast IP that the communication
    # layer uses 
    export PISCES_MCIP=
    # the unique name of the agent
    export AGENT_NAME=remote1
    java -classpath 
        "$JUNIT_HOME/junit.jar:../pisces.jar:$JUNIT_TEST_CLASSPATH" \
        org.pisces.RemoteTestRunnerAgent \
        -name $AGENT_NAME -com multicast \
        -mcp $PISCES_MCP  -mcip $PISCES_MCIP
    The executable script for a Pisces agent that uses a JMS provider is similar in most ways; the JMS communication layer takes as a parameter the name of a loader class that returns a ConnectionFactory of your JMS provider.
  2. Configuring and Running Pisces-Enabled Test Suites

    After configuring and running all of the relevant agents, we still need to configure and run our Pisces-enabled test suite.

    First, we have to add the communication layer configuration to the beginning of our TestSuite, as shown below:

    // create the multicast communication layer
    MulticastRemoteTestCommunicationLayer com =
        new MulticastRemoteTestCommunicationLayer(
           RemoteTestRunner.name ,"",6767);
    // set the communication layer 

    Then, we need to create an instance of a TestSuite and add a test to it to be run remotely. This is achieved by specifying the JUnit class (with an optional test method), as well as the name of the agent that should run this specific test.

    Here is an example of how this is done:

    // create a regular TestSuite
    TestSuite suite = 
        new TestSuite("Test for org.pisces.testers");
    // create a wrapper for a regular case 
    RemoteTestCase rtc = 
            ,"someTestMethod", "remote1"); 
    // add test case to TestSuite 

The next section of this article will provide an example of a distributed test scenario and the code of its TestSuite.

Example: Concurrent Login Test

Let's say we have a web application. A user logs in, gets a service, and then logs out. We want to make sure that our security module prevents a single user from concurrently logging in on two different computers.

We have created a MyTestCase test case object, as shown before, that has two test methods. The first, testLogin(), makes sure we can log in for the first time, and a second one, testLoginFails(), makes sure that the second login fails.

Here is what a test suite that utilizes Pisces might look like:

public class RemoteTestSuiteExample extends TestSuite {
    public static Test suite() {
        // configure and start the com layer
        JMSRemoteTestCommunicationLayer com = null;
        try {

            com = 
                new JMSRemoteTestCommunicationLayer
                        new MantaConnectionFactory());
        } catch (JMSException e) {
            throw new RuntimeException(e);

        // do the JUnit thing
        TestSuite suite = 
            new TestSuite("Test for org.pisces.testers");
        // run method testLogin in class MyTestCase
        // on remote agent remote1
        RemoteTestCase rtc =  
                "testLogin", "remote1");

        // run method testLoginFails in class MyTestCase
        // on remote agent remote2
        RemoteTestCase rtc1 = 
                "testLoginFails", "remote2");

        return suite;

If all goes well, the remote agent (called remote1) tries to log in and succeeds. When the other agent (called remote2) tries to log in with the same user and password, it fails because the user is already logged in on a different computer.

This test could have been more complex; for example, by adding a testLogOut() method and performing other security checks, but as an example I wanted to keep it simple.

In this example, I have utilized a serverless JMS provider called MantaRay, an open source messaging middleware package that is based on peer-to-peer technology. In the latest release of Pisces, you can find several examples that utilize a JMS communication layer and other examples that use a multicast communication layer.


Bugs are not fun, but with the help of JUnit, a widely used open source framework for automated testing, the task of running multiple test cases becomes much easier. Many projects extend the functionality of JUnit, but up until now, tests always had to be limited to one machine. With the help of Pisces and some "out of the box" thinking, we can now finally create complex and distributed test scenarios.


Amir Shevat is a senior software developer with eight years of experience in computing.

Return to ONJava.com.

Copyright © 2017 O'Reilly Media, Inc.