"Nobody likes bugs." Most articles about testing utilities start with this sentence. And it's true--we would all like our code to act exactly as we planned it to work. But like a rebellious child, when we release our code into the world it has a tendency to act as if it has a will of its own. Fortunately, unlike in parenthood, there are things we can do to make our code behave exactly as we would like.
There are many tools designed to help up test, analyze, and debug programs. One of the most well-known tools is JUnit, a framework that helps software and QA engineers test units of code. Almost everyone that encounters JUnit has a strong feeling about it: either they like it or they don't. One of the main complaints about JUnit is that it lacks the ability to test complex scenarios.
This complaint can be addressed by doing some "out of the box" thinking. This article will describe how JUnit can perform complex test scenarios by introducing Pisces, an open source JUnit extension that lets you write test suites composed of several JUnit tests, each running on a remote machine serially or in parallel. The Pisces extension will allow you to compose and run complex scenarios while coordinating all of them in a single location.
|
Related Reading 
JUnit Pocket Guide |
Two basic objects in JUnit are TestCase and
TestSuite. The TestCase provides a set of
utility methods that help run a series of tests. Methods such as
setup and teardown create the test
background at the beginning of every test and take it down at the
end, respectively. Other utility methods do a variety of tasks, such
as performing checkups while the test is running, asserting that
variables are not null, comparing variables, and
handling exceptions.
Developers that want to create a test case need to subclass the
TestCase object, override the setup and
teardown methods, and then add their own test methods
while conforming to the naming convention of
testTestName.
Here is what a simple TestCase subclass might look
like:
public class MyTestCase extends TestCase {
/**
* call super constructor with a test name
* string argument.
* @param testName the name of the method that
* should be run.
*/
public MyTestCase(String testName){
super(testName);
}
/**
* called before every test
*/
protected void setUp() throws Exception {
initSomething();
}
/**
* called after every test
*/
protected void tearDown() throws Exception {
finalizeSomething();
}
/**
* this method tests that ...
*/
public void testSomeTest1(){
...
}
/**
* this method tests that ...
*/
public void testSomeTest2 (){
...
}
}
TestSuite is composed of several
TestCases or other TestSuite objects. You
can easily compose a tree of tests made out of several
TestSuites that hold other tests. Tests that are added
to a TestSuite run serially; a single thread executes
one test after another.
ActiveTestSuite is a subclass of
TestSuite. Tests that are added to an
ActiveTestSuite run in parallel, with every test run
on a separate thread. One way to create a test suite is to inherit
TestSuite and override the static method
suite().
|
Here is what a simple TestSuite subclass might look
like:
public class MyTestSuite extends TestSuite {
public static Test suite() {
TestSuite suite =
new TestSuite("Test suite for ...");
// add the first test
MyTestCase mtc =
new MyTestCase("testSomeTest1");
suite.addTest(mtc);
// add the second test
MyTestCase mtc2 =
new MyTestCase("testSomeTest2");
suite.addTest(mtc2);
return suite;
}
}
Running a test or a test suite is easy, as there are several GUIs you can use, starting with the GUI provided by JUnit and going all the way to IDEs such as Eclipse.
Figure 1 shows how the Eclipse IDE presents a TestSuite.
Figure 1. JUnit integration with Eclipse
Because it is not the main subject of this article and because there are many articles about JUnit, this article will only provide a brief overview of the basic concepts of JUnit. See the Resources section for articles that cover JUnit in greater depth.
JUnit Strengths and Weaknesses
JUnit is an easy-to-use and flexible open source testing framework. As with every other project, it has many strengths but also some weaknesses. By using the automated JUnit testing framework, which does not need human intervention, one can easily accumulate a large number of JUnit tests and ensure that old bugs are not reproduced. In addition, JUnit helpfully integrates with build utilities, such as Ant, and with IDE utilities, such as Eclipse.
JUnit's weaknesses are also well known. It supports only synchronous testing and offers no support for call backs and other asynchronous utilities. JUnit is a black-box testing framework, so testing problems that do not directly affect functionality (such as memory leaks) is very hard. Additionally, it does not provide an easy-to-use scripting language, so you must know Java to use JUnit.
Another big limitation is that JUnit tests are limited to only one JVM. This limitation becomes a big issue when one is trying to test complex and distributed scenarios. The rest of this article covers this problem and ways to solve it.
|
Why do we need to test complex distributed scenarios?
It is possible to coordinate several tests running on different
JVMs using only JUnit and Ant. Ant can run tasks both serially and
in parallel (using the <parallel> tag), and can
be configured to stop if any of these tasks fail.
There are several limitations to this approach. First, using JUnit together with Ant still limits you to one box and one operating system. Second, as you accumulate many tests, the Ant XML file grows ever bigger and more unmanageable. Third, we found that in some cases the forked JVMs remained up and running even after the Ant task had terminated.
The Pisces project came into the world because of these limitations and in order to fully equip JUnit with the ability to test complex and distributed scenarios. The next sections of this article will introduce this open source project.
Project Pisces is an innovative open source project that extends the JUnit framework. Like many other extensions, Pisces adds functionality to JUnit while keeping the way you use it exactly the same.
At the heart of Pisces lies the ability to run JUnit tests on a
remote JVM on the same computer, or on a separate one. These remote
tests are wrapped in JUnit tests that are run locally so that the
developer or QA person performs regular (local) tests using regular
JUnit GUI tools. The wrapper object for the remote tests is called
RemoteTestCase and it extends the TestCase object.
Figure 2 shows the remote test and its wrapper.
Figure 2. The remote test and its wrapper
On every remote location we run a Pisces Agent application with a unique agent name. That agent performs the actual JUnit test and returns the result to the local test. Now, once we can run a remote test wrapped in a local test, we can create a more complex scenario that is composed of several remote tests. Figure 3 shows a test suite composed of several remote tests.
Figure 3. A test suite composed of several remote tests
|
Every agent runs a set of JUnit tests and each test may write information to the default out. Because it is important that the tester or developer gets this information while the test is running, the default out is copied to the main local test suite's console. This process eliminates the need to go and look at each of the agent consoles in the test suite.
Communication between the agents and the main local test is a complex matter, and Pisces must be able to work in a range of different environments and network topologies. In order to accommodate this issue, Pisces has a pluggable communication layer. Each of these implementations solves the problems that occur in their specific network environments.
Two basic communication layers are provided by Pisces by default--a multicast implementation, which is easy to configure but works only in a LAN environment, and a JMS implementation, which requires the installation of Message-Oriented Middleware (MOM) but works in most environments.
Configuring and Running Pisces Agents
As I mentioned before, the Pisces test suite is composed of several JUnit tests running on remote agents. Each agent is a Java application that, according to instructions it receives from the main test runner, runs the JUnit test and reports its result and the default out printouts back to the main test runner.
The easiest way to run an agent application is to configure and run the relevant executable script in the scripts folder provided in the Pisces build. Alternatively, you can also build your own script based on those already provided. The script allows users to configure general parameters for the agent, such as a unique identifier, and for the communication layer, a multicast IP and port.
For example, a script for running an agent with a multicast communication layer on Linux OS might look like this:
#!/bin/sh
# the folder were the agent can find junit.jar
export JUNIT_HOME=/opt/junit3.8.1
# the folders were the agent can find
# the junit tests
export JUNIT_TEST_CLASSPATH=../examples/
# the multicast port that the communication
# layer uses
export PISCES_MCP=6767
# the multicast IP that the communication
# layer uses
export PISCES_MCIP=228.4.19.76
# the unique name of the agent
export AGENT_NAME=remote1
java -classpath
"$JUNIT_HOME/junit.jar:../pisces.jar:$JUNIT_TEST_CLASSPATH" \
org.pisces.RemoteTestRunnerAgent \
-name $AGENT_NAME -com multicast \
-mcp $PISCES_MCP -mcip $PISCES_MCIP
The executable script for a Pisces agent that uses a JMS provider
is similar in most ways; the JMS communication layer takes as a
parameter the name of a loader class that returns a
ConnectionFactory of your JMS provider.Configuring and Running Pisces-Enabled Test Suites
After configuring and running all of the relevant agents, we still need to configure and run our Pisces-enabled test suite.
First, we have to add the communication layer configuration to
the beginning of our TestSuite, as shown below:
// create the multicast communication layer
MulticastRemoteTestCommunicationLayer com =
new MulticastRemoteTestCommunicationLayer(
RemoteTestRunner.name ,"228.4.19.76",6767);
// set the communication layer
RemoteTestRunner.setCom(com);
|
Then, we need to create an instance of a
TestSuite and
add a test to it to be run remotely. This is achieved by specifying
the JUnit class (with an optional test method), as well as the name
of the agent that should run this specific test.
Here is an example of how this is done:
// create a regular TestSuite
TestSuite suite =
new TestSuite("Test for org.pisces.testers");
// create a wrapper for a regular case
RemoteTestCase rtc =
RemoteTestCase.wrap(RemoteTestTestCase.class
,"someTestMethod", "remote1");
// add test case to TestSuite
suite.addTest(rtc);
The next section of this article will provide an example of a
distributed test scenario and the code of its
TestSuite.
Let's say we have a web application. A user logs in, gets a service, and then logs out. We want to make sure that our security module prevents a single user from concurrently logging in on two different computers.
We have created a MyTestCase test case object, as
shown before, that has two test methods. The first,
testLogin(), makes sure we can log in for the first
time, and a second one, testLoginFails(), makes sure
that the second login fails.
Here is what a test suite that utilizes Pisces might look like:
public class RemoteTestSuiteExample extends TestSuite {
public static Test suite() {
// configure and start the com layer
JMSRemoteTestCommunicationLayer com = null;
try {
com =
new JMSRemoteTestCommunicationLayer
(RemoteTestRunner.name,
new MantaConnectionFactory());
} catch (JMSException e) {
throw new RuntimeException(e);
}
RemoteTestRunner.setCom(com);
// do the JUnit thing
TestSuite suite =
new TestSuite("Test for org.pisces.testers");
// run method testLogin in class MyTestCase
// on remote agent remote1
RemoteTestCase rtc =
RemoteTestCase.wrap(MyTestCase.class,
"testLogin", "remote1");
suite.addTest(rtc);
// run method testLoginFails in class MyTestCase
// on remote agent remote2
RemoteTestCase rtc1 =
RemoteTestCase.wrap(MyTestCase.class,
"testLoginFails", "remote2");
suite.addTest(rtc1);
return suite;
}
}
If all goes well, the remote agent (called remote1)
tries to log in and succeeds. When the other agent (called
remote2) tries to log in with the same user and password,
it fails because the user is already logged in on a different
computer.
This test could have been more complex; for example, by adding a
testLogOut() method and performing other security
checks, but as an example I wanted to keep it simple.
In this example, I have utilized a serverless JMS provider called MantaRay, an open source messaging middleware package that is based on peer-to-peer technology. In the latest release of Pisces, you can find several examples that utilize a JMS communication layer and other examples that use a multicast communication layer.
Bugs are not fun, but with the help of JUnit, a widely used open source framework for automated testing, the task of running multiple test cases becomes much easier. Many projects extend the functionality of JUnit, but up until now, tests always had to be limited to one machine. With the help of Pisces and some "out of the box" thinking, we can now finally create complex and distributed test scenarios.
Amir Shevat is a senior software developer with eight years of experience in computing.
Return to ONJava.com.
Copyright © 2007 O'Reilly Media, Inc.