Black Box Web Testing with HttpUnit

White or Black?

Automated software tests are crucial for IT projects. They enable continuous modifications to an existing code base without the fear of damaging existing functionality. They are executed at will and don't carry the costs and inconsistencies associated with manual tests.

There are two fundamental approaches for automated software tests:

• White Box, where an application is tested from the inside using its internal application programmatic interface (API).
• Black Box, where an application is tested using its outward-facing interface (usually a graphical user interface).

I consider White Box tests as a nice-to-have feature, while Black Box tests are a necessity. White Box tests suffer from the following shortcomings:

Related Reading Webmaster in a Nutshell By Stephen Spainhour, Robert Eckstein Table of Contents Index Sample Chapter Read Online--Safari Search this book on Safari:   Only This Book All of Safari Code Fragments only

• Creating a false sense of stability and quality. An application can often pass all unit tests and still appear unavailable to users due to a non-functioning component (a database, connection pool, etc.). This is a critical problem, since it makes the IT application team look incompetent.

• Inability to validate use-case completeness because they use a different interface to the application. White Box tests focus on pleasing the wrong crowd -- developers. The right crowd to please in the real world is application users, because they fund IT projects. It is more important to understand how a user views an application than how a developer does. Developers and users use different interfaces to interact with an application; developers use a programmatic API, while users use a GUI. Application users care mostly about the inputs and outputs on their GUI screen. The only technique for testing these is interacting with the application via the same interface used by the users.

Sometimes White Box tests are sufficient. In general, I am no great supporter of White Box tests. White Box unit tests are sufficient to test programmatic interfaces (API). For example, a unit test is sufficient to test a square root function.

Black Box tests are useful in evaluating the overall health of the application and its ability to provide value to users. Most Black Box tests emulate the sequence of interactions between a user to an application in its exact order. Failure of a Black Box test indicates that a user received insufficient value from the application.

The focus of this article is Black Box tests of web browsers, perhaps the most common user interface in today's IT environment. The article demonstrates a simple yet effective way to perform a Black Box test of a web site using an open source package called HttpUnit.

Figure 1. Where White Box and Black Box tests interface to an application

Emulating User Activity via Black Box Tests

Black Box tests emulate user interaction with an application. Keeping the tests closely aligned with the manner in which users interact with the application is the key for quality tests.

Historically, writing Black Box tests was difficult. Applications used different GUI technologies and communication protocols; thus, the potential for tool reuse across applications was low. Commercial testing-tool packages relied on capturing mouse clicks and keystrokes, which proved brittle to change. The combination of wide usage of HTTP/HTML standards and an emerging number of open source projects makes Black Box testing an easier task. Emulating user activity on web sites can be done in several manners:

• Automating Internet Explorer via the SHDocVw COM interface.
• Using a commercial testing tool.
• Using an open source Java package like HttpUnit or jWebUnit.

HttpUnit is the focus of this article for several reasons:

• HttpUnit covers the useful spectrum of HTTP/HTML standards functionality, including forms, frames, JavaScript, SSL, cookies, headers. HttpUnit can be used to test a web site written in any programming language.
• HttpUnit is written in Java, so the test cases can use the entire spectrum of Java functionality.
• HttpUnit executes within a JVM as a standard Java library and can be integrated easily with Ant scripts and daemon test agents.
• HttpUnit is an open source project.

Swing and AWT applications are not suitable for HttpUnit tests. Those can be tested with sibling tools like jfcUnit.

Installing HttpUnit

Running HttpUnit will require the following CLASSPATH entries:

• httpunit.jar
• js.jar
• junit.jar
• nekohtml.jar
• xercesImpl.jar
• xmlParserAPIs.jar

A Word of Warning About Client-side JavaScript

HttpUnit supports client-side JavaScript. However, JavaScript is always an enigma. Sometimes it is appropriate to turn off support for client-side JavaScript by either removing js.jar from the CLASSPATH or calling the HttpUnit method HttpUnitOptions.setExceptionsThrownOnScriptError( false );.

Writing an HttpUnit Test Case

Site Under Test

The demonstration of HttpUnit's capabilities requires a web page accessible to the public. Ironically, I selected Dice.com. In this day and age it is likely that Java developers will know this resource. (If you detect an underlying pain in my writing, you can probably guess that I do know Dice.)

We shall write a Black Box test using HttpUnit to test a use case on the Dice web site.

To be clear: the Dice web site is not written in Java. HttpUnit Black Box tests are compatible with any HTML/HTTP-based web site. Furthermore, the test is performed against a code base completely hidden from us. This is not to be taken lightly in scenarios of delegation and design by contract, where the tester and coder are not the same. The set is ready now for testing a real-world use case:

The Dice Job-Seeker Use Case:

The following are the steps of the use case tested:

1. Navigate to www.dice.com.
2. Click on the Find a Job button.
3. Set the form data.
      • Set the Full Text Search.
      • Set the Days back to search.
      • Set the Jobs per Page.
4. Click on the Search button.
5. View the Result in the next page.
6. Click on a Job Opening link.
7. View a Job Opening.

If all of the above work, the test case is considered successful.

HttpUnit Main Classes

The main HttpUnit classes are:

• WebConversation, which emulates a web browser talking to a single site. It keeps track of the session context on the client side by holding cookies.
• WebRequest, which represents an HTTP Request sent to the server.
• WebResponse, which represents an HTTP Response returned by the server.
• WebForm, which represents an HTML form.

A typical HttpUnit interaction consists of creating a WebConversation instance, submitting a WebRequest, and navigating and validating the result returned by the WebResponse.

The Dice Job-Seeker HttpUnit Black Box Test

Step 1

The first thing to do is open a WebConversation with Dice:

URL serverUrl                = new URL(url);
WebConversation conversation = new WebConversation();
WebRequest request           = new GetMethodWebRequest(serverUrl,"");
WebResponse response         = conversation.getResponse(request);
assertNotNull("response of url: " + url, response)

Notice that the implementation of this HttpUnit test case uses the JUnit framework. The JUnit framework provides assertions, test runners, and Ant integration. Don't be misled by the name JUnit; our Black Box tests are not unit tests, we just use some of JUnit's capabilities.

Step 2

So far, the test uses very little knowledge about the structure of the web site. This is about to change. Here is the HttpUnit code used to click on the Find a Job image button:

WebImage image =
    response.getImageWithSource("images/topnav_findajob_off.gif");
assertNotNull("find a job impage",image);

WebLink findJobLink = image.getLink();
assertNotNull("find a job link",findJobLink);

System.out.println("found job seeker link");
response = findJobLink.click();
assertNotNull("job seeker page",response);

System.out.println("got job seeker page");

The first line in the code above searches for the image using the call response.getImageWithSource("images/topnav_findajob_off.gif");

The first question that comes to mind is, "How does a test coder find the name of the image?" The answer lies in HTML. To discover the image name, the test coder must use the web browser's View Source functionality. (Remember to View Source within the boundaries of the frame holding the HTML control you seek.)

Using the link associated with the image, the script opens the Job Seekers page. Like many Dice pages, this page uses HTML frames. The code section below selects the ENGINE_WIN frame:

response = response.getSubframeContents("ENGINE_WIN");
assertNotNull("got the engine frame",response);

The ENGINE_WIN frame uses HTML forms to submit user input to the Dice job search engine. The first necessary step is finding the input form within the page. This can be done by either the form name or form number within the containing page:

WebForm findJobsForm = response.getForms()[0];
assertNotNull("find jobs form",findJobsForm);

Step 3

The next step is populating the form with the job search criteria. The names of the form parameters are found again by viewing the page source.

findJobsForm.setParameter("FREE_TEXT",openSearchText);
findJobsForm.setParameter("DAYSBACK",daysBack);
findJobsForm.setParameter("NUM_PER_PAGE",jobsPerPage);

Step 4

After the form is populated, all that is left is calling the submit() method.

response = findJobsForm.submit();
assertNotNull("got search result frame",response);

Step 5

The job opening HTML links resulting from the job search form submission are sent to the same frame that originated the form submission. After receiving these links, the test script needs to filter out links not related to job openings:

WebLink resultLinks[] = response.getLinks();
assertNotNull("search links", resultLinks);
System.out.println("got search links back");

// count only the job opening links
int openingCounter = 0;
for(int i=0; i < resultLinks.length; i++)
{
    String url =  resultLinks[i].getURLString();
    if(url.indexOf("/jobsearch/servlet/JobSearch?op") != -1)
    {
        openingCounter++;
        System.out.println("found job opening: " + resultLinks[i].asText());
    }
}

Step 6

If the number of job openings found is greater than 0, the user can click on the job openings link:

if(openingCounter > 0)
{
    response = resultLinks[0].click();
    assertNotNull("job opening page",response); 
}

Congratulations! You just finished a complete Black Box test of a commercial web site by emulating an entire use case. Assuming the Dice.com site doesn't change soon, running the test will produce output similar to the following:

[java] .found job seeker link
[java] got job seeker page
[java] got engine frame
[java] got the find jobs search form
[java] got search result
[java] got search links back
[java] found job opening: Embedded Linux Guru
[java] found job opening: Visionary Software Design Manager
[java] found job opening: Senior Software Performance Engineer
[java] found job opening: Senior Java - Websphere Developer
[java] found job opening: Configuration and Deployment

Sadly, you might notice that the number of job openings for Java professionals is smaller than one would hope. (It's not a great time to jump ship.)

Please be considerate about running tests against someone else's site.

Integrating HttpUnit Tests Into a Build Script

A J2EE build script generally executes in the following order:

Figure 2. Software development lifecycle

The build tool of choice in many J2EE projects is Ant. Ant's feature list is out of the scope of this article, but of interest to testers are its features for reporting the success or failure of build tasks via email, and for executing the JUnit script. An Ant build script file is bundled with this article.

Ant Launch-and-Forget Build Script

The Ant mail logger enables SMTP email reporting of Ant build target successes and failures. Configuring the Ant mail logger requires several simple components:

• A property file holding the mailer information.
• An entry in an Ant build file to import the property file.
• Several .jars of Java classes enabling SMTP communication.

Invoke Ant using the command:

$ ant -logger org.apache.tools.ant.listener.MailLogger

Here is sample build.properties file used to set the Ant mail logger configuration:

MailLogger.mailhost=smtphost@domain.com
MailLogger.from=BuildAdmin@domain.com
MailLogger.failure.notify=true
MailLogger.success.notify=true
MailLogger.failure.to=FailureRecipient@domain.com
MailLogger.success.to=SuccessRecipient@domain.com
MailLogger.failure.subject=Ant Build Failure Subject
MailLogger.success.subject=Ant build Success Subject

Here is a sample entry in an Ant XML build file used to import the mailer properties:

<property file="build.properties"/>

Running the Ant XML build file bundled with this project will send an email message to the designated recipient.

Beware that reporting build failures with the Ant mail logger can be vicious. I've seen corporations whose build script notifies the entire galaxy whenever a build or test failure occurs. Use this feature with caution. (See an Anti-Pattern named: Email is Dangerous.)

HttpUnit Black Box Test Usage

Let's conclude with a real world example where Black Box tests saved the day. We had a three-tier web application. Our hosting app server was clustered and monitored. A user reported that the application was not responding. The developer assigned to the case successfully used the application.

Further exploration revealed that one node of the cluster contained an invalid configuration for the JDBC connection pool. Users assigned to that node faced an unresponsive application.

The lesson we learned was that application server monitoring can't guarantee application health. Application health must be checked via a Black Box test executed periodically by a daemon thread.

Black Box tests can also serve well as stress tests. HttpUnit scripts are very valuable for tests of concurrent access to a web site. The tests are a good representation of client behavior.

Usage Advice

• Attempt to mimic the user web-browser runtime behavior as closely as possible. Black box test scripts should not create environment entries and data not present in the runtime environment of the emulated user.
• Just do the necessary minimum. If you find yourself writing an artificial intelligence engine to dynamically test HTML field names, you have gone a tad too far. Black box tests must be simple and concrete.

Conclusion

Black Box tests are the most valuable type of tests available in the project team arsenal. HttpUnit Black Box tests can serve for use case completeness validation, application availability validation, stress testing, and regression tests.

Software packages like HttpUnit enable developers to automate Black Box tests quickly, while covering the entire functionality offered by today's web sites. Integration between Ant build scripts and HttpUnit is a best practice for protecting developers from hastily rolling unfinished changes to production.

Giora Katz-Lichtenstein is a J2EE professional consulting to the financial industry in New York City.

Return to ONJava.com.

Copyright © 2007 O'Reilly Media, Inc.