ONJava.com    
 Published on ONJava.com (http://www.onjava.com/)
 See this if you're having trouble printing code examples


Got Project Automation? Got Project Automation?

by Mike Clark
11/10/2004

Editor's Note: In his new book, Pragmatic Project Automation, Mike Clark gives you soup-to-nuts recipes for automating your software project: creating one-step builds with Ant, scheduling continuous builds with CruiseControl, generating software releases at the push of a button, installing and deploying applications with ease, and monitoring builds and running programs via email, RSS, your cell phone, and, yes, even lava lamps. The recipes include working examples that make it easy for beginners to follow along, while more advanced topics teach the old hands something new. In this article, he presents an overview of the benefits that automating your project can bring.

You're on the hook to deliver a software release for a critical demo tomorrow morning. The suits in sales are frothing at the mouth to show off your company's new whiz-bang application to some very important people with deep pockets. Just as you're finding your rhythm behind the keyboard, your boss stops by to remind you that this demo could make or break the project. No pressure!

One-Step Build and Test

It's almost noon before you type in the last line of code for those "must-have" demo features. Your favorite IDE says that your code compiles and passes its unit tests. But will your code work as expected when it's integrated with the rest of the system? To find out, you update your local workspace to get in sync with the files currently in the version control system. Then you run the project's one-step build process:

      
  $ ant

That command compiles all of the source files and runs all of the unit tests using the recipe listed in the following Ant build file:


  <project name="whizbang" default="test" basedir=".">

    <property name="build.prod.dir" location="build/prod"/>
    <property name="build.test.dir" location="build/test"/>
    <property name="src.dir"        location="src"/>
    <property name="test.dir"       location="test"/>
    <property name="vendor.lib.dir" location="vendor/lib"/>

    <path id="project.classpath">
      <pathelement location="${build.prod.dir}" />
      <pathelement location="${build.test.dir}" />
      <fileset dir="${vendor.lib.dir}">
        <include name="*.jar"/>
      </fileset>
    </path>

    <target name="prepare">
      <mkdir dir="${build.prod.dir}"/>
      <mkdir dir="${build.test.dir}"/>
    </target>

    <target name="compile" depends="prepare">
      <javac srcdir="${src.dir}" destdir="${build.prod.dir}">
        <classpath refid="project.classpath" />
      </javac>
    </target>

    <target name="compile-tests" depends="compile">
      <javac srcdir="${test.dir}" destdir="${build.test.dir}">
        <classpath refid="project.classpath" />
      </javac>
    </target>

    <target name="test" depends="compile-tests">
      <junit haltonfailure="true">
        <classpath refid="project.classpath" />
        <formatter type="brief" usefile="false" />
        <batchtest>
          <fileset dir="${build.test.dir}"
            includes="**/*Test.class" />
        </batchtest>
      </junit>
    </target>

  </project>

Related Reading

Pragmatic Project Automation
How to Build, Deploy, and Monitor Java Applications
By Mike Clark

As you're writing code, you frequently hit the convenient Build button in your IDE to make sure everything compiles. You've also become addicted to seeing a happy green bar when all of your JUnit tests pass, so you use the JUnit test runner integrated into your IDE. But not everyone on the team shares your love for this IDE, and you wouldn't want to have to fire up the IDE every time someone wanted to create a build. By using a build file that's externalized from your IDE, everyone on the team can consistently build and test the project in one step. (The project down the hall uses Maven to create one-step builds.)

No surprise to you, the build succeeds and you're once again reminded that you are the world's greatest programmer. Not only does this build process give you confidence in your code, it also gives you confidence that the project can be built outside of your IDE.

Feeling rather chuffed, you check in the files you've changed and duck out for a quick bite to eat. You still have a lot to do to prepare for the demo, and you need to leave early to make the first inning of your son's championship tee-ball game. The clock is ticking ...

Bubble Trouble

On the way back to your cube after lunch, you notice that the project's red lava lamp is boiling. Uh oh! When you left for lunch, the green lamp was happily bubbling. While you were gone, the scheduled build process on your project's dedicated build machine tried to build and test the code currently in the version control repository. But something went terribly wrong.

Running builds continuously on your project is easy because of the fact that you can create a build from the command line in one step. This mean you can easily train a computer to do that for you all day long. Otherwise, you'd have to park a developer in front of a command line to run the build file every time a food pellet appeared. Instead, you have CruiseControl configured to automatically create builds on an interval on your project's dedicated build machine, as directed by the following config.xml file:


  <cruisecontrol>
 
    <project name="whizbang" buildafterfailed="false">

      <bootstrappers>
        <currentbuildstatusbootstrapper
          file="logs/whizbang/currentbuildstatus.txt" />
      </bootstrappers>

      <modificationset quietperiod="30">
        <cvs localworkingcopy="checkout/whizbang" />
      </modificationset>

      <schedule interval="300">
        <ant buildfile="cc-build.xml" />
      </schedule>

      <log dir="logs/whizbang">
        <merge dir="checkout/whizbang/junit-results" />
      </log>
   
      <publishers>

        <currentbuildstatuspublisher
          file="logs/whizbang/currentbuildstatus.txt" />

        <!-- email publisher -->
        <!-- RSS publisher -->
        <!-- lava lamp publisher -->

      </publishers>

    </project>

  </cruisecontrol>

The config.xml file tells CruiseControl to wake up every five minutes and check your project's CVS repository to see if a build is necessary. Only if someone on your team has changed an existing file or added a new file to the version control repository will CruiseControl attempt to create a build. It defers how to create a build for your project to an Ant (or Maven) build file. You have CruiseControl configured to run an Ant build file called cc-build.xml, as follows:


  <project name="cc-build" default="build" basedir="checkout">

    <target name="build">
      <delete dir="whizbang" />
      <cvs command="co whizbang" />
      <ant antfile="build.xml" dir="whizbang" />
    </target>

  </project>

The cc-build.xml file bootstraps the build process by deleting the copy of your project used during the last build and checking out a fresh copy of your project from the CVS repository. Then it automatically runs the same build.xml file that you run from the command line to compile and test the project. After running the build file, CruiseControl publishes the build results to all registered publishers. (The project down the hall that uses Maven also uses CruiseControl, but it's configured to watch for modifications in some other version control system that you're happy not to be using.)

Doing all of this work every five minutes is a tireless job, which is exactly why you're glad CruiseControl does it for you. It seemed like overkill when you originally set it up, but you've learned to appreciate the value of timely feedback. The five-minute schedule just compiles all of the code and runs unit tests as a quick sanity check. You also have CruiseControl configured to run a comprehensive suite of system and performance tests on a less frequent interval. If the five-minute build fails, the problem is no more than five minutes old. This makes it easier for you to find and fix the problem, which saves you precious time. And if changes haven't been committed in the last five minutes, then CruiseControl keeps sleeping.

Hark, the build has failed! It's a good thing CruiseControl lit up the red lava lamp, because you may have overlooked the build failure email in your overflowing email inbox. Anxious to get to the bottom of the problem, you pull up the build status web page and find that in your haste you forgot to check in a new file. This is embarrassing, but at least you can easily fix the build now before problems compound into a nightmare debugging session right before the demo.

After checking in the missing file, you force a quick build on the build machine just to make sure everything is golden. Glancing over your shoulder you see the red lava lamp go dark and the green lamp come back to life. Crisis averted; life is good.

Quick Release

A little while later, everyone on the team has checked in all of their code for the demo. Now you're ready to create a distribution file and deploy it to the demo server. But you only have a few minutes to spare before leaving for the big game and the release procedure involves the following tedious steps:

  1. Test the code in the mainline directory.
  2. Create a release branch in version control.
  3. Check out the contents of the release branch.
  4. Build and test the code in the release branch.

    (Fix any problems.)

  5. Package all of the files for the release into a distribution file.
  6. Test the contents of the distribution file.
  7. Tag the release branch in version control.
  8. Send the distribution file to QA.

And those are just the steps you can remember! Indeed, releasing your software used to be a time-consuming and error-prone process. Consequently, you didn't release new versions of your software very often. But you got tired of having to remember all of the steps in the release procedure and the stress of having to correctly type in all of the required commands. Now your project's step-by-step release procedure is automated (and even documented) with a pair of push-button release scripts.

The demo will take a bit of preparation, and you want a stable working area isolated from activity on the mainline. But you don't want to freeze the mainline and block everyone from starting on the next release. The solution is to create a release branch in your version control repository. The first script handles steps 1 to 4 of the release procedure, which you run with a version number:

   
  $ release_branch 2_7_1

The script runs successfully, telling you that the release branch was created and all of the code compiled and passed its tests. If there were problems, you would have made changes in the release branch directory, tested those changes, and committed the changes to the release branch. You might also have merged the changes onto the mainline by running another script.

Once you have a release branch, and you've fixed any problems, you're ready to actually generate a release. To do that, you run another script that handles steps 4 to 8, given a version number:


  $ release_generate 2_7_1

The result of running that script is a self-contained distribution file--the same file that a customer might install or deploy. You're almost there; only one more step left before you're outta here.

Dirty Deploy Details

Deploying the application to the demo server is another multi-step manual procedure that you're sure to do wrong, even if you weren't racing against the clock. But because your team deploys software frequently--and deploying it reliably and accurately every time is important--you've automated the deployment steps. All of the dirty deployment details are carried out for you when you run the deployment script:


  $ deploy

Behind the scenes, that script transfers the distribution file to the demo server and unpacks all of the deployment modules into their respective directories. But before the script actually fires up the application server, it has one more handy trick up its sleeve.

You wouldn't want to automatically deploy the application, only to have it fail to start because of a stupid configuration problem. Your application has a number of configuration values that need to be properly set before the application will start cleanly. So before starting the application server, the script runs a suite of diagnostic tests to quickly pinpoint any potential problems with the deployment.

Specifically, you've noticed that fouling up the database configuration is a common deployment mistake. You've lost too much of your hairline debugging that problem, so last week you wrote the following diagnostic test using JUnit:


  public class DiagnosticTests extends junit.framework.TestCase {

    public void testDatabaseConnection() {

      Database database = new Database();

      try {

        database.connect();

      } catch(RuntimeException e) {
        fail("Unable to connect to the database '" +
             database.getURL() + "'. " +
             "Please check the 'database.url' property.");
      }
    }
  }

Related Reading

JUnit Pocket Guide
By Kent Beck

The testDatabaseConnection() method attempts to connect to a database using an instance of the project's Database class. That class reads configurable values, such as the database.url property, from a configuration file. If the diagnostic test can't connect to the database, then it's likely that your application will suffer the same fate when it's run. Therefore, if the connect() method throws an exception when called by the diagnostic test, the fail() method is called to print out a helpful message intended to help you fix the problem.

Thankfully, the deployment script reports no problems. That tells you that the application was deployed, all of the diagnostic tests passed, and the application server was started. Your demo is live!

Tattletale Monitor

You click through a few pages of the web application as a quick sanity check. It looks great, but you'd like to sleep well tonight knowing that it will still look great for the suits driving the demo tomorrow. The project is on the line, and you've learned that bad things can happen to demos overnight.

Not to worry. You reach inside of your automation toolbox for a program that scrapes a web site every couple of minutes looking for bad words such as "Error" or "Exception." If any of those words show up, or the web site becomes unavailable, the monitor will send an SMS message to your cell phone. That way, if the application crashes, you'll have the maximum amount of time to fix it before the demo. Before you sprint to the door, you hook up the monitor to the demo site:

   
  $ monitor http://demoserver:8080/whizbang

Down at the tee-ball field, you watch your son step up to the plate. Meanwhile, back in the office, your faithful monitoring program is running unattended. Your cell phone is close by, but it never rings and you sleep like a baby because the demo goes off without a hitch.

Onward and Upward

The demo was so successful that customers are lining up at the door for copies of the application. Posting the distribution file on the company's public web site or burning it to multiple CDs would slow down your team, if it weren't for automation scripts that handle those tasks, too. And if someone happens to (ahem) report a bug, you can easily re-create the demo from version control. When the bug has been fixed, you can generate a new release at the press of a button.

Automation helped reduce the risk of a failed demo by notifying you early when problems occurred, wherever you were. Automation also saved you time and guaranteed consistent results by giving you repeatable ways to build and deploy your software. As you continue to prepare demos and release new software, this automation will pay for itself many times over.

Pragmatic Project Automation

Unfortunately, this story isn't the reality for most projects, maybe not even your project. Many teams try to do all these project chores by hand, but people just aren't as good at repetitive tasks as computers. Those teams risk running a procedure differently the one time it matters, on one machine but not another, or doing it just plain wrong. And frankly, you know you've got better things to do than run continuous builds, follow multi-step checklists, copy files around on servers, and monitor running programs. But how do you quickly and effectively pull all of these pieces together?

It may seem, from this article, that automating your current or next project involves a lot of work. Thankfully, you don't need to automate all of your project's procedures to start realizing the benefits of automation today. Each project chore you automate is an investment that pays off immediately and increases in value over time. You can quickly get started with each individual step by using freely available tools such as Ant, Maven, CruiseControl, JUnit, and simple scripts. The book Pragmatic Project Automation shows you how to put your computer to work doing your project's repetitive tasks the same way, time after time, without bothering you. That means you'll have more time and energy to do the really exciting--and challenging--stuff like writing quality code.

Mike Clark is a consultant, author, speaker, and programmer.


Return to the ONJava.com

Copyright © 2009 O'Reilly Media, Inc.