Let’s start from the beginning, what is Flexmojos? Flexmojos is a maven plugin for Adobe Flex development. You can get acquainted with it here.

Some Background

When I started Flexmojos I just wanted a decent maven plugin to build my project. I never considered making it public or turning it into a open source project. At the time, I just wanted something for me and for me alone. As I continued to work on my own solution, I realized more people had problems building Flex projects using Maven. So I decided to share my project. This is how the Flexmojos project began, and once the project had some momentum we stared running into challenges that required rigorous integration testing.

After a few releases, Flexmojos started to present backward compatibility issues. Something that worked on 1.0-alpha-2 suddenly stopped working on 1.0-alpha-3. And this started to happen more often during new features inclusion. The obvious fix for that was to create a bunch of unit tests. But mojos aren’t exactly the easiest things to unit test. So I had to keep testing manually, at that point Maven and integration tests had too much complexity for my limited Maven API knowledge. So I stopped including new features and concentrated on getting Flexmojos stable. Eventually I managed to get 1.0 release out.

I saw how much time, releases and code I needed to get 1.0 stable and I wanted to get 2.0 stable w/o that much work. But how? Creating a good set of automated tests. But again, how to test mojos? At this point, there wasn’t much information about testing Maven plugins. Googling here and poking around there I found maven-verifier.

maven-verifier tests are run using JUnit or TestNG, and provide a simple class allowing you to launch Maven and assert on its log file and built artifacts. It also provides a ResourceExtractor, which extracts a Maven project from your src/test/resources directory into a temporary working directory where you can do tricky stuff with it.

That was perfect for my needs. Using maven-verifier I found an easy way to test Flexmojos. In fact, I found a very good way to test Flexmojos, however when I started to write more complex tests it proved to be not as easy as I expected. We’ll talk more about that later. Using maven verifier is very cool too because it allowed me to write integration tests using TestNG, so I could use asserts, inherit, annotations and all the stuff available for unit testing that I was used to using.

How are Flexmojos test organized

Flexmojos only has integration tests. Unit testing maven plugins is not easy and Flexmojos wasn’t designed in a particularly modular way. If it were, it would have more unit tests.

Since Flexmojos is a Maven plugin I needed it installed into a Maven repository to run the integration tests, for that reason Flexmojos has all integration tests grouped into a single project, called flexmojos-test-harness.

The flexmojos-test-harness creates a complete environment to run the integration tests: – Download and unpack Maven 2.0.9 from central; – Overwrite Maven settings.xml with one that contains extra repositories ( check it here ), with all required artifacts; – Install Flexmojos artifacts into a fake local repository; this allow us to – Instrument Flexmojos artifacts using Emma; – Add emma.jar to Maven libs directory (required to enable test coverage).

This project sets up everything necessary to run Flexmojos (install maven), it sets up Emma (instrumenting and add emma.jar) and makes sure the right version of the artifacts are used during tests (filling maven repo).

Motivation for Flexmojos Integration Testing

Q. Why download and unpack maven from central, instead of using the maven already present on machine? A. First and most important reason: Instrumented classes need emma.jar on classpath. If not present, it will throw ClassDefNotFoundExceptions. So I can tamper with this Maven instance created by Flexmojos adding emma.jar. And that is a good reason to ensure that Flexmojos can manage a separate Maven installation that is isolated from all of the variation of developers who might be testing Flexmojos. At any moment I can set another version to be sure that Flexmojos is still working.

Q. Why overwrite settings.xml? A. First of all, it was was necessary to add the Flexmojos repository, so that maven could download the flex compiler from the server. This modified settings.xml does include a reference to an isolated, local Maven repository, so that anyone running the integration tests will retrieve artifacts already available on current machine, this dramatically reduces the time required to run the integration tests. Also notice that this settings.xml blocks all snapshots from being downloaded, that is nice because it prevents instrumented Flexmojos being overwritten with non-instrumented versions from maven central.

Q. Why put the artifacts on a fake local repository? A. That is about isolation… if I used the machine’s local repository, Flexmojos artifacts would get instrumented by Emma. But that would break Flexmojos. It’s required to add emma.jar on Maven classpath just to run instrumented Flexmojos. But shipping instrumented Flexmojos gives users no advantage, quite the opposite, it would generate coverage reports every time, wasting CPU, disk and memory usage. Another reason is that would break the GPG signature for Flexmojos artifacts. So, creating a isolated local repository allowed me to run instrumented tests without messing with built artifacts.

How to Configure Integration Tests

Q. How do you download and unpack maven from central? A. To copy and unpack artifacts is used the maven-dependency-plugin. This plugin “provides the capability to manipulate artifacts. It can copy and/or unpack artifacts from local or remote repositories to a specified location.”

58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

<execution>
  <id>extract-maven</id>
  <phase>generate-test-resources</phase>
  <goals>
    <goal>unpack</goal>
  </goals>
  <configuration>
    <artifactItems>
      <artifactItem>
        <groupId>org.apache.maven</groupId>
        <artifactId>apache-maven</artifactId>
        <version>${maven.version}</version>
        <type>tar.gz</type>
        <classifier>bin</classifier>
      </artifactItem>
    </artifactItems>
    <outputDirectory>${project.build.directory}</outputDirectory>
  </configuration>
</execution>

Read more about unpack at plugin site. Unpack goal documentation and more examples.

Q. How does settings.xml know where to find the local Maven repository? A. The settings.xml used on tests has some variables, namely ${localRepository.basedir}, that links to default repository location, and ${maven.repo.local}, that has the location when default-value is overwritten. This variables are interpolated using maven-resource-plugin. To enabled the variables interpolation is only necessary to declare the resources with the filtering option set to true, like this:

43
44
45
46
47
48
49
50
51
52
53
54
55

<build>
  <testResources>
    <testResource>
      <directory>src/test/resources</directory>
      <filtering>true</filtering>
      <includes>
        <include>settings.xml</include>
      </includes>
      <targetPath>${fake-maven}/conf</targetPath>
    </testResource>
...
  </testResources>
</build>

That was not declared on maven-resource-plugin configuration, but on build section, under resources or testResources (Flexmojos ITs case).

Q. How does the resources directory get interpolated? And why isn’t maven-resource-plugin used? A. That is a performance question…. maven-resource-plugin would always interpolate all files, and there are several files under resources, that alone was taking 30 seconds to a minute. But when running individual tests, not all files are required, so Flexmojos implemented its own interpolation, which runs before the test. Take a look at AbstractFlexmojosTests.getProject()

Code coverage

For the last part of this blog, I show how Flexmojos generates a test coverage report. Flexmojos uses emma4it-maven-plugin (introduced on part 2 of this blog series).

Emma4it has a goal to merge all available Emma binaries into a single one.

76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

<plugin>
  <groupId>org.sonatype.maven.plugin</groupId>
  <artifactId>emma4it-maven-plugin</artifactId>
  <version>1.3</version>
  <executions>
    <execution>
      <id>merge</id>
      <phase>verify</phase>
      <goals>
        <goal>merge</goal>
      </goals>
      <configuration>
        <searchPath>${basedir}</searchPath>
      </configuration>
    </execution>
...
  </executions>
</plugin>

This way, it is possible to combine coverage results of unit and integration tests, it becomes more relevant on Flexmojos because we have being improving UT tests coverage, especially in some new components that are very easy to unit-test (like the new generation and testing API).

In part 4 we will talk a bit about ITs on Nexus. Nexus does have a very particular setup where the approach used for Flexmojos proved to be inefficient.

See you in the next post.

During the first run we noticed some problems.Each time Nexus started, VM would use an additional 45Mb of memory. Soon we realized that our embedder was not working right. Stopping Nexus was not removing it from memory.

Setting all fields to null after stopping helped somewhat. But now instead of 45Mb of leak we were getting 40Mb, which was still too much when we had more then 200 tests to run. It was time to profile our tests.

At Sonatype we use yourkit for profiling, so I will demonstrate how to use yourkit to profile tests executed by maven. You should be able to use the same principle with any other profile tool. 1 – First, download, get a license and install yourkit. 2 – Create a new profile to hold yourkit-related configuration:

  <profiles>
...
    <profile>
      <!-- http://www.yourkit.com/docs/80/help/agent.jsp -->
      <id>yourkit-profile</id>
 
      <properties>
        <yourkit.home>C:\Arquivos de programas\YourKit Java Profiler 8.0.13</yourkit.home>
      </properties>
 
    </profile>
  </profiles>

I put the yourkit.home in pom.xml, but you can have it in settings.xml as well.

3 – Add yourkit agent to surefire inside the profile:

...
    <profile>
...
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
              <argLine>-agentpath:"${yourkit.home}\bin\win32\yjpagent.dll"</argLine>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

There are other ways to do this you can see here.

4 – Launch yourkit UI.

Yourkit UI

5 – Launch maven using yourkit profile: $ mvn clean test -P yourkit-profile -Dyourkit.home={your yourkit installation home}

As soon as yourkit UI detects your application, it will begin collecting profiling data.

Test running

Optional, using the API

Regular profiling is not very useful to find memory leaks on unit tests. A much more efficient approach is to take memory snapshots programmatically. To do so, we need yourkit API.

On the same profile, add this dependency:

    <profile>
...
      <dependencies>
        <dependency>
          <groupId>com.yourkit</groupId>
          <artifactId>yjp</artifactId>
          <version>8.0.13</version>
          <scope>system</scope>
          <systemPath>${yourkit.home}\lib\yjp.jar</systemPath>
        </dependency>
      </dependencies>
...
    </profile>

Note that you should not move yjp.jar to your local maven repository.

Now we can create a com.yourkit.api.Controller instance and take memory snapshos when we need to.

To avoid compilation problems during regular builds (when you are not profiling) I suggest doing that using reflection, like this:

    // profiling with yourkit, activate using -P youtkit-profile
    private static Object profiler;
 
    private static void startProfiler()
    {
        Class controllerClazz;
        try
        {
            controllerClazz = Class.forName( "com.yourkit.api.Controller" );
        }
        catch ( Exception e )
        {
            log.info( "Profiler not present" );
            return;
        }
 
        try
        {
            profiler = controllerClazz.newInstance();
            controllerClazz.getMethod( "captureMemorySnapshot" ).invoke( profiler );
        }
        catch ( Exception e )
        {
            fail( "Profiler was active, but failed due: " + e.getMessage() );
        }
    }
 
    private static void takeSnapshot()
    {
        if ( profiler != null )
        {
            try
            {
                profiler.getClass().getMethod( "forceGC" ).invoke( profiler );
                profiler.getClass().getMethod( "captureMemorySnapshot" ).invoke( profiler );
            }
            catch ( Exception e )
            {
                fail( "Profiler was active, but failed due: " + e.getMessage() );
            }
        }
    }

That way you won’t need to change the code to run in non-profiling mode.

You can see the full example in place at nexus-test-harness.

If you are still reading and are curious what was causing the memory leak on Nexus tests, we discovered that jetty creates Shutdown Hook that can hold a reference to all servers (depends on how you configure it). That shutdown hook stops jetty instances when the VM is going down. So it was just a matter of removing my jetty instances from it when I manually stopped Nexus.

We will be using Emma to measure and report the coverage. EMMA is an open-source toolkit for measuring and reporting Java code coverage.

Unit test coverage

Let’s start by generating a coverage report for unit tests only. This will require 2 new plugins to be added to our pom. The first one:

<plugin>
  <groupId>org.sonatype.maven.plugin</groupId>
  <artifactId>emma-maven-plugin</artifactId>
  <version>1.1</version>
  <executions>
    <execution>
      <phase>process-classes</phase>
      <goals>
        <goal>instrument</goal>
      </goals>
    </execution>
  </executions>
</plugin>

This plugin will instrument classes at target/generated-class/emma/classes, this won’t change the original classes at target/classes. Then, we must tell surefire plugin to look for classes at new location:

  org.apache.maven.plugins
  maven-surefire-plugin
  2.4.3
 
    ${project.build.directory}/generated-classes/emma/classes

At this point, we do 50% of code coverage. Mainly because the psvm ( public static void main ) is never executed.

Here is the EMMA text report. Emma does produce xml and html reports in addition to this simple text report:

[EMMA v2.0.5312 report, generated Fri Jun 12 15:43:28 BRT 2009]
-------------------------------------------------------------------------------
OVERALL COVERAGE SUMMARY:
 
[class, %]	[method, %]	[block, %]	[line, %]	[name]
100% (1/1)	67%  (2/3)!	56%  (10/18)!	50%  (4/8)!	all classes

Note: at this point no report is generated, we will configure report generation after integration tests coverage reports are in place. Once you run Surefire, you may notice the coverage.ec in the project root directory. This file contains the coverage result in a binary format used by Emma.

Integration test coverage

So far, we have the unit tests results. As I demonstrated in the previous article in this series, the integration tests need the project artifact to run, so, we need to instrument this jar in order to get coverage result on ITs. Let’s see the pom:

  org.sonatype.maven.plugin
  emma4it-maven-plugin
  1.3
 
      instrument
 
        instrument-project-artifact
 
        true

Note that is using a different plugin: emma4it-maven-plugin. This plugin provides some additional goals for instrumenting integrations tests. For this project we use instrument-project-artifact goal. This goal will add Emma instrumentation into the project jar. This is specific to tests running on integration-test phase, because it does requires that the packaged artifact be instrumented.

There is also the configuration appendEmma. When true, this goal will shade the Emma jar into the instrumented jar. This is very useful to test jars that will be launched from command line, like this project.

[EMMA v2.0.5312 report, generated Fri Jun 12 16:05:45 BRT 2009]
-------------------------------------------------------------------------------
OVERALL COVERAGE SUMMARY:
 
[class, %]	[method, %]	[block, %]	[line, %]	[name]
100% (1/1)	100% (3/3)	89%  (16/18)	88%  (7/8)	all classes

At this point we have a very decent code coverage, almost 90%, 7 of 8 lines of code, and on this project it is not possible to get 8/8. And why is that? System.exit(). when this line is invoked nothing else is executed, that includes Emma instrumented code, so Emma won’t be able to know if System.exit() was executed, our integration tests catch that, because we check the exit code of the process, but Emma can’t do the same.

Test Coverage Report

At this point we have all coverage data stored at coverage.ec, in Emma binary format. Let’s now convert that into something that is more readable.

Add the following execution on emma4it-maven-plugin:

  report
post-integration-test
 
    report
 
        ${project.build.sourceDirectory}

Now the text, HTML, and XML reports are generated at target/site/emma.

Emma HTML coverage report

That is it for part 2. On the next part, I will show some ideas about testing more complex projects like flexmojos and nexus.

The application

Let’s define a very simple application to be tested. The application will return an exit code equal to the number of command-line parameters. Running the program with no parameters will produce an exit code equal to zero, two parameters exit code two, and so on.

The application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

package org.sonatype.simpleclientapp;
 
public class Main
{
    public static void main( String[] args )
    {
        System.exit( execute( args ) );
    }
 
    public static int execute( String[] args )
    {
        if ( args == null )
        {
            return 0;
        }
 
        return args.length;
    }
}

This class is very simple to test, in the next section you will see how you can write a simple integration test using JUnit.

Creating and running integration tests

Let’s create a simple integration test using JUnit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package org.sonatype.simpleclientapp;
 
import java.io.File;
 
import junit.framework.TestCase;
 
public class MainIT
    extends TestCase
{
 
    public void testExecute()
        throws Exception
    {
        assertEquals( 0, execute( new String[] {} ) );
        assertEquals( 1, execute( new String[] { "one" } ) );
        assertEquals( 6, execute( new String[] { "one", "two", "three", "four", "five", "six" } ) );
    }
 
    private int execute( String[] args )
        throws Exception
    {
        File jar = new File( "target/simple-client-app-1.0-SNAPSHOT.jar" );
 
        String[] execArgs = new String[args.length + 3];
        System.arraycopy( args, 0, execArgs, 3, args.length );
        execArgs[0] = "java";
        execArgs[1] = "-jar";
        execArgs[2] = jar.getCanonicalPath();
        Process p = Runtime.getRuntime().exec( execArgs );
        p.waitFor();
        return p.exitValue();
    }
 
}

Note that this test needs the packaged jar (simple-client-app-1.0-SNAPSHOT.jar), which isn’t created when surefire runs the unit tests.

To run this we need a different approach. By different approach I mean a different plugin: the Failsafe Maven Plugin. The Failsafe Plugin is a fork of the Surefire plugin designed to run integration tests.

The Failsafe plugin goals are designed to run after the package phase, on the integration-test phase.

The Maven lifecycle has four phases for running integration tests:

• pre-integration-test: on this phase we can start any required service or do any action, like starting a database, or starting a webserver, anything… you can think on this as Junit.setUp()
• integration-test: failsafe will run the test on this phase, so after all required services are started.
• post-integration-test: time to shutdown all services… you can think of this as Junit.tearDown()
• verify: failsafe runs another goal that interprets the results of tests here, if any tests didn’t pass failsafe will display the results and exit the build.

Configuring Failsafe in the POM:

23
24
25
26
27
28
29
30
31
32
33
34
35

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>failsafe-maven-plugin</artifactId>
        <version>2.4.3-alpha-1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

By default, the Surefire plugin executes **/Test*.java, **/*Test.java, and **/*TestCase.java test classes. The Failsafe plugin will look for **/IT*.java, **/*IT.java, and **/*ITCase.java. If you are using both the Surefire and Failsafe plugins, make sure that you use this naming convention to make it easier to identify which tests are being executed by which plugin.

In the next part of this series, I will talk about test coverage and demonstrate a method of including coverage of both Unit Tests and Integration Tests in a report that can be generated for a Maven project.

[media id=1 width=320 height=70] INTERVIEW TRANSCRIPT

Tim O’Brien: So, Rich could you introduce yourself?

Rich Seddon: Yeah, my name is Rich Seddon and I’ve been brought on at Sonatype for the purpose of producing a set of automated tests for the M2 Eclipse Product and the reason for doing this is that there’s a lot of performance improvements and bugs that we want to fix. But if we don’t have the automated test in place we won’t know if we’re breaking things. And it’s got to be automated because of the number of platforms that we support and the number of operating systems that we run on is very large and the manual testing process is really much, much more than a small company can possibly handle.

TO: So, can you give me a sense of what’s the difference between what you do and something like a basic JUnit test.

RS: Well, what I’m actually doing here is a full set of system tests for M2 eclipse so I actually in the test harness bring up the M2 eclipse on various platforms and drive the UI through a set of scenarios and verify along the way that no errors have occurred and that the state of the UI is as expected and the state of the project under test is as expected….you might say that what I’m doing is trying to capture, in an automated fashion, the types of usage that our users will actually go through from day to day.

TO: What kind of tools are you using for this?

RS: Right now I’m using uumm; I’m kind of using Hybrid approach. There are sort of two approaches to doing eclipse UI I testing. One of them is to drive it with a sort of UI robot where the robot pushes buttons and manipulates trees and things like that. And the other approach is sort of a white box testing approach where you bring up in a regular old unit test, eclipse, and you drive eclipse at the API level to the unit test. Both of these tests have their advantages and disadvantages.

The reason that I got attracted to this position is that, in the last few years, it seems there’s been a few sort of hybrid approaches come along one of them is called SWT POD (sp?) PPTP by eclipse and then there was Abbott. Then there’s a commercial company, Instantiations has a product called “Window Tester”. And these allow you to do both kinds of testing, UI Robot testing and API testing at the same time. What they do is they sort of record your actions as your drive eclipse through a set of scenarios and produce a JUnit test that you can then edit.

TO: Is this run as a part of the Maven build for m2eclipse?

RS: Yeah, it is run in….. integration using the Maven OSGI plugin. There’s an OSGI test runner plug-in that actually brings up the eclipse, loads in the unit test, and then within the running eclipses project runs the unit test.

TO: So, what sorts of things have you found through the testing that we wouldn’t have found if we hadn’t been doing this?

RS: Well, I’ve found quite a few things. There’s been certainly some deadlock situations that I don’t think we would have been able to reproduce. We would have got reports of them, but we never would have been able to reproduce them in-house because they don’t happen that often. But, when you have these tests running continuously and you’re capturing the output of them when things go wrong then that allows us to find intermittent bugs which is very difficult to do otherwise with manual testing.

The other thing that I’m finding though is that because I’m actually driving eclipse through an automated scenario, it forces me to pay very close attention to eclipse itself. So, I actually think even the manual testing that I’ve been doing has benefited from this process because I have to really think very carefully about how I’m driving eclipse through various scenarios in order to be able to produce these tests.

TO: I notice that sometimes the bugs that people find in a plug-in for eclipse have nothing to do with the plug-in itself. Are you testing different distributions of eclipse; like different combinations of plug-ins?

RS: I am not doing so much of that yet. I’ve just got started about a month ago, but it’s on the road map. So, I plan to have these automated tests— Right now they’re just running in Eclipse 3.4 and they’re running essentially with all the of the required plug-ins that M2Eclipse could ever never need: WTP, Subclipse and all kinds of things like that. But yes, it’s on the road map for maybe in a month or two, I’m going to start expanding the set of platforms and have different sets of plug-ins and different combinations of plug-ins. Certainly Eclipse 3.3, possibly Eclipse 3.2, IBM RAD and of course get running on different OS; These are all on the road map. And, once I’ve got the automated test running, I mean this is the advantage of the automation right, I can really easily expand it to be able to get good quality testing on all these different platforms.

TO: Well, thanks for taking the time to talk to us. And, we’ll check with you in a few months to see what progress you’ve made.

RS: All right, sounds good.