In the third part of this series I will talk about how integration tests (and unit tests) are done on Flexmojos. If you haven't been following along, you should read the first and second posts of this series.
Let's start from the beginning, what is Flexmojos?
Flexmojos is a maven plugin for Adobe Flex development. You can get acquainted with it here.
Some Background
When I started Flexmojos I just wanted a decent maven plugin to build my project. I never considered making it public or turning it into a open source project. At the time, I just wanted something for me and for me alone. As I continued to work on my own solution, I realized more people had problems building Flex projects using Maven. So I decided to share my project. This is how the Flexmojos project began, and once the project had some momentum we stared running into challenges that required rigorous integration testing.
After a few releases, Flexmojos started to present backward compatibility issues. Something that worked on 1.0-alpha-2 suddenly stopped working on 1.0-alpha-3. And this started to happen more often during new features inclusion. The obvious fix for that was to create a bunch of unit tests. But mojos aren't exactly the easiest things to unit test. So I had to keep testing manually, at that point Maven and integration tests had too much complexity for my limited Maven API knowledge. So I stopped including new features and concentrated on getting Flexmojos stable. Eventually I managed to get 1.0 release out.
I saw how much time, releases and code I needed to get 1.0 stable and I wanted to get 2.0 stable w/o that much work. But how? Creating a good set of automated tests. But again, how to test mojos? At this point, there wasn't much information about testing Maven plugins. Googling here and poking around there I found maven-verifier.
maven-verifier tests are run using JUnit or TestNG, and provide a simple class allowing you to launch Maven and assert on its log file and built artifacts. It also provides a ResourceExtractor, which extracts a Maven project from your src/test/resources directory into a temporary working directory where you can do tricky stuff with it.
That was perfect for my needs. Using maven-verifier I found an easy way to test Flexmojos. In fact, I found a very good way to test Flexmojos, however when I started to write more complex tests it proved to be not as easy as I expected. We'll talk more about that later. Using maven verifier is very cool too because it allowed me to write integration tests using TestNG, so I could use asserts, inherit, annotations and all the stuff available for unit testing that I was used to using.
How are Flexmojos test organized
Flexmojos only has integration tests. Unit testing maven plugins is not easy and Flexmojos wasn't designed in a particularly modular way. If it were, it would have more unit tests.
Since Flexmojos is a Maven plugin I needed it installed into a Maven repository to run the integration tests, for that reason Flexmojos has all integration tests grouped into a single project, called flexmojos-test-harness.
The flexmojos-test-harness creates a complete environment to run the integration tests:
- Download and unpack Maven 2.0.9 from central;
- Overwrite Maven settings.xml with one that contains extra repositories ( check it here ), with all required artifacts;
- Install Flexmojos artifacts into a fake local repository; this allow us to
- Instrument Flexmojos artifacts using Emma;
- Add emma.jar to Maven libs directory (required to enable test coverage).
This project sets up everything necessary to run Flexmojos (install maven), it sets up Emma (instrumenting and add emma.jar) and makes sure the right version of the artifacts are used during tests (filling maven repo).
Motivation for Flexmojos Integration Testing
Q. Why download and unpack maven from central, instead of using the maven already present on machine?
A. First and most important reason: Instrumented classes need emma.jar on classpath. If not present, it will throw ClassDefNotFoundExceptions. So I can tamper with this Maven instance created by Flexmojos adding emma.jar. And that is a good reason to ensure that Flexmojos can manage a separate Maven installation that is isolated from all of the variation of developers who might be testing Flexmojos. At any moment I can set another version to be sure that Flexmojos is still working.
Q. Why overwrite settings.xml?
A. First of all, it was was necessary to add the Flexmojos repository, so that maven could download the flex compiler from the server. This modified settings.xml does include a reference to an isolated, local Maven repository, so that anyone running the integration tests will retrieve artifacts already available on current machine, this dramatically reduces the time required to run the integration tests. Also notice that this settings.xml blocks all snapshots from being downloaded, that is nice because it prevents instrumented Flexmojos being overwritten with non-instrumented versions from maven central.
Q. Why put the artifacts on a fake local repository?
A. That is about isolation... if I used the machine's local repository, Flexmojos artifacts would get instrumented by Emma. But that would break Flexmojos. It's required to add emma.jar on Maven classpath just to run instrumented Flexmojos. But shipping instrumented Flexmojos gives users no advantage, quite the opposite, it would generate coverage reports every time, wasting CPU, disk and memory usage. Another reason is that would break the GPG signature for Flexmojos artifacts. So, creating a isolated local repository allowed me to run instrumented tests without messing with built artifacts.
How to Configure Integration Tests
Q. How do you download and unpack maven from central?
A. To copy and unpack artifacts is used the maven-dependency-plugin. This plugin "provides the capability to manipulate artifacts. It can copy and/or unpack artifacts from local or remote repositories to a specified location."
<execution>
<id>extract-maven</id>
<phase>generate-test-resources</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.apache.maven</groupId>
<artifactId>apache-maven</artifactId>
<version>${maven.version}</version>
<type>tar.gz</type>
<classifier>bin</classifier>
</artifactItem>
</artifactItems>
<outputDirectory>${project.build.directory}</outputDirectory>
</configuration>
</execution>
Read more about unpack at plugin site. Unpack goal documentation and more examples.
Q. How does settings.xml know where to find the local Maven repository?
A. The settings.xml used on tests has some variables, namely ${localRepository.basedir}, that links to default repository location, and ${maven.repo.local}, that has the location when default-value is overwritten. This variables are interpolated using maven-resource-plugin.
To enabled the variables interpolation is only necessary to declare the resources with the filtering option set to true, like this:
<build>
<testResources>
<testResource>
<directory>src/test/resources</directory>
<filtering>true</filtering>
<includes>
<include>settings.xml</include>
</includes>
<targetPath>${fake-maven}/conf</targetPath>
</testResource>
...
</testResources>
</build>
That was not declared on maven-resource-plugin configuration, but on build section, under resources or testResources (Flexmojos ITs case).
Q. How does the resources directory get interpolated? And why isn't maven-resource-plugin used?
A. That is a performance question.... maven-resource-plugin would always interpolate all files, and there are several files under resources, that alone was taking 30 seconds to a minute. But when running individual tests, not all files are required, so Flexmojos implemented its own interpolation, which runs before the test. Take a look at AbstractFlexmojosTests.getProject()
Code coverage
For the last part of this blog, I show how Flexmojos generates a test coverage report. Flexmojos uses emma4it-maven-plugin (introduced on part 2 of this blog series).
Emma4it has a goal to merge all available Emma binaries into a single one.
<plugin>
<groupId>org.sonatype.maven.plugin</groupId>
<artifactId>emma4it-maven-plugin</artifactId>
<version>1.3</version>
<executions>
<execution>
<id>merge</id>
<phase>verify</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<searchPath>${basedir}</searchPath>
</configuration>
</execution>
...
</executions>
</plugin>
This way, it is possible to combine coverage results of unit and integration tests, it becomes more relevant on Flexmojos because we have being improving UT tests coverage, especially in some new components that are very easy to unit-test (like the new generation and testing API).
In part 4 we will talk a bit about ITs on Nexus. Nexus does have a very particular setup where the approach used for Flexmojos proved to be inefficient.
See you in the next post.
