The Butler Dispute, Round 2

Nearly 4 month have passed by since the renaming of Hudson project to Jenkins, which marks the climax of the dispute between the old Hudson developers and the guys from Oracle and Sonatype.

Jenkins has made a great job since then, delivering 15 releases in a weekly schedule. Build number 1.400 was hit in March, wich is not a particular significant release but shows how well things go on. The way from Hudson to Jenkins is as easy as it could be, and it seems like many users are going it.

Indeed there are a lot of reasons why to choose Jenkins over Hudson, just to name a few:

• Support by the fabulous Hudson core development team – with Kohsuke Kawaguchi, the creator of Hudson, and other brave guys.
• Strong community activity – measured in figures like commit counts and mailing list traffic, see this post for some numbers.
• Most of the plugins moved over to Jenkins – 5 of the top 5 and 19 of the top 25 plugins continue primary development with Jenkins, see here for some statistics.
• High quality and regular releases – the weekly schedule led to 15 high quality releases, each of them providing a couple of bug fixes and new features (see changelog). Moreover, a few weeks ago, Jenkins governance board proposed to start another release line for most stable baselines with a 3-months schedule.

Even the Hudson board seems to have observed that Jenkins outperforms Hudson in many ways, at least they are thinking about "how to make it more attractive for plug-in developers to support both Hudson and Jenkins" (see this message on Hudson-Dev list). The author's perception is that "Hudson also appears to be slowing down development wise" and "another place where Hudson appears to be slowing down, is when you compare changelogs". Some of the ideas deal with copying approaches that are working fine for the Jenkins project.

Hence, it seems Jenkins is the winner of the battle and has in fact benefited from the fork.... until today.

Because today, Oracle submitted a proposal to move Hudson to the Eclipse Foundation. This is, well, somewhat astonishing since that means Oracle will lose both control and the Hudson trademarks – which was the main background of the original dispute with the community.

As part of the proposal, other big players have announced support for the project, including IBM, VMware, Tasktop and Intuit. That means, moving the Hudson project to Eclipse will for sure result in higher attention and more resources (developers).

Does this change anything? Will Jenkins be the unlucky loser, after all? I don't think so. The heavens didn't really smile on Hudson since the fork (kind of bad karma) and I don't see why the move to Eclipse should change that. It's all about people, not code.

Moreover, Jenkins has been invited by Sonatype to reunite with Hudson. But... why should they do that? Jenkins is a vibrant project today, so what is the benefit? Also, there have been some deep disappointments on personal level that are not forgotten yet.

It's going to be interesting!

Read More

Jenkins: Pimp It Up!

Some days ago, I started to review what plugins are available for Jenkins, my favorite CI server. I haven't done so for a long time, so I was somewhat surprised to see a full universe of plugins (380+) listed in the Wiki...

There is next to everything you can imagine. Among the plugins I would like to suggest for consideration are these:

• Dependency Graph View Plugin – Shows the dependency graph of the Jenkins projects using graphviz. This greatly helps to keep track of dependencies between all your Jobs.
  
  



• Disk Usage Plugin – This plugin calculates and records disk usage (space for builds and workspace) per project and per build, and can display trend graphs.
  
  



• Global Build Stats Plugin – can be used to gather and display global build result statistics, monitoring over time, and show nice graphics.
  
  



• And, of course, all the static analyzing plugins that scan result files of several static code analysis tools and visualize the results as trend graphs:
  • Checkstyle Plug-in
  • Cobertura Plugin
  • FindBugs Plugin
  • JavaNCSS Plugin
  • PMD Plugin
  • Task Scanner Plugin
  • Warnings Plugin
  
  
  

Have fun!

Read More

DocBook with Maven Issue

We are using DocBook for writing technical documentation for all our projects and in-house frameworks. We are actually quite happy with this approach, especially because we are able to automatically publish the docs in a number of formats, including HTML and PDF. To do so, we use the docbkx-maven-plugin in the project's nightly build.

So, all has been in best order... until I decided to upgrade this docbkx-maven-plugin from version 2.0.8 to current version 2.0.11 (due to some issues we had). After doing so, the document conversion issues an error which breaks the build:

[ERROR] Failed to execute goal com.agilejava.docbkx:docbkx-maven-plugin:2.0.11:generate-pdf (pdf) on project builddoc-ma
ven-plugin: Failed to transform to PDF: org.apache.fop.fo.ValidationException: null:30:723: Error(30/723): fo:table-body
 is missing child elements.
[ERROR] Required Content Model: marker* (table-row+|table-cell+)

Well, this is somewhat unexpected because I didn't change anything but the plugin version, and I don't see any reason it could not work as before. In particular, we are still using the same docbook version in our POM. Here is the relevant snippet:

<plugin>
  <groupId>com.agilejava.docbkx</groupId>
  <artifactId>docbkx-maven-plugin</artifactId>
  <version>2.0.11</version>
  <dependencies>
    <!-- the DocBook XML DTD and catalog files (see http://www.oasis-open.org/docbook) -->
    <dependency>
      <groupId>org.docbook</groupId>
      <artifactId>docbook-xml</artifactId>
      <version>4.4</version>
      <scope>runtime</scope>
    </dependency>
  </dependencies>

  <executions>
    <execution>
      <id>pdf</id>
      <goals>
        <goal>generate-pdf</goal>
      </goals>
      <phase>post-site</phase>
      <configuration>
        ...
      </configuration>
    </execution>
    ...
  </executions>

  <configuration>
    <htmlStylesheet>css/html.css</htmlStylesheet>
    <htmlCustomization>${basedir}/src/doc/xsl/html_chunk_customization.xsl</htmlCustomization>
    <foCustomization>${basedir}/src/doc/xsl/fopdf_customization.xsl</foCustomization>
    ...
  </configuration>
</plugin>

It's important to understand that we are using the advanced customizing capabilities of DocBook, i.e. we customized the stylesheets used for rendering HTML and PDF. The created custom stylesheets contain an import to urn:docbkx:stylesheet, and in the Maven POM the htmlCustomization and foCustomization properties point to those custom stylesheets. This is how it's supposed to be, and this is how it worked all along.

I found out that the error message is correct when building with plugin version greater than 2.0.8, since the for-each element indeed does not return any element which results in an empty fo:table-body. In fact, none of the xsl:value-of in our customized stylesheet returned any value any more....

So here is why: since docbkx-maven-plugin version 2.0.9, the plugin is using namespaced stylesheets. That is, we must use a namespace in our custom stylesheet to be able to select any docbook element! See this or this post for related comments.

Thus, all I have to do is to add the docbook namespace declaration at the top and add the docbook namespace prefix to all references to element names in my customization layer. See highlighted lines in this XSL snippet:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:fo="http://www.w3.org/1999/XSL/Format"
                xmlns:db="http://docbook.org/ns/docbook"
                exclude-result-prefixes="date"
                version="1.0">

  <xsl:template name="book.titlepage.separator">
    <fo:block>
      <fo:table table-layout="fixed" width="163mm">
        ...
        <fo:table-body text-align="left">
          <xsl:for-each select="/db:book/db:bookinfo/db:revhistory/db:revision">
            ...
          </xsl:for-each>
        </fo:table-body>
      </fo:table>
    </fo:block>
  </xsl:template>
  ...
</xsl:stylesheet>

Well, that did the trick – after spending a couple of hours of investigation... I think that issue should be clearly noted with the docbkx-maven-plugin Maven plugin, because in the end it is an incompatability between versions 2.0.8 and 2.0.9. Alas, I did not find this information on the plugin's Changes Report page. At least, nothing that pointed me (not being a DocBook expert) into this direction... :-(

Read More

The Way From Hudson To Jenkins

Some time has gone by since the Hudson/Jenkins fork... and there has been even more talk in the community. However, slowly the dust settles, everybody is getting back to business. And finally, we decided to switch from Hudson to Jenkins! This is about why and how.

Why move to Jenkins?

But wait: who has forked, anyways? Is it Jenkins that forked Hudson, or is it Hudson that did the fork of Jenkins? There is some evidence that the community just did a rename of the project (due to trademark conflicts), and after that Oracle forked Jenkins, using the Hudson name they claim holding the trademark on.

You may think this question is a purely theoretical one, but actually it's not. I'll have to legitimate the decision to move to Jenkins to my stakeholders, and using a fork would be a "smell". Project forks are usually not as good as the "original", are possibly done out of selfish reasons, are considered to harm the community etc. Hence, not moving to a fork but instead following the "real" project is a good reason for the move to Jenkins.

An even better one is "project vibrancy", that is the pace of development and level of support provided by the community. This is usually measured by indicators such as the number of commits, the mailing list traffic, the quantity, quality and regularity of releases etc. See this post for such an analysis on commit counts and mailing lists post counts. This is more than four weeks old now and covers not more than two weeks, but nevertheless the result is obvious: Jenkins moves much faster than Hudson does, and community is much more agile. This is confirmed by following the dev mailing lists of both: for Hudson, most of the relevant posts are by either Oracle or Sonatype engineers – seems the Hudson community has become pretty small... Moreover, as this post shows, most of the top plugins will continue primary development under Jenkins.

Last not least, I really respect Kohsuke Kawaguchi (the original creator of Hudson) and what he has done for us. I feel ashamed by how Oracle is dealing with him and the rest of the core team, that's why I have a strong tendency to follow the "good guys" with Jenkins.

As I blogged before, Maven integration is probably one of the most important features of any CI server (at least for me). I guess Sonatype is doing better with Maven integration – it's "The Maven Company", right? – and they are working with Oracle on Hudson. At least, they are putting huge efforts into rock-solid integration. However, after having seen a Sonatype Webinar about their plans with Hudson, I'm not that convinced any more. Current features looked a bit awkward and also does the GWT based UI they are using. So, from my point of view, this point is not yet decided.

Counting it all together, there are some really good reasons to move from Hudson to Jenkins, so we did.

How to upgrade

Now... how do you actually migrate from Hudson to Jenkins? Well, it couldn't be easier. There is a Wiki page about Upgrading from Hudson to Jenkins. To make it short, the involved steps are:

1. Backup your current installation – just for the good feeling.


2. Change Update Site: In your Hudson, go to Manage Hudson > Plugin Management > Advanced > Update Site and enter "http://updates.jenkins-ci.org/update-center.json" as URL for Jenkins update site.


3. Choose to upgrade automatically on Manage Hudson page, just as you did so many times to update Hudson. This will download the new JAR.


4. Restart Hudson, eh, Jenkins.... and there it is!

That's it. Took less than 5 min! Hudson indeed is a drop-in replacement, so you usually do not have to change anything (environment variables, system properties, start scripts, job configuration etc).

Well, there is only one thing: the name of the WAR file is still hudson.war! Is Oracle aware of this? ;-)

Read More

The Butler Dispute

I thought it would be time to resurrect my blog, after not having posted for a couple of months. That was mainly because I have been really busy with some interesting stuff I should post about in the future, like Xtext upgrade...

Oracle vs. the Community

But today, I just bumped the ongoing dispute between Hudson and Jenkins guys. We are using Hudson since around 2008, coming from Cruise Control. We really liked the web interface, being able to setup everything by just using your browser. Of course, the features also have been impressive since then. Setting up a build farm is just fun with Hudson.

You probably know that there has been a fork of Hudson which is named Jenkins (others say Hudson has been renamed to Jenkins and then forked into Hudson). This all started with the Hudson team being unhappy with the infrastructure provided by java.net (which is driven by Oracle since Sun acquisition), due to its poor reliability. The community talked about moving parts of the project to other servers, and first candiate was issue tracking. Suddenly, the project is locked due to the migration of java.net projects to new Kenai infrastructure, which was announced by Oracle but somehow missed by the project owners. Frustrated by the migration, the community decides to move code to GitHub and mailing list to Google Groups. See "Who's driving this thing?" for the facts.

This is the point where Oracle steps in, claiming to have a trademark on the name. If the project decides to move, it must be using another name: "Because it is open source, we can't stop anybody from forking it. We do however own the trademark to the name so you cannot use the name outside of the core community. We acquired that as part of Sun." (BTW, that might not be true after all). Later he stated that "the final decision of what to do w.r.t. infrastructure belongs to Oracle".

Guess what: this really concerned the community. There have been some talks between key community members and Oracle representives, in an attempt to agree on a "proposal for a stable structure and arrangement" which later would be proposed to the community. But, with no success. That finally led to the decision of the community to move to GitHub and at the same time rename the project to another butler's name: "Jenkins".

Of course, Oracle's view on the subject is a bit different...

Welcome Jenkins!

Well, so now you have the choice: use Hudson, or use Jenkins. You know, competition is usually a good thing, so let the race begin. The majority of the community seems to have made the switch to Jenkins (given the blogs and mailing list traffic). This is because Oracle's behavior is not quite understood and does not cast a positive light on their comprehension of Oracle's role in the Hudson community.

However, Oracle is putting enormous resources (people and hardware) into the Hudson project. And what's even more important, Sonatype is helping to drive Hudson to the next level. Sonatype? Right, that's the company behind Maven and all the great Maven tools like Nexus and m2eclipse.

Maven Support – the Killer Feature?

One of the most important features (for me, but also for possibly the majority of other users) is Maven 3 support. Sure, Hudson/Jenkins already support Maven 3 since version 1.392 (end of 2010, see changelog). But hey, Sonatype entered the scene, and they will surely do better.

Sonatype, too, have put some full-time engineers into the Hudson project, making sure that "Hudson users can look forward to a long, bright future". See this or this post. Sonatype in the end of the day wants to earn money with Hudson (and Maven), so I expect to see outstanding features related to Maven 3 support, Eclipse integration and workflow extensions for Hudson. See here for some of their ideas.

Well, this really makes a thrilling game. I honestly appreciate what Kohsuke Kawaguchi and others have built up with Hudson from the ground up, and would like to see them win on the "evil company that pushed them out of the project". And by the way, Sonatype seems to be in good companion when talking about being evil – they removed the oldest commiter of m2eclipse from the project a year ago.

So, is this again the good vs. evil story? I don't know. In the end, both projects will have their users. And they will learn and benefit from each other. So let's wait and see.... Time will tell. It's going to be an interesting year, though!

Read More

Maven 3 and Plugin Mysteries

You probably know that Maven 3 has landed. Before testing it with our projects, I was curious about the plugins that are defined in the Maven master POM's pluginManagement section and hence are locked-down with respect to their version. Since all projects inherit from this master POM, they will use the respective version of the plugins if not explicitely overwritten anywhere in the project's POM hierarchy.

Maven 3 is a bit more strict concerning automatic version resolution of invoked plugins. Other than Maven 2, it will always use the latest release (i.e. non-SNAPSHOT) version of a plugin if there was no explicit version specified in the POM or on the command line. Moreover, it will issue a warning when missing plugin versions are detected "to encourage the addition of plugin versions to the POM or one of its parent POMs". This is to increase reproducability of builds.

Thus, in Maven 3 the desired build stability is ensured by urging the POM author to give explicit plugin versions, and doesn't any more rely on a full list of plugins (with versions) defined in the master POM. That's why I expected to find a small or even empty pluginManagement section. Well, let's see.

To find out what's in the pluginManagement of master POM, you just have to create a minimal POM and show the effective POM (that results from the application of interpolation and inheritance, including master POM and active profiles) by calling help:effective-pom for this simple project.

So, what do we get? The following list shows the plugin versions that are defined in the Maven 2.2.1 master POM, the Maven 3 master POM, as well as the most recent version of those plugins.

Well, we can see some interesting details here:

• The number of plugins defined in the master POMs pluginManagement section is drastically less for Maven 3 than for Maven 2.2.1 – that's what we expected. However, there are still a few.


• Which pluings are listed and which are not? It seems like the plugins for the most basic lifecycle phases (like clean, install, deploy) are predefined, but others are not (like compile or jar). Is there any policy?


• What is really odd: for some of the plugins that are predefined, there is a newer version available than is listed in the Maven 3 master POM (colored red). Why could that be? I have not checked, but Maven 3 is out for a few days now, so I suspect for most of those plugins the new version has been available before. Is that intentionally? Are the new versions not considered "good" or "stable" by the Maven guys? Or did they just forgot to upgrade? Or did not found it important in any kind?


• Another thing I can't explain: when you look on the Maven 3 Project Plugin Management site, there are listed a lot more plugins, and some are even of other version than what we got by showing the effective POM for a minimal project POM. How could this be? I have no clue...

In a previous post, I have listed the plugins predefined by Maven 3.0-alpha5. Interestingly, there have been a lot more of them (like for Maven 2.2.1), but the "stale version" question was the same...

Read More

World of Versioning

Today, we had a discussion on how to name a hotfix release of our framework product, built with Maven (you knew I'm a fan of Maven, didn't you?). It's a very basic question, but still an interesting one and it opens a whole universe of ideas, opinions and rules...

The previous versions of our product had been named like this:

1.3.0. 1.3.1 ... 1.4.0, 1.4.1, ... 1.5.0, 1.5.1, 1.5.2, ... 1.5.6

They all are based on a release plan and contain bugfixes as well as improvements and new features. For each of those versions, we have written release notes and built a site.

Now, what do we do when there is the need to release a bugfix version of a regular release we built a few days ago? There are some options:

1. 1.5.7 – i.e. increment last number; however, this doesn't seem to fit well because the bugfix release is of another character than standard releases
2. 1.5.6.1 – i.e. add an additional numerical identifier
3. 1.5.6.a – i.e. add another non-numerical identifier
4. 1.5.6-patch1 – i.e. add another qualifier describing it's actually a patch release

When searching the Net for version number rules in the Maven world, you'll stumble upon the DefaultArtifactVersion class in the core of Maven which expects that version numbers will follow a specific format:

<MajorVersion [> . <MinorVersion [> . <IncrementalVersion ] ] [> - <BuildNumber | Qualifier ]>

Where MajorVersion, MinorVersion, IncrementalVersion and BuildNumber are all numeric and Qualifier is a string. If your version number does not match this format, then the entire version number is treated as being the Qualifier (see Versions Maven Plugin).

This means, options 1 and 4 of above would be a viable alternative in the Maven world. However, note that there is some discussion about this Maven schema. It suffers from inconsistent/unintuitive parsing, lexically sorting of qualifiers and some other flaws. This would yield to unexpected comparison results especially when using Maven SNAPSHOT versions. The Proposal given on that page seems to be integrated with Maven 3.

Actually, we wouldn't have this discussion if the third level would not be named Incremental version in Maven world, but rather bugfix version or patch version. There is a Semantic Versioning Specification (SemVer) that recommends this version schema:

A normal version number MUST take the form X.Y.Z where X, Y, and Z are integers. X is the major version, Y is the minor version, and Z is the patch version. Each element MUST increase numerically. For instance: 1.9.0 < 1.10.0 < 1.11.0.

There are some rules describing when to increase which part. The main idea is to use the first numerical (major version) to indicate backwards incompatible changes to the public API, and in contrast the last numerical (patch version) suggests that only backwards compatible bug fixes have been introduced.

This SemVer schema is fully compatible with Maven (regardless of SNAPSHOT versions). If we had used this, we would probably have ended up in a "higher" version number like 5.4.0, but now the upcoming patch would have the version number 5.4.1 without any consideration.

By the way, a lot of public recommendations for software versioning follow this <major>.<minor>.<patch> schema. See this question and Wikipedia for more information on Software Versioning.

So. What do we do now? We'll release a version 1.5.6-patch1 for the patch, but think about changing our versioning according to SemVer, i.e. to upgrade the major number when introducing incompatible changes, and the minor number in most other cases.

Read More