The Butler Dispute, Round 2

Nearly 4 month have passed by since the renaming of Hudson project to Jenkins, which marks the climax of the dispute between the old Hudson developers and the guys from Oracle and Sonatype.

Jenkins has made a great job since then, delivering 15 releases in a weekly schedule. Build number 1.400 was hit in March, wich is not a particular significant release but shows how well things go on. The way from Hudson to Jenkins is as easy as it could be, and it seems like many users are going it.

Indeed there are a lot of reasons why to choose Jenkins over Hudson, just to name a few:

• Support by the fabulous Hudson core development team – with Kohsuke Kawaguchi, the creator of Hudson, and other brave guys.
• Strong community activity – measured in figures like commit counts and mailing list traffic, see this post for some numbers.
• Most of the plugins moved over to Jenkins – 5 of the top 5 and 19 of the top 25 plugins continue primary development with Jenkins, see here for some statistics.
• High quality and regular releases – the weekly schedule led to 15 high quality releases, each of them providing a couple of bug fixes and new features (see changelog). Moreover, a few weeks ago, Jenkins governance board proposed to start another release line for most stable baselines with a 3-months schedule.

Even the Hudson board seems to have observed that Jenkins outperforms Hudson in many ways, at least they are thinking about "how to make it more attractive for plug-in developers to support both Hudson and Jenkins" (see this message on Hudson-Dev list). The author's perception is that "Hudson also appears to be slowing down development wise" and "another place where Hudson appears to be slowing down, is when you compare changelogs". Some of the ideas deal with copying approaches that are working fine for the Jenkins project.

Hence, it seems Jenkins is the winner of the battle and has in fact benefited from the fork.... until today.

Because today, Oracle submitted a proposal to move Hudson to the Eclipse Foundation. This is, well, somewhat astonishing since that means Oracle will lose both control and the Hudson trademarks – which was the main background of the original dispute with the community.

As part of the proposal, other big players have announced support for the project, including IBM, VMware, Tasktop and Intuit. That means, moving the Hudson project to Eclipse will for sure result in higher attention and more resources (developers).

Does this change anything? Will Jenkins be the unlucky loser, after all? I don't think so. The heavens didn't really smile on Hudson since the fork (kind of bad karma) and I don't see why the move to Eclipse should change that. It's all about people, not code.

Moreover, Jenkins has been invited by Sonatype to reunite with Hudson. But... why should they do that? Jenkins is a vibrant project today, so what is the benefit? Also, there have been some deep disappointments on personal level that are not forgotten yet.

It's going to be interesting!

Read More

The Way From Hudson To Jenkins

Some time has gone by since the Hudson/Jenkins fork... and there has been even more talk in the community. However, slowly the dust settles, everybody is getting back to business. And finally, we decided to switch from Hudson to Jenkins! This is about why and how.

Why move to Jenkins?

But wait: who has forked, anyways? Is it Jenkins that forked Hudson, or is it Hudson that did the fork of Jenkins? There is some evidence that the community just did a rename of the project (due to trademark conflicts), and after that Oracle forked Jenkins, using the Hudson name they claim holding the trademark on.

You may think this question is a purely theoretical one, but actually it's not. I'll have to legitimate the decision to move to Jenkins to my stakeholders, and using a fork would be a "smell". Project forks are usually not as good as the "original", are possibly done out of selfish reasons, are considered to harm the community etc. Hence, not moving to a fork but instead following the "real" project is a good reason for the move to Jenkins.

An even better one is "project vibrancy", that is the pace of development and level of support provided by the community. This is usually measured by indicators such as the number of commits, the mailing list traffic, the quantity, quality and regularity of releases etc. See this post for such an analysis on commit counts and mailing lists post counts. This is more than four weeks old now and covers not more than two weeks, but nevertheless the result is obvious: Jenkins moves much faster than Hudson does, and community is much more agile. This is confirmed by following the dev mailing lists of both: for Hudson, most of the relevant posts are by either Oracle or Sonatype engineers – seems the Hudson community has become pretty small... Moreover, as this post shows, most of the top plugins will continue primary development under Jenkins.

Last not least, I really respect Kohsuke Kawaguchi (the original creator of Hudson) and what he has done for us. I feel ashamed by how Oracle is dealing with him and the rest of the core team, that's why I have a strong tendency to follow the "good guys" with Jenkins.

As I blogged before, Maven integration is probably one of the most important features of any CI server (at least for me). I guess Sonatype is doing better with Maven integration – it's "The Maven Company", right? – and they are working with Oracle on Hudson. At least, they are putting huge efforts into rock-solid integration. However, after having seen a Sonatype Webinar about their plans with Hudson, I'm not that convinced any more. Current features looked a bit awkward and also does the GWT based UI they are using. So, from my point of view, this point is not yet decided.

Counting it all together, there are some really good reasons to move from Hudson to Jenkins, so we did.

How to upgrade

Now... how do you actually migrate from Hudson to Jenkins? Well, it couldn't be easier. There is a Wiki page about Upgrading from Hudson to Jenkins. To make it short, the involved steps are:

1. Backup your current installation – just for the good feeling.


2. Change Update Site: In your Hudson, go to Manage Hudson > Plugin Management > Advanced > Update Site and enter "http://updates.jenkins-ci.org/update-center.json" as URL for Jenkins update site.


3. Choose to upgrade automatically on Manage Hudson page, just as you did so many times to update Hudson. This will download the new JAR.


4. Restart Hudson, eh, Jenkins.... and there it is!

That's it. Took less than 5 min! Hudson indeed is a drop-in replacement, so you usually do not have to change anything (environment variables, system properties, start scripts, job configuration etc).

Well, there is only one thing: the name of the WAR file is still hudson.war! Is Oracle aware of this? ;-)

Read More

The Butler Dispute

I thought it would be time to resurrect my blog, after not having posted for a couple of months. That was mainly because I have been really busy with some interesting stuff I should post about in the future, like Xtext upgrade...

Oracle vs. the Community

But today, I just bumped the ongoing dispute between Hudson and Jenkins guys. We are using Hudson since around 2008, coming from Cruise Control. We really liked the web interface, being able to setup everything by just using your browser. Of course, the features also have been impressive since then. Setting up a build farm is just fun with Hudson.

You probably know that there has been a fork of Hudson which is named Jenkins (others say Hudson has been renamed to Jenkins and then forked into Hudson). This all started with the Hudson team being unhappy with the infrastructure provided by java.net (which is driven by Oracle since Sun acquisition), due to its poor reliability. The community talked about moving parts of the project to other servers, and first candiate was issue tracking. Suddenly, the project is locked due to the migration of java.net projects to new Kenai infrastructure, which was announced by Oracle but somehow missed by the project owners. Frustrated by the migration, the community decides to move code to GitHub and mailing list to Google Groups. See "Who's driving this thing?" for the facts.

This is the point where Oracle steps in, claiming to have a trademark on the name. If the project decides to move, it must be using another name: "Because it is open source, we can't stop anybody from forking it. We do however own the trademark to the name so you cannot use the name outside of the core community. We acquired that as part of Sun." (BTW, that might not be true after all). Later he stated that "the final decision of what to do w.r.t. infrastructure belongs to Oracle".

Guess what: this really concerned the community. There have been some talks between key community members and Oracle representives, in an attempt to agree on a "proposal for a stable structure and arrangement" which later would be proposed to the community. But, with no success. That finally led to the decision of the community to move to GitHub and at the same time rename the project to another butler's name: "Jenkins".

Of course, Oracle's view on the subject is a bit different...

Welcome Jenkins!

Well, so now you have the choice: use Hudson, or use Jenkins. You know, competition is usually a good thing, so let the race begin. The majority of the community seems to have made the switch to Jenkins (given the blogs and mailing list traffic). This is because Oracle's behavior is not quite understood and does not cast a positive light on their comprehension of Oracle's role in the Hudson community.

However, Oracle is putting enormous resources (people and hardware) into the Hudson project. And what's even more important, Sonatype is helping to drive Hudson to the next level. Sonatype? Right, that's the company behind Maven and all the great Maven tools like Nexus and m2eclipse.

Maven Support – the Killer Feature?

One of the most important features (for me, but also for possibly the majority of other users) is Maven 3 support. Sure, Hudson/Jenkins already support Maven 3 since version 1.392 (end of 2010, see changelog). But hey, Sonatype entered the scene, and they will surely do better.

Sonatype, too, have put some full-time engineers into the Hudson project, making sure that "Hudson users can look forward to a long, bright future". See this or this post. Sonatype in the end of the day wants to earn money with Hudson (and Maven), so I expect to see outstanding features related to Maven 3 support, Eclipse integration and workflow extensions for Hudson. See here for some of their ideas.

Well, this really makes a thrilling game. I honestly appreciate what Kohsuke Kawaguchi and others have built up with Hudson from the ground up, and would like to see them win on the "evil company that pushed them out of the project". And by the way, Sonatype seems to be in good companion when talking about being evil – they removed the oldest commiter of m2eclipse from the project a year ago.

So, is this again the good vs. evil story? I don't know. In the end, both projects will have their users. And they will learn and benefit from each other. So let's wait and see.... Time will tell. It's going to be an interesting year, though!

Read More

Fix Foreign Code

Well, finally, I'm back! I have been busy working on-site for a customer of my company, helping to fix their project and increase quality to successfully conduct the rollout. Additionally, I spent my evenings working as a release manager and keeper of the Maven based infrastructure for several projects developed in-house. So this was more than a fulltime job and unfortunately no time was left to read or write blog posts :-(

However, that project assignment is nearly over now and I intend to write more regularly now about my findings, trials and tribulations.

Foreign Code Dilemma

My main task when working for our customer was to fix bugs and improve quality of their application, which was nearly completely implemented with respect to use cases and business requirements. This is a situation that might be known to most developers: you are thrown into a project you don't know much of, lots of source code is already implemented, quality is, well, varying, and some important milestone or release date is right ahead. This is what I call the Foreign Code Dilemma.

What do you do to quickly get up to speed and rescue the project? Well, there are some things that I find quite useful in situations like this. In no particular order...

Introduce Continuous Integration

It should be common sense these days that Continuous Integration (CI) is able to improve software quality and reduce integration issues as well as overall risks. CI is a software development practice where changes are integrated frequently – usually at least daily – and the result is verified by an automated build and test to detect issues as quickly as possible. The distinguished article about Continuous Integration by Martin Fowler is a must-read.

Fortunately, the customer's project already provided automated Ant build scripts to checkout, build and test the software. Moreover, they were running on a Cruise Control server each night, so we were quite close.

The first thing I did was to move to Hudson, the best integration server available today (if you'd ask me). The transition was quite smooth and done within a few hours, including setting up a brand new build server. If you're still using Cruise Control, you really should consider to move over to Hudson... I think I should post about the cool distribution feature of Hudson soon.

One issue with the project was the build time: a full build takes 3-4 hours, mainly due to long-running unit and selenium test cases. Of course, this inhibits doing real CI. All we could do for now was to split up the build into the four main tasks, creating a Hudson job for each of them: (1) checkout & compile & package, (2) static code checks, (3) unit tests, (4) selenium tests. Since (1) and (2) are running rather quick (about 10 min) those jobs qualify for CI builds. This is not perfect but still better than doing no CI at all.

Introduce Test Cases

Test cases are an essential part of a software development project these days, and I always consider a tasks not being finished unless there are test cases ensuring that the functionality is implemented correctly. I'm sure you agree ;-)

The project I was working on had lots of JUnit test cases, as well as hundreds of Selenium tests checking the web application in the browser. That's not bad, really. Nevertheless, there were two issues:

• The number of test cases does not tell anything about the test coverage. For example, the Selenium tests all did test a "happy day" scenario, moving through the wizard pages of the web application straight from the first to the end. But does it still work, for instance, if you step to the forth page, choose some options on that page, step back two pages, change an option, and go to the forth page again? Nobody tested.


• Selenium web tests are slow, which is no surprise taken the fact that the tests are running in a browser and need to connect to the deployed web application. In my project, the full test suite took more than 3 hours to run... What's even worse is that some of the JUnit tests have not been designed as unit tests, i.e. they required a full service stack to run successfully, as such being more an integration than a unit test. As a consequence, those tests require to startup all services which takes a lot of time.

Thus, the task for this project actually was not to introduce, but to improve unit tests: increase code coverage and separate unit from integration tests. This way, unit tests can be run within CI builds, providing a quick result for the quality of committed code.

Introduce Code Metrics

When more than a few people are working on a project, establishing a coding standard is usually a rewarding idea. It helps you to be comfortable with the sources of anybody else from your team, and when doing code comparison you don't see differences all the time that are just caused by reformatting, hiding the significant changes.

If you have defined such coding standards, you need to check them. Checkstyle is the tool of choice. Here is what you should do:

• Define a Checkstyle configuration to be used for your project. Discuss the rules with developers and stakeholders.
• Run Checkstyle with your CI and/or nightly builds to create a report, including a list of violations for defined rules.
• Establish Checkstyle within your IDE of choice to provide immediate feedback to the developers before they commit.
• Define which exceptions to the rule are acceptable (should not be more than a dozen or so) and suppress them permanently, using Checkstyle suppression filters.
• Get rid of all remaining violations, which might take a few days of effort. Still, this investment will pay off.
• Once the number of Checkstyle violations is "small" (meaning less than 10, ideally zero), make sure it remains small.
• Establish a team culture where committing code with Checkstyle violations is anything else but cool.

That works quite well in my experience. For the mentioned project, we already had common Eclipse formatting settings, but Checkstyle helped to further improve the code and people adopted it right from the start.

The Debugger is Your Best Friend

When you have to fix bugs in code you have never seen before, use the debugger as much as possible. To find the hot spot, you usually don't have to read or understand the whole class or even hierarchies of classes. Thus, it'll save you a lot of time when you don't start with code reviews but use the debugger to find the piece of code to blame.

BTW, the same applies to the look and feel of web applications. Instead of consulting lots of layout code and stylesheets, use browser tools like Firebug to debug pages, styles and JavaScript code (including Ajax requests) right in the displayed page.

Of course, this approach is not appropriate when fixing larger design issues...

Don't Be Shy!

When using this toolset, you shouldn't be shy. If you think some code needs refactoring, do so – maybe not a week before going live, but you get the point. The CI build should give you immediate feedback if the change could be integrated, and the tests will tell you if everything still works. Take your chance to improve the code. If your change anyhow is causing an issue, fix it, add another test and don't be discouraged!

Read More

Concurrent Builds with Hudson

Multiple Build Executors

We are using the Hudson Continuous Integration Server for our integration builds and are quite happy with it. It is fast, stable, feature-rich, extensible, well integrated with Maven and has an appealing user interface.

One of the nice features that we are using regularly is the Build Executor setting that allows you to specify the number of simultaneous builds. This is useful to increase throughput of Hudson on multi-core processor systems, where the number of executors should (at least) match the number of available cores.

However, Maven isn't really designed for running multiple instances simultaneously since the local respository isn't multi-process safe. The chance for conflicts seems small (multiple processes must access the same dependency at the same time, at least one of them writing). However, in praxis, we encounter this type of concurrency issue at least once a day now, which is starting to hurt us! The build is failing with a message like this:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

GET request of: some/group/some-artifact-1.2.3-SNAPSHOT.jar from my-repo failed
  some.group:some-artifact:jar:1.2.3-SNAPSHOT
...

Caused by I/O exception: ...some-artifact-1.2.3-SNAPSHOT.jar.tmp (The requested operation cannot be performed on a file with a user-mapped section open)

or this:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Error copying temporary file to the final destination: Failed to copy full contents from ...some-artifact-1.2.3-SNAPSHOT.jar.tmp to ...\some-artifact-1.2.3-SNAPSHOT.jar

The reason is, the JAR file is locked by another process that is executing some long-lasting test cases, for instance. At the same time, a second build tries to download a new version of this snapshot into the local repository, which is done with the help of the mentioned .tmp file.

Safe Maven Repository

The only way to avoid this type of issue is to use separate local Maven Repositories for each of the processes. You can tell Maven to use a custom local repository location by specifying the localRepository setting in your settings.xml file.

In Hudson, this is even more convenient. There is a checkbox Use private Maven repository in the advanced part of the Build section of Maven projects. Just click that to setup a private local Maven repo for that project. You should consider to do so when you run into the described issue now and then.

Obviously using private repos will increase the total amount of disk space due to caching the same dependencies in multiple places. Additionally, the first build will take significantly more time because everything has to be downloaded once. However, both consequences are well acceptable given the better stability and isolation of projects.

Instead of clicking the Hudson checkbox for all your projects, you should consider to setup the local Maven repo in your settings.xml instead. This has a number of advantages:

• You don't have to setup the option for each and every project, but have it in a central place.
• You can use a common root for all local Maven repos, like d:/maven-repo. This allows you to easily purge all your local repositories from time to time, in order to reduce disk space as well as validate the content (i.e. make sure the build is still running in a clean environment and all required artifacts are in your corporate Maven repository).

For instance, here is what works fine for us:

<localRepository>d:/builds/.m2/${env.JOB_NAME}/repository</localRepository>

This is using a Hudson environment variable (JOB_NAME) to create subfolders for the actual projects aka jobs. See here for a list of available variables.

Oh yes, what I suggest is also encouraged by Brian Fox in his Maven Continuous Integration Best Practices blog post, so you should consider twice to adopt this best practice :o)

Read More