GEF3D goes Git, Maven/Tycho, and Hudson

or
Standing on the shoulders of giants

Abstract: Setting up a continuous integration build based on Git, Maven/Tycho, and Hudson is surprisingly easy. I assume that this is no real news for most readers. However, I ways very skeptical about that, especially because of all the project dependencies. So, this posting is meant for readers hesitating to set up an automatic build system because they think that it would be too complicated, just as I thought until .. well, until now.

One of the things on my todo list of 2011 was to set up a continuous integration build for GEF3D. I did set up such a system several years ago, using some XML files describing a module and its dependencies, and an XSL Transformation generating Ant scripts based on these module descriptions. In order to run nightly builds, cron was used -- yeah, good old times. I remember trying Maven back then, and I was so much disappointed that I wrote my own tools. Due to this experience, I had lots of respect setting up a build system for GEF3D.

In the beginning, Miles helped me to set up a Buckminster based build. He had some experience with this tool because his build system for the AMP project is based on Buckminster as well. Since Miles has switched to Git, he suggested the GEF3D project to switch as well. This would simplify the set up, as we only have to deal with one version control system. I filed a bug report for moving GEF3D from SVN to Git. Since I had read in the Git Migration guide that when moving to Git it would also be a good opportunity to refactor the structure of the project, we decided to introduce folders separating plugins, features, examples, and so on. Somehow my report got forgotten... As it was very complicated to configure the Buckminster build, and due to other things, I also didn't push it.

Migrate to Git

So, as the year is reaching its end, I decided to give it another try -- despite all these unknown tools such as Buckminster, Git, and Hudson. I started with Git. The first giants I have to give kudos to are the Eclipse Git guys, and Stefan, who wrote a nice blog posting about how he moved CDO to Git. The basic idea of Stefan's approach is really simple:

1. Create a Git repository locally and use svn2git to migrate the code. Note that there exist different svn2git tools. I used https://github.com/nirvdrum/svn2git, while Stefan used https://github.com/schwern/svn2git.git! The import is a single command, in my case
   

   svn2git https://dev.eclipse.org/svnroot/technology/org.eclipse.gef3d --authors users.txt

   users.txt is a list of the committers, with username = prename surname <email>. Very simple, indeed. Since GEF3D is not that big, the migration requires only 10 minutes, and the created Git repository has a size about 9 MB.
   
2. Then I refactored the project structure, simply done via command line: git mv is your friend here. (Kristian helped me with little problems, as I'm a Git rookie as well ;-) ).
   
3. Commit all changes to the local Git repository, pack the ".git" repository, upload it to developer.eclipse.org and let the webmaster unpack it at git.eclipse.org.
   

Stefan had to do much more work, as he had to create four repositories by extracting things from the existing CDO project, which is much bigger than the GEF3D project, of course. You will find the GEF3D git repository at git://git.eclipse.org/gitroot/gef3d/org.eclipse.gef3d.git

Maven/Tycho

During my research about some Git questions, I stumbled over the GMF tooling project (well, I know that project for a long time, but I didn't cared about the "internal" structure). It uses the same project structure as the GEF3D team had decided to be used for GEF3D, and it uses Tycho. Although I had my (bad) experiences with Maven, I gave Tycho a try. And again, I was very much surprised at how easy it is to set up a complete build system with Maven/Tycho. I used the GMF tooling poms as a template, and after a couple of minutes (not hours!) I had a build system which could build most parts of GEF3D. Kudos to the Tycho giant team! Besides some special Eclipse packaging things, e.g., telling Maven how to handle an Eclipse plugin (and probably much more hidden under the hud), one really nice feature is the ability to use P2 repositories as Maven repositories. GEF3D has dependencies to GEF, GMF and EMF. To resolve these dependencies, I only had to define a single repository:

<repository>
    <id>Galileo</id>
    <layout>p2</layout>
    <url>http://download.eclipse.org/releases/galileo</url>
</repository>  

This is so cool!
Unfortunately, LWJGL (the OpenGL wrapper library used by GEF3D) does not provide a p2 repository, but an old style update site instead. That is, its update site provides only the site.xml file, and no p2 metadata. By accident, I'm the guy maintaining the LWJGL update site build script. It is an Ant based build, added to the overall LWJGL build system. Since LWJGL does not use Maven, and no Eclipse at all, I could not rely on the Tycho or p2 publisher to build the p2 metadata. In order to keep the overhead for the LWJGL project low, I wrote my own Ant task creating the missing p2 metadata for an old style update site. If you ever need something like this, the code of this Ant task is available from the LWJGL SVN -- it is a plain Java Ant task without any Eclipse dependencies. It can also be used standalone in order to create the content.xml/jar and artifact.xml/jar from a bunch of plugins, features, and a site.xml. At the moment, the official LWJGL update site is not updated yet and I'm using a personal mirror for GEF3D. But Brian, who maintaines the LWJGL update site, will probably update it soon.

Remark: The LWJGL update site provides the LWJGL plugin, which basically bundles LWJGL as an Eclipse plugin. Additionally, source and documentation bundles are provided, and a tool bundle with an information view (showing the OpenGL settings of your graphics card), and a library for easily configuring standalone LWJGL apps. And thank you very much, Brian, for maintaining the update site at lwjgl.org!

I also had to fight with Maven and Tycho to get the source and documentation bundles build (seems as if some tiny things were changed when Tycho moved to Eclipse, so you have to compare the settings in the documentation with actual poms). Thanks to Chris' Minerva project, and the GMF tooling project, I could solve these issues. The Minerva project also demonstrates how to configure tests (simple JUnit tests, and plugin tests with SWT bot) -- and it was easy to configure this for GEF3D as well. Although I'm still curious about Buckminster, I was really surprised how well Maven/Tycho works. And since it is already working, I won't switch to Buckminster. However, I could imagine that if you have special requirements, it would be easier to configure that with Buckminster. I'm currently tutoring a student setting up a Buckminster build system for a research project -- I'm looking forward to comparing the results.

Hudson

Eventually, I had to set up a Hudson job for the GEF3D build. Miles had already prepared that job, and I only had to configure GEF3D's git repository, and Maven. I tried this first on a locally installed Hudson ("installed" sounds like a lot of work, actually it is only downloading a war and start Hudson via java -jar hudson.war). Again, setting up a Hudson job for a Maven based build system is really easy. All you have to do is to specify your code repository (git in my case), and the parameters passed to Maven (which usually is "clean install"). That's it. Well, at the moment I have some problems building the javadoc API reference, as the Javadoc at hudson.eclipse.org apparently behaves a little bit different as the Javadoc on my local machine. But at the moment I can ignore that problem, and I'm sure this can be solved soon.

Summary

I was really surprised at how easy it was to migrate from SVN to Git, to set up a build system with Maven/Tycho, and to configure the job with Hudson. As a matter of fact it was that easy, that I probably will use Git, Maven/Tycho, and Hudson for new projects right from the start (I know, that's what all the agile guys tell you to do... but I didn't dare to actually do it). I was particularly surprised at how good Maven works with Eclipse thanks to Tycho -- the Tycho team did a really great job here! According to Leonard, there's a crack in everyting... and I'm a little bit nervous about configuring special requirements with Maven, such as integrating code transformation tools. I've found some blog posts about getting Xtext/Xtend work with Maven -- seems as if there are reasons to being nervous... but that's how the light comes in :-D