Setup a Github Triggered Build Machine for an Eclipse Project

Disclaimer 1: This blog post literally is a "web log", i.e., it is my log about setting up a Jenkins machine with a job that is triggered on a Github pull request. A lot of parts have been described elsewhere, and I link to the sources I used here. I also know that nowadays (e.g., new Eclipse build infrastructure) you usually do that via docker -- but then you need to configure docker, in which case you will need the same knowledge. However, since this describes a full setup for building an Eclipse project via Jenkins, it might be interesting to others as well.

Disclaimer 2: Setting up a continuous integration pipeline (which we will do here) is a kind of DevOps job. One problem with devops is that, as in my case, developers often do not have that much operations knowledge. So if some operator finds any silly things here, please leave me a comment -- I'm eager to learn.

Motivation

Why did I need a Github triggered build in the first place? I'm working as a professor at the Hamburg University of Applied Sciences, teaching programming, software engineering and computer science stuff in general. I am currently teaching a special class at the university: the “Eclipse N4JS project”. In this kind of class, which students in the 5th semester are supposed to attend, the students are expected to do a more real project. These projects are derived from research, e.g. cyber systems or autonomous cars. In my case, they have to work on a really real project: Eclipse N4JS. The students become a Scrum team (9 people), and they are working on smaller yet real-world tasks.

Since Eclipse N4JS is a real programming language used in production for a couple of years yet, it seems very risky to let students work on that stuff. Even more if you know that the Eclipse N4JS IDE nightly build is actually used in production. This is only possible because we have a very extensive test suit with 94.920 tests.

So, the students are eventually supposed to do real pull-request. For that I needed to set up a Jenkins job running all tests on that pull-requests in order to assure that it can be merged to master without breaking anything. This blog post is my log how I set up that server.

Setup a Github triggered build machine

In the following, I describe how to setup a Linux (Ubuntu 18.04.3 LTS) machine with all required software, in particular Jenkins, and how to configure Github etc., to start a build whenever a pull request is created against a repository. The pull request performs a pre-merge, that is, it runs your build with the pull request already merged to master (for the build only)! In short, we will

• install Java, Maven, and Jenkins
• install some JavaScript software (needed for smee client)
• setup Github to trigger our build (via smee.io)
• install Xvnc to enable UI tests
• write a Jenkins build file eventually running our maven build (for Eclipse N4JS)

Description

I roughly followed "Automated Jenkins builds on GitHub pull request" by Karolis

Requirements

I have a dedicated Linux Ubuntu 18.04.3 LTS machine (16 GB memory for my large test suite :-) ) set up on a VM (done by my friendly admin Peter R. -- thank you!). In order to be able to use english descriptions etc., I switched to english and also adjusted some language settings accordingly:

First, switch to english:

sudo vi /etc/default/locale

Changed this file to

LANG=en_US.UTF-8
LANGUAGE="en_US:en" 

You also need the admin to enable port 8080 on your machine, so that others can reach it in case you are behind a firewall. We will install Jenkins to listen to port 8080. Of course, you can use any other port.

Install required fundamental software

First we install some software that is kind of fundamental: git, java, maven, and ant.

1. Install git, if it is not installed yet:

   sudo apt install git
   

2. Install Java 11:

   sudo apt install openjdk-11-jdk-headless
   

   Note: Java 11 is support by Jenkins since 2.164. We need the JDK, since Jenkins needs a JDK! Ensure that the path exists:

   JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
   

3. Install Maven and Ant:

   sudo apt install maven
   sudo apt install ant
   

   Check version and maven home:

   mvn --version
   ant -version
   

   This will printout something like

   Apache Maven 3.6.0
   Maven home: /usr/share/maven 
   

   and

   Apache Ant(TM) version 1.10.5 compiled on March 28 2019 
   

   Note that ant is installed to /usr/share/ant-- although you probably do not need that path later on.

   Alternatively you could install them manually, e.g.

   sudo apt install unzip
   wget "http://ftp.fau.de/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.zip"
   unzip apache-maven-3.6.1-bin.zip
   sudo mv apache-maven-3.6.1 /opt/
   

   Ant

   wget "http://us.mirrors.quenda.co/apache//ant/binaries/apache-ant-1.9.14-bin.zip"
   unzip apache-ant-1.9.14-bin.zip
   mv apache-ant-1.9.14 /opt/
   

   Note that you need to add the path to the bin files of Maven and Ant to the PATH in scripts started later.
4. In order to be able to install the Smee client later on, you need node.js (and npm). There are some hints found at nodesource's github page.

   sudo apt install curl
   curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash -
   sudo apt-get install -y nodejs
    

5. In my case, I also needed yarn. This is only needed by my build (i.e. the tests), so you could skip that step if you do not need yarn. There is an issue which you might run into (as I did), described in a Yarn Github issue 2821. Thus, ensure the right yarn is installed: If calling yarn emits ERROR: There are no scenarios; must have at least one. do the following:

   sudo apt remove cmdtest 
   curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
   echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
   sudo apt update
   sudo apt install yarn
   

Setup Smee and Smee client

If you are not admin of your Github repository, and if you do not want to drill to many holes in your firewall, then you probably need Smee.io. Smee.io is a "Webhook payload delivery service". That is, you tell Github to send events to smee.io, and you can use the Smee client to retrieve the these events and forward them internally.

Steps

1. Setup a channel at smee.io, simply by clicking "start new channel" at smee.io. Write down the URL you get there, as this is the URL you need to configure at github.
2. Install Smee client on your build machine.

   sudo npm install --global smee-client
   

   I assume the following warning can be ignore, since smee is now in version 5.1: Warning: superagent@3.8.3: by default, therefore you may need to add it yourself (e.g. GitHub blocks requests without a User-Agent header). This notice will go away with v5.0.2+ once it is released. Note that /usr/bin/smee actually is located at /usr/lib/node_modules/smee-client/bin/smee.js.
3. With this client, you then configure the internal fowarding of the event, i.e.

   smee -u https://smee.io/.... --path /github-webhook/ --port 8080
   

   This is the theory. In practice you want at least to run smee as a service. So this is what we do in this setp: set up smee as service We write a init script and a systemctl service description for that. Here is /etc/init.d/smee

   #! /bin/sh
   
   PATH=/bin:/usr/bin:/sbin:/usr/sbin
   DAEMON=/usr/bin/smee
   PIDFILE=/var/run/smee.pid
   
   test -x $DAEMON || exit 0
   
   . /lib/lsb/init-functions
   
   case "$1" in
     start)
    log_daemon_msg "Starting smee" "smee"
    start_daemon -p $PIDFILE $DAEMON
    log_end_msg $?
       ;;
     stop)
    log_daemon_msg "Stopping smee" "smee"
    killproc -p $PIDFILE $DAEMON
    log_end_msg $?
       ;;
     force-reload|restart)
       $0 stop
       $0 start
       ;;
     status)
       status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
       ;;
     *)
       echo "Usage: /etc/init.d/smee {start|stop|restart|force-reload|status}"
       exit 1
       ;;
   esac
   
   exit 0
   

   Additionally you need the /etc/systemd/system/smee.service file. This contains the fowarding with your URL

   [Unit]
   Description=smee.io webhook delivery for my build
   After=network.target
   StartLimitIntervalSec=0
   
   [Service]
   Type=simple
   Restart=always
   RestartSec=1
   User=jenkins
   ExecStart=/usr/bin/smee -u https://smee.io/«YOUR SMEE PATH» --path /github-webhook/ --port 8080
   
   [Install]
   WantedBy=multi-user.target
   

   In order to start smee, call it similar to jenkins:

   sudo /etc/init.d/smee start
   

   In order to check if it worked, call

   sudo /etc/init.d/smee status
   

   This should output something similar to

   smee.service - smee.io webhook delivery for my build
      Loaded: loaded (/etc/systemd/system/smee.service; disabled; vendor preset: enabled)
      Active: active (running) since Thu 2019-08-22 07:04:29 UTC; 7min ago
    Main PID: 15314 (node)
       Tasks: 11 (limit: 4915)
      CGroup: /system.slice/smee.service
              └─15314 node /usr/bin/smee -u https://smee.io/... --path /github-webhook/ --port 8080
    
   Aug 22 07:04:29 ... systemd[1]: Started smee.io webhook delivery for my build
   Aug 22 07:04:29 ... smee[15314]: Forwarding https://smee.io/... to http://127.0.0.1:8080/github-webhook/
   Aug 22 07:04:30 ... smee[15314]: Connected https://smee.io/...
   

   You may also use sudo journalctl -u smee (or add an -f for realtime reporting) checking the output in the systemctl log -- this will then also display the events smee received from the Github webhook!

   Note that if you are playing around, you may need to stop it via sudo /etc/init.d/smee stop.

See below for some troubleshooting hints.

Use GHEF

If you are the admin of the github repository, you can adjust the event being sent to your Jenkins via smee.io. Since I am not the admin, I would have to write a bug report every time I want to adjust something. Besides, the settings provided by GitHub do not fit my needs: I want to have events for all push events, and I want to have an update when a new pull request has been opened. I did not find any setting for that on GitHub -- the "only push events" does not send anything when a new pull request has been opened.

When you get all events, your Jenkins starts too many jobs. E.g., a job is triggered only if someone wrote a comment. For that reason, I wrote a small script acting as a proxy between the smee client and Jenkins. This proxy, called GHEF for "GitHub Event Filter", only forwards push and pull-request opened events to Jenkins. The script is written in JavaScript, and it is to be installed similarly as the smee client. It needs to be installed via

sudo npm install --global ghef

I run GHEF on port 3000, thus, smee forwards to 3000. GHEF then forwards to port 8080 (i.e. Jenkins).

You then need to provide init script also similar to the smee client. So, the events are processed as follows:

1. GitHub emits event to smee.io
2. The smee client gets the event from smee.io, forwarding it to GHEF
3. GHEF processes the event, in case of push or pull-request opened it is forwared to Jenkins, otherwise ignore
4. Jenkins finally gets the event, and the GitHub plugin triggers a build

The syslog shows this process:

systemd[1]: Started GitHub Event Filter for N4JS webhook.
ghef[28779]: GitHub Event Filter 0.0.5 listening on port 3000, forwarding to localhost:8080!
...
ghef[26459]:   Ignoring github event: create
Oct 29 12:26:40 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200
Oct 29 12:30:44 projn4js ghef[26459]: Forwarding #1534 github event: pull_request, action: opened
Oct 29 12:30:44 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200
Oct 29 12:30:46 projn4js ghef[26459]:   Ignoring github event: status
Oct 29 12:30:46 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200
...

Note that port 3000 is only available locally.

Prepare GitHub

Add Webhook

At github.com: Go to Settings > Webhooks

• Payload URL: https://smee.io/«YOUR SMEE PATH»
• Content Type: application/json
• no secret
• Send me everthing

If you are not admin of your repository, e.g., in case of an Eclipse project, you need to file a bugzilla. In this bugzilla, tell the admin your Smee URL. Note that you need to use your own Github account to access the repository, see next section.

Create Token

In order to enable your build to communicate with the Github API, credentials are required. This is done by creating a Github API token with the right credentials.

Goto Personal access tokens at GitHub.

The following rights are required (maybe less, please drop me comment if you know details!):

- repo
- repo:status
-  repos_deployment
-  public_repo
- write:packages
- read:packages
- admin:org
 - write:org
 - read:org
- notifications 
- admin:public_key
 - read:public_key
- admin:repo_hook
 - read:repo_hook
- write:discussion
 - read:discussion

This token is used as password when you set up the GitHub organization!

Install Jenkins on Ubuntu

We are now ready to install Jenkins.

Descriptions:

For that, I also had a look at the following descriptions:

• Jenkins Installation Chapter
• "How To Install Jenkins on Ubuntu 18.04" at linuxize.com

Steps

1. Install Jenkins:

   wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
   sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
   sudo apt-get update
   sudo apt-get install jenkins
   

   I got an error: jenkins : Depends: daemon but it is not installable Solved this by running

   sudo add-apt-repository universe
   

   and then

   sudo apt-get install jenkins
   

   again. This installed Jenkins 2.176.2 (note: Java 11 support since 2.164) In order to find out where everything has been installed, have a look at /etc/default/jenkins -- in this file, all settings are defined.
2. Check that Jenkins is running:

   systemctl status jenkins
   

   and in general control Jenkins via

   sudo /etc/init.d/jenkins restart
   

   Usage: /etc/init.d/jenkins {start|stop|status|restart|force-reload}
3. Log in the first time with initial password. The initial password can be found via

   sudo cat /var/lib/jenkins/secrets/initialAdminPassword
   

   Then open your Jenkins (http:\\yourserver:8080) with this password. Note: I got a security exception, probably due to an interrupted first start. Fix explained on stackoverflow did not work, since the config file is overridden. You may fix this from the web interface. I removed Jenkins (and removed the /var/lib/jenkins folder) and reinstalled it.
4. Install suggested plugins
5. Created new admin user.
6. Setup global tools, that is, Java, Maven and Ant. For that, goto Manage Jenkins > Global Tool Configuration in your Jenkins web UI and setup these tools accordingly:
   1. setup JDK: Manage Jenkins > Global Tool Configuration with JAVA_HOME as listed above
   2. Maven and ant may be installed automatically (using name mvn and ant), or you configure them using the paths of your installation above. the important thing is to use the correct name, in my case mvn and ant.

XVNC

Since I need to run UI tests, I needed a virtual display. Actually, this was the hardest part of the setup, and unfortunately I did not find much help on the internet.

Descriptions

There are some descriptions available.

1. Xvnc Jenkins Plugin Page
2. "Continuous Integration with Hudson/Jenkins" from the LSDevLinux team

Steps

1. Install xvnc:

   sudo apt-get install vnc4server fluxbox
   sudo su - jenkins
   vncserver
   pass: xxxxxxxx (doesn't matter)
   

   The password, logs, and xstartup script are all found at /var/lib/jenkins/.vnc. This is the place to look in case of problems.

   Alas this did not work, in the log file (e.g., /var/lib/jenkins/.vnc/«servername»:«displaynumber».log) I found the line /var/lib/jenkins/.vnc/xstartup: x-terminal-emulator: not found

   For the UI tests, you do not need a terminal emulator, so just remove the line starting xterm from the xstartup script.

   I also had to create a file .Xmodmap in /var/lib/jenkins. I did this as user jenkins (sudo -su - jenkins) and simply touched (touch .Xmodmap)it. Also see notes below (if something is not working...)

   Eventually I ended up with the following startup script:

   [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
   [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
   vncconfig -iconic &
   

   (and everything else commented out)

   Note that there is an alternative VNC, TigerVNC. I also tried that one, since it is the one installed on the old Eclipse build infrastructure. But I changed back to vnc4server (since I got the same problems with TigerVNC...).

2. Install xvnc plugin via Jenkins plugin manager.
3. This step I only added later, I am not quite sure if it is actually required. When I compared my build output with the output of a build which was working, I noticed that the vncserver is started slightly differently. So I configured the vncstartup in the global Jenkins configuration, i.e. I set the vncserver command line to

   /usr/bin/vncserver :$DISPLAY_NUMBER -geometry 1024x768 -depth 24 -ac -noreset
   

   This should do the trick, unfortunately I run into problems. That is, my tests were not able to connect to the virtual display. I still do not really know why, and the internet was not helpful at this point (if you know a good description, let me know!). There are a couple of possible things that can go wrong: vncserver is not starting correctly or is immediately shutdown, security settings are wrong, or -- as I figured out eventually -- the global environment variable DISPLAY is not correctly forwarded to your test.

   Due to these problems I was not able to successfully run my UI tests. I tried a lot, and I even installed even Gnome in my desperation. This caused my vncserver to not even start anymore with a XIO: fatal IO error 11 (Resource temporarily unavailable) on X server... in my logs. I then uninstalled Gnome (sudo apt-get remove gnome), which did not fix that problem, removed and re-installed vnc4server and Fluxbox. Did not work. Eventually, after removing gdm and gnome-settings-daemon, and after reconfiguring vnc4server and Fluxbox (sudo dpkg-reconfigure fluxbox and sudo dpkg-reconfigure vnc4server) it worked again.

Setup Pipeline

Back to the triggered build. You need the Github Jenkins plugin, if you have installed the recommended plugins on first start-up of Jenkins (see above), this is already installed.

We will set up a pipeline build on Jenkins, which is triggered (indirectly) via Github.

Descriptions

For this, I followed the following tutorials:

• "GitHub Integration: Webhooks" by cloudbees
• "Triggering builds with webhooks behind a secure firewall" at Jenkins blog
• "GitHub Branch Source Plugin" by cloudbees
• Jenkins GitHub Branch Source Plugin, the plugin is installed with the recommended plugins.

Steps

1. Add a new credential via the Jenkins Web UI at Credentials> System > Global credentials (unrestricted). Use your GitHub user account and the previously created token as password.
2. Create a new GitHub organization (which will then spawn jobs). Go to Jenkins > New Item > GitHub Organization. I got an exception here (Caused by: java.lang.NoClassDefFoundError: com/cloudbees/plugins/credentials/CredentialsProvider), similar to the one described as Jenkins issue 58071.
   • I tried to install the Blue Ocean plugin as this seems to fix the problem for some people. -- That did not help in my case.
   • Restarted Jenkins (sudo /etc/init.d/jenkins restart) helped, though!
3. In the job, go to Configure and configure the job:
   • Credentials: user name with password, the password is the token!
   • Owner: eclipse (or whatever organization owns the repository)
   • Behaviors: Add > Filter by regex: «repository name» (since you do not want to get events for everything)
   • Discover pull requests from origin
     • Merging the pull request with the current target branch...
   • Discover pull requests from forks
     • Merging the pull request with the current target branch...
     • Trust: Nobody
   • Pipeline Jenkinsfile: «Name of your Jenkins file»
   The organization is scanned, the project is detected, with some PRs. Now, all what is left to do is to write the Jenkins file and merge it to your repository.

Jenkinsfile

Actually, I did not write the following Jenkinsfile from scratch but reused the file used for my project (N4JS) on the Eclipse N4JS JIPP Instance. This file is found in the N4JS repository

However, I had to modify it a bit, due to a problem with my setup, see below for details.

/*
 * Copyright (c) 2019 HAW Hamburg
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Based on eclipse-nightly.jenkinsfile
 */
pipeline {
    agent any

    environment {
        NODEJS_PATH= '/usr/bin' // '/shared/common/node-v10.15.3-linux-x64/bin'
        YARN_PATH  = '/usr/bin' // '/shared/common/yarn/1.15.2/bin'
        MAVEN_OPTS = '-Xmx4G'
        JAVA_HOME  = '/usr/lib/jvm/java-11-openjdk-amd64'
        TIMESTAMP  = new Date().format("yyyyMMddHHmm")
   }

    stages {
        stage('Build and Test') {
            steps {
                echo 'Starting Xvnc'
                wrap([$class: 'Xvnc', takeScreenshot: false, useXauthority: true]) {
                    echo 'Building and testing, using display ' + DISPLAY
                    script {
                     def envvars = [ 
                      '-DDISPLAY=localhost'+DISPLAY,
                            '-DargLine="-DDISPLAY=localhost'+DISPLAY+'"'
                     ].join(' ')
                        def options = [
                            '--batch-mode',
                            //'--quiet',
                            '--update-snapshots',
                            '--show-version',
                            '-Dtycho.localArtifacts=ignore',
                            // for adjusting the script, maybe uncomment the following two lines:
                             '-Dmaven.test.failure.ignore',
                             "--fail-at-end",
                            '-DWORKSPACE=' + env.WORKSPACE,
                            '-DexcludeJRE'                               
                        ].join(' ')
                        def profiles = [
                            'buildProduct',
                            'execute-plugin-tests',
                            'execute-plugin-ui-tests',
                            'execute-ecma-tests',
                            'execute-accesscontrol-tests',
                            'execute-smoke-tests'
                         //'execute-hlc-integration-tests'
                        ].join(',')

                        sh """\
                            pwd
                            git log -n 1
                            npm version
                        """
                        sh "mvn ${envvars} clean verify -P${profiles} ${options}"
                        // for tests, you may want to run less tests:
                        // sh "mvn clean verify ${options}"
                        // and for quick testing: only
                        // sh "mvn -o clean"

                      // sh "ls -Ral builds/org.eclipse.n4js.product.build/target/repository/"
                    }
                } // end wrap
            } // end steps
        } // end stage
    }

    post {
        always {
            junit '**/surefire-reports/**/*.xml'
        }
        cleanup {
            // Execute after every other post condition has been evaluated, regardless of status
            // See https://jenkins.io/doc/book/pipeline/syntax/#post
            echo 'Cleaning up workspace'
            // Not today: deleteDir()
        }
    }
}

Some notes on the script:

• In order to run UI tests, you need to run the tests with Xvnc running (see wrap ...).
• In my case, it did not work because the DISPLAY environment variable was not correctly passed to the maven surefire tests. So I added the evnvars in the script and passed them to maven in the shell call. This took me several days to figure out!

One final note: Of course your build is only triggered if it find that Jenkinsfile, since we defined that earlier. The first time your build is then triggered should be the pull request in which you ask for merging the Jenkinsfile. And this is a great pull request for testing it :-)

When it works, you will be rewarded with a message at Github similar to this one:

Some hints in case something does not work

Do you get notifications?

In order to check that smee is really working, try the following:

// content of index.js
const http = require('http')
const port = 3000

const requestHandler = (request, response) => {
  console.log(request.url)
  response.end('Hello Node.js Server!')
}

const server = http.createServer(requestHandler)

server.listen(port, (err) => {
  if (err) {
    return console.log('something bad happened', err)
  }

  console.log(`server is listening on ${port}`)
})

and then

smee -u https://smee.io/«YOUR SMEE PATH» --path /github-webhook/ --port 3000

This should output any push on your system. Then stop that Smee and reuse the original one.

Emulate requests

Note that you can manually post requests to smee and see if it is working (so you do not need github for that) via

curl --header "Content-Type: application/json" --request POST --data '{ "key": "val" }' https://smee.io/«YOUR SMEE PATH»

Easier: Or you can redeliver a request, either via Smee (on the Smee website) or Github (on the webhook settings).

Forgot your password?

In case you forgot your Jenkins admin password, have a look at this [stackoverflow thread][https://stackoverflow.com/questions/6988849/how-to-reset-jenkins-security-settings-from-the-command-line]. For the impatient reader: Update the password hash at /var/lib/jenkins/users/username/config.xml with

<passwordHash>#jbcrypt:$2a$10$razd3L1aXndFfBNHO95aj.IVrFydsxkcQCcLmujmFQzll3hcUrY7S</passwordHash>

Once you have done this, just restart Jenkins and log in using this password: test. Of course, you need to change it ASAP!

Update Jenkins

You may get security warning on the management page http://«server»:8080/manage

In order to update Jenkins, follow these steps:

sudo apt-get update
sudo apt-get install jenkins

Jenkins will be restarted automatically. After that, update plugins in the update center. Simply select "compatibles" at the very bottom of the page and activate the uddate.

You may want to manually restart Jenkins after that via sudo /etc/init.d/jenkins restart.

Thanks: Thank you Yoosiba for some hints regarding my XVNC problem (and for writing the initial Jenkins-file for the Eclipse nightly)! And thank you, mmews-n4, for review.

JShell in Eclipse

Java 9 introduced a new command line tool: JShell. This is a read–eval–print loop (REPL) for Java with some really nice features. For programmers I would assume writing a test is the preferred choice, but for demonstrating something (in a class room for example) this is a perfect tool if you are not using a special IDE such as BlueJ (which comes with its own REPL).
The interesting thing about running JShell inside Eclipse is to use a project's classpath. This is how to run JShell inside Eclipse out of the box (given Java >=9 and Eclipse -- I have used 2019-09 -- are installed), using the currently selected project as the base line:

• Select "Run > External Tools > External Tools Configuration"
• Enter the following settings:
  • Name: JShell
  • Location: /usr/bin/jshell -- or whereever your jshell command is found on your system
  • Working Directory: ${project_loc}
  • Arguments: --class-path "${project_classpath}"

You can of course change these settings according to your needs. Now, when you run the tool ("Run > External Tools > JShell" or via the toolbar item), a console view is opened with the shell. It is almost like working with JShell from command line except tab etc. is not working (instead the cursor moves around).
For people not aware of JShell, there is a very good tutorial by Robert Field, and tons of other web pages about it.
For the impatient reader, some hints:

• Inside JShell, you have some extra commands available after a prefix "/", most notable "/help"
• If you change anything in your code (in the Eclipse editor), this is not necessarily reflected in JShell. You need to call "/reset" first!
• You can define methods without a class, i.e. you can define functions. Syntax is similar to methods, though.

JShell as external tool inside Eclipse

Note: There is also a rather old discussion at Stackoverflow in which the "external tool" solutions is explained as well. Apparently it seems that there were some problems fixed in the meantime, because in older posts you will find that some people had problems with this solution.

Two new Eclipse projects: Xpect and Xsemantics

Yesterday, two new Eclipse projects had been created:

• Eclipse Xpect
• Eclipse Xsemantics

Both project are around for quite some time and they both are based on Eclipse Xtext, the famous framework to create editors with all state-of-the-art features (parser, linker, validators, content assist, etc.) for your own textual DSLs, simply based on a grammar.

Xpect is written by Moritz Eysholdt, who is also a committer of Xtext. It is a unit- and integration-testing framework to be used for Xtext-based languages. Instead of writing fiddly JUnit tests, you can simply write things like

// XPECT errors --> "cannot divide two strings"
"hello" / "world"

assuming your language supports division, strings, and types.

And types is the topic of Xsemantics, written by Lorenzo Bettini, author of the book "Implementing DSLs with Xtext and Xtend". It is a DSL (implemented in Xtext itself) for writing type systems, reduction rules, interpreters and general relation rules for languages implemented in Xtext. So you can write rules like that:

rule subtypeUnion_Left
 G|- UnionTypeExpression U <: TypeRef S
from {
 U.typeRefs.forall[T| G |- T <: S]
}

Well, you probably need to know type theory a little bit to see the beauty in that.

Both of these frameworks are heavily used by Eclipse N4JS: it contains more than 10.000 Xpect tests (e.g., all specification tests) and a Java 8 like type system defined in Xsemantics, and this is also why I act as kind of "assisting project lead" to help the original authors to bring these great tools to Eclipse. The authors and the N4JS team at enfore are now working on bringing the code to Eclipse as soon as possible, to simplify the usage of these frameworks for all Xtext users!

Keep on modelling!

I once said that modelling without graphical editors is like Tour de France without mountains. Well, I have to correct myself: a great test suite and a complicated type system are just as exciting ;-)

BoF @ ECE2016: Eclipse and AsciiDoc

At NumberFour, we use AsciiDoc for our specifications and documentation. We are even working on producing AsciiDoc API documentation from JSDoc-like comments (similar to JavaDoc) for our JavaScript language N4JS. Since we ran into a couple of problems with AsciiDoctor, I thought that it might be interesting to learn what other people in the Eclipse Community think about AsciiDoc(tor) and if we might be able to "join forces" to overcome certain obstacles. So I organized a "Birds of Feather" about "Documentation with AsciiDoc and Eclipse" at EclipseCon 2016 Europe. We were about 10 participants and this blog post serves more or less as a kind of meeting minutes of this BoF combined with our findings at NumberFour.

AsciiDoc is a lightweight markup language, similar to Markdown. The original AsciiDoc was written in Python, but Asciidoctor, probably the most popular tool nowadays, is written in Ruby with a variant AsciidoctorJ which provides Java support (via JRuby) and Asciidoctor.js for JavaScript (via Opal). From an Asciidoc document (.adoc) you can either create HTML, PDF or even EPub. Unfortunately, the PDF support is lacking some major features (e.g., footnotes are not supported yet), but there is a common workaround to produce DocBook XML files from AsciiDoctor and then use FOP to eventually create a beautiful PDF. Lars Vogel wrote a nice tutorial about AsciiDoc and AsciiDoctor. One of the great things about AsciiDoc is that GitHub provides basic support for AsciiDoc as well!

Writing AsciiDoc

The first question is how to write AsciiDoc. One solution is to use stand-alone editors such as Sublime (there is an AsciiDoc plug-in for syntax highlighting and coce completion which, together with the preview plug-in, is a really great solution).

But usually, you do not want to switch editors when you are already using Eclipse. Fortunately, there's an existing AsciiDoc editor which is part of Mylyn Wiki Text. This editor has some syntax highlighting, an outline view and even a preview.

This editor seems to be a good starting point for further support of AsciiDoc on the Eclipse platform, although it might have some shortcomings with regard to features and design. Regardless, the BoF participants (including Torkild, a Mylyn Docs committer) decided to use the mylyn-docs developers mailing list for further communication about AsciiDoc and Eclipse.

Apparently, having a good editor, possibly even with WYSIWYG or at least WYSIWYM support is one of the most important things required.

Building AsciiDoc

There are several ways of building AsciiDoc. The most simple one is about using the AsciiDoctor tools directly from the command line. There is also a Maven plugin provided by the AsciiDoctor project.

We haven't discussed that topic any further though. It seems as if the existing tools are sufficient even though there are some shortcomings when using custom macros as described below. You may have a look at the N4JS documentation pom.xml to get an idea of how this could look like.

Converting to AsciiDoc

During the BoF, the question came up how to convert existing documentation written in other formats to Asciidoc. One solution for that is Pandoc, which we (at NumberFour) are using to convert our LaTeX specification to Asciidoc. Jeremie Bresson has already blogged about the BoF and even provided a solution for using Mylyn Wikitext to convert Eclipse MediaWiki to Asciidoc. Thank you very much, Jeremie!

Using AsciiDoc as a Single Source

I already mentioned above that you can generate HTML and PDF from AsciiDoc. However, HTML is not HTML. We use AsciiDoc to generate

• the public web site (i.e., gh-pages)
• the Eclipse help
• the PDF specification

from a single asciidoc file. To give you an idea, have a look at an adoc source file and the generated web page. If you download the N4JS IDE, you can also have a look at the generated Eclipse help containing the same content. The PDF is not publicly available yet, we are currently working on migrating our language specification from LaTeX to AsciiDoc.

In order to generate Eclipse help, a table of contents file is required. At the moment, we generate them via the geneclipsetoc-maven-plugin Maven plugin, see our pom.xml for our configuration.

You may also use the Mylyn Docs to generate code, which is described in the Mylyn Docs Eclipse help. This also provides support for generating EPub, the first steps towards this support has already been described 2011 in Torkild's blog! It’s now a stable part of Mylyn Docs and has been for a few years. Currently it is in maintenance mode as there are few bugs and there is not much to add – apart for EPUB 3.0 support. You can find an EPUB examples (and presentations about Mylyn Docs in general) on Torkild's GitHub page.

Customizing AsciiDoc

Although AsciiDoc provides a lot of features ("macros" in Asciidoc terminology) for technical documentation, such as source blocks, warning blocks etc., we (at NumberFour) missed some functionality. We required additional support for

1. definitions -- a typical feature required in specifications
2. BibTex cites and bibliographies -- we already have a large BibTex database which we want to reuse
3. todos -- well, also a typical feature required in specifications and documentations
4. links to source code -- e.g., for adding links to either GitHub, or, in case of Eclipse help, to the files in the workspace
5. larger documents, e.g., includes similar to LaTeX's chapter folders

One problem we found is that Ruby-based custom macros calling native code cannot be used with AscidoctorJ. E.g., there already exists a custom Ruby macro which provides support for BibTex, but this does not work with AscidoctorJ. We want to write the macros in Java, since we plan to provide some editor support in Eclipse as well -- and then Java would be the better solution. Thus we already wrote our own Java-based, BibTex macro (fixing some smaller AsciidoctorJ problems on the way) and we are also working on solutions for the other topics. If you are interested in that, or if you have written Java-based macros for AsciiDoc as well -- let me know! We haven't open sourced our custom macros yet since they are still under (heavy) development. But we will probably do that once they are in a state to make them public.

Edit: Updated description of EPUB support -- thank you Torkild for the information!

From Xcore/ecore to OmniGraffle

Some years ago I wrote a small tool for creating OmniGraffle UML diagrams directly from Java source code. Visualizing Java is nice, but since I'm often use ecore/Xcore to define my models, I wanted a tool to also nicely visualize EMF based models.

I have now extended my tool, j2og, to also create UML class diagrams from ecore or Xcore models. Below you see a (manually layouted) version of an automatically generated diagram of the ecore library example.

j2og does not layout the diagram, since OmniGraffle provides some nice layout algorithms anyway. When creating the diagram, you can tweak the output with several settings. For example

1. show or hide attribute and operation compartments
2. show context, optionally grayed out -- the context are classifiers defined in an external package
3. show package names, omit common package prefixes etc.
4. and more

Note that besides OmniGraffle, you can open the diagrams with other tools (Diagrammix, Lucidchart) as well. See the j2og github page for details. You can install the tool via update site or Eclipse marketplace link.

The following image (click to enlarge) is the result of exporting a large Xcore model defining the AST of N4JS, a statically typed version of JavaScript. I have exported it and applied the hierarchy layout algorithm -- no other manual tweaks were applied. Of course, this diagram is probably too large to be really useable, but it is a great start to document (parts) of the model. Well, in case of an AST you probably prefer using an EBNF grammar ;-)

PS: Of course you could use ecoretools to create an UML diagram. I usually need the diagrams for documentation purposes. In that case, OmniGraffle simply is so much better since it is easier to use and the diagrams look so much nicer, (sorry, ecoretools).

N4JS: Emphasising the Java in JavaScript

For the last three years, I have worked at NumberFour leading a team which creates a type-safe extension for ECMAScript (aka JavaScript) called N4JS. I'm happy to tell you that yesterday, NumberFour announced N4JS to go open source (press release, PDF). You can find the project home page at:

http://numberfour.github.io/n4js/

N4JS bridges the strengths of JavaScript and Java; the result is a typed JavaScript superset that is dynamic, flexible and type-safe. This first release mainly targets Node.js developers, enabling them to create large and maintainable projects.

The idea behind N4JS is to bring Java's type system to JavaScript and to make JavaScript really as type-safe as Java. This is probably also the main difference to TypeScript, which focuses on ease of transition from untyped to typed JavaScript. I don't want to go into the details here, we already created a longer description of the differences to TypeScript. 

As a result, N4JS brings many features known from Java to ECMAScript. It is based on ECMAScript 2015 (almost all new features are supported, missing things will be added soon). It therefore supports classes as in ECMAScript 2015 (or Java). It also introduces interfaces with default methods similar to Java 8. Also the generics are quite similar to Java, including support for generic types, generic methods, and wildcards (i.e. use-site variance). We have actually ported the type inference algorithm for type variables introduced with Java 8 to N4JS. Of course we had to adjust the type system to match the semantics of ECMAScript. N4JS therefore also supports union and intersection types or "this"-type (great for builder pattern).

The most notable thing about N4JS is that it not only supports Java's nominal typing but also TypeScript's structural typing. It even supports different variants of structural typing, e.g., only taking fields into account. We also have created a short introduction to this very special feature of N4JS.

There are many other bells and whistles in N4JS such as dependency injection (as in JSR330/Google Guice) or testing with annotations (as in JUnit). And this is only the beginning. With the foundation finished, we are looking forward to adding more sophisticated features such as more program analysis.

Although we call the current release a "public alpha", N4JS is pretty stable already. We are using N4JS internally for over a year now and we have a rather large code base (and more than 70.000 tests!). Of course, there are a lot of known issues (we will migrate them to GitHub issues soon)...

N4JS comes with an Eclipse-based IDE. Our goal is to provide the same IDE support for N4JS as JDT for Java. You can either download the N4JS IDE as a standalone product or use the update site to add N4JS to your Eclipse Mars installation. We also provide an Oomph script to set-up an Eclipse IDE ready for developing N4JS itself. See the GitHub pages for source code and details.

Well, this has become a rather long posting already. I will post more about N4JS, its features and new developments in the future either here or on the N4JS developer blog. Last but not least, I want to thank the people behind N4JS. In particular I want to thank the N4JS team at NumberFour -- you did a great job and I'm looking forward to working on N4JS and other things with you! In place of the many helpers I want to thank Sebastian Zarnekow -- you did an amazing job (readers may want to look at the grammar file to get an idea of what kind of magic he had to add to Xtext to enable that)!

Update (14.3.2016, 19:00): The N4JS Eclipse project proposal is now public!

j2og -- Java To OmniGraffle, Updated

Three years ago, I wrote a small plugin called "j2og" (for Java To OmniGraffle) which can create OmniGraffle drawings from your existing Java files. At that time, I used AppleScript to create the drawing. Unfortunately this mechanism was a little bit fragile, and somehow it didn't work with new versions of OmniGraffle / Mac OS X. Instead of fixing the AppleScript problem, I rewrote the export.

The new version now directly exports to OmniGraffle files. That means OmniGraffle is no longer needed to export your files to an OmniGraffle drawing. Actually, there are several tools which can open the exported drawings now:

• OmniGraffle, of course; only available for Mac OS X
• Diagrammix, via "Import"; only available for Mac OS X
• Lucidchar, not tested yet; available for Mac OS X and Windows

The overall functionality was not changed. E.g., you can export classes and interfaces with or without attributes or operations, attributes can be transformed to associations, context of classes can be exported as well. The screenshot shows a sample drawing exported from the j2og sources. Since j2og does not layout the drawing (I was too lazy to write an layout algorithm), you have to manually layout the diagram. E.g., use the auto-layout of OmniGraffle as I did for the sample screenshot.

You can either install j2og from the Eclipse marketplace or the update site (see j2og homepage at github for details).