Disclaimer 1: This blog post literally is a "web log", i.e., it is my log about setting up a Jenkins machine with a job that is triggered on a Github pull request. A lot of parts have been described elsewhere, and I link to the sources I used here. I also know that nowadays (e.g., new Eclipse build infrastructure) you usually do that via docker -- but then you need to configure docker, in which case you will need the same knowledge. However, since this describes a full setup for building an Eclipse project via Jenkins, it might be interesting to others as well.
Disclaimer 2: Setting up a continuous integration pipeline (which we will do here) is a kind of DevOps job. One problem with devops is that, as in my case, developers often do not have that much operations knowledge. So if some operator finds any silly things here, please leave me a comment -- I'm eager to learn.
Motivation
Why did I need a Github triggered build in the first place? I'm working as a professor at the Hamburg University of Applied Sciences, teaching programming, software engineering and computer science stuff in general. I am currently teaching a special class at the university: the “Eclipse N4JS project”. In this kind of class, which students in the 5th semester are supposed to attend, the students are expected to do a more real project. These projects are derived from research, e.g. cyber systems or autonomous cars. In my case, they have to work on a really real project: Eclipse N4JS. The students become a Scrum team (9 people), and they are working on smaller yet real-world tasks.
Since Eclipse N4JS is a real programming language used in production for a couple of years yet, it seems very risky to let students work on that stuff. Even more if you know that the Eclipse N4JS IDE nightly build is actually used in production. This is only possible because we have a very extensive test suit with 94.920 tests.
So, the students are eventually supposed to do real pull-request. For that I needed to set up a Jenkins job running all tests on that pull-requests in order to assure that it can be merged to master without breaking anything. This blog post is my log how I set up that server.
Setup a Github triggered build machine
In the following, I describe how to setup a Linux (Ubuntu 18.04.3 LTS) machine with all required software, in particular Jenkins, and how to configure Github etc., to start a build whenever a pull request is created against a repository. The pull request performs a pre-merge, that is, it runs your build with the pull request already merged to master (for the build only)! In short, we will
- install Java, Maven, and Jenkins
- install some JavaScript software (needed for smee client)
- setup Github to trigger our build (via smee.io)
- install Xvnc to enable UI tests
- write a Jenkins build file eventually running our maven build (for Eclipse N4JS)
Description
I roughly followed "Automated Jenkins builds on GitHub pull request" by Karolis
Requirements
I have a dedicated Linux Ubuntu 18.04.3 LTS machine (16 GB memory for my large test suite :-) ) set up on a VM (done by my friendly admin Peter R. -- thank you!). In order to be able to use english descriptions etc., I switched to english and also adjusted some language settings accordingly:
First, switch to english:
sudo vi /etc/default/locale
LANG=en_US.UTF-8
LANGUAGE="en_US:en"
You also need the admin to enable port 8080 on your machine, so that others can reach it in case you are behind a firewall. We will install Jenkins to listen to port 8080. Of course, you can use any other port.
Install required fundamental software
First we install some software that is kind of fundamental: git, java, maven, and ant.
- Install git, if it is not installed yet:
sudo apt install git - Install Java 11:
Note: Java 11 is support by Jenkins since 2.164. We need the JDK, since Jenkins needs a JDK! Ensure that the path exists:sudo apt install openjdk-11-jdk-headlessJAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64 - Install Maven and Ant:
Check version and maven home:sudo apt install maven sudo apt install ant
This will printout something likemvn --version ant -version
andApache Maven 3.6.0 Maven home: /usr/share/maven
Note that ant is installed toApache Ant(TM) version 1.10.5 compiled on March 28 2019/usr/share/ant-- although you probably do not need that path later on.Alternatively you could install them manually, e.g.
Antsudo apt install unzip wget "http://ftp.fau.de/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.zip" unzip apache-maven-3.6.1-bin.zip sudo mv apache-maven-3.6.1 /opt/
Note that you need to add the path to the bin files of Maven and Ant to the PATH in scripts started later.wget "http://us.mirrors.quenda.co/apache//ant/binaries/apache-ant-1.9.14-bin.zip" unzip apache-ant-1.9.14-bin.zip mv apache-ant-1.9.14 /opt/ - In order to be able to install the Smee client later on, you need node.js (and npm). There are some hints found at
nodesource's github page.
sudo apt install curl curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash - sudo apt-get install -y nodejs - In my case, I also needed yarn. This is only needed by my build (i.e. the tests), so you could skip that step if you do not need yarn. There is an issue which you might run into (as I did), described in a Yarn Github issue 2821. Thus, ensure the right yarn is installed: If calling yarn emits
ERROR: There are no scenarios; must have at least one.do the following:sudo apt remove cmdtest curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list sudo apt update sudo apt install yarn
Setup Smee and Smee client
If you are not admin of your Github repository, and if you do not want to drill to many holes in your firewall, then you probably need Smee.io. Smee.io is a "Webhook payload delivery service". That is, you tell Github to send events to smee.io, and you can use the Smee client to retrieve the these events and forward them internally.
Steps
- Setup a channel at smee.io, simply by clicking "start new channel" at smee.io. Write down the URL you get there, as this is the URL you need to configure at github.
- Install Smee client on your build machine.
I assume the following warning can be ignore, since smee is now in version 5.1: Warning: superagent@3.8.3: by default, therefore you may need to add it yourself (e.g. GitHub blocks requests without a User-Agent header). This notice will go away with v5.0.2+ once it is released. Note thatsudo npm install --global smee-client/usr/bin/smeeactually is located at/usr/lib/node_modules/smee-client/bin/smee.js. - With this client, you then configure the internal fowarding of the event, i.e.
This is the theory. In practice you want at least to run smee as a service. So this is what we do in this setp: set up smee as service We write a init script and a systemctl service description for that. Here issmee -u https://smee.io/.... --path /github-webhook/ --port 8080/etc/init.d/smeeAdditionally you need the#! /bin/sh PATH=/bin:/usr/bin:/sbin:/usr/sbin DAEMON=/usr/bin/smee PIDFILE=/var/run/smee.pid test -x $DAEMON || exit 0 . /lib/lsb/init-functions case "$1" in start) log_daemon_msg "Starting smee" "smee" start_daemon -p $PIDFILE $DAEMON log_end_msg $? ;; stop) log_daemon_msg "Stopping smee" "smee" killproc -p $PIDFILE $DAEMON log_end_msg $? ;; force-reload|restart) $0 stop $0 start ;; status) status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $? ;; *) echo "Usage: /etc/init.d/smee {start|stop|restart|force-reload|status}" exit 1 ;; esac exit 0/etc/systemd/system/smee.servicefile. This contains the fowarding with your URLIn order to start smee, call it similar to jenkins:[Unit] Description=smee.io webhook delivery for my build After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=jenkins ExecStart=/usr/bin/smee -u https://smee.io/«YOUR SMEE PATH» --path /github-webhook/ --port 8080 [Install] WantedBy=multi-user.target
In order to check if it worked, callsudo /etc/init.d/smee start
This should output something similar tosudo /etc/init.d/smee status
You may also usesmee.service - smee.io webhook delivery for my build Loaded: loaded (/etc/systemd/system/smee.service; disabled; vendor preset: enabled) Active: active (running) since Thu 2019-08-22 07:04:29 UTC; 7min ago Main PID: 15314 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/smee.service └─15314 node /usr/bin/smee -u https://smee.io/... --path /github-webhook/ --port 8080 Aug 22 07:04:29 ... systemd[1]: Started smee.io webhook delivery for my build Aug 22 07:04:29 ... smee[15314]: Forwarding https://smee.io/... to http://127.0.0.1:8080/github-webhook/ Aug 22 07:04:30 ... smee[15314]: Connected https://smee.io/...sudo journalctl -u smee(or add an-ffor realtime reporting) checking the output in the systemctl log -- this will then also display the events smee received from the Github webhook!Note that if you are playing around, you may need to stop it via
sudo /etc/init.d/smee stop.
Use GHEF
If you are the admin of the github repository, you can adjust the event being sent to your Jenkins via smee.io. Since I am not the admin, I would have to write a bug report every time I want to adjust something. Besides, the settings provided by GitHub do not fit my needs: I want to have events for all push events, and I want to have an update when a new pull request has been opened. I did not find any setting for that on GitHub -- the "only push events" does not send anything when a new pull request has been opened.
When you get all events, your Jenkins starts too many jobs. E.g., a job is triggered only if someone wrote a comment. For that reason, I wrote a small script acting as a proxy between the smee client and Jenkins. This proxy, called GHEF for "GitHub Event Filter", only forwards push and pull-request opened events to Jenkins. The script is written in JavaScript, and it is to be installed similarly as the smee client. It needs to be installed via
sudo npm install --global ghef
You then need to provide init script also similar to the smee client. So, the events are processed as follows:
- GitHub emits event to smee.io
- The smee client gets the event from smee.io, forwarding it to GHEF
- GHEF processes the event, in case of push or pull-request opened it is forwared to Jenkins, otherwise ignore
- Jenkins finally gets the event, and the GitHub plugin triggers a build
systemd[1]: Started GitHub Event Filter for N4JS webhook. ghef[28779]: GitHub Event Filter 0.0.5 listening on port 3000, forwarding to localhost:8080! ... ghef[26459]: Ignoring github event: create Oct 29 12:26:40 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200 Oct 29 12:30:44 projn4js ghef[26459]: Forwarding #1534 github event: pull_request, action: opened Oct 29 12:30:44 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200 Oct 29 12:30:46 projn4js ghef[26459]: Ignoring github event: status Oct 29 12:30:46 projn4js smee[21884]: POST http://127.0.0.1:3000/github-webhook/ - 200 ...Note that port 3000 is only available locally.
Prepare GitHub
Add Webhook
At github.com: Go toSettings > Webhooks
- Payload URL: https://smee.io/«YOUR SMEE PATH»
- Content Type: application/json
- no secret
- Send me everthing
Create Token
In order to enable your build to communicate with the Github API, credentials are required. This is done by creating a Github API token with the right credentials.
Goto Personal access tokens at GitHub.
The following rights are required (maybe less, please drop me comment if you know details!):
- repo
- repo:status
- repos_deployment
- public_repo
- write:packages
- read:packages
- admin:org
- write:org
- read:org
- notifications
- admin:public_key
- read:public_key
- admin:repo_hook
- read:repo_hook
- write:discussion
- read:discussion
Install Jenkins on Ubuntu
We are now ready to install Jenkins.
Descriptions:
For that, I also had a look at the following descriptions:Steps
- Install Jenkins:
I got an error:wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add - sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list' sudo apt-get update sudo apt-get install jenkinsjenkins : Depends: daemon but it is not installableSolved this by running
and thensudo add-apt-repository universe
again. This installed Jenkins 2.176.2 (note: Java 11 support since 2.164) In order to find out where everything has been installed, have a look atsudo apt-get install jenkins/etc/default/jenkins-- in this file, all settings are defined. - Check that Jenkins is running:
and in general control Jenkins viasystemctl status jenkins
Usage:sudo /etc/init.d/jenkins restart/etc/init.d/jenkins {start|stop|status|restart|force-reload} - Log in the first time with initial password. The initial password can be found via
Then open your Jenkins (sudo cat /var/lib/jenkins/secrets/initialAdminPasswordhttp:\\yourserver:8080) with this password. Note: I got a security exception, probably due to an interrupted first start. Fix explained on stackoverflow did not work, since the config file is overridden. You may fix this from the web interface. I removed Jenkins (and removed the /var/lib/jenkins folder) and reinstalled it. - Install suggested plugins
- Created new admin user.
- Setup global tools, that is, Java, Maven and Ant. For that, goto
Manage Jenkins > Global Tool Configurationin your Jenkins web UI and setup these tools accordingly:- setup JDK:
Manage Jenkins > Global Tool ConfigurationwithJAVA_HOMEas listed above - Maven and ant may be installed automatically (using name mvn and ant), or you configure them using the paths of your installation above. the important thing is to use the correct name, in my case mvn and ant.
- setup JDK:
XVNC
Since I need to run UI tests, I needed a virtual display. Actually, this was the hardest part of the setup, and unfortunately I did not find much help on the internet.
Descriptions
There are some descriptions available.Steps
- Install xvnc:
The password, logs, and xstartup script are all found atsudo apt-get install vnc4server fluxbox sudo su - jenkins vncserver pass: xxxxxxxx (doesn't matter)/var/lib/jenkins/.vnc. This is the place to look in case of problems.Alas this did not work, in the log file (e.g.,
/var/lib/jenkins/.vnc/«servername»:«displaynumber».log) I found the line/var/lib/jenkins/.vnc/xstartup: x-terminal-emulator: not foundFor the UI tests, you do not need a terminal emulator, so just remove the line starting xterm from the xstartup script.
I also had to create a file
.Xmodmapin /var/lib/jenkins. I did this as user jenkins (sudo -su - jenkins) and simply touched (touch .Xmodmap)it. Also see notes below (if something is not working...)Eventually I ended up with the following startup script:
(and everything else commented out)[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources vncconfig -iconic &Note that there is an alternative VNC, TigerVNC. I also tried that one, since it is the one installed on the old Eclipse build infrastructure. But I changed back to vnc4server (since I got the same problems with TigerVNC...).
- Install xvnc plugin via Jenkins plugin manager.
- This step I only added later, I am not quite sure if it is actually required. When I compared my build output with the output of a build which was working, I noticed that the vncserver is started slightly differently. So I configured the vncstartup in the global Jenkins configuration, i.e. I set the vncserver command line to
/usr/bin/vncserver :$DISPLAY_NUMBER -geometry 1024x768 -depth 24 -ac -noresetThis should do the trick, unfortunately I run into problems. That is, my tests were not able to connect to the virtual display. I still do not really know why, and the internet was not helpful at this point (if you know a good description, let me know!). There are a couple of possible things that can go wrong: vncserver is not starting correctly or is immediately shutdown, security settings are wrong, or -- as I figured out eventually -- the global environment variable DISPLAY is not correctly forwarded to your test.
Due to these problems I was not able to successfully run my UI tests. I tried a lot, and I even installed even Gnome in my desperation. This caused my vncserver to not even start anymore with a
XIO: fatal IO error 11 (Resource temporarily unavailable) on X server...in my logs. I then uninstalled Gnome (sudo apt-get remove gnome), which did not fix that problem, removed and re-installed vnc4server and Fluxbox. Did not work. Eventually, after removing gdm and gnome-settings-daemon, and after reconfiguring vnc4server and Fluxbox (sudo dpkg-reconfigure fluxboxandsudo dpkg-reconfigure vnc4server) it worked again.
Setup Pipeline
Back to the triggered build. You need the Github Jenkins plugin, if you have installed the recommended plugins on first start-up of Jenkins (see above), this is already installed.
We will set up a pipeline build on Jenkins, which is triggered (indirectly) via Github.
Descriptions
For this, I followed the following tutorials:
- "GitHub Integration: Webhooks" by cloudbees
- "Triggering builds with webhooks behind a secure firewall" at Jenkins blog
- "GitHub Branch Source Plugin" by cloudbees
- Jenkins GitHub Branch Source Plugin, the plugin is installed with the recommended plugins.
Steps
- Add a new credential via the Jenkins Web UI at
Credentials> System > Global credentials (unrestricted). Use your GitHub user account and the previously created token as password. - Create a new GitHub organization (which will then spawn jobs). Go to
Jenkins > New Item > GitHub Organization. I got an exception here (Caused by: java.lang.NoClassDefFoundError: com/cloudbees/plugins/credentials/CredentialsProvider), similar to the one described as Jenkins issue 58071.- I tried to install the Blue Ocean plugin as this seems to fix the problem for some people. -- That did not help in my case.
- Restarted Jenkins (
sudo /etc/init.d/jenkins restart) helped, though!
- In the job, go to Configure and configure the job:
- Credentials: user name with password, the password is the token!
- Owner: eclipse (or whatever organization owns the repository)
- Behaviors: Add > Filter by regex: «repository name» (since you do not want to get events for everything)
- Discover pull requests from origin
- Merging the pull request with the current target branch...
- Discover pull requests from forks
- Merging the pull request with the current target branch...
- Trust: Nobody
- Pipeline Jenkinsfile: «Name of your Jenkins file»
Jenkinsfile
Actually, I did not write the following Jenkinsfile from scratch but reused the file used for my project (N4JS) on the Eclipse N4JS JIPP Instance. This file is found in the N4JS repository
However, I had to modify it a bit, due to a problem with my setup, see below for details.
/*
* Copyright (c) 2019 HAW Hamburg
* All rights reserved. This program and the accompanying materials
* are made available under the terms of the Eclipse Public License v1.0
* which accompanies this distribution, and is available at
* http://www.eclipse.org/legal/epl-v10.html
*
* Based on eclipse-nightly.jenkinsfile
*/
pipeline {
agent any
environment {
NODEJS_PATH= '/usr/bin' // '/shared/common/node-v10.15.3-linux-x64/bin'
YARN_PATH = '/usr/bin' // '/shared/common/yarn/1.15.2/bin'
MAVEN_OPTS = '-Xmx4G'
JAVA_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
TIMESTAMP = new Date().format("yyyyMMddHHmm")
}
stages {
stage('Build and Test') {
steps {
echo 'Starting Xvnc'
wrap([$class: 'Xvnc', takeScreenshot: false, useXauthority: true]) {
echo 'Building and testing, using display ' + DISPLAY
script {
def envvars = [
'-DDISPLAY=localhost'+DISPLAY,
'-DargLine="-DDISPLAY=localhost'+DISPLAY+'"'
].join(' ')
def options = [
'--batch-mode',
//'--quiet',
'--update-snapshots',
'--show-version',
'-Dtycho.localArtifacts=ignore',
// for adjusting the script, maybe uncomment the following two lines:
'-Dmaven.test.failure.ignore',
"--fail-at-end",
'-DWORKSPACE=' + env.WORKSPACE,
'-DexcludeJRE'
].join(' ')
def profiles = [
'buildProduct',
'execute-plugin-tests',
'execute-plugin-ui-tests',
'execute-ecma-tests',
'execute-accesscontrol-tests',
'execute-smoke-tests'
//'execute-hlc-integration-tests'
].join(',')
sh """\
pwd
git log -n 1
npm version
"""
sh "mvn ${envvars} clean verify -P${profiles} ${options}"
// for tests, you may want to run less tests:
// sh "mvn clean verify ${options}"
// and for quick testing: only
// sh "mvn -o clean"
// sh "ls -Ral builds/org.eclipse.n4js.product.build/target/repository/"
}
} // end wrap
} // end steps
} // end stage
}
post {
always {
junit '**/surefire-reports/**/*.xml'
}
cleanup {
// Execute after every other post condition has been evaluated, regardless of status
// See https://jenkins.io/doc/book/pipeline/syntax/#post
echo 'Cleaning up workspace'
// Not today: deleteDir()
}
}
}
- In order to run UI tests, you need to run the tests with Xvnc running (see wrap ...).
- In my case, it did not work because the DISPLAY environment variable was not correctly passed to the maven surefire tests. So I added the
evnvarsin the script and passed them to maven in the shell call. This took me several days to figure out!
When it works, you will be rewarded with a message at Github similar to this one:
Some hints in case something does not work
Do you get notifications?
In order to check that smee is really working, try the following:// content of index.js
const http = require('http')
const port = 3000
const requestHandler = (request, response) => {
console.log(request.url)
response.end('Hello Node.js Server!')
}
const server = http.createServer(requestHandler)
server.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err)
}
console.log(`server is listening on ${port}`)
})
smee -u https://smee.io/«YOUR SMEE PATH» --path /github-webhook/ --port 3000
Emulate requests
Note that you can manually post requests to smee and see if it is working (so you do not need github for that) viacurl --header "Content-Type: application/json" --request POST --data '{ "key": "val" }' https://smee.io/«YOUR SMEE PATH»
Forgot your password?
In case you forgot your Jenkins admin password, have a look at this [stackoverflow thread][https://stackoverflow.com/questions/6988849/how-to-reset-jenkins-security-settings-from-the-command-line]. For the impatient reader: Update the password hash at/var/lib/jenkins/users/username/config.xml with
<passwordHash>#jbcrypt:$2a$10$razd3L1aXndFfBNHO95aj.IVrFydsxkcQCcLmujmFQzll3hcUrY7S</passwordHash>
Update Jenkins
You may get security warning on the management page http://«server»:8080/manageIn order to update Jenkins, follow these steps:
sudo apt-get update
sudo apt-get install jenkins
You may want to manually restart Jenkins after that via sudo /etc/init.d/jenkins restart.
Thanks: Thank you Yoosiba for some hints regarding my XVNC problem (and for writing the initial Jenkins-file for the Eclipse nightly)! And thank you, mmews-n4, for review.
