Adding a RESTful Client to Jenkins Pipelines

Roughly a year ago, I worked on a project to simplify the numerous Jenkins build configurations we had to maintain using Jenkins Pipelines , which greatly improved our overall build quality and ease of introducing new builds/projects. However, one thing that repeatedly popped up was our reliance on cURL for making calls to external APIs - escaping quotes in the wrong place, checking whether the build was running on a Linux or Windows slave, and various others. I decided to fix this by adding a Groovy REST client to the workflow-libs, so that Jenkins could GET/POST/DELETE without dropping to a shell, and along the way ran into a number of problems that I've decided to write down for the next time I forget all about them.

If you're just looking for the code, check out this repo .

Jenkins Setup

First of all, you'll need Jenkins, naturally. I prefer using Docker for local development, so let's fetch the latest release:

docker pull jenkinsci/jenkins:2.68-alpine

Make a directory for the Jenkins home folder, so we can keep our changes on restarts, expose a port for the web interface, and let's get started:

mkdir -p ~/jenkins
docker run -p 8080:8080 -d -v /home/incognitjoe/jenkins:/var/jenkins_home --name jenkins  jenkins

Note: if you intend to use workflow-libs as a git repository, expose the SSHD port you configure in the Jenkins control panel as well as 8080.

Use docker logs jenkins to check the initial password it sets on startup, log in, and install the recommended plugins, which include all of the Pipeline plugins. After creating your user account, you should be greeted with a totally empty Jenkins. Restart the docker container just to make sure everything's applied correctly.

Now, if you browse to ~/jenkins, you should see a workflow-libs folder. Read the pipeline docs for more information about this, but basically, we're going to adding some Groovy scripts in here that can be used from Jenkins jobs.

Development setup

See the gradle-related files here for my setup. We're going to be using Jodd as the base HttpClient.

Why Gradle?

I don't like Maven.

Why Jodd, instead of the much more common HttpBuilder?

The Jenkins runtime is a harsh and unforgiving land. Initially I used HttpBuilder, and while GETs performed totally fine, have a snippet of the stack trace that happened when I tried to POST something:

[Pipeline] End of Pipeline
java.lang.UnsupportedOperationException: Refusing to marshal org.codehaus.groovy.runtime.MethodClosure for security reasons
    at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:452)
    at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
    at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
    at com.thoughtworks.xstream.converters.reflection.SerializableConverter$1.writeToStream(SerializableConverter.java:140)
    at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.writeObjectOverride(CustomObjectOutputStream.java:84)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at java.util.HashMap.internalWriteEntries(HashMap.java:1785)
    at java.util.HashMap.writeObject(HashMap.java:1362)
    at sun.reflect.GeneratedMethodAccessor148.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135)
Caused: com.thoughtworks.xstream.converters.ConversionException: Could not call groovyx.net.http.StringHashMap.writeObject() : Refusing to marshal org.codehaus.groovy.runtime.MethodClosure for security reasons
---- Debugging information ----
message             : Could not call groovyx.net.http.StringHashMap.writeObject()
cause-exception     : java.lang.UnsupportedOperationException
cause-message       : Refusing to marshal org.codehaus.groovy.runtime.MethodClosure for security reasons
-------------------------------

I couldn't figure out any way around this without an enormous amount of effort, so finding Jodd was a lifesaver.

Where on earth are your tests?

So, here's the thing: Jenkins will allow us to use @Grab to install dependencies at runtime, since workflow-libs isn't compiled ahead of time. However, it seems that there's a slight issue with trying to test scripts that use @Grab with Spock. Considering you'll likely encounter errors within the Jenkins runtime that you can't reproduce locally no matter what(see above: that marshalling error is never going to happen on your local machine), any tests you write for this stuff are going to be limited in value - that's why we're using a Jenkins instance as our testbed instead.

Workflow-libs Explained

Let's create two directories in workflow-libs, src/ and vars/. Anything we add to src is a Library function, while vars are Globals that can be accessed from any Jenkins Pipeline job. Libraries can be any valid Groovy code(mostly - nested closures seem to cause endless amounts of grief, including silently failing to execute - your mileage may vary), but cannot access the Jenkins DSL - so, if you want to drop to the shell with a Jenkins sh command, that should go into a script in vars. More information is available here . In particular, check the note about vars loading as singletons, since that can lead to surprising behaviour.

With all that said, our HttpClient should only need URLs and content to send, without needing to access environment variables and the like, so let's create src/org/incognitjoe/JenkinsHttpClient.groovy and add the following:

#! /usr/bin/groovy
package org.incognitjoe

import groovy.json.JsonBuilder
@Grab("org.jodd:jodd-http:3.8.5")
import jodd.http.HttpRequest

/**
 * Helper class for making REST calls from a Jenkins Pipeline job.
 */
class JenkinsHttpClient {

    private HttpRequest httpRequest
    private String userAgent = 'Jenkins'

    JenkinsHttpClient() {
        httpRequest = new HttpRequest()
    }

    /**
     * GET method
     * @param url
     * @return response body as String
     */
    def get(String url) {
        def resp = httpRequest.get(url)
                .header("User-Agent", userAgent)
                .send()
        return resp.bodyText()
    }

    /**
     * POST method, convert body Map to application/json.
     * @param url
     * @param body
     * @return response body as String
     */
    def postJson(String url, Map<?, ?> body) {
        String jsonbody = new JsonBuilder(body).toString()
        def resp = httpRequest.post(url)
                .header("User-Agent", userAgent)
                .contentType('application/json')
                .body(jsonbody)
                .send()
        return resp.bodyText()
    }

    /**
     * DELETE method
     * @param url
     * @return
     */
    def delete(String url) {
        def resp = httpRequest.delete(url)
                .header("User-Agent", userAgent)
                .send()
        return resp
    }
}

Nothing enormously complicated here. Now, to actually use this, we'll create vars/testingStuff.groovy and add the following:

// Change your package name to match your src structure
import org.incognitjoe.JenkinsHttpClient

def print_ghibli_films() {
    JenkinsHttpClient http = new JenkinsHttpClient()
    println(http.get('https://ghibliapi.herokuapp.com/films/'))
}

def notifySlack(String slackHookUrl, Map<?, ?> postBody) {
    JenkinsHttpClient http = new JenkinsHttpClient()
    http.postJson(slackHookUrl, postBody)
}

Might as well create vars/testingStuff.txt as well, and add whatever help text you want - this file is used to display help information in the Jenkins interface.

...and we're done. Now, how do we actually use these functions? Well, the filename of anything in vars is available to Pipeline jobs as a step, so we can create a Pipeline job in Jenkins, and enter this script:

// Master node
node() {
    testingStuff.print_ghibli_films()
}

If everything's lined up correctly (see Common Errors and Gotchas below), running that build should print out a bunch of information from the Studio Ghibli API. Replace and extend as required.

Common Errors and Gotchas

If you're reading this far, presumably you're making changes to the Pipeline files since film data isn't all that useful to you, so there's a couple of things you should watch out for:

script security settings: Jenkins will block a pretty considerable number of method calls by default, which you'll either need to review and approve as they happen in the control panel, or you can throw caution to the wind and install the Permissive Script Security plugin.
closures not executing: this comes up in odd places, but if you have a few nested closures and you're getting null values where you'd expect anything else at all, try rewriting your code to not use closure composition.
'No such DSL method': this error can be a bit overwhelming, but if you're absolutely sure you haven't made a typo when calling your custom functions, then you're likely looking for an argument mismatch, e.g. you tried to call a function that requires two args but only gave one, or you passed a List where a String was expected.
SSL errors: remember, any certs you need to trust have to be added to the Jenkins JVM keystore, not the system trust.