Building a JIRA integration with Clojure & Atlassian Connect

Atlassian JIRA is the market leading issue-tracking and agile project management tool, and is in heavy use at companies around the globe.

For CTX, my team search app I wanted to build a slick integration to JIRA to help my customers to search their JIRA issues alongside their Slack messages, Trello cards, files, GitHub issues and emails.

When it came to actually building the integration, I was pleasantly surprised by the APIs available, and especially the Atlassian Connect API and the simple integration patterns it promotes.

While there’s lots of documentation — and even a short blog series about integrating Bitbucket with a Clojure application, I couldn’t find a single, simple tutorial that covered Connect, the webhook interface and the main JIRA API in one place.

So I wrote one.

The full source of the application we’re building is available on GitHub here.

What are we building?

We’re going to build a JIRA integration in Clojure, that listens for issues being created and mutated and creates a basic activity feed of changes.

To do this, we need a Clojure web application, listening on an HTTPS endpoint.  We’ll register the application with JIRA, and every time a user creates or updates a ticket, JIRA will send us an event on a Webhook.

We’ll record these in an in-memory database, and write a little front-end application that polls our app to display a list of changes to issues.

What is Atlassian Connect?

Connect provides a simple framework for integrating with Atlassian applications like JIRA and Confluence. It leverages OAuth2 and JWT, along with some simple integration patterns, to give developers a nice and consistent way to talk to the Atlassian API suite.

In its’ most basic form, all you need to do is publish a Descriptor listing the endpoints you provide and the services you require, point your Atlassian application at it, and then deploy some code on the listed endpoints that talks HTTP / JSON.

Pre-conditions

First, install Leiningen, and make sure it’s in your PATH.  And get yourself a Java JDK if you haven’t got one.

Check it all works:

$ lein --versionLeiningen 2.8.1 on Java 9.0.4 Java HotSpot(TM) 64-Bit Server VM

Basic project setup

Let’s create a basic Clojure app skeleton using the Compojure template.

$ lein new compojure connect-example$ cd connect-example$ ls-rw-r--r--  1 rory  Users  273 20 Mar 16:21 README.md-rw-r--r--  1 rory  Users  483 20 Mar 16:21 project.cljdrwxr-xr-x  3 rory  Users  102 20 Mar 16:21 resources  drwxr-xr-x  3 rory  Users  102 20 Mar 16:21 src  drwxr-xr-x  3 rory  Users  102 20 Mar 16:21 test$ lein ring server-headlessRetrieving lein-ring/lein-ring/0.9.7/lein-ring-0.9.7.pom from clojars  ...2018-03-20 16:24:40.501:INFO:oejs.Server:jetty-7.6.13.v20130916  2018-03-20 16:24:40.550:INFO:oejs.AbstractConnector:Started [email protected]:3000

In a second tab, use curl to test it's running:

$ curl http://localhost:3000Hello World

Great! We’ve got a running app.

Import basic dependencies

Now we need to add some libraries to our Clojure project to let us do the basics of HTTP comms, the Connect flow and so on.

Open up the file project.clj in an editor of your choice, and make it look like this:

(defproject connect-example "0.1.0-SNAPSHOT"  :min-lein-version "2.0.0"  :dependencies [[org.clojure/clojure "1.9.0"]                 [compojure "1.6.0"]                 [ring/ring-defaults "0.3.1"]                 [ring/ring-json "0.4.0"]                 [clj-connect "0.2.4"]]  :plugins [[lein-ring "0.9.7"]]  :ring {:handler connect-example.handler/app}  :resource-paths ["resources"]  :profiles  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]                        [ring/ring-mock "0.3.0"]]}})

The libraries in the :dependencies clause give us HTTP serving (ring), routing (compojure) and a simple wrapper around the Atlassian Connect API & underlying JWT technology (clj-connect)

Connecting to JIRA

JIRA will only talk to our application over HTTPS, so the first hurdle is how to get it on the internet.

Without setting up a server, or using something like Heroku, this can be annoyingly time consuming, so here’s a simple way round it: ngrok.

ngrok lets you create an externally accessible SSL endpoint to test your integration software. You just run it like this;

$ ngrok tls 3000ngrok by @inconshreveable    Session Status                online  Account                       Rory Gibson  Version                       2.2.8  Region                        United States (us)  Web Interface                 http://127.0.0.1:4040  Forwarding                    http://9a69173f.ngrok.io -> localhost:3000  Forwarding                    https://9a69173f.ngrok.io -> localhost:3000Connections                   ttl     opn     rt1     rt5     p50     p90                                0       0       0.00    0.00    0.00    0.00

and it will create a tunnel, and give you a URL — e.g. https://9a69173f.ngrok.io - you can use to access your software, running on your development machine but accessible over SSL on the internet.

Now that we have a service that’s discoverable externally, we can generate our OAuth credentials in JIRA.

Serving the Descriptor

Now we’ve got all the plumbing sorted, we can serve up the Connect Descriptor.

This is a simple JSON file that declares the application’s capabilities and dependencies, so that JIRA (and other Atlassian applications) know how to talk to it.

Create a file at <project>/resources/public/connect.json and paste the following content into it (ensure that the baseUrl property matches your ngrok URL):

{    "key": "connect-example",    "name": "connect-example",    "description": "Demo JIRA / Clojure integration",    "baseUrl": "https://9a69173f.ngrok.io",    "authentication": {        "type": "jwt"    },    "lifecycle": {        "installed": "/installed"    },    "modules": {        "oauthConsumer": {            "clientId": "PLACEHOLDER"        },        "webhooks": [            {                "event": "*",                "url": "/recv"            }        ]            },    "scopes": ["READ"]}

Handling lifecycle callbacks

The Connect API works by loading your descriptor, then making a call to /installed on your app as a "lifecycle callback" (there are others, like Uninstalled, that we're ignoring here).

All it wants is a 200 response.

We’ll use Compojure and Ring to create a simple handler that serves this function (and provides basic API plumbing)

(ns connect-example.handler  (:require [compojure.core :refer :all]            [compojure.route :as route]            [ring.middleware.defaults :refer [wrap-defaults api-defaults]]            [ring.middleware.json :refer [wrap-json-body wrap-json-response]]            [ring.util.response :refer [response]]))(defroutes app-routes  (POST "/installed" [] {:status 200 :body "OK"}))(def app  (-> app-routes    wrap-json-body    wrap-json-response    (wrap-defaults api-defaults)))

Now that we have this much, we can wire it up to JIRA in the Cloud. Save everything and restart the app (lein ring server-headless)

Go and login to your JIRA instance and click Settings (the cog icon) > Add-Ons.

Go to the tiny Settings link at the bottom of the page.

and ensure that Enable development mode and Enable private listings are both ticked.

Now you should see a link to Upload add-on — click this and paste in the fully-qualified HTTPS URL to your descriptor (e.g. https://abcd1234.ngrok.com/connect.json)

Handling events

Once Connect has hooked up to your app, you’ll start receiving the events we specified in the Descriptor whenever they occur.  All we need is some handlers and some logic on our side.

Add 2 routes to the routing block — one for JIRA to call when an event happens, and one for us to call to list events.

(POST "/recv" req (recv req))(GET "/events" req (response (events req)))

and the functions for them to call.

recv gets the payload, extracts the data we want and stores it in an atom in the server (instead of a DB, this is only a demo :)

(def event-store (atom '()))(defn persist!  "Save event to local in-memory structure"  [e]  (swap! event-store conj e)  (println e) ;; debug info for development  e)(defn transform  "Get the fields we care about from the JIRA payload"  [e1]  (let [id        (-> e1 :issue :key)        user      (-> e1 :user :displayName)        timestamp (-> e1 :timestamp)        type      (-> e1 :webhookEvent)        summary   (-> e1 :issue :fields :summary)]    {:user user :id id :timestamp timestamp :summary summary :type type}))(defn handle-event  [raw]  (-> raw    clojure.walk/keywordize-keys    transform    persist!))(defn events  [_]  @event-store)(defn recv  "Handler function for all events"  [req]  (handle-event (:body req))  {:status 200 :body "OK"})

Restart your app

Hit Ctrl-C then lein ring server-headless again.

If you change an Issue in your JIRA (say by moving it from To Do to Done in the agile view) you should see some output in your terminal as JIRA calls our app and we save the data, and dump it to screen with the println above.

Hit the /events endpoint to check we're saving the data and you should see something like this:

$ curl http://localhost:3000/events[{"summary":"My issue 1,"timestamp":1521891438142,"type":"jira:issue_updated","id":"TEST-1"}]

Adding a UI

That’s great! We have an inbound and outbound API.  But most non-developers don’t like to use curl for everything, so we'd better add a UI.

We’ll build something in ClojureScript, so that we can use the same paradigms and dialect across front and back-end codebases.

$ mkdir -p src/clj$ mv src/connect-example src/clj$ mkdir -p src/cljs/connect-example$ touch resources/public/index.html

Now edit your project.clj to the following (pulling in the new directories and adding the ClojureScript libraries and build configuration)

(defproject connect-example "0.1.0-SNAPSHOT"  :min-lein-version "2.0.0"  :dependencies [[org.clojure/clojure "1.9.0"]                 [org.clojure/clojurescript "1.10.217"]                 [compojure "1.6.0"]                 [ring/ring-defaults "0.3.1"]                 [ring/ring-json "0.4.0"]                 [rum "0.11.2"]                 [clj-connect "0.2.4"]                 [cljs-ajax "0.7.3"]]  :plugins [[lein-ring "0.9.7"]            [lein-figwheel "0.5.13"]]  :source-paths ["src/clj"]  :resource-pahts ["resources" "target/cljsbuild"]  :ring {:handler connect-example.handler/app}  :profiles  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]                        [ring/ring-mock "0.3.2"]]}}  :clean-targets ^{:protect false} [:target-path "out" "resources/public/cljs"]  :figwheel {:css-dirs ["resources/public/css"]             :server-port 3000             :ring-handler connect-example.handler/app}  :cljsbuild {              :builds [{:id "dev"                        :source-paths ["src/cljs"]                        :figwheel true                        :compiler {:main "connect-example.core"                                   :asset-path "cljs/out"                                   :output-to  "resources/public/cljs/main.js"                                   :output-dir "resources/public/cljs/out"}                        }]              })

Create and fill in your resources/public/index.html

<!DOCTYPE html>  <html>    <head></head>  <body>    <div id="content"></div>    <script src="app.js" type="text/javascript"></script>  </body></html>

and src/cljs/connect-example/core.cljs...

(ns connect-example.core)(let [el (.getElementById js/document "content")]  (set! (.-innerText el) "Hello World"))

Which will, when it’s loaded, find the div with ID content and put an informative message into it.

Now let’s try to run it, using the awesome Figwheel plugin;

$ lein figwheelRetrieving figwheel/figwheel/0.5.13/figwheel-0.5.13.pom from clojars  Retrieving org/clojure/core.async/0.3.443/core.async-0.3.443.pom from central  ...Figwheel: Validating the configuration found in project.clj  Spec Warning:  missing an :output-to option - you probably will want this ...  Figwheel: Configuration Valid ;)  Figwheel: Starting server at http://0.0.0.0:3449  Figwheel: Watching build - dev  Figwheel: Cleaning build - dev  Figwheel: Starting server at http://0.0.0.0:3449  Figwheel: Watching build - dev  Compiling "main.js" from ["src/cljs"]...  Successfully compiled "main.js" in 1.95 seconds.  Launching ClojureScript REPL for build: dev  Figwheel Controls:            (stop-autobuild)                ;; stops Figwheel autobuilder          (start-autobuild [id ...])      ;; starts autobuilder focused on optional ids          (switch-to-build id ...)        ;; switches autobuilder to different build          (reset-autobuild)               ;; stops, cleans, and starts autobuilder          (reload-config)                 ;; reloads build config and resets autobuild          (build-once [id ...])           ;; builds source one time          (clean-builds [id ..])          ;; deletes compiled cljs target files          (print-config [id ...])         ;; prints out build configurations          (fig-status)                    ;; displays current state of system          (figwheel.client/set-autoload false)    ;; will turn autoloading off          (figwheel.client/set-repl-pprint false) ;; will turn pretty printing off  Switch REPL build focus:          :cljs/quit                      ;; allows you to switch REPL to another build    Docs: (doc function-name-here)    Exit: Control+C or :cljs/quit Results: Stored in vars *1, *2, *3, *e holds last exception objectPrompt will show when Figwheel connects to your application

Load up a browser tab pointed to http://localhost:3000 and you should see, in your browser, the message "Hello World".

Hello World

Some React magic

ClojureScript web development is heavily focussed on the React ecosystem.

Several React bindings exist; one of the easiest to get started with is Rum.

Add the following to the :dependencies key in your project.clj:

[cljs-ajax "0.7.3"][rum "0.11.2"]

Modify core.cljs so it includes the require for Rum and a component definition (which creates a React component under the hood), then mounts it on the document body.

(ns connect-example.core  (:require [rum.core :as rum :refer [defc mount]))(defc label [text]  [:div {:class "label"} text])(mount (label "Hello again") js/document.body)

Restart Figwheel (Ctrl-C in your terminal then lein figwheel again) and refresh your browser, and that's it - you're seeing a React rendered application!

A simple list-based UI

Now we have the basic structure to put a React UI on screen, let’s fill it in with something that’s actually useful.

To match the wireframe above, we’ll have an H1 for the title:

(defc heading []  [:h1 "JIRA Activity Feed"])

And a component to iterate over a list of JIRA issues — so we’ll call the display of each result an “issue”. We’ll create some dummy data to render initially, too.

(defc issue  [{:keys [id summary type timestamp] :as issue}]  [:div.issue   [:div.id id]   [:div.info    [:div.title summary]    [:div.detail type]    [:div.date timestamp]]])(defc issues  [is]  [:div.issues   (map issue is)])(def fake-issues '({:id "FOO-1" :summary "The Title" :type "jira:issue_updated" :timestamp 123456789}))(defc content  [data]  [:div.container   (heading)   (if data     (issues data)     [:div "No activity found in feed."])])(mount (content fake-issues) (.getElementById js/document "content"))

Save everything and reload your browser and you should see a very basic list and title.

It could probably do with some styling, so let’s tweak our index.html as follows:

<!DOCTYPE html>  <html>    <head>    <link rel="stylesheet" href="/site.css" />  </head>  <body>    <div id="content"></div>    <script src="cljs/main.js" type="text/javascript"></script>  </body></html>

Create a file resources/public/css/site.css with some simple styles:

body {    font-family: sans-serif;}.container {  margin-left: auto;  margin-right: auto;  padding-top: 30px;  width: 800px;}div.issue {    font-size: 16px;  width: 600px;  border: solid black 1px;  padding: 1em;  margin-bottom: 1em;}.issue .id {  font-weight: bold;  float: left;  width: 75px;  height: 50px;  text-align: center;  padding: 1em .5em 0 0;}.issue .info {  height: 100%;}.issue .summary {  font-weight: bold;  padding-bottom: 0.5em}

Restart lein figwheel again and reload and everything should be perfect.

NB: once you get to this stage, Figwheel’s most awesome benefit kicks in — hot auto-reload of ClojureScript and CSS files. Just save in your editor and the browser updates on its’ own!

AJAX comms

Now let’s get the UI to query the server side for our JIRA information…

Add a new require so the top of core.cljs looks like this:

(ns connect-example.core  (:require [rum.core :as rum]            [ajax.core :refer [GET]]))

and then let’s add a function to fetch and display the data (and actually call that function to do the display)

Remove the existing mount form and add this to the end of core.cljs:

(defn handler [data]  (mount (content data) (.getElementById js/document "content")))(fetch-issues handler)

Finally, let’s get the browser app to poll the server every 5 seconds to update the feed.

(js/setInterval  #(fetch-issues handler)  5000)

What have we got?

We’ve built a dynamically-updating ClojureScript React application that’s showing live changes from JIRA.

It turns out it’s surprisingly easy to integrate a Clojure app with JIRA through Atlassian Connect.

Evidently, what we’ve built here is only the barest beginnings of a production app; in real life you’d want to check the validity of inbound events using the JWT functionality built into the Connect API, and you’d at least want a proper database of some sort to persist your data into.

And remember — the full working source to this application is on GitHub here for your reference: https://github.com/getctx/connect-example

One more thing…

If you like what you see here, please check out our application, CTX, which lets you and your team search your apps like JIRA, Trello, Slack, GitHub, Google Drive, email and more in one shot. We offer a free trial with no credit card required.  Thanks!

Originally published at getctx.io on March 24, 2018.