jar-copier version 0.3.0 released

We just released a new version of jar-copier, 0.3.0, that includes:

  • Better reporting of misconfiguration.
  • Thoroughly testing misconfiguration reporting.
  • Added the possibility to manually specify the jars (not java-agents).

The main change is the last item, which was planned but now it became clear that some people actually wanted it.

Advertisements

Using New Relic with Clojure

This post describes how I got New Relic to run with my Clojure project. I’m using Heroku but most of what I say here should be applicable in other environments and I’ll try to point you in the right direction when it doesn’t. Please, feel free to comment with improvements or corrections.

There are already a few articles out there about this same subject but none of them gave me a complete picture, which is what I’m attempting here. I’ll cite my references at the end.

Enabling New Relic

There are two ways to enable New Relic in a Heroku app. One is through a command like this one:

heroku addons:create newrelic:wayne

The other way is through the dashboard of the app in question. It doesn’t matter which way you use, but after enabling it, your dashboard should have an entry for New Relic, like this:

Heroku Dashboard with New Relic entry

Click on “New Relic APM :: Newrelic”, which will take you to your New Relic welcome screen (unless you already have projects). If you are not using Heroku, creating a New Relic account will land you at a similar-looking page:

Fresh New Relic Dashboard

Click on APM – that’s Application Performance Management, i.e. what most people have in mind when they say “New Relic”. That will take you to the page in which you choose which programming language or framework you are using:

Getting started with New Relic

Sadly, no Clojure, so just click on Java.

If you already have a New Relic project, to create a new one, search for the “Add More” link:

New Relic Add More

Once you click on Java, you’ll see the instructions to install the Java agent in a Java project:

Installing the Java Agent

In the rest of this post I’ll explain how to get the Java agent running in a Clojure project. But first, the configuration.

Configuration

New Relic has a licence key that your app will use to both identify and authenticate itself. If you scroll down you’ll see a button to reveal that key:

Download Java agent

On that page click “Download the Java agent”. It’s a zip file containing some documentation, a bunch of jar files, and the one file you care about: newrelic.yml. Copy that file to the root of your Clojure project.

In that file, you’ll find the licence key specified in a line like this:

  license_key: 'bab654434b8672dc0580ab7f88bf9c7984dd81bd'

Having credentials in your source code repository is a bad idea. Your app should be able to be open source without your server and services being compromised. Even if your app is not open source, you may get developers and designers that you don’t trust working on it or you might use third party services to run tests, perform static analysis, etc, etc. that you don’t want to have the keys of your kingdom.

Remove the licence so that that line looks like this:

  license_key: ''

and instead make sure you export an environment variable named NEW_RELIC_LICENSE_KEY  that will be automatically picked up by New Relic to authenticate to their server:

NEW_RELIC_LICENSE_KEY="bab654434b8672dc0580ab7f88bf9c7984dd81bd"

Heroku sets this up automatically when you add the plug-in, and you can see this environment variable in your applications settings:

Heroku app's settings with New Relic

Another alternative would be to provision the servers with a full copy of newrelic.yml, an approach I use when deploying Rails applications with Capistrano but I’ve never used it with a Clojure or Java application.

In the config file, also search for the lines defining the name of your application:

common: &default_settings
  # ...
  app_name: My Application
  # ...

development:
  <<: *default_settings
  app_name: My Application (Development)

test:
  <<: *default_settings
  app_name: My Application (Test)

production:
  <<: *default_settings

staging:
  <<: *default_settings
  app_name: My Application (Staging)

and replace all mentions of “My Application” with the proper name of your application. Leave the extra bits between parenthesis as it is, so you can identify whether a report or alert is referring to production, staging or something else.

Java Agent

New Relic for Java runs as a Java agent. You can learn more about them in the java.lang.instrument documentation but in short, they are jars that are loaded by the JVM independently and around your own application. Leiningen has support for Java Agents by adding:

:java-agents [[com.newrelic.agent.java/newrelic-agent "3.20.0"]]

to your project.clj file. You can find the latest version of newrelic-agent in The Central Repository. If at this point you run your app or the repl, you’ll probably see some output like this:

Sep 4, 2015 15:27:35 +0100 [29537 1] com.newrelic INFO: New Relic Agent: Loading configuration file "newrelic.yml"
Sep 4, 2015 15:27:35 +0100 [29537 1] com.newrelic ERROR: license_key is empty in the config. Not starting New Relic Agent.

New Relic is being loaded and it’s complaining about the lack of a key. That’s a good thing.

When you deploy to production, things won’t go as smooth. Java won’t be able to find the newrelic-agent.jar in the CLASSPATH so it’ll just silently skip the agent. There’s a command line for java  that will help it locate the jar: -javaagent:newrelic-agent.jar. The problem with that is, where’s the jar? Nowhere to be seen.

The official recommendation from New Relic and other blog posts is to copy the jar file to your source tree and then reference it from there. That has two problems:

  • Having big blobs in source control is not very nice.
  • That jar is out of the loop of dependencies, so it won’t ever be upgraded unless someone remembers to explicitly do it. That’s bad.

To solve this problem I created a library called jar-copier that copies a Java agent jar into a configured directory so that you can point to it with the command line. This works in Heroku because when you deploy, the source code is checked out from git and then the uberjar is built on that same server, so jar-copier has an opportunity to run and place the jar in the correct place.

If you are shipping ready-made uberjars, you’ll have to find another way to provision your server with a copy of the agent jar in a well known location that you can point to (or that’s in the CLASSPATH ).

jar-copier is a Leiningen plug-in, so you need to add it to your list of plug-ins like this:

:plugins [; other plugins
          [com.carouselapps/jar-copier "0.2.0"]]

then, to make sure it’s run automatically, your project needs:

:prep-tasks ["javac" "compile" "jar-copier"]

and finally, you need to configure jar-copier to copy Java agents and know the destination, for example:

:jar-copier {:java-agents true
             :destination "resources/jars"}

A full example might look like this:

(defproject proclodo-spa-server-rendering "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.7.0"]]
  :plugins [[jar-copier "0.1.0"]]
  :prep-tasks ["javac" "compile" "jar-copier"]
  :java-agents [[com.newrelic.agent.java/newrelic-agent "3.20.0"]]
  :jar-copier {:java-agents true
               :destination "resources/jars"})

The last step to get the Java agent running in Heroku is to pass the right -javaagent attribute with the following command:

heroku config:set JVM_OPTS="-javaagent:resources/jars/com.newrelic.agent.java/newrelic-agent.jar $(heroku config:get JVM_OPTS)"

The next time you deploy to Heroku, in your logs, you should see:

2015-09-04T16:13:33.968914+00:00 heroku[web.1]: Starting process with command `java -javaagent:resources/jars/com.newrelic.agent.java/newrelic-agent.jar -cp target/project-x.jar clojure.main -m project-x.server`
2015-09-04T16:13:36.569062+00:00 app[web.1]: Sep 4, 2015 16:13:36 +0000 [3 1] com.newrelic INFO: New Relic Agent: Loading configuration file "newrelic.yml"
2015-09-04T16:13:36.783532+00:00 app[web.1]: Sep 4, 2015 16:13:36 +0000 [3 1] com.newrelic INFO: Agent Host: ee4b7ef8-03a0-4b76-ab17-c6850aa462ec IP: 172.16.177.186
2015-09-04T16:13:36.783663+00:00 app[web.1]: Sep 4, 2015 16:13:36 +0000 [3 1] com.newrelic INFO: New Relic Agent v3.20.0 is initializing...
2015-09-04T16:13:37.788277+00:00 app[web.1]: Sep 4, 2015 16:13:37 +0000 [3 1] com.newrelic.agent.deps.org.reflections.Reflections WARN: given scan urls are empty. set urls in the configuration
2015-09-04T16:13:37.964165+00:00 app[web.1]: Sep 4, 2015 16:13:37 +0000 [3 1] com.newrelic.agent.deps.org.reflections.Reflections INFO: Reflections collected metadata from input stream using serializer com.newrelic.agent.deps.org.reflections.serializers.JsonSerializer
2015-09-04T16:13:38.452178+00:00 app[web.1]: Sep 4, 2015 16:13:38 +0000 [3 11] com.newrelic INFO: Instrumentation com.newrelic.instrumentation.spring-aop-2 is disabled. Skipping.
2015-09-04T16:13:38.482551+00:00 app[web.1]: Sep 4, 2015 16:13:38 +0000 [3 10] com.newrelic INFO: Instrumentation com.newrelic.instrumentation.servlet-user is disabled. Skipping.
2015-09-04T16:13:38.472009+00:00 app[web.1]: Sep 4, 2015 16:13:38 +0000 [3 13] com.newrelic INFO: Instrumentation com.newrelic.instrumentation.grails-async-2.3 is disabled. Skipping.
2015-09-04T16:13:38.484518+00:00 app[web.1]: Sep 4, 2015 16:13:38 +0000 [3 10] com.newrelic INFO: Instrumentation com.newrelic.instrumentation.jcache-datastore-1.0.0 is disabled. Skipping.
2015-09-04T16:13:47.074297+00:00 app[web.1]: Sep 4, 2015 16:13:47 +0000 [3 1] com.newrelic.agent.RPMServiceManagerImpl INFO: Configured to connect to New Relic at collector.newrelic.com:443
2015-09-04T16:13:47.253668+00:00 app[web.1]: Sep 4, 2015 16:13:47 +0000 [3 1] com.newrelic INFO: Setting protocol to "https"
2015-09-04T16:13:47.253045+00:00 app[web.1]: Sep 4, 2015 16:13:47 +0000 [3 1] com.newrelic INFO: Setting audit_mode to false
2015-09-04T16:13:51.881568+00:00 app[web.1]: Sep 4, 2015 16:13:51 +0000 [3 1] com.newrelic.agent.config.ConfigServiceImpl INFO: Configuration file is /app/newrelic.yml
2015-09-04T16:13:51.944223+00:00 app[web.1]: Sep 4, 2015 16:13:51 +0000 [3 1] com.newrelic INFO: Agent class loader: sun.misc.Launcher$AppClassLoader@14dad5dc
2015-09-04T16:13:51.944000+00:00 app[web.1]: Sep 4, 2015 16:13:51 +0000 [3 1] com.newrelic INFO: New Relic Agent v3.20.0 has started
2015-09-04T16:13:51.956967+00:00 app[web.1]: Sep 4, 2015 16:13:51 +0000 [3 1] com.newrelic INFO: Premain startup complete in 16,175ms
2015-09-04T16:14:11.919171+00:00 app[web.1]: Sep 4, 2015 16:14:11 +0000 [3 27] com.newrelic INFO: Display host name is ee4b7ef8-03a0-4b76-ab17-c6850aa462ec for application project-x
2015-09-04T16:14:13.510515+00:00 app[web.1]: Sep 4, 2015 16:14:13 +0000 [3 27] com.newrelic INFO: Collector redirection to collector-175.newrelic.com:443
2015-09-04T16:14:14.082876+00:00 app[web.1]: Sep 4, 2015 16:14:14 +0000 [3 27] com.newrelic INFO: Agent 3@ee4b7ef8-03a0-4b76-ab17-c6850aa462ec/project-x connected to collector.newrelic.com:443
2015-09-04T16:14:14.082997+00:00 app[web.1]: Sep 4, 2015 16:14:14 +0000 [3 27] com.newrelic INFO: Reporting to: https://rpm.newrelic.com/accounts/1075850/applications/10426069
2015-09-04T16:14:14.082654+00:00 app[web.1]: Sep 4, 2015 16:14:14 +0000 [3 27] com.newrelic INFO: Agent run id: 45253420734418464
2015-09-04T16:14:14.101758+00:00 app[web.1]: Sep 4, 2015 16:14:14 +0000 [3 27] com.newrelic INFO: Real user monitoring is enabled with auto instrumentation for application "project-x"
2015-09-04T16:14:14.101607+00:00 app[web.1]: Sep 4, 2015 16:14:14 +0000 [3 27] com.newrelic INFO: Using RUM version 686 for application "project-x"

Reporting data

By this point, the Java agent is running and possibly reporting some data, but you are not instrumenting your code. I found a bunch of libraries that you can use to instrument your code and by far my favorite is new-reliquary because it provides Ring middleware.

At the time of this writing, the released version, 0.1.5, doesn’t support reporting data as a web transaction. New Relic will categorize the data as Non-Web, which is not what you want. Thankfully there’s already a pull request to fix this and we released it as com.carouselapps/new-reliquary 0.1.5. We are likely to request it to be deleted once it’s released properly, but until then, feel free to use our copy.

After adding it to your project, you need to add the Ring middleware. In the file where you set up your middleware add the following requires:

(ns handler
  (:require ;... other requires
            [new-reliquary.ring :refer [wrap-newrelic-transaction]]
            [ring.middleware.params :refer [wrap-params]]))

and then wrap your handler in them:

(-> handler
    wrap-params
    wrap-newrelic-transaction)

My full app definition looks like this:

(def app
  (let [handler (wrap-defaults #'routes site-defaults)]
    (if (env :dev)
      (-> handler
          wrap-exceptions
          wrap-reload)
      (-> handler
          wrap-params
          wrap-newrelic-transaction))))

which doesn’t enable New Relic in development. Yours is probably quite different though. And that’s it, you should now be seeing your performance reports in New Relic. You can see a full example in proclodo-spa-server-rendering’s

Shameless plug time! At Carousel Apps we not only use New Relic, we have a constant dashboard displaying it so we never lose sight of our current performance and servers. We’ve created a product to display dashboards like this called Screensaver Ninja. It displays websites, including New Relic, as your screensaver, so you can turn all your computers into information Screensaver Ninjadisseminators. It’s also great for permanent screens, as it displays many web sites in rotation, keeping them fresh (useful even for self updating websites, like New Relic, because sometimes they crash) as well as keeping the computer secure by using the screensaver lock.

My References

In the process of getting New Relic to work with Clojure running on Heroku, I found the following blog posts:

and this libraries:

Feel free to explore them.

jar-copier 0.1.0 released

jar-copier is a Leiningen plug in to copy jars from your dependencies to your source tree. It’s a very small simple utility that proved to be necessary to have a sane setup with Java agents (New Relic for example).

It’s very simple to use. Put [jar-copier “0.1.0”]  into the :plugins  vector on your project.clj. To run this plug in, execute:

$ lein jar-copier

If you want the task to run automatically, which is recommended, add:

:prep-tasks ["javac" "compile" "jar-copier"]

and it’ll be invoked every time you build your uberjar.

You need to configure the plug-in in your project.clj like this:

:jar-copier {:java-agents true
             :destination "resources/jars"}

:java-agents instruct this plug in to automatically copy any jars that are specified as Java agents. :destination  specifies where to copy them to.

For example, from proclodo-spa-server-rendering:

(defproject proclodo-spa-server-rendering "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.7.0"]]
  :plugins [[jar-copier "0.1.0"]]
  :prep-tasks ["javac" "compile" "jar-copier"]
  :java-agents [[com.newrelic.agent.java/newrelic-agent "3.20.0"]]
  :jar-copier {:java-agents true
               :destination "resources/jars"})

Photo by Maik Meid