servers

Isomorphic JavaScript (with ClojureScript) for pre-rendering single-page-applications, part 2

In part 1 I covered the basic problem that SPA (single page applications) face and how pre-rendering can help. I showed how to integrate Nashorn into a Clojure app. In this second part, we’ll get to actually do the rendering as well as improving performance. Without further ado, part 2 of isomorphic ClojureScript.

Rendering the application

Now to the fun stuff! It would be nice if we had a full browser running on the server where we could throw our HTML and JS and tell it go! but unfortunately I’m not aware of such thing. What we’ll do instead is call a JavaScript function that will do the rendering and we’ll inject that into our response HTML.

The function to convert a path into HTML will be called render-page and it’ll be in core.cljs:

(defn ^:export render-page [path]
  (reagent/render-to-string [(parse-path path)]))

We need to mark this function as exportable because JavaScript optimizations can be very aggressive even removing dead code and since this code is called dynamically from Clojure, it’ll look like it’s unused and it’ll be removed.

render-page  is similar to mount-root but instead of causing the result to be displayed to the user, it just returns it. The former takes the path as an argument while the latter reads it from the local state which is in turn set by Pushy by reading the current URL.

To invoke that function, we’ll go back to handler.clj, just after we define js-engine we’ll define a function called render-page:

render-page (fn [path]
              (.invokeMethod
                ^Invocable js-engine
                (.eval js-engine "projectx.core")
                "render_page"
                (object-array [path])))

and instead of sending a message about the application is loading, we just call it:

[:div#app [:div (render-page path)]]

That extra div is not necessary, it’s there only because projectx.core/current-page adds it and without it you’ll get a funny error in the browser:

React error about server rendering not matching

Aside from that little trip into the internals of React, which is interesting, we now have a snappy, pre-rendered application… that is… if you can wait 3 seconds or so for it to load:

Server side scripting taking too long to load

That is not good, not good at all. We have a serious performance problem here, we need to get serious about fixing it.

Performance

The first step to fix any performance problems is making sure you have one, as premature optimization is the root of all evil. I think we are at this point with this little project. The second step is measuring the problem: we need a good repeatable way of measuring the problem that allows us to actually locate it and and verify it was fixed.

To measure the performance behaviour of this app I’m going to use one of Heroku’s bigger instances, the Performance-L, which is a dedicated machine with 14GB of RAM. The reason is that I don’t want out of memory or my virtual CPU affected by other instances to muddy my measurements. That unacceptable 3 seconds load time was measured in that type of server.

To perform the load and the measurement of the response I’m going to use the free version of BlazeMeter, an web application to trigger load testing which I’m falling in love with. The UI is great. I’m going to hit the home and the about page with their default configuration which includes up to 20 virtual users:

BlazeMeter configuration

In all the tests I’m going to make a few requests to the application manually after any restart to make sure the application is not being tested in cold. Ok… go!

Performance with naive script engine

That is terrible! Under load it behaves so much worst! 17.1s response time. Now that we have a way to measure how horrendous our application is behaving, we need to pin-point which bit is causing this. The elephant in the room is of course server-side JavaScript execution.

Disabling the server side JavaScript engine causes load times to go down:

Load time without scripting engine

but what we really care about is the load testing:

Load testing without script engine

40ms vs 17000ms, that’s a big difference! The scripting engine is definitely the problem, so, what now?

Optimizing time

Now it’s time to find optimizations. Poking around Nashorn it seems the issue is that it has a very slow start. We already know that browsers spend a lot of time parsing and compiling JavaScript and the way we are using Nashorn, we are parsing and compiling all our JavaScript in every request. Clearly we should re-use this compiled JavaScript.

Re-using Nashorn is not straightforward because it’s not thread safe while our server is multi-threaded. JavaScript just assumes that there’s one and only thread and when developing Nashorn they decided to respect that and not make any other assumptions, which leads to a non-thread-safe implementation. We need to re-use Nashorn engines, but never at the same time by two or more threads.

Nashorn does provides a way to have binding sets, that is, the state of a program, separate from the Nashorn script engine, so that you could use the same engine with various different states. Unfortunately this is very poorly documented. Fortunately, ClojureScript is immutable, so we don’t have much to worry about breaking state.

After a lot of experimentation and poking, I came up with an acceptable solution using a pool. My choice was to use Dirigiste through Aleph‘s Flow. To do that, we extract the creation of a JavaScript engine into its own function:

(defn create-js-engine []
  (doto (.getEngineByName (ScriptEngineManager.) "nashorn")
    (.eval "var global = this")
    (.eval (-> "public/js/server-side.js"
               io/resource
               io/reader))))

Then we define the pool. In Dirigiste, each object in the pool is associated to a key, so that effectively it’s a pool of pools. We don’t need this functionality, so we’ll have a single constant key:

(def js-engine-key "js-engine")

and without further ado, the pool:

(def js-engine-pool
  (flow/instrumented-pool
    {:generate   (fn [_] (create-js-engine))
     :controller (Pools/utilizationController 0.9 10000 10000)}))

flow is aleph.flow and Pools is io.aleph.dirigiste.Pools. In this pool you can have different controllers which create new objects in different ways. The utilization controller will attempt to have the pool at 0.9, the first arg, so that if we are using 9 objects, there should be 10 in the pool. The other two args is the maximum per key and the total maximum and they are set two numbers that are essentially infinite.

The reason for such a big pool is that you should never run out of JavaScript engines. If your server is getting too many requests for the amount of RAM, CPU or whatever limit you find, it should be throttled by some other means, not by an arbitrary pool inside it. Normally you’ll throttle it by limiting the amount of worker threads you have or something like that.

The function render-page was promoted to be top level and now takes care of taking a JavaScript engine from the pool and returning it when done:

(defn render-page [path]
  (let [js-engine @(flow/acquire js-engine-pool js-engine-key)]
    (try (.invokeMethod
           ^Invocable js-engine
           (.eval js-engine "projectx.core")
           "render_page"
           (object-array [path]))
         (finally (flow/release js-engine-pool js-engine-key js-engine)))))

The function to render the app now doesn’t create any engines, it just uses the previous method:

(defn render-app [path]
  (html
    [:html
     [:head
      [:meta {:charset "utf-8"}]
      [:meta {:name    "viewport"
              :content "width=device-width, initial-scale=1"}]
      (include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
     [:body
      [:div#app [:div (render-page path)]]
      (include-js "js/app.js")]]))

Let’s load test this new solution:

Load testing with script engine pool

That is a big difference. It’s almost as fast as no server side scripting! You can find this change in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx/… as well as the full final project: https://github.com/carouselapps/isomorphic-clojurescript-projectx/tree/nashorn

Future

There are a few problems or potential problems with this solution that I haven’t addressed yet. One of those is that at the moment I’m not doing anything to have Nashorn generate the same cookies or session as we would have in the real browser.

This pool works well when it’s under constant use, but for many web apps that do not see than level of usage, the pool will kill all script engines which means every request will have to create a fresh one. Solving this might require creating a brand new controller, a mix between Dirigiste’s Pools.utilizationController  and Pools.fixedController.

A big thanks to DomKM for his Omelette app, that was a source of inspiration.

Another approach worth considering is to implement the rendering system in portable Clojure (cljc), the common language between Clojure and ClojureScript and have it run natively on the server, without the need of a JavaScript engine. I’m very skeptical of this working in the long run as it means none of your rendering function can ever use any JavaScript or if they do, you need to implement Clojure(non-Script) equivalents.

This approach is being explored by David Tanzer and he wrote a blog post about it: Server-Side and Client-Side Rendering Using the Same Code With Re-Frame. David’s approach is to use Hiccup to do the rendering on the server side, where React and Reagent are not available. I personally prefer to steer clear of template engines that are not safe by default, like Hiccup at the time of  this writing, as they make XSS inevitable. The only reason why I’m using it in projectx is because that’s what the template provided and I wanted to do the minimum amount of changes possible.

Another optimization I briefly explored is not doing the server side rendering for browsers that don’t need it, that is, actual browser being used by people, like Chrome, Firefox, Safari, even IE (>10). The problem is that many bots do identify themselves as those types of browsers and Google gets very unhappy when its bots see a different page than the browsers, so it’s dangerous to perform this optimization except, maybe, for pages that you can only see after you log in.

In conclusion I’m happy enough with this solution to start moving forward and using it, although I’m sure it’ll require much tweaking an improvement. Something I’m considering is turning it into a library, but this library would make quite a bit of assumptions about your application, how things are rendered, compiled, etc. What’s your opinion, would you like to see this code expressed as a library or are you happy to just copy and paste?

Photo by Jared Tarbell

servers

Isomorphic JavaScript (with ClojureScript) for pre-rendering single-page-applications, part 1

I don’t think I have found the ultimate solution for this problem yet but I have reached a level in which I’m comfortable sharing what I have because I believe it’ll be useful for other people tackling the same problem.

The reason why I doubt this is the ultimate solution is because it has not been battle tested enough for my taste. I haven’t used it in big applications and I haven’t used in production, maintaining it for months or years.

The problem

We are building SPAs, that is, single page applications. Think Google Maps or GMail. When you request the page, you get a relatively small HTML and a huge JavaScript app. This browser app then renders the page and from now on reacts to your interactions, requesting more data from the server whenever needed but never reloading the whole web page.

The reason to develop an application like this is that the user experience ends up being much better. The app feels faster, snappier, more alive. Reloading the whole page, parsing CSS, JavaScript and HTML is slow, but rendering a snippet of HTML is fast. Furthermore, once you have a full app on the client you can start taking advantage of it, performing, for example, validation, storing data than you won’t request again, etc. which saves talking to the server, making the user experience much better.

The problem, though, is that in the initial request you are not sending any content and many web consumers won’t run JavaScript to render your application. I’m talking about search engine bots, snippet generation bots (like the one Facebook, LinkedIn and Twitter use). Even though it seems Google’s bot is executing some JavaScript, it might not be wise to depend on it.

Snippet and image generated by Facebook

Snippet and image generated by Facebook

The solution is to run the client side of the application on the server up to the point of waiting for user interaction, generating the HTML that matches that page, and shipping that to the browser. This also help with the fresh page experience as the user will quickly get some content instead of having to wait for a lot of JavaScript to be parsed, compiled and executed (take a look at GMail and how long it takes to load and show you content).

GMail loading

GMail loading…

JavaScript, on the server

Running the client JavaScript on the server is often referred to as isomorphic JavaScript, meaning, same form, that is, same code, running on both server and client. There are several server-side (no windows, headless) JavaScript implementations to chose from:

When choosing my approach I was looking for a simple solution, one with the least moving parts to make it easier to deploy and more stable over time. Nashorn was an immediate winner as it ships with Java 8 and it’s well integrated, hiding away secondary processes and inter-process communication (if it’s happening at all, I’m not sure, and this is good).

Nashorn came with two big issues though:

  • It’s slow to create new Nashorn instances (this might be true for all JS implementations).
  • The documentation is not great.

I think I have overcame both of this issues, so, without further ado, let’s jump in. You can create a new script engine like this:

(.getEngineByName (ScriptEngineManager.) "nashorn")

ScriptEngineManager has many methods to get a script engine, some use the mime type, or the extension, and with those, you may or may not get Nashorn. I prefer to explicitly request Nashorn as it should be available on all Java 8 installations and I don’t believe we can transparently switch JavaScript implementations as they might be too different.

Once you have a script engine, evaluating code is very easy:

(.eval js-engine "var hello = 'world'")

The method eval can also take files, streams, etc. Invoking a JavaScript method is a bit more involved:

(.invokeMethod ^Invocable js-engine
               js-object
               "method_name"
               (object-array [arg1 arg2 arg3])

That will invoke the method method_name in the JavaScript object js-object which you can obtain this way:

(.eval js-engine "object_name")

There’s a lot more to Nashorn but that’s all we are going to use for implementing server-side JavaScript/ClojureScript.

The application

We’ll start from a reagent application created by:

lein new reagent projectx

which you can start by running:

lein figwheel

You can find all the code for this little application in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx. When you visit the app, you’ll briefly see this:

ClojureScript has not been compiled

That page, which you can find in handler.clj, is the actual HTML sent to the browser, before the ClojureScript/JavaScript kicks in:

(def home-page
  (html
   [:html
    [:head
     [:meta {:charset "utf-8"}]
     [:meta {:name "viewport"
             :content "width=device-width, initial-scale=1"}]
     (include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
    [:body
     [:div#app
      [:h3 "ClojureScript has not been compiled!"]
      [:p "please run "
       [:b "lein figwheel"]
       " in order to start the compiler"]]
     (include-js "js/app.js")]]))

Or in actual HTML:

<html>
<head>
    <meta charset="utf-8"/>
    <meta content="width=device-width, initial-scale=1" name="viewport"/>
    <link href="css/site.css" rel="stylesheet" type="text/css"/>
</head>
<body>

ClojureScript has not been compiled!

please run lein figwheel in order to start the compiler

http://js/app.js </body> </html>

In production, you’ll normally want to show a message about the application being loaded. Here we are going to try to replace it with the actual rendered application.

After seeing that page briefly, ClojureScript gets compiled to JavaScript, served to the browser, executed and it renders the homepage, which looks like this:

Rendered homepage

This template conveniently ships with two pre-built pages, the home page and the about page. Click in the link to go to the about page and you’ll see its content but no request was sent to the server. All content was shipped before and the rendering happens client side:

About page with Network traffic

If we request that URL, we’ll se the same loading message and then the about page is going to be shown, but there’s a problem. The server doesn’t know that the about page was being requested because the fragment, the bit after the # in the URL, is not sent to the server.

Proper URLs

The reason why a fragment is used that way is because we don’t want to send a request to the server when we click a link and that’s what browsers do when you go from /blah#bleh to /blah#blih. Thankfully HTML 5 comes to the rescue with its history API. You can learn more about it in Dive into HTML5: Manipulating History for Fun & Profit. If you are wondering whether it’s safe to use this feature already, all current browsers support it (except Opera Mini) and IE since version 10:History Browsers support 2015-09

To move forward with server side rendering of SPAs you need to switch to HTML5 History, which is implemented in ClojureScript by a library called Pushy. While you are at it, I also recommend to switch to an bidirectional routing library like bidi or silk. To make the long story short, you can look at the diff to implement bidi and Pushy in projectx.

Now that the we are using sane URLs, we need to process them on the server side. In the file handler.clj we’ll find the main HTML template, the routes and the app:

(def home-page
  (html
   [:html
    [:head
     [:meta {:charset "utf-8"}]
     [:meta {:name "viewport"
             :content "width=device-width, initial-scale=1"}]
     (include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
    [:body
     [:div#app
      [:h3 "ClojureScript has not been compiled!"]
      [:p "please run "
       [:b "lein figwheel"]
       " in order to start the compiler"]]
     (include-js "js/app.js")]]))

(defroutes routes
  (GET "/" [] home-page)
  (resources "/")
  (not-found "Not Found"))

(def app
  (let [handler (wrap-defaults #'routes site-defaults)]
    (if (env :dev) (-> handler wrap-exceptions wrap-reload) handler)))

home-page will stop being a constant as it’ll be a function on the path and while we are at it, let’s rename it to something more appropriate, like render-app:

(defn render-app [path]
  (html
    [:html
     [:head
      [:meta {:charset "utf-8"}]
      [:meta {:name    "viewport"
              :content "width=device-width, initial-scale=1"}]
      (include-css (if (env :dev) "css/site.css" "css/site.min.css"))]
     [:body
      [:div#app
       [:h3 "ClojureScript has not been compiled!"]
       [:p "please run "
        [:b "lein figwheel"]
        " in order to start the compiler"]]
      (include-js "js/app.js")]]))

The reason why it’s taking the path and not the full URL is that the ClojureScript part of this app works with paths instead of URLs and we’ll need them to be consistent. This is due to how Pushy and likely HTML5 History behave.

The routes will now pass the path to render-app:

(defroutes routes
  (GET "*" request (render-app (path request)))
  (resources "/")
  (not-found "Not Found"))

The function that turns the request into a path is similar to ring.util.request/request-url:

(defn- path [request]
  (str (:uri request)
       (if-let [query (:query-string request)]
         (str "?" query))))

When this change is done, you should see no effect in the running application at all. If you want to confirm things are working properly, you could add this to the render-app  function:

[:p path]

and you’ll see the path the server sees before the ClojureScript kicks in. You can see the diff for this step in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx/….

The JavaScript engine

Now things get interesting. The render-app method needs to run some JavaScript, so it’ll create the script engine. First, we need to import it (and also require clojure.java.io , which we’ll be using soon):

(ns projectx.handler
  (:require ; ...
           [clojure.java.io :as io])
  (:import [javax.script ScriptEngineManager]))

After creating the engine, we need to define the variable global because Nashorn doesn’t specify it and reagent needs it. Once that’s done, we are ready to load the JavaScript code:

(defn render-app [path]
  (let [js-engine (doto (.getEngineByName (ScriptEngineManager.) "nashorn")
                    (.eval "var global = this")
                    (.eval (-> "public/js/app.js"
                               io/resource
                               io/reader)))]
    ; ...

It doesn’t yet render anything, but let’s give it a try, let’s see it load the code or… well… fail:

javax.script.ScriptException: ReferenceError: "document" is not defined in <eval> at line number 2

What’s happening here is that app.js is referring document and Nashorn implements JavaScript, but it’s not a browser, it doesn’t have the global, window or document global objects. Let’s look at the offending file:

var CLOSURE_UNCOMPILED_DEFINES = null;
if(typeof goog == "undefined") document.write('http://js/out/goog/base.js');
document.write('http://js/out/cljs_deps.js');
document.write('if (typeof goog != "undefined") { goog.require("projectx.dev"); } else { console.warn("ClojureScript could not load :main, did you forget to specify :asset-path?"); };');

This is a generated JavaScript file that is loaded by our small HTML file. It in turns causes the rest of the JavaScript files to be loaded but the mechanism it uses works in a browser, not in Nashorn. This is where things get hard.

From the project definition, this is how app.js  is built:

:cljsbuild {:builds {:app {:source-paths ["src/cljs" "src/cljc"]
                           :compiler {:output-to     "resources/public/js/app.js"
                                      :output-dir    "resources/public/js/out"
                                      :asset-path   "js/out"
                                      :optimizations :none
                                      :pretty-print  true}}}}

It’s built with no optimizations. One of the optimizations, called whitespace, puts all the JavaScript in a single file, so there’s no document trick to load them, but sadly, it will not work in Figwheel.

The solution I came up with, a hack, is to have two builds. One called app which is what I consider the JavaScript app itself and the other one called server-side, which is the one prepared to run on the server:

:cljsbuild {:builds {:app {:source-paths ["src/cljs" "src/cljc"]
                           :compiler     {:output-to     "resources/public/js/app.js"
                                          :output-dir    "resources/public/js/app"
                                          :asset-path    "js/app"
                                          :optimizations :none
                                          :pretty-print  true}}
                     :server-side {:source-paths ["src/cljs" "src/cljc"]
                                   :compiler     {:output-to     "resources/public/js/server-side.js"
                                                  :output-dir    "resources/public/js/server-side"
                                                  :optimizations :whitespace}}}}

For sanity’s sake, I changed the output of app to go to the directory called app, instead of out. Running Figwheel will auto-compile app, but not server-side; for that, you also need to run lein cljsbuild auto. Now the application loads with no errors.

We also need to properly configure server-side for the dev and uberjar profiles:

:cljsbuild {:builds {:app         {:source-paths ["src/cljs" "src/cljc"]
                                   :compiler     {:output-to  "resources/public/js/app.js"
                                                  :output-dir "resources/public/js/app"
                                                  :asset-path "js/app"}}
                     :server-side {:source-paths ["src/cljs" "src/cljc"]
                                   :compiler     {:output-to     "resources/public/js/server-side.js"
                                                  :output-dir    "resources/public/js/server-side"
                                                  :optimizations :whitespace}}}}

:profiles {:dev     {;...
                     :cljsbuild    {:builds {:app         {:source-paths ["env/dev/cljs"]
                                                           :compiler     {:optimizations :none
                                                                          :source-map    true
                                                                          :pretty-print  true
                                                                          :main          "projectx.dev"}}
                                             :server-side {:compiler {:optimizations :whitespace
                                                                      :source-map    "resources/public/js/server-side.js.map"
                                                                      :pretty-print  true}}}}}

           :uberjar {;...
                     :cljsbuild   {:jar    true
                                   :builds {:app         {:source-paths ["env/prod/cljs"]
                                                          :compiler     {:optimizations :advanced
                                                                         :pretty-print  false}}
                                            :server-side {:compiler     {:optimizations :advanced
                                                                         :pretty-print  false}}}}}}

You might have notice that we are not including env/dev/cljs  and env/dev/cljs  for server-side. That is because those files call projectx.core/init!, which triggers the whole application to start working, which depends on global objects, like window, which are not present in Nashorn.

With this, even the uberjar loads properly and creates JavaScript engines, but so far, we are not doing any server side rendering. That’s the next step. You can see the full diff for this change in GitHub: https://github.com/carouselapps/isomorphic-clojurescript-projectx/….

To be continued…

//s3.amazonaws.com/downloads.mailchimp.com/js/mc-validate.js// Part 2 has now been published.

Photo by Jared Tarbell

copy

jar-copier 0.1.0 released

jar-copier is a Leiningen plug in to copy jars from your dependencies to your source tree. It’s a very small simple utility that proved to be necessary to have a sane setup with Java agents (New Relic for example).

It’s very simple to use. Put [jar-copier “0.1.0”]  into the :plugins  vector on your project.clj. To run this plug in, execute:

$ lein jar-copier

If you want the task to run automatically, which is recommended, add:

:prep-tasks ["javac" "compile" "jar-copier"]

and it’ll be invoked every time you build your uberjar.

You need to configure the plug-in in your project.clj like this:

:jar-copier {:java-agents true
             :destination "resources/jars"}

:java-agents instruct this plug in to automatically copy any jars that are specified as Java agents. :destination  specifies where to copy them to.

For example, from proclodo-spa-server-rendering:

(defproject proclodo-spa-server-rendering "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.7.0"]]
  :plugins [[jar-copier "0.1.0"]]
  :prep-tasks ["javac" "compile" "jar-copier"]
  :java-agents [[com.newrelic.agent.java/newrelic-agent "3.20.0"]]
  :jar-copier {:java-agents true
               :destination "resources/jars"})

Photo by Maik Meid

Refactoring: Consolidate Duplicate Conditional Fragments; in Java, Python and Ruby

I’m reading the book Refactoring and one of the refactorings it shows is called “Consolidate Duplicate Conditional Fragments” and it shows an example in Java:

if (isSpecialDeal()) {
  total = price * 0.95;
  send();
} else {
  total = price * 0.98;
  send();
}

is refactored into

if (isSpecialDeal()) {
  total = price * 0.95;
} else {
  total = price * 0.98;
}
send();

If you do it in Python it’s actually quite similar:

if isSpecialDeal():
  total = price * 0.95
  send()
else:
  total = price * 0.98
  send()

is refactored into

if isSpecialDeal():
  total = price * 0.95
else:
  total = price * 0.98
send()

But in Ruby it’s different. In Ruby, like in Lisp, everything is an expression, everything has a value (maybe there are exceptions, I haven’t found them). Let’s look at it in Ruby:

if isSpecialDeal()
  total = price * 0.95
  send()
else
  total = price * 0.98
  send()
end

is refactored into

total = if isSpecialDeal()
  price * 0.95
else
  price * 0.98
end
send()

Or if you want it indented in another way:

total = if isSpecialDeal()
              price * 0.95
            else
              price * 0.98
            end
send()

We can push it one step further:

total = price * if isSpecialDeal()
                          0.95
                        else
                          0.98
                        end
send()

Of these three languages, only Ruby can manage to have inside each branch of the if only what changes depending on the condition and nothing else. In this simple case you could use the ternary operator, :?, but if the case wasn’t simple, Ruby would be at an advantage.

I’m reading Refactoring: Ruby Edition next.

Are dynamic languages just a temporary workaround?

This can unleash so much hate mail, but here it goes, my inbox is ready!

Are dynamic languages just a temporary workaround? I’m not sure! I’m switching between the two types of languages all the time: Java, Python, C#, JavaScript. I’ll try to make the long story short.

Statically typed languages, like Java and C#, are nice because when you do

blah.bleh()

you know that blah’s class has a bleh method, at compile time. But better than that, when you typed “blah.” you get a list of methods, and you already know whether there’s a bleh method or not, and if you typed bleh and it doesn’t exist, the IDE lets you know, no need to wait for the compiler. Also you can do very deterministic refactoring, renaming all “bleh” for “bluh” for example.

Statically typed languages are not nice because they are very verbose and require a lot of boilerplate (if you’ve used Haskell, just bear with me for now please), so you end up with things like:

List[Car] cars = new List[Cars]();
foreach (Car car in cars) {
    car.Crash();
}

How many “cars” do you read there? And that’s a nice example. There are worse. So come dynamically typed languages and you can write:

cars = []
for car in cars:
    car.crash()

You have less cars, and less (no) lists. That means you are more productive. You start chunking out code faster without having to stop and think “What type of object will this or that method return?”. But crash() can crash your application instead of just the car because you can’t know if it exists until run-time. That might be OK or not, testing and whatnot.

Then comes C# 3.0 where you can do:

var cars = new List[Car]();
foreach (var car in cars) {
    car.crash();
}

And you can see that syntactically it got closer to Python, which is what gives you the productivity. Don’t know the type? type “var”. But semantically, it’s still statically typed, like the previous statically typed example. You know that car is going to be of some class that has a crash method. You can actually know car’s class at compile time, no need to run it.

That’s called type inference. You don’t have to specify the type where the compiler is capable of inferring it for you. C# type inference system is still very limited (but better than Java’s). Let’s see an example in another language

cars = []
map crash cars

That means, create a list called cars, call the function crash on each car. Would you say that that is a statically typed language? or a dynamic one? I’d say it is dynamic, but it is static. Very static. It’s Haskell. Haskell’s compiler will infer all the types for you. It’s amazing, you’ll write code as robust as with C#, but as terse as Python’s (Monads will then kill your productivity, but that’s another story).

In Python 3 you can define types for arguments. They are mostly useless, but it’s an interesting direction. I think the best it can do is that when a program crashes it’ll tell you: “function blah expected an int, but got a float, not sure if that was the problem, but you might want to look into that”.

Now, my question is, are dynamically typed languages just a temporary workaround? As our compilers get better, our computers faster, will statically typed languages keep giving us as many or more reassurances about our code and utilities while at the same time they become as simple and terse as dynamically typed languages? Or will dynamically typed languages start to gain types and over time be more static without the programmers that use them ever noticing?

My question is, will we in the future, 50 or 100 years, look back and said “Dynamically typed languages where a temporary workaround to statically languages being painful to use when human beings and their toy computers were so primitive?” in the same way we can say today that “non-lexical scope was a limitation we had and have due to the limitations of computer hardware 30 years ago”.

Reviewed by Daniel Magliola. Thank you!

NetBeans could make the Ruby on Rails experience great

NetBeans could make the Ruby on Rails experience great for the vast majority of developers who are using Windows, where installing Ruby, Rails, PHP, MySQL, Python, etc is always a pain and the end result is ugly. But it falls short in some important ways which turned my experience with it into a nightmare.

The reason I say “for developers using Windows” is because I believe that for everybody else, the experience is great already. Or as good as it can be and NetBeans can be an excellent IDE, but not improve the installation and managing experience.

This is my story, my rant.

I downloaded the latest NetBeans and installed it. When creating my first Ruby project, I encountered the first problem. Ruby chocked on my username, which was “J. Pablo Fernández”. You could say it was my fault. Windows 7 asked for my name and I typed it. I wasn’t aware it was asking for my username. Even then I would have typed the same, because Windows 7 doesn’t distinguish between usernames and names, and in the 21st century, computers should be able to deal with any character anywhere.

I know it’s not NetBeans’ fault, it’s Ruby’s. But! Can you imagine a Software Engineer telling Steve Jobs “oh, copying files in a Mac behaves weirdly because it uses rsync and that’s its behavior, you see, it makes sense because…”? Of course Steve would have interrupted: “You’ve failed me for the last time”. The next developer would have patched rsync, trying to get the patch upstream, or creating an alternate rsync or stop using rsync.

I’ve spent many hours creating another user, migrating to it, which in Windows is like 100 times harder than it should.

Hours later, as soon as I created a project I got a message saying that I should upgrade gem, Ruby’s package manager, because the current version was incompatible with the current Rails version. By then I had already played with NetBeans’ gem interface telling it to upgrade everything, it should have upgraded gem as well, not just the gems. Every single developer out there running NetBeans must be encountering this error, and indeed there are quite a few threads about it on forums.

Trying to upgrade gem with NetBeans was impossible. I think what they did to install and upgrade gems in NetBeans is excellent, but failing to upgrade gem itself was a huge drawback. This one was NetBeans’ fault. Neverfear, let’s do it from the command line.

When doing it from the command line I encountered another error:

\NetBeans was unexpected at this time.

Looking around it seems it’s because of the spaces in “Program Files (x86)”. That means that the command line environment for Ruby that NetBeans installs is broken for everybody. I repeat: everybody. The answer: install it somewhere else.

Well, I have two things to say about it: first, fix the freaking thing, Ruby, gem, whatever. Paths can have spaces and all kind of weirdness. It’s a big world full of people speaking languages that can’t be represented with ASCII and people that believe computers should do our bidding, instead of the other way around. “If I want spaces you better give me spaces, useless lump of metal and silicon”.

Second, if you know one of your dependencies is broken, try to avoid triggering the broken behavior or at least warn the user about it. “We see you picked C:\Program Files (x86)\ to install NetBeans, which is pretty standard, but you know, Ruby is broken and can’t work in there, not even JRuby, so if you plan to use those at all, please consider installing it somewhere else.”

I uninstalled NetBeans, or tried to. The uninstaller didn’t work. I deleted it and tried to install it on C:\ProgramFilesx86, which failed because some other directory created by NetBeans somewhere else existed from the previous installation, which halted the installation. I started a dance of run installer, remove dir, run installer, remove dir, run installer… until it worked.

Once I finished I found out that NetBeans installed in C:\ProgramFilesx86\Netbeans 6.7.1. Yes, that’s a space. Oh my…

As a bonus, NetBeans can’t automatically find Sun’s JDK in its default directory. I had to point to it by hand. Sun was, as usually, absolutely disrespectful of the platform conventions and installed its crap in C:\Sun. I would have picked another place but I thought “I’m sure some stupid program will want to pick that shit from there”. Silly me.

12 hours have passed and I still haven’t been able to write a single line of source code. I contemplated installing Ruby by hand, but it’s so ugly that I decided I’m not going to use Windows for this. I’m going to work on another platform where installing Ruby is trivial and where I would probably never touch NetBeans because I have other editors.

I know there’s a lot not really related to NetBeans here, for example, the fact that working with Python, or Ruby or MySQL in Windows is a pain; but it’s a great opportunity for NetBeans. There are developers wanting to use those languages and environments and if NetBeans makes it easy for them, they will pick NetBeans not because of its editor, but because of everything else (which is what I was hoping to get out of NetBeans).

Aside from doing some usability tests, the people working on NetBeans should learn from the people working on Ubuntu (not the people working on Evolution) and instead of asking me for debugging traces when I report a simple obvious bug and then tell me it’s not their fault, they should submit those bugs upstream, to Ruby, gem, or whatever. Whenever someone like me submits that bug to NetBeans they should mark it as duplicate of an existing open bug that points to the upstream bug. I would have followed that link and told the Ruby developers “wake up!”. As it is, I didn’t. It’s too much work for me.

Reviewed by Daniel Magliola. Thank you!

Printing the class-path in Clojure

Let’s compare how we print the class-path in Clojure and how we do it on Java.

In Clojure:

(println (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))

In Java:

import java.net.URL;
import java.net.URLClassLoader;

public class PrintClasspath {
    public static void main(String[] args) {
        //Get the System Classloader
        ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();

        //Get the URLs
        URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();

        for(int i=0; i&lt; urls.length; i++)
        {
            System.out.println(urls[i].getFile());
        }
    }
}

To be fair, the output is not the same, but the effect is.