No-hashes bidirectional routing in re-frame with bidi and pushy

Note: this is a bidi counterpart to the silk article: No-hashes bidirectional routing in re-frame with silk and pushy. The content is very similar, only the code changes.

I recently replaced secretary with bidi and pushy in a re-frame project that was created fresh out of leiningen template and this is how I did it, but first, the reason.

What I like about bidi is that it’s bidirectional. It not only parses URLs into data structures but also generates URLs from data structures. This is not unique to bidi, silk also does it and feature-wise they are almost equivalent. On bidi’s website you can find this table comparing various routing libraries:

Library clj cljs Syntax Isomorphic? Self-contained? Extensible?
Compojure Macros
Moustache Macros
RouteOne Macros
Pedestal Data
gudu Data
secretary Macros
silk Data
fnhouse Macros
bidi Data

I don’t have a strong reason to chose bidi over silk, both seem like excellent choices. I also added pushy to the mix so I could have nice URLs, such as /about  instead of /#/about. The reason for this is that it looks better, follows the semantics of URLs better and allows for server side rendering, to be covered in a future article. It uses HTML5 pushState which by now it’s widely supported.

Let’s get started

I’m going to use a fresh project for this article, but the information here is applicable to any similar situation. Let’s get started:

lein new re-frame projectx +routes

The first step is to get rid of secretary and include bidi and pushy by replacing

[secretary "1.2.3"]

with

[bidi "1.20.3"]
[kibu/pushy "0.3.2"]

The Routes

The routes.cljs  file will be completely re-written. The namespace declaration will include bidi and pushy:

(ns projectx.routes
  (:require [bidi.bidi :as bidi]
            [pushy.core :as pushy]
            [re-frame.core :as re-frame]))

The first step is to define the routes we want. One of the designing features of bidi is that routes are data structures, not function/macro calls:

(def routes ["/" {""      :home
                  "about" :about}])

The app-routes function that used to define functions is replaced by one that sets up pushy:

(defn app-routes []
  (pushy/start! (pushy/pushy dispatch-route parse-url)))

parse-url  is a very simple function that uses bidi/match-route to turn a URL into a data structure representing it:

(defn- parse-url [url]
  (bidi/match-route routes url))

Then dispatch-route is called with that structure:

(defn- dispatch-route [matched-route]
  (let [panel-name (keyword (str (name (:handler matched-route)) "-panel"))]
    (re-frame/dispatch [:set-active-panel panel-name])))

The :handler of the matched-route will be :home or :about and panel-name  is constructed to be :home-panel  or :about-panel so that we can dispatch setting the active panel the same way secretary used to do it. And that’s the core of it. You now have to change URLs in the view to be /about  and / instead of /#/about and /#/ respectively.

Generating URLs

The whole point of using bidi was to not hard-code the URLs, for that I added this function to routes.cljs:

(def url-for (partial bidi/path-for routes))

which allowed me to replace the previous URLs with:

(routes/url-for :about)

and

(routes/url-for :home)

This works fine until you try to re-load the about page and you get a 404 Not Found error. This is because the server doesn’t know about the /about  URL or any other URL the client-side might support. What you need to do is just send serve the application no matter what the URL and let the client do the dispatching (with any exceptions you might have for APIs and whatnot).

I’m actually not quite sure how you do it with a figwheel-only project, probably by setting a ring handler. With compojure I ended up creating a route like this:

(routes (ANY "*" [] (layout/render "app.html")))

Something else you might consider is passing the whole matched-route  structure to the handler, so that the handler has access to path and query attributes.

And that’s it! Now you have beautiful bi-directional no-hash client-side routing.

Summary

For your reference, the full routes.cljs :

(ns projectx.routes
  (:require [bidi.bidi :as bidi]
            [pushy.core :as pushy]
            [re-frame.core :as re-frame]))

(def routes ["/" {""      :home
                  "about" :about}])

(defn- parse-url [url]
  (bidi/match-route routes url))

(defn- dispatch-route [matched-route]
  (let [panel-name (keyword (str (name (:handler matched-route)) "-panel"))]
    (re-frame/dispatch [:set-active-panel panel-name])))

(defn app-routes []
  (pushy/start! (pushy/pushy dispatch-route parse-url)))

(def url-for (partial bidi/path-for routes))

and the full views.cljs:

(ns projectx.views
    (:require [re-frame.core :as re-frame]
              [projectx.routes :as routes]))

;; --------------------
(defn home-panel []
  (let [name (re-frame/subscribe [:name])]
    (fn []
      [:div (str "Hello from " @name ". This is the Home Page.")
       [:div [:a {:href (routes/url-for :about)} "go to About Page"]]])))

(defn about-panel []
  (fn []
    [:div "This is the About Page."
     [:div [:a {:href (routes/url-for :home)} "go to Home Page"]]]))

;; --------------------
(defmulti panels identity)
(defmethod panels :home-panel [] [home-panel])
(defmethod panels :about-panel [] [about-panel])
(defmethod panels :default [] [:div])

(defn main-panel []
  (let [active-panel (re-frame/subscribe [:active-panel])]
    (fn []
      (panels @active-panel))))

No-hashes bidirectional routing in re-frame with silk and pushy

Update: there’s a version of this article that uses bidi instead of silk: No-hashes bidirectional routing in re-frame with bidi and pushy. The content is very similar, only the code changes.

I recently replaced secretary with silk and pushy in a re-frame project that was created fresh out of leiningen template and this is how I did it, but first, the reason.

What I like about silk is that it’s bidirectional. It not only parses URLs into data structures but also generates URLs from data structures. This is not unique to silk, bidi also does it and feature-wise they are almost equivalent. On bidi’s website you can find this table comparing various routing libraries:

Library clj cljs Syntax Isomorphic? Self-contained? Extensible?
Compojure Macros
Moustache Macros
RouteOne Macros
Pedestal Data
gudu Data
secretary Macros
silk Data
fnhouse Macros
bidi Data

I don’t have a strong reason to chose silk over bidi, both seem like excellent choices. I also added pushy to the mix so I could have nice URLs, such as /about  instead of /#/about. The reason for this is that it looks better, follows the semantics of URLs better and allows for server side rendering, to be covered in a future article. It uses HTML5 pushState which by now it’s widely supported.

Let’s get started

I’m going to use a fresh project for this article, but the information here is applicable to any similar situation. Let’s get started:

lein new re-frame projectx +routes

The first step is to get rid of secretary and include silk and pushy by replacing

[secretary "1.2.3"]

with

[com.domkm/silk "0.1.1"]
[kibu/pushy "0.3.2"]

The routes

The routes.cljs  file will be completely re-written. The namespace declaration will include silk and pushy:

(ns projectx.routes
  (:require [clojure.set :refer [rename-keys]]
            [domkm.silk :as silk]
            [pushy.core :as pushy]
            [re-frame.core :as re-frame]))

The first step is to define the routes we want. One of the designing features of silk is that routes are data structures, not function/macro calls:

(def routes (silk/routes [[:home [[]]]
                          [:about [["about"]]]]))

The app-routes function that used to define functions is replaced by one that sets up pushy:

(defn app-routes []
  (pushy/start! (pushy/pushy dispatch-route parse-url)))

parse-url  is a very simple function that uses silk/arrive to turn a URL into a data structure representing it:

(defn- parse-url [url]
  (silk/arrive routes url))

Then dispatch-route is called with that structure:

(defn- dispatch-route [matched-route]
  (let [matched-route (sanitize-silk-keywords matched-route)
        panel-name (keyword (str (name (:name matched-route)) "-panel"))]
    (re-frame/dispatch [:set-active-panel panel-name])))

The :name of the matched-route will be :home or :about and panel-name  is constructed to be :home-panel  or :about-panel so that we can dispatch setting the active panel the same way secretary used to do it.

sanitize-silk-keywords  just simplifies the namespaced keywords silk produces:

(defn- sanitize-silk-keywords [matched-route]
  (rename-keys matched-route {:domkm.silk/name    :name
                              :domkm.silk/pattern :pattern
                              :domkm.silk/routes  :routes
                              :domkm.silk/url     :url}))

And that’s the core of it. You now have to change URLs in the view to be /about  and / instead of /#/about and /#/ respectively.

Generating URLs

The whole point of using silk was to not hard-code the URLs, for that I added this function to routes.cljs:

(def url-for (partial silk/depart routes))

which allowed me to replace the previous URLs with:

(routes/url-for :about)

and

(routes/url-for :home)

This works fine until you try to re-load the about page and you get a 404 Not Found error. This is because the server doesn’t know about the /about  URL or any other URL the client-side might support. What you need to do is just send serve the application no matter what the URL and let the client do the dispatching (with any exceptions you might have for APIs and whatnot).

I’m actually not quite sure how you do it with a figwheel-only project, probably by setting a ring handler. With compojure I ended up creating a route like this:

(routes (ANY "*" [] (layout/render "app.html")))

Something else you might consider is passing the whole matched-route  structure to the handler, so that the handler has access to path and query attributes.

And that’s it! Now you have beautiful bi-directional no-hash client-side routing.

Summary

For your reference, the full routes.cljs :

(ns projectx.routes
  (:require [clojure.set :refer [rename-keys]]
            [domkm.silk :as silk]
            [pushy.core :as pushy]
            [re-frame.core :as re-frame]))

(def routes (silk/routes [[:home [[]]]
                          [:about [["about"]]]]))

(defn- parse-url [url]
  (silk/arrive routes url))

(defn- sanitize-silk-keywords [matched-route]
  (rename-keys matched-route {:domkm.silk/name    :name
                              :domkm.silk/pattern :pattern
                              :domkm.silk/routes  :routes
                              :domkm.silk/url     :url}))

(defn- dispatch-route [matched-route]
  (let [matched-route (sanitize-silk-keywords matched-route)
        panel-name (keyword (str (name (:name matched-route)) "-panel"))]
    (re-frame/dispatch [:set-active-panel panel-name])))

(defn app-routes []
  (pushy/start! (pushy/pushy dispatch-route parse-url)))

(def url-for (partial silk/depart routes))

and the full views.cljs:

(ns projectx.views
    (:require [re-frame.core :as re-frame]
              [projectx.routes :as routes]))

;; --------------------
(defn home-panel []
  (let [name (re-frame/subscribe [:name])]
    (fn []
      [:div (str "Hello from " @name ". This is the Home Page.")
       [:div [:a {:href (routes/url-for :about)} "go to About Page"]]])))

(defn about-panel []
  (fn []
    [:div "This is the About Page."
     [:div [:a {:href (routes/url-for :home)} "go to Home Page"]]]))

;; --------------------
(defmulti panels identity)
(defmethod panels :home-panel [] [home-panel])
(defmethod panels :about-panel [] [about-panel])
(defmethod panels :default [] [:div])

(defn main-panel []
  (let [active-panel (re-frame/subscribe [:active-panel])]
    (fn []
      (panels @active-panel))))
macro-key

Macros, the Lisp advantage

Learning about macros in Lisps was one of my biggest whoa-moments in my programming career and since then I’ve given presentations about them to audiences ranging from 1 to 100 people. I have a little script that I follow in which I implement a custom form of the if-conditional. Unfortunately, I don’t think I’ve managed to generate many whoa-moments. I’m probably not doing macros justice. It’s not an easy task as they can be complex beasts to play with.

As we are experimenting with Clojure, I eventually needed a tool that I knew was going to be a macro, and I built a simple version of it. The tool is assert_difference. I don’t know where it first appeared, but Rails ships with one and not satisfied with that one I built one a few years ago. In the simplest case it allows you to do this:

assert_difference("User.count()", 1) do
  add_user_to_database()
end

assert_difference("User.count()", 0) do
  modify_user_on_the_database()
end

assert_difference("User.count()", -1) do
  remove_user_from_the_database()
end

The problem

Do you see what’s wrong there? Well, wrong is a strong word. What’s not as good as it could be? It’s the fact that User.count is expressed as a string and not code, when it is code. The reason for doing that is that we don’t want that code to run, we want to have it in a way that we can run it, run the body of the function (adding users, removing users, etc) and run it again comparing it with the output of the previous one.

There’s no (nice) way in Ruby or many languages to express code and not run it. Rephrasing that, there’s no (nice) way to have code as data in Ruby as in most languages. I’m not picking on Ruby, I love that language, I’m just using it because it’s the language I’m most familiar with; what I’m saying is probably true about any programming language you chose.

One of the things you’ll hear repeated over and over in the land of the Lisp, be it Common Lisp, Scheme or Clojure is that code is data. And that’s what we want here, we want to have a piece of code as data. We want this:

assert_difference(User.count(), 1) do
  add_user_to_database()
end

assert_difference(User.count(), 0) do
  modify_user_on_the_database()
end

assert_difference(User.count(), -1) do
  remove_user_from_the_database()
end

which in Lisp syntax it would look like this:

(assert-difference (user-count) 1
  (add-user-to-database))

(assert-difference (user-count) 0
  (modify-user-on-the-database))

(assert-difference (user-count) -1
  (remove-user-from-the-database))

and this, ladies and gentlemen, is not only easy, it’s good practice.

Enter the macro

I achieved it with a simple 4-line macro and here it is:

(defmacro assert-difference [form delta & body]
  `(let [count# ~form]
     ~@body
     (assert-equal (+ count# ~delta) ~form)))

If you are not familiar with Clojure that will be very hard to read, so, let me help you:

The first line defines the macro with the name assert-difference  and getting 3 or more parameters with the first one called form , the second delta  and all other parameters, as a list, in body. So, in this example:

(assert-difference (user-count) 1
  (add-user-to-database))

we end up with:

  • form => (user-count)
  • delta => 1
  • body => [(add-user-to-database)]

Note that the parameters to the macro didn’t get the value of calling (user-count), it got the code itself, unexecuted, represented as data that we can inspect and play with, not an unparsed string.

The body of the macro is a bit cryptic because it’s a template. The backtick at the beginning just identifies it as a template and ~  means “replace this variable with the parameter”. ~@  is a special version of ~  that we have to use because body contains a list of statements instead of a single one. That means that:

`(let [count# ~form]
    ~@body 
    (assert-equal (+ count# ~delta) ~form))

turns into:

(let [count# (user-count)]
   (add-user-to-database)
   (assert-equal (+ count# 1) (user-count)))

Is it starting to make sense? count#  is a variable that is set to (user-count), then we execute the body, that is (add-user-to-database) and then we execute  (user-count) again and compare it count#  plus delta. This is the code that’s emitted by the macro, this is the code that actually gets compiled and executed.

If you are wondering about why the variable name has a hash at the end, imagine that variable was just named count  instead and the macro was used like this:

(let [count 10]
  (assert-difference (user-count) 1
    (add-users-to-database (generate-users count)))

That snippet defines count, but then the macro defines count again, by the time you reach (generate-users count) that count  was masked by the macro-generated one. That bug can be very hard to debug. The hash at the ends makes it into a uniquely named variable, something like count__28766__auto__ , that is consistent with the other mentions of count#  within that macro.

Isn’t that beautiful?

The real solution

The actual macro that I’m using for now is:

(defmacro is-different [[form delta] & body]
  `(let [count# ~form]
     ~@body
     (is (= (+ count# ~delta) ~form))))

which I’m not going to package and release yet like I did with assert_difference because it’s nowhere near finished and I’m not going to keep on improving it until I see the actual patterns that I like for my tests.

You might notice that it doesn’t use assert-equal. That’s a function I made up because I believe it was familiar for non-clojurians reading this post. When using clojure.test, you actually do this:

(is (= a b))

There’s one and only one thing to remember: is . Think of is  as a generic assert and that’s actually all that you need. No long list of asserts like MiniTest has: assert, assert_block, assert_empty, assert_equal, assert_in_delta, assert_in_epsilon, assert_includes, assert_instance_of, assert_kind_of, assert_match, assert_nil, assert_operator, assert_output, assert_predicate, assert_raises, assert_respond_to, assert_same, assert_send, assert_silent and, assert_throws.

In a programming language like Ruby we need all those assertions because we want to be able to say things such as “1 was expected to be equal to 2, but it isn’t” which can only be done if you do:

assert_equal(1, 2)

but not if you do

assert(1 == 2)

because in the second case, assert doesn’t have any visibility into the fact that it was an equality comparison, it would just say something like “true was expected, but got false” which is not very useful.

Do you see where this is going? is  is a macro, so it has visibility into the code, it can both run the code and use the code as data and thus generate errors such as:

FAIL in (test-users) (user.clj:12)
expected: 1
  actual: 2
    diff: - 1
          + 2

which if you ask me, is a lot of very beautiful detail to come out of just

(is (= 1 2))

But language X!

When talking to people about this, I often get rebuttals that this or that language can do it too and yes, other languages can do some things like this.

For example, we could argue that this is possible in Ruby if you encase the code in an anonymous function when passing it around, such as:

assert_difference(->{User.count}, 1) do
  add_user()
end

but it’s not as nice and we don’t see it very often. What we see is 20 assert methods like in MiniTest. To have an impact in the language, these techniques need to be easy, nice, first class citizens, the accepted style. Otherwise they might as well not exist.

An even better syntax for blocks in Ruby might help with the issue and indeed what you are left with is a different approach to incredible flexibility and it already exists. It’s called Smalltalk and there’s some discussion about closures, what you have in Lisp,  and objects, what you have in Smalltalk, being equivalent.

I’m also aware of a few languages having templating systems to achieve things such as this, like Template Haskell, but they are always quite hard to use and left for the true experts. You rarely see them covered in a book for beginners of the language, like macros tend to be covered for Lisp.

There are also languages that have a string based macro system, like C and I’ve been told that Tcl does as well. The problem with this is that it’s from hard to impossible to build something that’s of medium complexity, to the point that you are recommended to stay away from it.

All of the alternative solutions I mention so far have a problem: code is not (nice) data. When a macro in Lisp gets a piece of code such as:

(+ 1 2)

that code is received as a list of three elements, containing + , 1  and 2 . If the macro instead received:

(1 2 +)

the code would be a list containing 1, 2 and +. Note that it’s not valid Lisp code, it doesn’t have to be because it’s not being compiled and executed. The output of a macro has to be valid Lisp code, the input can be whatever and thus, making a macro that changes the language from prefix notation to suffix notation, like in the last snippet of code, is one of the few first exercises you do when learning to make macros.

What makes it so easy to get code as data and work with it and then execute the data as code is the fact that Lisp’s syntax is very close to the abstract syntax tree of the language. The abstract syntax tree of Lisp programs is something Lisp programmers are familiar with intuitively while most programmers of other languages have no idea what the AST looks like for their programs. Indeed, I don’t know what it looks like for any of the Ruby code I wrote.

Most programmers don’t know what an AST actually is, but even the Lisp programmers that don’t know what an AST is have an intuition for what the ASTs of their programs are.

This is why many claim Lisp to be the most powerful programming language out there. You could start thinking of another programming language that has macros that receive code as data and that their syntax is close to the AST and if you find one of those, congratulations, you found a programming language of the Lisp family because pretty much those properties make it a member of the Lisp family.