Clojure Macros — Lessons from unspoken symbols

At Swym, we use Clojure for a lot of things. And invariably, we have ended up with use cases where macros are deemed necessary. In this post, I try to take through the lessons I learned (read mistakes) along the way.

Source — https://www.videoblocks.com/video/hacker-text-terminal-fake-data-scroll-computer-terminal-display-computing-random-text-generations-scrolling-down-the-page-vxglayqceimdkqwzw

I started off by creating a simple macro using the macro _defmacro_.

Note — The code output is commented out so anyone can copy paste to a repl without issues, but the syntax highlight doesn’t show it as commented out.

Not really a macro

(defmacro not-really-macro [a] (do (println a) a));#'user/not-really-macro

(macroexpand `(not-really-macro "test"));test;"test"

(not-really-macro "test");test;"test"

_not-really-macro_ is just a println code that will execute as soon you call it. So not really a macro. Moving on

Lesson 0 — Not everything defined with **_defmacro_** qualifies as a macro.

Maybe a macro

;; without list, using syntax quoting(defmacro wrong-macro [a] `(do (println a) a))

(macroexpand `(wrong-macro "test"));(do (clojure.core/println user/a) user/a)

(wrong-macro "test");CompilerException java.lang.RuntimeException: No such var: user/a, ...

May have been a macro if it worked. The error is clear enough, there is no a defined in the user namespace. Which is weird as I expect that to come from the input arg to the macro, right? Hmm, what if I switch the namespace and try expanding

(ns outerspace);nil;in outerspace now

(macroexpand `(user/wrong-macro "test"));(do (clojure.core/println user/a) user/a)

Now, that is interesting. The namespace of _a_ didn’t change. So the macro expansion is not picking up the arg passed.

Enter Unquote — Variable capture

Since I quoted the code, I need to _unquote_ to access the symbols outside the _quote_.

(in-ns 'user)(defmacro ok-macro [a] `(do (println ~a) ~a));#'user/ok-macro

(macroexpand `(ok-macro "test"));(do (clojure.core/println "test") "test")

(ok-macro "test");"test";test

A double WooHoo! moment, both the macro expansion and the actual output was correct.

Lesson 1 — When in quoted code, access outside references using unquote.

More un/quoting — figuratively and “literal”ly

So, there are clojure functions _quote_, _unquote_ and _unquote-splicing_. Trying to use them in the macro,

(defmacro forcequote-macro [a] (quote (do (println ~a) ~a)));#'user/forcequote-macro

(macroexpand `(forcequote-macro "test"));(do (println (clojure.core/unquote a)) (clojure.core/unquote a))

(forcequote-macro "test");CompilerException java.lang.RuntimeException: Unable to resolve symbol: a in this context...

Huh? Why not?

Lesson 2 — ` is not the literal shortcut to quote.

To confirm that lesson 2 is actually true,

(= 'a (quote a));true

(= `a (quote a));false

Success! Sort of.

Lesson 3 — ‘ is the literal shortcut to quote.

Now, doing the equivalent for _unquote_.

(defmacro forceunquote-macro [a] `(do (println (unquote a)) (unquote a)));#'user/forceunquote-macro

(macroexpand `(forceunquote-macro "test"));(do (clojure.core/println (clojure.core/unquote user/a)) (clojure.core/unquote user/a))

(forceunquote-macro "test");CompilerException java.lang.RuntimeException: No such var: user/a, compiling...

Hmm, there seems to be a pattern to this “madness” (or so I called it).

Lesson 4 — ~ is not the literal shortcut to unquote.

So, the unquoting and quoting were not working when called by reference, meaning it was too late to identify the symbols that are used inside the macro. Literals had to be used, no two ways about it.

Symbol generation

If the problem is the symbol, why not just generate a symbol I needed on demand inside that macro and referenced that instead. Here it goes

;; gensym(defmacro sym-gen-macro [a](let [dyn-a (gensym a)]`(let [~dyn-a ~a](println ~dyn-a)~dyn-a)));#'user/sym-gen-macro

(macroexpand `(sym-gen-macro "test"));(let* [test1663 "test"] (clojure.core/println test1663) test1663)

(sym-gen-macro "test");test;"test"

_gensym_ to the rescue, the macro worked! But that didn’t seem right. It works, but at what cost. Every time the macro was invoked, there is a new symbol created and the reference is updated to a inside the let anyway. So nope, definitely not it.

Lesson 5 — **_gensym_** cannot solve your problem of unquoting.

Use the source

After confirming there is no getting away from those mystery literals, moved to unquote-splicing. Very powerful in using entire body of args to be passed

(defmacro expand-body [& body]`(println ~@body));#'user/expand-body

(macroexpand `(expand-body "test1" "test2"));(clojure.core/println "test1" "test2")

(expand-body "test1" "test2");test1 test2;nil

Worked well, getting the hang of it now. Now I tried using the definition given here. It didn’t work without literals as expected, but I was in for a rude but interesting shock.

(source unquote);(def unquote);nil

(source unquote-splicing);(def unquote-splicing);nil

(source macroexpand));(defn macroexpand; "Repeatedly calls macroexpand-1 on form until it no longer; represents a macro form, then returns it. Note neither; macroexpand-1 nor macroexpand expand macros in subforms."; {:added "1.0"; :static true}; [form]; (let [ex (macroexpand-1 form)]; (if (identical? ex form); form; (macroexpand ex))));nil

(source quote);Source not found;nil

As you can see, unquote and unquote-splicing are _def__’_s just symbols, unlike the other _defn_‘s. So, of course I can’t use them instead of the literals, duh.

Lesson 6 — Not all literals have an equivalent **_defn_**.

Lesson 7 — Use ~@ to take a list of args expand inside the macro

Extra — Try (source defn), it is an interesting read.

Getting into the inner circle — Creating inner args

I tried adding a new symbol which would prepend to the input string.

(defmacro innersym-macro [a]`(let [dyn-a# (str "Prepend-" ~a)](println dyn-a#)dyn-a#));#'user/innersym-macro

(macroexpand `(innersym-macro "test"));(let* [dyn-a__1749__auto__ (clojure.core/str "Prepend-" "test")] (clojure.core/println dyn-a__1749__auto__) dyn-a__1749__auto__)

(innersym-macro "test");Prepend-test;"Prepend-test"

That is getting close to being awesome!

Lesson 8 — Use # — to create symbols inside the quote-d code block, also known as autogensym.

I will have some Destructuring, please? Please?

Pushing my luck, I tried destructuring the inner level args.

(defmacro innerdestructure-macro [a]`(let [{:keys [prepend#] :as aprepender#} {:prepend "Prependtext" :append "Appendtext"}dyn-a# (str prepend# ~a (:append aprepender#))](println dyn-a#)dyn-a#));#'user/innerdestructure-macro

(macroexpand `(innerdestructure-macro "test"));(let* [map__1853 {:prepend "Prependtext", :append "Appendtext"} map__1853 (if (clojure.core/seq? map__1853) (clojure.lang.PersistentHashMap/create (clojure.core/seq map__1853)) map__1853) aprepender__1844__auto__ map__1853 prepend__1843__auto__ (clojure.core/get map__1853 :prepend__1843__auto__) dyn-a__1845__auto__ (clojure.core/str prepend__1843__auto__ "test" (:append aprepender__1844__auto__))] (clojure.core/println dyn-a__1845__auto__) dyn-a__1845__auto__)

(innerdestructure-macro "test");testAppendtext;"testAppendtext"

The destructuring didn’t work, as _prepend#_ got treated as a nil, but the direct get key worked well.

Lesson 9 — Destructuring doesn’t work in the first level of quoted code block

Using intern-ness

Now, using the awesome macro skills acquired so far, I ventured into creating dynamic symbols inside namespaces whenever a macro is executed.

(defmacro interning-macro [a]`(let [{:keys [prepend#] :as aprepender#} {:prepend "Prependtext" :append "Appendtext"}dyn-a# (str prepend# ~a (:append aprepender#))](intern*ns*'~'ooh-fn(fn [oohargs#](println oohargs# dyn-a#)oohargs#))));#'user/interning-macro

(macroexpand `(interning-macro "test"));(let* [map__1995 {:prepend "Prependtext", :append "Appendtext"} map__1995 (if (clojure.core/seq? map__1995) (clojure.lang.PersistentHashMap/create (clojure.core/seq map__1995)) map__1995) aprepender__1985__auto__ map__1995 prepend__1984__auto__ (clojure.core/get map__1995 :prepend__1984__auto__) dyn-a__1986__auto__ (clojure.core/str prepend__1984__auto__ "test" (:append aprepender__1985__auto__))] (clojure.core/intern clojure.core/*ns* (quote ooh-fn) (clojure.core/fn [oohargs__1987__auto__] (clojure.core/println oohargs__1987__auto__ dyn-a__1986__auto__) oohargs__1987__auto__)))

(ns outerspacestar)(user/interning-macro "star");#'outerspacestar/ooh-fn

(ns outerspaceplanet)(user/interning-macro "planet");#'outerspaceplanet/ooh-fn

(in-ns 'user)(outerspacestar/ooh-fn {:a 10});{:a 10} starAppendtext;{:a 10}

(outerspaceplanet/ooh-fn {:b 20});{:b 20} planetAppendtext;{:b 20}

That was awesome! Having some internal references from when the macro was instantiated. This comes handy in creating repeatable modules with configuration changes. Many lessons in this one

Lesson 10 — ***ns*** — refers to current namespace where the code is executing.

Lesson 11 — **_(intern somens '~'symname <<symdefinition>>)_** is equivalent to adding **_(def symname symdefinition)_** in that somens namespace_._

Lesson 12 — **_(def x (fn []))_ = _(defn x [])_**.

Extra — Checkout Protocols sometime, if you haven’t already i.e.

Namespaces inside interns inside macros inside namespace

Now, going for the limit-breaker of my understanding — How about loading a namespace inside the intern of current namespace generated from a macro

(defn resolvable-fn1 [](println "resolved1"));#'user/resolvable-fn1

(defn resolvable-fn2 [](println "resolved2"));#'user/resolvable-fn2

(defmacro interning-resolve-macro [a]`(let [{:keys [prepend#] :as aprepender#} {:prepend "Prependtext" :append "Appendtext"}dyn-a# (str prepend# ~a (:append aprepender#))](intern*ns*'~'resolvens-fn(fn [rargs#]((ns-resolve (symbol "user") (symbol "resolvable-fn1")))((ns-resolve '~'user '~'resolvable-fn2))(println rargs# dyn-a#)rargs#))))#'user/interning-resolve-macro

(macroexpand `(interning-resolve-macro "test"));(let* [map__2276 {:prepend "Prependtext", :append "Appendtext"} map__2276 (if (clojure.core/seq? map__2276) (clojure.lang.PersistentHashMap/create (clojure.core/seq map__2276)) map__2276) aprepender__2266__auto__ map__2276 prepend__2265__auto__ (clojure.core/get map__2276 :prepend__2265__auto__) dyn-a__2267__auto__ (clojure.core/str prepend__2265__auto__ "test" (:append aprepender__2266__auto__))] (clojure.core/intern clojure.core/*ns* (quote resolvens-fn) (clojure.core/fn [rargs__2268__auto__] ((clojure.core/ns-resolve (clojure.core/symbol "user") (clojure.core/symbol "resolvable-fn1"))) ((clojure.core/ns-resolve (quote user) (quote resolvable-fn2))) (clojure.core/println rargs__2268__auto__ dyn-a__2267__auto__) rargs__2268__auto__)))

(ns outerspacestar)(user/interning-resolve-macro "star")#'outerspacestar/resolvens-fn

(ns outerspaceplanet)(user/interning-resolve-macro "planet")#'outerspaceplanet/resolvens-fn

(in-ns 'user)(outerspacestar/resolvens-fn {:a 100});resolved1;resolved2;{:a 100} starAppendtext;{:a 100}

(outerspaceplanet/resolvens-fn {:b 200});resolved1;resolved2;{:b 200} planetAppendtext;{:b 200}

Adding on to aforementioned awesomeness, resolving symbols can be done in interesting ways to successful results.

Lesson 13 — **_(symbol "xyz")_** = **_'xyz_**.

Lesson 14 — In macro world — **_(symbol "xyz")_** = **_'~'xyz_**.

Lesson 15 — Before a **_ns-resolve_** is called, the ns needs to have been loaded. So better do a **_(require 'nssymbol)_** before using **ns-resolve**.

It was getting a little messy to see all those dynamic symbols in one go. Got me to try out _macroexpand-1_.

(macroexpand-1 `(interning-resolve-macro "test"))

;(clojure.core/let [{:as aprepender__2266__auto__, :keys [prepend__2265__auto__]} {:prepend "Prependtext", :append "Appendtext"} dyn-a__2267__auto__ (clojure.core/str prepend__2265__auto__ "test" (:append aprepender__2266__auto__))] (clojure.core/intern clojure.core/*ns* (quote resolvens-fn) (clojure.core/fn [rargs__2268__auto__] ((clojure.core/ns-resolve (clojure.core/symbol "user") (clojure.core/symbol "resolvable-fn1"))) ((clojure.core/ns-resolve (quote user) (quote resolvable-fn2))) (clojure.core/println rargs__2268__auto__ dyn-a__2267__auto__) rargs__2268__auto__)))

Neater.

Lesson 16 — **macroexpand-1** goes 1 level of expansion and macroexpand goes to all levels and expands every. single. macro.

Extrapolations

Some good practices that evolved out of the lessons learnt

1. Keep destructuring to a limit inside the quoted block to avoid confusion.
2. Keep core functions that can be outside the macro, outside it. There is no need for those functions to be created every time the macro is loaded
3. For the heavy weight macros that initialize a bunch of interns on namespaces, keep the number of interns and actual instantiation to a judicious limit. The higher this number, the time to startup could get that much slower.
4. On the flip side, if there is an increase in startup time, take a look at the require‘s that force load of macros and help cut down to the only mandatory startup namespaces.
5. If you are doing a lot of intern ns generations, it is probably time to give Protocols a look and see if that works better. There is a small chance you might like it :)

Macros sure are powerful in many ways, allowing for data to become the code, executable and everything. But (of course there is a “but”) the documentation around it is kind of shrouded in mystery and (for a lack of better word) not simple. Hopefully the lessons from this post help some of those mysteries reveal to the uninitiated.

I am sure I have missed a point or two, so please feel free to correct me wherever necessary in the comments section below. Also, do share your experiences with Clojure macros below, would love to know them!

References and useful links

1. https://www.braveclojure.com/writing-macros/ — Deep and useful when facing the macro music i.e. errors from macros
2. https://stackoverflow.com/questions/3667403/what-is-the-difference-between-defn-and-defmacro
3. http://stackoverflow.com/questions/4571042/can-someone-explain-clojures-unquote-splice-in-simple-terms
4. https://clojure.org/reference/special_forms
5. https://clojure.org/reference/protocols
6. https://www.braveclojure.com/multimethods-records-protocols/ — Another one from braveclojure, it is filled with Clojure goodness!