Some of the sources for my Clojure/conj 2018 slides

all-these-class-files.org

What Are All These Class Files Even About?, and Other Stories

Latex Prelude

Introduction

Title Slide

\LARGE What Are All These Class Files Even About?
\large \vspace{10pt} and Other Stories \

\small \vspace{60pt}

by Gary Fredericks
at Clojure/conj 2018

About Me

\huge

• @gfredericks_
• DRW
• etc.

What Are We Doing?

\huge Trying to understand the relationship between Clojure’s bytecode and the dynamic runtime

Why Would We Do That?

\huge

• It’s interesting to know how things work
• It’s probably useful for faster debugging (stack traces might make more sense!)

Is This Hard?

\huge Yes

• Clojure code can be compiled and loaded in lots of ways
• There’s a parallel in-memory representation of things in the code
• Not even a clear line between compile-time and runtime

Caveats

\huge

• Lots of things are missing
• Some things are intentional lies
• Some things are accidental lies

Roadmap

\huge

• require
• require, but moreso
• compile
• require after compile
• :reload, etc.

Part 1: require

let’s just load some code

\frame{\subsectionpage}

Code

src/my/ns.clj

(ns my.ns
  (:require
   [my.other-ns :as other]))

(defn assoc-foo
  [m]
  (assoc m :foo other/foo))

\vspace{10pt} src/my/other_ns.clj

(ns my.other-ns)

(def foo "foo")

Let’s start a repl

\huge

$ clj
Clojure 1.10.0-RC2
user=>

Namespace Graph

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-1.pdf}

Require

user=> (require 'my.ns)

(ns my.ns ...)

(ns my.ns
  (:require
   [my.other-ns :as other]))

(macroexpand '(ns my.ns ...))

;; mildly edited for readability
(do
  (in-ns 'my.ns)
  (with-loading-context
    (refer 'clojure.core)
    (require '[my.other-ns :as other]))
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.ns))
      nil)))

(macroexpand '(ns my.ns ...))

;; mildly edited for readability
(do
  ;; ___________
  (in-ns 'my.ns)
  ;; ^^^^^^^^^^^
  (with-loading-context
    (refer 'clojure.core)
    (require '[my.other-ns :as other]))
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.ns))
      nil)))

(in-ns 'my.ns)

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-2b.pdf}

(macroexpand '(ns my.ns ...))

;; mildly edited for readability
(do
  (in-ns 'my.ns)
  (with-loading-context
    ;; __________________
    (refer 'clojure.core)
    ;; ^^^^^^^^^^^^^^^^^^
    (require '[my.other-ns :as other]))
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.ns))
      nil)))

(refer 'clojure.core)

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-2.pdf}

(macroexpand '(ns my.ns ...))

;; mildly edited for readability
(do
  (in-ns 'my.ns)
  (with-loading-context
    (refer 'clojure.core)
    ;; ________________________________
    (require '[my.other-ns :as other]))
    ;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.ns))
      nil)))

(ns other.ns ...)

(ns my.other-ns)

(macroexpand '(ns my.other-ns ...))

;; mildly edited for readability
(do
  ;; _____________________
  (in-ns 'my.other-ns)
  (with-loading-context
    (refer 'clojure.core))
  ;; ^^^^^^^^^^^^^^^^^^^^^
  (if (.equals 'my.other-ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.other-ns))
      nil)))

(in-ns 'my.other-ns) and (refer 'clojure.core)

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-3.pdf}

(macroexpand '(ns my.other-ns ...))

;; mildly edited for readability
(do
  (in-ns 'my.other-ns)
  (with-loading-context
    (refer 'clojure.core))
  ;; ______________________________________________
  (if (.equals 'my.other-ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.other-ns))
      nil)))
  ;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(def foo "foo")

(ns my.other-ns)

(def foo "foo")

(def foo "foo")

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-4.pdf}

(def foo "foo")

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-5.pdf}

(macroexpand '(ns my.ns ...))

;; mildly edited for readability
(do
  (in-ns 'my.ns)
  (with-loading-context
    (refer 'clojure.core)
    ;; ________________________________
    (require '[my.other-ns :as other]))
    ;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (dosync
       (commute @#'*loaded-libs* conj 'my.ns))
      nil)))

:as other

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-6.pdf}

(defn assoc-foo ...)

(ns my.ns
  (:require
   [my.other-ns :as other]))

(defn assoc-foo
  [m]
  (assoc m :foo other/foo))

(defn assoc-foo ...)

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-7.pdf}

(defn assoc-foo ...)

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-8.pdf}

So what was all that then?

All top-level forms in clojure code are for side effects, often in the namespace graph

Part 2: require, but moreso

where was the bytecode there?

\frame{\subsectionpage}

The Repl

\huge

$ clj
Clojure 1.10.0-RC2
user=> (require 'my.ns)

The API

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-normal.pdf}

main -> repl

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-repl.pdf}

(require 'my.ns) as a compiled class

public final class user$eval1
    extends clojure.lang.AFunction {
  // #'clojure.core/require
  public static final clojure.lang.Var const__0;
  // 'my.ns
  public static final clojure.lang.AFn const__1;
  // const__0 = RT.var("clojure.core","require");
  // const__1 = Symbol.intern(null,"my.ns");
  public static {};
  // super()
  public user$eval1();
  // const__0.getRawRoot().invoke(const__1);
  public static java.lang.Object invokeStatic();
  // call .invokeStatic()
  public java.lang.Object invoke();
}

RT.var("clojure.core","require");

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-1.pdf}

Classloading

\small

package clojure.lang;

import java.net.URLClassLoader;
// ...etc.

public class DynamicClassLoader
  extends URLClassLoader{

  static ConcurrentHashMap<String, Reference<Class>>
    classCache =
        new ConcurrentHashMap<String, Reference<Class>>();


  // ... lots of overridden methods
}

require

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-require.pdf}

Evaling src/my/ns.clj

(ns my.ns
  (:require
   [my.other-ns :as other]))

(defn assoc-foo
  [m]
  (assoc m :foo other/foo))

Further macroexpand (ns my.ns ...)

\small

;; mildly edited for readability
(do
  (in-ns 'my.ns)
  ((fn loading__6621__auto__ []
     (clojure.lang.Var/pushThreadBindings
      {clojure.lang.Compiler/LOADER
       (.getClassLoader
        (.getClass loading__6621__auto__))})
     (try
       (refer 'clojure.core)
       (finally
         (clojure.lang.Var/popThreadBindings)))))
  (if (.equals 'my.ns 'clojure.core)
    nil
    (do
      (clojure.lang.LockingTransaction/runInTransaction
       (fn [] (commute @#'*loaded-libs* conj 'my.ns)))
      nil)))

Functions calling functions

\large

public class my.ns$eval154
  extends clojure.lang.AFunction {
  // ...
  public static Object invokeStatic(){
    if(!clojureCoreSym.equals(myNsSym)){
      LockingTransaction.runInTransaction(
        new my.ns$eval154$fn__155();
      );
    }
  }
}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-1.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-2.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-3.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-4.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-5.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-6.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-7.pdf}

Side Effects

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-8.pdf}

So what was all that then?

• Each function literals is compiled to a dedicated class that can be instantiated into functions
• At the repl and during normal code loading, top-level forms are
  • compiled to bytecode for a 0-arg function
  • loaded via the DynamicClassLoader
  • instantiated and .invoke()‘d, likely performing side effects on the namespace graph or returning a value to the repl

Part 3: compile

let’s do some AOT

\frame{\subsectionpage}

(compile 'my.ns)

\LARGE

$ clj
Clojure 1.10.0-RC2
user=> (compile 'my.ns)
Syntax error macroexpanding
  clojure.core/ns at (ns.clj:1:1).
No such file or directory

(compile 'my.ns)

\LARGE

$ mkdir classes

$ clj
Clojure 1.10.0-RC2
user=> (compile 'my.ns)
my.ns

All These Class Files

$ find classes/ -type f

classes/clojure/core/specs/alpha$fn__222.class
classes/clojure/core/specs/alpha$fn__235.class
classes/clojure/core/specs/alpha$fn__285$fn__289.class
classes/clojure/core/specs/alpha__init.class
classes/clojure/core/specs/alpha$fn__249$fn__255.class
classes/clojure/core/specs/alpha$fn__180.class
classes/clojure/core/specs/alpha$fn__237$fn__241.class
classes/clojure/core/specs/alpha$fn__249$fn__253.class
classes/clojure/core/specs/alpha$fn__212.class
classes/clojure/core/specs/alpha$fn__197.class
... 47 more ...

All These Class Files

$ find classes/ -type f | grep my

classes/my/ns$assoc_foo.class
classes/my/ns$fn__304.class
classes/my/ns$loading__6706__auto____298.class
classes/my/ns__init.class
classes/my/other_ns$fn__302.class
classes/my/other_ns$loading__6706__auto____300.class
classes/my/other_ns__init.class

The API

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-compile.pdf}

All These Class Files

$ find classes/ -type f | grep my

classes/my/ns$assoc_foo.class
classes/my/ns$fn__304.class
classes/my/ns$loading__6706__auto____298.class
classes/my/ns__init.class
classes/my/other_ns$fn__302.class
classes/my/other_ns$loading__6706__auto____300.class
classes/my/other_ns__init.class

Top Level Side Effects of my.ns

• (in-ns 'my.ns)
• call ns-helper-fn-1
• pass ns-helper-fn-2 to that transaction locking thing
• instantiate my.ns$assoc_foo
• call .setMeta on #'assoc-foo
• call .bindRoot on #'assoc-foo

my.ns__init

• public class my.ns__init
  • static fields
    • 2 vars (#'in-ns, #'assoc-foo),
    • 3 constants (metadata, ~’my.ns~, ~’clojure.core~)
  • static init
    • calls __init0()
    • calls load()
  • __init0
    • initializes the five static fields
  • load
    • call (in-ns 'my.ns)
    • initialize my.ns$loading__6706__auto____298 and .invoke()
    • check (= 'clojure.core 'my.ns)
    • call LockingTransaction.runInTransaction(new my.ns$fn__304())
    • set metadata on #'assoc-foo
    • initialize my.ns$assoc_foo, set root binding of #'assoc-foo

Order of Events

- (compile 'my.ns)
  - Compiler.compile
    - Compiler.compile1('(ns my.ns ...))
      - writes two helper classes
      - (require 'my.other-ns)
        - Compiler.compile
          - Compiler.compile1('(ns my.other-ns ...))
            - writes two helper classes
          - Compiler.compile1('(def foo "foo"))
          - writes init class for my.other-ns
    - Compiler.compile1('(defn assoc-foo ...))
      - writes assoc_foo class
    - writes init class for my.ns

So What Was All That Then?

compile

• operates on namespaces
• evals all of the code into the current JVM
• writes .class files for all classes, including an *__init class that performs all the top-level side effects

Part 4: require after compile

doing it all over again

\frame{\subsectionpage}

The Repl

\huge

$ clj
Clojure 1.10.0-RC2
user=> (require 'my.ns)
nil

The API

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-require-compiled.pdf}

(Class/forName "my.ns__init")

my.ns__init<clinit>

• looks up vars and constants
• calls (in-ns)
• calls (require 'my.other-ns)
  • my.other_ns__init<clinit>
    • looks up vars and constants
    • calls (in-ns)
    • sets meta and root of #'foo
• initializes my.ns$assoc_foo
• sets meta and root of #'assoc-foo

So What Was All That Then?

• require-ing a namespace when there’s a class file available results in loading the *__init class, which triggers all the top-level side effects that mutate the namespace graph

Part 5: :reload, etc.

let’s change something

\frame{\subsectionpage}

user=> (def foo "bar")

user=> (require 'my.ns)
nil
user=> (in-ns 'my.other-ns)
#object[clojure.lang.Namespace 0x4ee37ca3 "my.other-ns"]
my.other-ns=> (def foo "bar")
#'my.other-ns/foo

(def foo "bar")

\hspace*{-23pt} \includegraphics[scale=0.35]{images/RT-inkscape-9.pdf}

(def foo "bar")

my.other-ns=> (my.ns/assoc-foo {})
{:foo "bar"}

(defn assoc-foo ...) (require 'my.ns :reload)

src/my/ns.clj

(ns my.ns
  (:require
   [my.other-ns :as other]))

(defn assoc-foo
  [m]
  (assoc m :foo other/foo
           :another "map-entry"))

The API

\hspace*{-23pt} \includegraphics[scale=0.23]{images/API-require.pdf}

What Happens?

• (in-ns) (already exists)
• (require 'my.other-ns) (NOOP)
• Compile the function, load into DynamicClassLoader as my.ns$assoc_foo
• Compiler looks up #'assoc-foo (already exists)
• (.setMeta #'assoc-foo ...) (no problem)
• (.bindRoot #'assoc-foo (new my.ns$assoc_foo)) perfect!

(defn assoc-foo ...) (require 'my.ns :reload)

my.other-ns=> (my.ns/assoc-foo {})
{:foo "bar" :another "map-entry"}

So What Was All That Then?

• the repl and (require :reload) do largely the same thing:
  • Use Compile.eval
    • creates classes
    • loads them into the dynamic classloader
    • invokes them, so that they perform modifications to the namespace graph, existing namespaces and vars

Epilogue

\frame{\sectionpage}

Edge Cases

• Weird things can happen with name collisions
• The unit of compilation isn’t quite a form – files have scope for certain dynamic vars, which the repl can’t honor

Bottom Lines

• There’s a namespace graph!
• ns, def, and other top level “commands” mutate the namespace graph
• Functions are compiled to classes in memory or in the file system
• in AOT, files can be compiled to *__init classes that perform the same mutations as evaling the file

Outroduction

This is it, it’s over now

Org Stuff

gen-API-svgs.clj

#!/usr/bin/env bash

":"; DEPS=\
'{:deps
  {dorothy {:mvn/version "0.0.7"}
   hiccup  {:mvn/version "1.0.5"}}}'
":"; exec clojure -Sdeps "$DEPS" "$0" "$@"

(ns user.all-API-svgs
  (:require
   [clojure.string :as string]
   [clojure.walk   :as walk]
   [dorothy.core   :as dot]
   [dorothy.jvm    :as djvm]
   [hiccup.core    :refer [html]]))

(def grays ["black" "#555555" "#AAAAAA"])

(defn color1
  [mode]
  (case mode
    :highlight (grays 0)
    :normal (grays 1)
    :dim (grays 2)))

(defn color2
  [mode]
  (case mode
    :highlight (grays 0)
    :normal (grays 0)
    :dim (grays 1)))

(defn function-node
  [mode node-id header & extra-lines]
  [node-id
   {:shape  "none"
    :margin "0"
    :fillcolor (color1 mode)
    :color (color2 mode)
    :fontcolor (color2 mode)
    :label  (html [:table {:border "0"}
                   (->> extra-lines
                        (map #(vector % false))
                        (cons [header true])
                        (map (fn [[s first?]]
                               (let [s (string/replace s "\n" "<br />")]
                                 [:tr [:td {:bgcolor
                                            (if first?
                                              (color1 mode)
                                              "white")}
                                       (if first? [:b [:font {:color "white"}
                                                       s]]
                                           s)]]))))])}])

(defn choice-node
  [mode node-id & lines]
  [node-id {:shape     "record"
            :color     (color1 mode)
            :fontcolor (color2 mode)
            :style     "rounded"
            :penwidth  (if (= :highlight mode) "3" "1")
            :label     (->> lines
                            (map #(string/replace % "\n" "\\n"))
                            (string/join "|")
                            (format "{%s}"))}])

(defn action-node
  [mode node-id label]
  (let [color (case mode
                (:normal :highlight) (grays 0)
                :dim (grays 2))]
    [node-id {:label     label
              :fillcolor "white" #_color
              :color     color
              :fontcolor (color2 mode)
              :penwidth  (if (= :highlight mode) "3" "1")}]))

(def cluster-attrs
  {:color     "#666666"
   :fontcolor "#333333"
   :penwidth  "5"
   :fontsize  "20pt"
   :fontname  "monospace"
   :labeljust "l"})

(defn the-graph
  [node-mode-fn edge-mode-fn]
  (let [partial-highlight (fn [f]
                            (fn [id & args]
                              (apply f
                                     (node-mode-fn id)
                                     id args)))
        function-node     (partial-highlight function-node)
        choice-node       (partial-highlight choice-node)
        action-node       (partial-highlight action-node)
        edge              (fn self
                            ([f t] (self f t {}))
                            ([f t attrs]
                             [f t
                              (case (edge-mode-fn f t)
                                :highlight
                                (assoc attrs :penwidth "5")

                                :normal attrs

                                :dim (assoc attrs :color (grays 2) :fontcolor (grays 2)))]))]
    (dot/digraph
     [(dot/graph-attrs {:ranksep "0.2"})
      (dot/node-attrs {:fontname "monospace"
                       :shape "rect"
                       :style "filled"})
      (dot/edge-attrs {:fontname "monospace"})
      (function-node :require "require")
      (function-node :use "use")
      (edge :require :load-libs)
      (edge :use :load-libs)
      (function-node :load
                     "load"
                     "supports relative paths"
                     "cyclic dep check")
      (edge :load :RT/load)

      (function-node :load-string "load-string" "opens Reader")
      (edge :load-string :load-reader)
      (edge :load-reader :cload {:weight "30"})
      (function-node :load-reader "load-reader")
      (function-node :load-file "load-file")
      (edge :load-file :cload-file)
      (function-node :eval "eval")
      (edge :eval :ceval)
      (function-node :compile "compile" "binds *compile-files*")
      (edge :compile :load-one)

      (dot/subgraph :cluster_private
                    [(assoc cluster-attrs :label "^:private")

                     (function-node :load-libs
                                    "load-libs"
                                    "loops through args"
                                    "supports ns/class prefixes")
                     (edge :load-libs :load-lib)
                     (function-node :load-lib
                                    "load-lib"
                                    "calls alias and refer")
                     (edge :load-lib :load-lib-check)
                     (choice-node :load-lib-check
                                  ":reload or :reload-all?"
                                  "require used and\nns already loaded?")
                     (edge :load-lib-check :NOOP)
                     (action-node :NOOP "NOOP")
                     (edge :load-lib-check :load-one)
                     (edge :load-lib-check :load-all)
                     (function-node :load-all
                                    "load-all"
                                    "manipulates *loaded-libs*\nto force reloading")
                     (edge :load-all :load-one)
                     (function-node :load-one
                                    "load-one"
                                    "convert ns name to \"path\""
                                    "maybe throw if ns\n not defined after"
                                    "maybe mark ns as loaded")])
      (edge :load-one :load)

      (dot/subgraph :cluster_compiler
                    [(assoc cluster-attrs :label "clojure.lang.Compiler")
                     (function-node :cload "load(rdr)*" "binds vars" "new classloader" "loops over forms")
                     (edge :cload :ceval {:label "each form"})
                     (function-node :cload-file "loadFile(filepath)" "opens Reader")
                     (edge :cload-file :cload)
                     (function-node :ceval "eval(form)*" "new classloader" "macroexpand" "analyze, eval")
                     (function-node :ccompile "compile(rdr)*" "binds vars" "loops over forms" "writes class file")
                     (edge :ccompile :ccompile1 {:label "each form"})
                     (function-node :ccompile1 "compile1(...)" "new classloader" "macroexpand" "analyze, eval")
                     (dot/subgraph [{:rank "same"} :ccompile1 :ceval])])

      (dot/subgraph :cluster_RT
                    [(assoc cluster-attrs :label "clojure.lang.RT")
                     (function-node :RT/load "load(path)*" "check for .clj, .cljc,\nand .class files")
                     (edge :RT/load :RT/load2check)
                     (choice-node :RT/load2check "not-stale class\nfile exists?")
                     (edge :RT/load2check :classforname {:label "yes"})
                     (action-node :classforname "Class.forName(...)")
                     (edge :RT/load2check :RT/load2check2 {:label "no"})
                     (choice-node :RT/load2check2 "is *compile-files*\nset?")
                     (edge :RT/load2check2 :RT/loadresourcescript3 {:label "no"})
                     (edge :RT/load2check2 :RT/compile {:label "yes"})
                     (function-node :RT/loadresourcescript3 "loadResourceScript\n(class, filename)*" "opens Reader")
                     (edge :RT/loadresourcescript3 :cload)
                     (function-node :RT/compile "compile(filename)" "opens Reader")
                     (edge :RT/compile :ccompile)
                     (edge :RT/clinit :RT/loadresourcescript3 {:label "user.clj"})
                     (function-node :RT/clinit "&lt;clinit&gt;")])

      (dot/subgraph :cluster_compile
                    [(assoc cluster-attrs :label "clojure.lang.Compile")
                     (function-node :Compile/main "main(namespaces)" "binds vars" "loops over args")])
      (edge :Compile/main :compile {:label "each namespace"})

      (dot/subgraph :cluster_main
                    [(assoc cluster-attrs :label "clojure.main")
                     (function-node :main "main")
                     (edge :main :repl {:label "--repl or\\nno args"})
                     (edge :main :choice {:label "script\\nor --init"})
                     (choice-node :choice "starts with @?")
                     (edge :choice :RT/loadresourcescript3 {:label "yes"})
                     (edge :choice :cload-file {:label "no"})
                     (function-node :repl "repl" "new classloader" "binds vars")])
      (edge :main :load-reader {:label "STDIN script"})
      (edge :main :eval {:label "--eval"})
      (edge :main :require {:label "--main"})
      (edge :repl :eval {:label "each form"})

      (dot/subgraph :cluster_genclass
                    [(assoc cluster-attrs :label "class from gen-class")
                     (function-node :gen-class/main "main(String[] args)")])
      (edge :gen-class/main :load {:label "(via clojure.lang.Util)"})

      (dot/subgraph [{:rank "same"}
                     :require
                     :use
                     #_:eval
                     :load
                     #_:load-file
                     :compile
                     #_:load-reader])])))

(doseq [[file-suffix highlighted edge-blacklist]
        [["normal" :all]
         ["require" #{:require
                      :load-libs
                      :load-lib
                      :load-lib-check
                      :load-one
                      :load
                      :RT/load
                      :RT/load2check
                      :RT/load2check2
                      :RT/loadresourcescript3
                      :cload
                      :ceval}]
         ["require-again" #{:require
                            :load-libs
                            :load-lib
                            :load-lib-check
                            :NOOP}]
         ["compile" #{:compile :load-one :load :RT/load :RT/load2check
                      :RT/load2check2 :RT/compile :ccompile :ccompile1}]
         ["require-compiled" #{:require :load-libs :load-lib :load-lib-check
                               :load-one :load :RT/load :RT/load2check
                               :classforname}]
         ["repl" #{:main :repl :eval :ceval} #{[:main :eval]}]
         ["nested-require-1" #{:main :eval :ceval}]
         ["nested-require-2" #{:require
                               :load-libs
                               :load-lib
                               :load-lib-check
                               :load-one}]
         ["nested-require-3" #{:load-one
                               :load
                               :RT/load
                               :RT/load2check
                               :RT/load2check2
                               :RT/loadresourcescript3
                               :cload
                               :ceval}]
         ["nested-require-4" #{:require
                               :load-libs
                               :load-lib
                               :load-lib-check
                               :load-one
                               :load}]
         ["nested-require-5" #{:load :RT/load
                               :RT/load2check
                               :RT/load2check2
                               :RT/loadresourcescript3
                               :cload :ceval}]
         ["new-classloaders" #{:cload :ceval :ccompile1 :repl} #{[:cload :ceval]}]]]
  (djvm/save! (dot/dot (the-graph
                        (if (= :all highlighted)
                          (constantly :normal)
                          #(if (highlighted %) :highlight :dim))
                        (if (= :all highlighted)
                          (constantly :normal)
                          (fn [f t]
                            (if (and (highlighted f)
                                     (highlighted t)
                                     (not ((or edge-blacklist #{}) [f t])))
                              :highlight
                              :dim)))))
              (format "dot-svgs/API-%s.svg" file-suffix)
              {:format :svg}))

(shutdown-agents)

namespace-graph.dot

digraph G {
  fontname=monospace;
  rankdir=LR;
  node[fontname=monospace,shape=rect,style=filled];

  subgraph vars {
    node[fillcolor="#88FF88"];
    assocfoovar[label="#'assoc-foo"];
    requirevar[label="#'require"];
    foovar[label="#'foo"];
    assocvar[label="#'assoc"];
  }

  subgraph functions {
    node[fillcolor="#4488DD",shape=oval];
    assocfn[label="#fn clojure.core$assoc__5401"];
    assocfoofn[label="#fn my.ns$assoc_foo"];
    requirefn[label="#fn clojure.core$require"];
  }

  subgraph namespaces_etc {
    node[margin=0,shape=none];
    allns[label=<
      <table border="1" cellspacing="0">
        <tr><td bgcolor="black"><font color="white"><b>clojure.lang.Namespace</b></font></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>static Map namespaces</b></font></td></tr>
        <tr><td align="left" port="core">'clojure.core</td></tr>
        <tr><td align="left" port="myns">'my.ns</td></tr>
        <tr><td align="left" port="myotherns">'my.other.ns</td></tr>
        <tr><td align="left">...17 more...</td></tr>
      </table>
    >];

    core[label=<
      <table border="1" cellspacing="0">
        <tr><td port="label" bgcolor="black"><font color="white"><b>#ns clojure.core</b></font></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.mappings</b></font></td></tr>
        <tr><td align="left" port="require">'require</td></tr>
        <tr><td align="left" port="assoc">'assoc</td></tr>
        <tr><td align="left" port="string">'String</td></tr>
        <tr><td align="left"><i>...896 more...</i></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.aliases</b></font></td></tr>
        <tr><td align="left" port="jio">'jio</td></tr>
      </table>
    >];

    myns[label=<
      <table border="1" cellspacing="0">
        <tr><td port="label" bgcolor="black"><font color="white"><b>#ns my.ns</b></font></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.mappings</b></font></td></tr>
        <tr><td align="left" port="assoc">'assoc</td></tr>
        <tr><td align="left" port="string">'String</td></tr>
        <tr><td align="left" port="assocfoo">'assoc-foo</td></tr>
        <tr><td align="left"><i>...754 more...</i></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.aliases</b></font></td></tr>
        <tr><td align="left" port="other">'other</td></tr>
      </table>
    >];

    myotherns[label=<
      <table border="1" cellspacing="0">
        <tr><td port="title" bgcolor="black"><font color="white"><b>#ns my.other-ns</b></font></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.mappings</b></font></td></tr>
        <tr><td align="left" port="foo">'foo</td></tr>
        <tr><td align="left"><i>...756 more...</i></td></tr>
        <tr><td bgcolor="#444444"><font color="white"><b>.aliases</b></font></td></tr>
        <tr><td><i>(empty)</i></td></tr>
      </table>
    >];

    # why the hell is this edge rendered so wrong?
    # myns:other -> myotherns:title[constraint=false];
    # {rank=same; myns; myotherns}
  }

  allns:core -> core:label;
  allns:myns -> myns:label;
  allns:myotherns -> myotherns:title;

  assocvar -> core:label[style="dashed",constraint=false];
  requirevar -> core:label[style="dashed",constraint=false];

  core:assoc -> assocvar;
  core:require -> requirevar;
  core:string -> string;

  myns:assoc -> assocvar;
  myns:string -> string;
  myotherns:foo -> foovar;
  myns:assocfoo -> assocfoovar;
  assocvar -> assocfn;
  requirevar -> requirefn;
  foovar -> foo;
  foo[label="\"foo\"",fillcolor="#FF8888",shape=oval];
  assocfoovar -> assocfoofn;
  foovar -> myotherns:title[style=dashed];
  assocfoovar -> myns:label[style=dashed,constraint=false];
  assocfoofn -> assocvar[constraint=false];
  assocfoofn -> foovar[constraint=false];
  string[label="#class String",shape=octagon,fillcolor=yellow];
}