Skip to content

Instantly share code, notes, and snippets.

@cjohansen

cjohansen/stasis.html

Created February 2, 2015 16:12
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cjohansen/45b73e6206597c51b1dc to your computer and use it in GitHub Desktop.
Save cjohansen/45b73e6206597c51b1dc to your computer and use it in GitHub Desktop.
Download ZIP
Raw
stasis.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="chrome=1">
<meta http-equiv="X-UA-Compatible" content="edge">
<title>Building static sites in Clojure with Stasis</title>
<meta name="author" content="Christian Johansen">
<meta name="robots" content="all">
<link href="http://feeds.feedburner.com/cjno-en" rel="alternate" type="application/rss+xml" title="cjohansen.no blog-feed">
<link rel="Shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="stylesheet" type="text/css" href="/stylesheets/cjohansen.css">
</head>
<body>
<div class="banner masthead">
<p>
<a href="/"><strong>cjohansen.no</strong> Programming, Free software</a>
</p>
</div>
<div class="article">
<h1>Building static sites in Clojure with Stasis</h1>
<p>
<a href="https://github.com/magnars/stasis">Stasis</a> is a static site
toolkit for Clojure. Unlike pretty much every other static site
generator, though, it is not an "opinionated framework", or packed full
with flavor-of-the-month templating languages and whatnot. It is just a
few functions that helps with creating websites that can be hosted as
static files, and developed against a live server (enabling layouts, and
various dynamic features to generate the pages). As its Readme states;
there are no batteries included. This post will take you through
creating your first Stasis site, and serving it with super-optimized
frontend assets, courtesy
of <a href="https://github.com/magnars/optimus">Optimus</a>. Both Stasis
and Optimus are written by my good friend and colleague,
<a href="http://emacsrocks.com">Mr. Emacs
Wizard</a>, <a href="https://github.com/magnars">Magnar Sveen</a>.
</p>
<p>
The source code for this post can be
found <a href="https://github.com/cjohansen/cjohansen-no/tree/blog-post">on
GitHub</a>. As I plan to evolve the code to become my new blog
(eventually), you should look for the <code>blog-post</code> tag, which
represents the code as per this post.
</p>
<h2>Who is this for?</h2>
<p>
This post may interest anyone looking to set up a static site of some
sort. The range of sites that can be successfully developed as static
sites are bigger than you might think. While Stasis is a Clojure tool,
only a very basic understanding of Lisp should be necessary to follow
along.
</p>
<p>
I'm hoping this post will not only show you the power of Clojure and
Stasis for building static web sites, but also give you a good
introduction to some very useful Clojure libraries. Maybe even to
Clojure itself. In particular, I will discuss using these libraries:
<a href="http://github.com/magnars/stasis">Stasis</a>,
<a href="http://github.com/magnars/optimus">Optimus</a>,
<a href="https://github.com/cgrand/enlive">enlive</a>,
<a href="https://github.com/weavejester/hiccup">hiccup</a>,
<a href="https://github.com/Raynes/cegdown">cegdown</a>,
<a href="https://github.com/bfontaine/clygments">clygments</a> and even
write some tests with <a href="https://github.com/marick/Midje">Midje</a>.
</p>
<div class="toc" id="toc"></div>
<h2 id="setup">Getting set up</h2>
<p>
First things first, let's get a project set up to serve our frontpage.
If you've never worked with
Clojure, <a href="http://leiningen.org/">install Leiningen</a>. Now
create your project:
</p>
<pre class="highlight shell"><code>lein new cjohansen-no
cd cjohansen-no</code></pre>
<p>
That creates an empty project for you. I will use this post to start the
new code base for my blog. You might want to give yours a different
name. Open project.clj, and add Stasis as a dependency. While you're at
it, add a description and tune the license to your desires. When you're
done, it should look something like this:
</p>
<pre class="highlight lisp"><code>(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "2.2.0"]])</code></pre>
<p>
Now we will add a truly static page (i.e. a file). Create and open
src/cjohansen_no/web.clj. This is the namespace we will use to define
our pages. Put the following content in it:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [stasis.core :as stasis]))
(defn get-pages []
(stasis/slurp-directory "resources/public" #".*\.(html|css|js)$"))</code></pre>
<p>
<code>#"..."</code> is Clojure syntax for regular expressions.
</p>
<p>
(Note that the namespace is called cjohansen-no, while the directory was
called cjohansen_no. If you misspell either of these, Clojure will bark
at you with a rather unfriendly message saying it can't find the
namespace. Remember to double check file/directory names and
namespaces).
</p>
<p>
The web.clj file pulls in stasis under the local name
<code>stasis</code>, and defines one function. Stasis will expect to
receive a map of <code>url => content</code> defining pages, so this is
what <code>get-pages</code> does. The content can be a function, in
which case it will be only be called when Stasis needs to serve this
particular page. This lazy loading becomes useful if your site is big
enough for it to become slow by loading everything in one go. For now,
we'll not worry about it.
</p>
<p>
<code>slurp-directory</code> is a Stasis function that creates a map of
files in a directory (recursively). Every file in the subtree that
matches the provided regular expression (i.e. every html, css and js
file under resources/public) will be included. The keys in the map (e.g.
the URLs), will be the file path relative
to <code>resources/public</code>. For example,
<code>resources/public/index.html</code> will be served
as <code>/index.html</code>.
</p>
<p>
Put the following in resources/public/index.html:
</p>
<pre class="highlight html"><code>&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;My blog&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;My blog&lt;/h1&gt;
&lt;p&gt;
Welcome to it.
&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;</code></pre>
<p>
As Stasis is a "no batteries included" framework, we need to do some
work to either export the static site or view it live through a web
server. As the file is already static, we'll set up the live server
first. Update project.clj to pull in a couple of dependencies and
configure a Leiningen plugin called ring:
</p>
<pre class="highlight lisp"><code>(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]]
:ring {:handler cjohansen-no.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})</code></pre>
<p>
<a href="https://github.com/ring-clojure/ring">Ring</a> is the defacto
HTTP toolkit for Clojure, and can be compared to Python's WSGI or Ruby's
Rack. We have added it as a dependency, and set up the <code>app</code>
function in the <code>cjohansen-no.web</code> namespace (e.g. the file
we created earlier) to be the entry-point for the web server. The
profile configuration loads some convenient Leiningen tasks for the
development profile (which is also the default profile).
</p>
<p>
A web application in Ring is just a function that receives as its only
argument a request hash and returns a map like the following:
</p>
<pre class="highlight lisp"><code>{:status 200
:headers {"Content-Type" "text/html"}
:body "Hello World"}</code></pre>
<p>
Stasis provides the <code>serve-pages</code> function to help us produce
this response. It returns a function that will look at the request and
find the corresponding page in our map. Add the <code>app</code>
function to the src/cjohansen-no/web.clj file:
</p>
<pre class="highlight lisp"><code>(def app (stasis/serve-pages get-pages))</code></pre>
<p>
Run the server:
</p>
<pre class="highlight shell"><code>lein ring server</code></pre>
<p>
This will pop up a browser displaying your static HTML file in all its
naked glory.
</p>
<h2 id="templating">Adding templating</h2>
<p>
Next we will add a page that is split between the content/body of the
page and the wrapping layout. The layout will be shared by many files,
so applying it in one central place saves us some work.
</p>
<p>
Add a partial page to resources/partials/about.html
</p>
<pre class="highlight html"><code>&lt;h1&gt;About this site&lt;/h1&gt;
&lt;p&gt;
It is a web page.
&lt;/p&gt;</code></pre>
<p>
Rather than using HTML manually to create the layout, we will use
the popular Clojure templating library called
<a href="https://github.com/weavejester/hiccup">Hiccup</a>. Hiccup
allows us to express HTML in a more compact form by using vectors,
keywords and maps. It is best illustrated with an example. Add Hiccup to
project.clj:
</p>
<pre class="highlight lisp"><code>:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]] ;; Like so</code></pre>
<p>
Alter the namespace form in src/cjohansen_no/web.clj to require the
<code>html5</code> function from Hiccup:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [hiccup.page :refer [html5]]
[stasis.core :as stasis]))</code></pre>
<p>
Add the following function to the same file:
</p>
<pre class="highlight lisp"><code>(defn layout-page [page]
(html5
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1.0"}]
[:title "Tech blog"]
[:link {:rel "stylesheet" :href "/styles/styles.css"}]]
[:body
[:div.logo "cjohansen.no"]
[:div.body page]]))</code></pre>
<p>
As you can see, Hiccup markup is mostly a leaner version of HTML. One
thing that makes Hiccup very cool is that is accepts elements like the
ones in our example, nested lists of elements, and nils. This means that
we can map over data structures inline in the Hiccup structure without
having to worry about nested lists creating nested markup structures.
Think of lists as
the <a href="https://developer.mozilla.org/en/docs/Web/API/DocumentFragment">DocumentFragments</a>
of Hiccup. We can also inline <code>if</code> forms without else forms
without worrying about dangling <code>nil</code>s causing weird
artefacts in the generated markup.
</p>
<p>
To use the new layout, we will add a page definition for our partial
page, and apply the layout on it. To read files from the app's resources
directory, require the <code>clojure.java.io</code> package:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [clojure.java.io :as io]
[hiccup.page :refer [html5]]
[stasis.core :as stasis]))</code></pre>
<p>
Add the about page function:
</p>
<pre class="highlight lisp"><code>(defn about-page [request]
(layout-page (slurp (io/resource "partials/about.html"))))</code></pre>
<p>
Finally, add the new page to the page map created
in <code>get-pages</code>:
</p>
<pre class="highlight lisp"><code>(defn get-pages []
(merge (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
{"/about/" about-page}))</code></pre>
<p>
Because we added a new dependency to project.clj, you now need to quit
the server (Ctrl+C) and restart it. When you have, /about/ should
present you with your fantastic new page-with-layout.
</p>
<h3>Serving all partials</h3>
<p>
Doing all this work for every new partial page seems a tad bit too
laborious. We will generalize the code we just added by re-writing it to
serve all partial pages under <code>resources/partials</code> in the
same way.
</p>
<p>
Remember how <code>(stasis/slurp-directory "resources/partials"
#".*\.html$")</code> will create a map with relative paths as keys and
the contents of the corresponding files as values? This is almost what
we want, except we now also want to wrap the content in the layout. To
achieve this, we will loop through the map and wrap all the values in a
function call that adds the layout. Like so:
</p>
<pre class="highlight lisp"><code>(defn partial-pages [pages]
(zipmap (keys pages)
(map layout-page (vals pages))))</code></pre>
<p>
This function accepts as input the map produced by Stasis'
<code>slurp-directory</code> and returns a map where the values have all
been wrapped in a layout. The <code>zipmap</code> function takes two
collections and returns a map. It builds each map entry by pulling an
entry from the first collection to use as the key, and an entry from the
second collection to use as the value. Now remove the
<code>about-page</code> function, and update <code>get-pages</code>
to look like this:
</p>
<pre class="highlight lisp"><code>(defn get-pages []
(merge (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
(partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))))</code></pre>
<p>
This will <em>almost</em> work. You will find our about page from before
at <a href="http://localhost:3000/about.html">/about.html</a>, instead
of /about. The reason is that <code>slurp-directory</code> uses the
relative file names as paths. We can fix this in one of two ways:
</p>
<ol>
<li>
Mapping the keys as well, to lose the extension. While this will
certainly work, it will produce odd results for any file called
index.html
</li>
<li>
Rename <code>resources/partials/about.html</code> to
<code>resources/partials/about/index.html</code>
</li>
</ol>
<p>
We will opt for option 2 in this case. I will show option 1 when we deal
with markdown files.
</p>
<h3>Path conflicts</h3>
<p>
Now that we have two page sources, and both of them create root-level
URLs, there is a risk that we end up with conflicts. This would happen
if someone added <code>about/index.html</code> to the public directory,
because it would conflict with the <code>about/index.html</code>
produced by the partial-pages function. Update <code>get-pages</code> to
the following to avoid the problem:
</p>
<pre class="highlight lisp"><code>(defn get-pages []
(stasis/merge-page-sources
{:public (stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:partials (partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))}))</code></pre>
<p>
The <code>merge-page-sources</code> function works pretty much
like <code>merge</code>, except it will throw an exception if either
source defines duplicate URLs. The map keys are useful to determine the
source of the conflict. For instance, if you were to
add <code>public/about/index.html</code>, you would get this error:
</p>
<pre><code>URL conflicts between :public and :partials: #{"/about/index.html"}</code></pre>
<p>(That last bit is Clojure notation for describing sets).</p>
<h2 id="markdown">Writing in markdown</h2>
<p>
Partial pages are nice, but being able to write in markdown would be
even better. <a href="https://clojars.org/me.raynes/cegdown">Cegdown</a>
is a Clojure wrapper for Pegdown, a popular Java library for rendering
markdown. Add it to your project.clj:
</p>
<pre class="highlight lisp"><code> :dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]] ; Like so</code></pre>
<p>
Now, add it to the namespace form in src/cjohansen_no/web.clj. While
we're at it, we will require the Clojure string library as well (we'll
use it shortly):
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))</code></pre>
<p>
Now we will add a function to render every page in
<code>resources/md</code> as markdown. It will be very similar to the
partials we did before, but now with markdown rendering as well. Because
we don't want ".md" as part of the URL for these pages, we will map the
keys as well.
</p>
<pre class="highlight lisp"><code>(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "/") (keys pages))
(map #(layout-page (md/to-html %)) (vals pages))))</code></pre>
<p>
The <code>#( )</code> form is a function literal. Inside
it, <code>%</code> refers to the first argument. In the above example,
the following are identical:
</p>
<pre class="highlight lisp"><code>#(str/replace % #"\.md$" "")
;; ...and:
(fn [path] (str/replace path #"\.md$" ""))</code></pre>
<p>
I wrote more on the anonymous function literal
in <a href="/clojure-to-die-for">a separate post</a>.
</p>
<p>
The final step is to add the new page source to our map:
</p>
<pre class="highlight lisp"><code>(defn get-pages []
(stasis/merge-page-sources
{:public
(stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:partials
(partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
:markdown
(markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))</code></pre>
<p>
Add the following to <code>resources/md/my-first-post.md</code>:
</p>
<pre class="highlight markdown"><code># My first post
It's pretty short for now.
</code></pre>
<p>
Restart the server again (we added new dependencies, remember?) When you
have, /my-first-post should present you with your brief, but lovely blog
post.
</p>
<h2 id="syntax-highlighting">Post-processing: Syntax highlighting</h2>
<p>
Any self-respecting tech blog needs nice syntax highlighting for code
blocks. When it comes to syntax
highlighting, <a href="http://pygments.org/">Pygments</a> is the bee's
knees. It supports just about any language you can think of, there's a
bunch of color themes around and it is stable and resillient. It is also
the library used to highlight code on
GitHub. <a href="https://github.com/bfontaine/clygments">Clygments</a>
is a Clojure interface to it (which uses Jython; Pygments is a Python
library).
</p>
<p>
We will add syntax highlighting as a post-processing step for HTML. This
way, we can support syntax highlighting for full static pages in the
public directory, partial pages and pages rendered from markdown. To do
this, we will use another templating library for
Clojure, <a href="https://github.com/cgrand/enlive">enlive</a>.
Actually, enlive is more than just a templating library. As you will
see, it can be used to transform documents in various interesting ways.
</p>
<h3>A code block</h3>
<p>
Let's add
a <a href="https://help.github.com/articles/github-flavored-markdown#fenced-code-blocks">fenced
code block</a> to our markdown file:
</p>
<pre class="highlight markdown"><code># My first post
It's pretty short for now. Here's our project.clj:
```clj
(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]]
:ring {:handler cjohansen-no.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
```
</code></pre>
<p>
In order for this work, we need to inform <code>cegdown</code> that we
want to enable the fenced code blocks extension. While we're at it,
we'll enable a couple of other useful extensions as well:
</p>
<pre class="highlight lisp"><code>(def pegdown-options ;; https://github.com/sirthias/pegdown
[:autolinks :fenced-code-blocks :strikethrough])
(defn render-markdown-page [page]
(layout-page (md/to-html page pegdown-options)))
(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "") (keys pages))
(map render-markdown-page (vals pages))))</code></pre>
<p>
Reloading the blog post will show you how the fenced code blocks are
rendered:
</p>
<pre class="highlight html"><code>&lt;pre&gt;&lt;code class="clj"&gt;...&lt;/code&gt;&lt;/pre&gt;</code></pre>
<p>
We will now use enlive to extract this piece of markup and replace it
with the version highlighted by clygments. First off, add the new
dependencies to project.clj (remember to restart the server!)
</p>
<pre class="highlight lisp"><code> :dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"] ; New
[clygments "0.1.1"]] ; New</code></pre>
<p>
Add a new namespace (i.e., file) to the project. Copy the following code
into <code>src/cjohansen_no/highlight.clj</code>:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.highlight
(:require [clojure.java.io :as io]
[clygments.core :as pygments]
[net.cgrand.enlive-html :as enlive]))</code></pre>
<p>
Enlive has many tricks up its sleave. Perhaps the most interesting one
is the somewhat confusingly named <code>sniptest</code>. It takes some
HTML as a string, and selector/function pairs. It will then;
</p>
<ol>
<li>Parse the HTML</li>
<li>Find all nodes matching the selector</li>
<li>Call the corresponding function once for every match</li>
<li>Replace the node with the result of calling the function</li>
<li>Return the transformed HTML as a string</li>
</ol>
<pre class="highlight lisp"><code>(defn highlight-code-blocks [page]
(enlive/sniptest page
[:pre :code] highlight
[:pre :code] #(assoc-in % [:attrs :class] "codehilite")))</code></pre>
<p>
This function will find every <code>code</code> element inside
a <code>pre</code> element and pass it through the
<code>highlight</code> function (to be shown). Then, it will add a class
name to the same elements. In practice, you would probably do both in
the <code>highlight</code> function, but this allows me to illustrate
how you can perform multiple transformations in one go. The "codehilite"
class name just happens to be class name used by the Pygments CSS themes
available <a href="https://github.com/richleland/pygments-css.git">here</a>
(we will include this later). Add the second function:
</p>
<pre class="highlight lisp"><code>(defn- highlight [node]
(let [code (->> node :content (apply str))
lang (->> node :attrs :class keyword)]
(pygments/highlight code lang :html)))</code></pre>
<p>
The dash in <code>defn-</code> means that this function is private, and
only referrable within the current namespace. The nodes that enlive
operate on are maps like this:
</p>
<pre class="highlight lisp"><code>{:tag :code
:attrs {:class "clj"}
:content [...]}</code></pre>
<p>
The content is a list of new nodes and/or strings.
</p>
<p>
<code>-&gt;&gt;</code> is the
<a href="http://clojuredocs.org/clojure_core/clojure.core/-%3E%3E">thread-last
macro</a>. It takes any number of arguments, threads values from left to
right; given <code>(-&gt;&gt; a (b 1 2) c)</code>, it will take the
value <code>a</code>, pass it as the last argument to <code>b</code>
(i.e. <code>(b 1 2 a)</code>), pass the return value of that expression
as the last argument to <code>c</code>, and finally return the result of
that. So the above line:
</p>
<pre class="highlight lisp"><code>(->> node :content (apply str))</code></pre>
<p>
Is the same as this:
</p>
<pre class="highlight lisp"><code>(apply str (:content node))</code></pre>
<p>
I wrote more on threading macros in
<a href="/clojure-to-die-for">a separate post</a>.
</p>
<p>
Using a keyword as a function is one way to look up that key in a map.
For instance, <code style="white-space: nowrap">(:name person)</code> in Clojure is the same as
<code>person[:name]</code> in Ruby.
</p>
<p>
<code>(:content node)</code> returns a vector of content (either strings
or nested elements). If we passed this vector to <code>str</code>, we
would get a string representation of the vector back. To avoid this, we
use <code>apply</code>, which unfolds the elements of the vector as
individual arguments. This example hopefully illustrates the difference:
</p>
<pre class="highlight lisp"><code>(str ["A" "B"]) ;; => "[\"A\" \"B\"]"
(apply str ["A" "B"]) ;; equivalent to (apply str ["A" "B"])
;; => "AB"</code></pre>
</p>
<p>
To preview the highlighting in the browser, we need to make some changes
to the web namespace. Start by updating the namespace form, at the top
of web.clj, to pull in our new dependency:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [cjohansen-no.highlight :refer [highlight-code-blocks]] ; This one
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))</code></pre>
<p>
Syntax highlighting should apply to all pages. A good place to do it is
between our old <code>get-pages</code> function and Stasis' rendering.
We will do this by adding a <code>prepare-pages</code> function:
</p>
<pre class="highlight lisp"><code>(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(highlight-code-blocks %) (vals pages))))</code></pre>
<p>
Again, we use zipmap to produce a new map where the keys are untouched,
but the values have been mapped. To make Stasis run through this,
rename <code>get-pages</code> to <code>get-raw-pages</code>, and add a
new get-pages:
</p>
<pre class="highlight lisp"><code>(defn get-raw-pages []
(stasis/merge-page-sources
{:public
(stasis/slurp-directory "resources/public" #".*\.(html|css|js)$")
:partials
(partial-pages (stasis/slurp-directory "resources/partials" #".*\.html$"))
:markdown
(markdown-pages (stasis/slurp-directory "resources/md" #"\.md$"))}))
(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (highlight-code-blocks %)) (vals pages))))
(defn get-pages []
(prepare-pages (get-raw-pages)))</code></pre>
<p>
Reloading the markdown page should show you that what we've done so far
both kinda worked and kinda didn't.
</p>
<p>
We are getting highlighted code. However, Pygments includes some
unwanted wrapping markup. As we're already inside a <code>pre</code>
element, the result is not quite as desired. To fix this we will use
enlive once more to massage the output from Pygments. The output
includes a wrapper div and a pre, let's extract just the code:
</p>
<pre class="highlight lisp"><code>(defn- extract-code
[highlighted]
(-> highlighted
java.io.StringReader.
enlive/html-resource
(enlive/select [:pre])
first
:content))
(defn- highlight [node]
(let [code (->> node :content (apply str))
lang (->> node :attrs :class keyword)]
(assoc node :content (-> code
(pygments/highlight lang :html)
extract-code))))
</code></pre>
<p>
Enlive's <code>select</code> function selects elements from a document,
but unlike <code>sniptest</code>, it does not accept a string. Instead,
we must go through its <code>html-resource</code> function, which only
accepts input streams. The end result is that we do a select on what
Pygments gives us in order to get just the highlighted code. Refreshing
the blog post shows that it works as expected.
</p>
<p>
The <code>-&gt;</code> is the thread-first macro. It works like
thread-last, except it threads values as the first argument to the next
function. The above example could be written without threading as well,
but most people find the threading form to be the easiest on the eyes:
</p>
<pre class="highlight lisp"><code>(-> highlighted
java.io.StringReader.
enlive/html-resource
(enlive/select [:pre])
first
:content)
;; Same as:
(:content (first (enlive/select (enlive/html-resource
(java.io.StringReader. highlighted))
[:pre])))
</code></pre>
<p>
To add some styling, pick a CSS file
from <a href="https://github.com/richleland/pygments-css.git">the
suggested themes repo</a>, and load it onto the page. Update
the <code>layout-page</code> in web.clj to look like this:
</p>
<pre class="highlight html"><code>(defn layout-page [request page]
(html5
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1.0"}]
[:title "Tech blog"]
[:link {:rel "stylesheet" :href "/pygments-css/autumn.css"}]]
[:body
[:div.logo "cjohansen.no"]
[:div.body page]]))</code></pre>
<h2 id="lazy">Lazy pages</h2>
<p>
As we're adding more features to our site, it is becoming apparent that
processing all the pages to completion on every request isn't ideal.
Fixing this is quite easy with Stasis, because we can give Stasis
functions instead of strings, and then Stasis will call the function to
build a particular page only when it needs to render that specific page.
</p>
<p>
To make our pages lazy, update <code>prepare-pages</code> to replace the
values with functions instead of strings of highlighted HTML. The
function should take one argument, the request map.
</p>
<pre class="highlight lisp"><code>(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (highlight-code-blocks %)) (vals pages))))</code></pre>
<p>
By having the function literal return a new function that takes one
argument, we have significantly improved performance for our development
server.
</p>
<h2 id="assets">Asset optimization</h2>
<p>
Now that we have a blog with syntax highlighting, we need to start
thinking about delivery. Fast webpages beat slow ones on all sorts of
metrics. One way to make our site faster is by employing various
frontend asset optimization techniques. For this purpose, there is
(among others) <a href="https://github.com/magnars/optimus">Optimus</a>.
We will use it to:
</p>
<ul>
<li>Concatenate CSS and JavaScript files</li>
<li>Minify CSS and JavaScript files</li>
<li>Serve CSS and JavaScript from cache-friendly URLs</li>
</ul>
<p>
A "cache friendly" URL is one that is unique every time the contents of
the URL changes. This way we can serve assets with aggressive cache
headers, and users will only need to download them once. The next time
we deploy, if the assets have changed, they will have a new URL. Optimus
facilitates this by providing some functions to help us link to assets.
</p>
<p>
We will start by adding Optimus as a dependency in project.clj. Remember
to restart the server after doing this.
</p>
<pre class="highlight lisp"><code>(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]] ; New
:ring {:handler cjohansen-no.web/app}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
</code></pre>
<p>
Now update the web.clj namespace form to require some functions from
Optimus:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [optimus.assets :as assets] ; New
[optimus.optimizations :as optimizations] ; New
[optimus.prime :as optimus] ; New
[optimus.strategies :refer [serve-live-assets]] ; New
[cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))</code></pre>
<p>
Instead of having Stasis serve the files in public, we will hand them to
Optimus as assets. We will define a separate function for these assets,
as it makes for a natural place to add further assets and/or bundles of
assets later:
</p>
<pre class="highlight lisp"><code>(defn get-assets []
(assets/load-assets "public" [#".*"]))</code></pre>
<p>
If your CSS files use <code>@import</code>, Optimus will (by default)
take care to inline the import, so there is no need to define bundles at
this point. Refer to
the <a href="https://github.com/magnars/optimus#readme">Optimus
readme</a> for more details.
</p>
<p>
To make our app use the new assets, we will change the <code>app</code>
function:
</p>
<pre class="highlight lisp"><code>(def app
(optimus/wrap (stasis/serve-pages get-pages)
get-assets
optimizations/all
serve-live-assets))</code></pre>
<p>
The call to <code>stasis/serve-pages</code> returns a function (a Ring
app, remember?) <code>optimus/wrap</code> returns another function with
the same signature that wraps the original one. We pass it the function
to get all our assets, optimization rules (a function) and a strategy
for serving the assets (also a function). <code>optimizations/all</code>
is a grab bag of every trick Optimus knows:
</p>
<ul>
<li>Minify JavaScript</li>
<li>Minify CSS</li>
<li>Inline CSS imports</li>
<li>Concatenate bundles</li>
<li>Add cache-bust expires headers (replace URL references with generated unique ones)</li>
<li>Add last-modified headers</li>
</ul>
<p>
You are free to pick and choose from this list if you want, but for most
cases, <code>optimizations/all</code> is what you want.
</p>
<p>
Lastly, we employed Optimus' <code>serve-live-assets</code> strategy,
which means that Optimus will read assets from disk on every request.
This is useful in development mode, but in a production setting, you
would typically use one that's less resource intensive,
like <code>serve-frozen-assets</code>.
</p>
<p>
Create a CSS file and make sure it gets included from the page layout.
</p>
<pre class="highlight css"><code>@import url(../pygments-css/autumn.css);
body {
font: 16px Helvetica, arial, freesans, clean, sans-serif;
line-height: 1.5;
margin: 0 10px;
}</code></pre>
<p>
Refreshing the blog in the browser should display the same page as
before. However, if you hit the CSS file directly, you will find that
Optimus has done what it can to optimize serving it.
</p>
<h3>Rewriting links</h3>
<p>
There is one final thing to take care of. In production, we can
configure our web server to serve assets with a far future expires
header. But in order for that to be safe, we need distinct URLs for
every change to the file. Let's add another require to the web namespace
form:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [optimus.assets :as assets]
[optimus.link :as link] ; New
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))</code></pre>
<p>
With this in place, we can use Optimus to generate the link to the CSS
file. However, to do that, it needs access to the request map, so we
need to change a few things. We will start with the
<code>layout-page</code> function:
</p>
<pre class="highlight lisp"><code>(defn layout-page [request page]
(html5
[:head
[:meta {:charset "utf-8"}]
[:meta {:name "viewport"
:content "width=device-width, initial-scale=1.0"}]
[:title "Tech blog"]
[:link {:rel "stylesheet" :href (link/file-path request "/styles/main.css")}]]
[:body
[:div.logo "cjohansen.no"]
[:div.body page]]))</code></pre>
<p>
Both the <code>partial-pages</code> and <code>markdown-pages</code> need
to pass the request to <code>layout-page</code>. If we change them to
return functions, Stasis will call those functions with the request.
</p>
<pre class="highlight lisp"><code>(defn partial-pages [pages]
(zipmap (keys pages)
(map #(fn [req] (layout-page req %)) (vals pages))))</code></pre>
<p>
Remember that <code>#( )</code> is a function literal, so the mapping
function here is a function that returns another function (which takes a
request map as its only argument). The markdown generation is similar,
but includes the additional step of running the content through cegdown:
</p>
<pre class="highlight lisp"><code>(defn markdown-pages [pages]
(zipmap (map #(str/replace % #"\.md$" "") (keys pages))
(map #(fn [req] (layout-page req (md/to-html % pegdown-options)))
(vals pages))))</code></pre>
<p>
Previously these maps contained strings, so we need to update their use
now that they're functions. We start with a new function:
</p>
<pre class="highlight lisp"><code>(defn prepare-page [page req]
(-> (if (string? page) page (page req))
highlight-code-blocks))</code></pre>
<p>
This function takes a page and a request. Because every page will go
through this function, some will be strings, and some will be functions.
If the page is a string, we leave it untouched, and if it's a function,
we call it with the request map and pipe the result through a series of
post-processing steps. There's currently only one step, but the
threading macro has set us up for easily adding more steps later. The
final piece of the puzzle is to update the <code>prepare-pages</code>
function:
</p>
<pre class="highlight lisp"><code>(defn prepare-pages [pages]
(zipmap (keys pages)
(map #(partial prepare-page %) (vals pages))))</code></pre>
<p>
Again, we use the function literal <code>#( )</code>. We also
use <code>partial</code>. This returns a new function that knows the
first argument to pass to <code>prepare-page</code>. When you call this
new function with one argument (a request),
the <code>prepare-page</code> function will be called with a page and a
request. Update the page in the browser, view source and note that
Optimus has now given our CSS file a nice and unique URL.
</p>
<h2 id="exports">Export to disk</h2>
<p>
So far we've only surfed the server version, but the whole point of this
exercise was to create something that can work as a static site. To dump
the file to disk, start by adding a custom Leiningen build alias in
project.clj:
</p>
<pre class="highlight lisp"><code>(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]]
:ring {:handler cjohansen-no.web/app}
:aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]} ; New
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}})
</code></pre>
<p>
This configures `lein build-site` as a command that will invoke
the <code>export</code> function in the <code>cjohansen-no.web</code>
namespace. Stasis gives us what we need to build this function:
</p>
<pre class="highlight lisp"><code>(def export-dir "dist")
(defn export []
(stasis/empty-directory! export-dir)
(stasis/export-pages (get-pages) export-dir))</code></pre>
<p>
While this won't technically fail, it also won't be the whole picture.
Had we not been using Optimus, this would be OK. Since we are using
Optimus, we want to make sure the export is optimized as well. The fix
is simple; tell Optimus to dump assets for us, and add an entry to
Stasis' request map extensions so that Optimus finds the assets. First
update the namespace form to require the Optimus <code>export</code>
library:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web
(:require [optimus.assets :as assets]
[optimus.export] ; New
[optimus.link :as link]
[optimus.optimizations :as optimizations]
[optimus.prime :as optimus]
[optimus.strategies :refer [serve-live-assets]]
[cjohansen-no.highlight :refer [highlight-code-blocks]]
[clojure.java.io :as io]
[clojure.string :as str]
[hiccup.page :refer [html5]]
[me.raynes.cegdown :as md]
[stasis.core :as stasis]))</code></pre>
<p>
Then update the export function:
</p>
<pre class="highlight lisp"><code>(defn export []
(let [assets (optimizations/all (get-assets) {})]
(stasis/empty-directory! export-dir)
(optimus.export/save-assets assets export-dir)
(stasis/export-pages (get-pages) export-dir {:optimus-assets assets})))</code></pre>
<p>
Now, on the command line, run <code>lein build-site</code>. After a
short while you will find your entire site ready to ship in
the <code>dist</code> directory. This can be directly rsynced to your
server.
</p>
<h2 id="testing">Testing and verification</h2>
<p>
Building sites like we've done in this post opens for various
interesting ways of programatically performing tests and health checks.
I will show you two simple, yet immensely useful tests we can add to a
site of this kind. You can of course also add unit tests for individual
functions, and doing so in a system composed of mostly pure functions is
very straight-forward, yet outside the scope of this post.
</p>
<h3>Testing for 200 OK</h3>
<p>
One nice test to put in a site like this is an integration test that
checks that every page renders without errors. We will
use <a href="https://github.com/marick/Midje">Midje</a> for our tests,
so let's update project.clj:
</p>
<pre class="highlight lisp"><code>(defproject cjohansen-no "0.1.0-SNAPSHOT"
:description "cjohansen.no source code"
:url "http://cjohansen.no"
:license {:name "BSD 2 Clause"
:url "http://opensource.org/licenses/BSD-2-Clause"}
:dependencies [[org.clojure/clojure "1.5.1"]
[stasis "1.0.0"]
[ring "1.2.1"]
[hiccup "1.0.5"]
[me.raynes/cegdown "0.1.1"]
[enlive "1.1.5"]
[clygments "0.1.1"]
[optimus "0.14.2"]]
:ring {:handler cjohansen-no.web/app}
:aliases {"build-site" ["run" "-m" "cjohansen-no.web/export"]}
:profiles {:dev {:plugins [[lein-ring "0.8.10"]]}
:test {:dependencies [[midje "1.6.0"]] ; New
:plugins [[lein-midje "3.1.3"]]}}) ; New
</code></pre>
<p>
We've added a test profile that includes the midje dependencies. Add the
following to test/cjohansen_no/web_test.clj:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web-test
(:require [cjohansen-no.web :refer :all]
[midje.sweet :refer :all]))
(fact
"All pages respond with 200 OK"
(doseq [url (keys (get-pages))]
(let [status (:status (app {:uri url}))]
[url status] => [url 200])))
</code></pre>
<p>
We simply call our <code>get-pages</code> function, loop the resulting
map, and call each page function with a request map consisting only of a
URL. The comparison is made with a vector of the URL and the status. The
reason for this is that the URL will be included in the error message if
this fails. This way we can know which pages fail. To run the tests:
</p>
<pre><code>lein with-profile test midje</code></pre>
<p>
Doing this will inform us that the generated core_test.clj fails. Just
delete it. Other than that, the test confirms that all is well with our
site. To keep the tests running while working on the site, run autotest:
</p>
<pre><code>lein with-profile test midje :autotest</code></pre>
<h3>Building a link checker with enlive</h3>
<p>
Another useful test to have in place is a link-checker. We will make one
that at least verifies that the internal links between pages in our app
are correct, and that they don't cause any unnecessary redirects (e.g.
from /about to /about/).
</p>
<p>
Enlive is very useful for these things. We will use the
<code>select</code> function to find all links, and then make sure that
the <code>href</code> attribute points to an existing URL if it is a
path (not a full URL, which is treated as an external link). First up is
the <code>link-valid?</code> function, which checks if a single link is
valid given a map of pages:
</p>
<pre class="highlight lisp"><code>(defn link-valid? [pages link]
(let [href (get-in link [:attrs :href])]
(or
(not (.startsWith href "/"))
(contains? pages href)
(contains? pages (str href "index.html")))))</code></pre>
<p>
The link is considered valid if the href attribute either points to a
URL that isn't a path within our app (relative paths are assumed not
used) or if it points to one of the pages in the map. We're lenient
enough to allow links to /about/ when we have /about/index.html. Since
we will use enlive to select all the links, update the test namespace
form to this:
</p>
<pre class="highlight lisp"><code>(ns cjohansen-no.web-test
(:require [cjohansen-no.web :refer :all]
[midje.sweet :refer :all]
[net.cgrand.enlive-html :as enlive]))</code></pre>
<p>
Then add the test itself:
</p>
<pre class="highlight lisp"><code>(fact
"All links are valid"
(let [pages (get-pages)]
(doseq [url (keys (get-pages))
link (-> (:body (app {:uri url}))
java.io.StringReader.
enlive/html-resource
(enlive/select [:a]))]
(let [href (get-in link [:attrs :href])]
[url href (link-valid? pages link)] => [url href true]))))</code></pre>
<p>
Again, we loop all the pages and get them. For each page, we select all
links, and expect all of them to pass the link checker. Again we make a
slightly strange comparison in the interest of having more than
true/false in the output if one of these fail. If you add an invalid
link to the markdown file now, running the tests will produce this:
</p>
<pre><code>FAIL "All links are valid" at (web_test.clj:21)
Expected: ["/my-first-post" "/about/" true]
Actual: ["/my-first-post" "/about/" false]</code></pre>
<p>
18 lines of code to verify all links on the site. Pretty nifty! Rather
than having numerous tests that load all the pages, it would probably be
a good way to change the structure of the tests such that we only load
each page once, and instead register various test functions we want to
run for each page. This is left as an exercise for the reader.
</p>
<h2 id="summary">Summary</h2>
<p>
I hope this post has shown you the power and flexibility of Stasis and
all the other tools. Perhaps it has even convinced you further of the
value of Clojure. I really do dislike number-of-lines-of-code jerkoffs,
but it is worth mentioning that we were able to build a reasonably
feature-complete technical blog in roughly 100 lines of code using a
simple, yet powerful "no batteries included" library like Stasis (which
itself clocks in at just over 100 lines of code). I hope you will
consider Clojure and Stasis for your next semi-static web project.
</p>
<p>
Big thanks to Magnar Sveen for proof-reading and correcting this post.
</p>
<div class="meta">
<p class="twitter"><a href="http://twitter.com/cjno">Follow me (@cjno) on Twitter</a></p>
<div class="contribute">
<h2>Discuss</h2>
<ul>
<li class="hackernews"><a href="https://news.ycombinator.com/item?id=7375425">Hacker News discussion</a></li>
<li class="reddit"><a href="http://www.reddit.com/r/Clojure/comments/202qs2/building_static_sites_in_clojure_with_stasis/">Reddit discussion</a></li>
</ul>
</div>
<div id="tweets" class="comments"></div>
</div>
</div>
<div class="banner footer">
<p>
<span>
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/" title="Creative Commons License">
<img alt="Creative Commons License" src="/images/cc-by-nc-sa.png">
</a>
2006 - 2014 <a href="mailto:christian@cjohansen.no">Christian Johansen</a>
</span>
</p>
</div>
<script src="/javascript/dist/cjohansen.js"></script>
<script type="text/javascript">var _gaq=_gaq||[];_gaq.push(["_setAccount","UA-20457026-1"]);_gaq.push(["_trackPageview"]);(function(b){var c=b.createElement("script");c.type="text/javascript";c.async=true;c.src="http://www.google-analytics.com/ga.js";var a=b.getElementsByTagName("script")[0];a.parentNode.insertBefore(c,a)})(document);</script>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment