Create a gist now

Instantly share code, notes, and snippets.

Lightning Fast WordPress: Caddy+Varnish+PHP-FPM

README

This gist assumes you are migrating an existing site for www.example.com — ideally WordPress — to a new server — ideally Ubuntu Server 16.04 LTS — and wish to enable HTTP/2 (backwards compatibile with HTTP/1.1) with always-on HTTPS, caching, compression, and more. Although these instructions are geared towards WordPress, they should be trivially extensible to other PHP frameworks, other FastCGI backends, and even non-FastCGI backends (using proxy in lieu of fastcgi in the terminal Caddyfile stanza).

Quickstart: Use your own naked and canonical domain names instead of example.com and www.example.com and customize the Caddyfile and VCL provided in this gist to your preferences!

These instructions target Varnish Cache 4.1, PHP-FPM 7.0, and Caddy 0.9.5. (I'm using MariaDB 10.1 as well, but that's not relevant to this guide.)

Architecture

What's the point of all this?

Why are they so many layers?

Why Caddy?

The primary goal of this architecture is to massively and immediately improve any given WordPress site with a quick and easy upgrade to HTTP/2, always-on HTTPS, and robust caching and compression middleware. To that end, we can envision a stack that looks something like:

                                                    +-----------+
          Request +->                               |  Dynamic  |
                                                 +-->  Content  |
+----------+     +----------+     +-----------+  |  |           |
|          +----->          +-----> Cache &   <--+  +-----------+
|  Client  |     | SSL/Edge |     | Compress  |
|          <-----+          <-----+           <--+  +-----------+
+----------+     +----------+     +-----------+  |  |  Static   |
                                                 +--+  Assets   |
         <-+ Response                               |           |
                                                    +-----------+

Varnish enables caching and compression of static assets as well as dynamic content and is the best in its class, so let's plug that in. This leaves us wanting for implementations of the following components:

  • SSL termination and certificate management
  • Request sanitization and XSS attack mitigation*
  • Request and error logging
  • IP-based whitelisting and rate-limiting
  • URL rewriting (in request headers)
  • URL canonicalization (in response bodies)
  • Static asset serving
  • Hand-off to Varnish via reverse proxy
  • Hand-off to PHP-FPM via FastCGI

Caddy does (*almost) all of these things right out of the box. It makes the "happy path" very happy. And although HAProxy might be better at the edge, and NGINX might have more features, Caddy is a nice fit for this use-case. Performance is still an open question, but so far it's held up under load quite capably for me.

Our architecture starts to take shape as we fill in the boxes:

                                                    +-----------+
          Request +->                               | FastCGI   |
                                                 +--> (PHP-FPM) |
+----------+     +----------+     +-----------+  |  |           |
|          +----->          +----->           <--+  +-----------+
|  Client  |     |  Caddy   |     |  Varnish  |
|          <-----+          <-----+           <--+  +-----------+
+----------+     +----------+     +-----------+  |  |  Static   |
                                                 +--+  Assets   |
        <-+ Response                                |           |
                                                    +-----------+

Ah, but there's a problem: Varnish can't talk to FastCGI or serve static assets from disk. We need to add at least one more component:

                                                             +-----------+
        Request +->                                          | FastCGI   |
                                                          +--> (PHP-FPM) |
+----------+   +---------+   +-----------+   +---------+  |  |           |
|          +--->         +--->           +--->         <--+  +-----------+
|  Client  |   |  Caddy  |   |  Varnish  |   |  Caddy  |
|          <---+         <---+           <---+         <--+  +-----------+
+----------+   +---------+   +-----------+   +---------+  |  |  Static   |
                                                          +--+  Assets   |
       <-+ Response                                          |           |
                                                             +-----------+

             |--------Distribution---------|--------Origination----------|                                                              

This should (hopefully) illustrate why we need so many different layers. It may help to think of the left side of the stack as content distrbution, and the right side of the stack as content origination. Keeping these two halves separate makes it easy to scale when you're ready by letting you swap in a content distribution network (CDN)—which is essentially caching and compression as a service with anycast and SSL termination—while keeping the same origin:

                Hypothetical Future State
                =========================

                                              +-----------+
        Request +->                           | FastCGI   |
                                           +--> (PHP-FPM) |
+----------+   +----------+   +---------+  |  |           |
|          +--->          +--->         <--+  +-----------+
|  Client  |   |   CDN    |   |  Caddy  |
|          <---+          <---+         <--+  +-----------+
+----------+   +----------+   +---------+  |  |  Static   |
                                           +--+  Assets   |
       <-+ Response                           |           |
                                              +-----------+

             |-Distribution-|--------Origination----------|                                                              

Alternatively, if you don't go the CDN route, you have the ability to add more Caddy, Varnish, and/or PHP-FPM instances distributed and load-balanced across multiple servers.

Caching of Static Assets

It is no longer en vogue to cache static assets in memory, the logic being that sendfile(2) and the filesystem cache can deliver files from disk faster than a userland process can copy bytes from memory. Unfortunately, you still have to deal with gzipping uncompressed content (or gunzipping compressed content) depending on the Accept-Encoding of the client, so sendfile(2) can't be used in all cases; only when dealing with "naturally" compressed static assets like images and videos. In a more perfect world, you might want an architecture that looks something like this:

                                                      +------------+
          Request +->                                 |  Dynamic   |
                                                   +-->  Content   |
+----------+     +------------+     +-----------+  |  |            |
|          +----->            +-----> Cache &   <--+  +------------+
|  Client  |     | SSL/Edge   |     | Compress  |
|          <-----+            <-----+           <--+  +------------+
+----------+     +-----^------+     +-----------+  |  |    Un-     |
                       |                           |  | Compressed |
        <-+ Response   |                           +--+   Static   |
                 +-----+------+                       |   Assets   |
                 | Compressed |                       +------------+
                 |   Static   |
                 |   Assets   |
                 +------------+

Unfortunately, Caddy doesn't give us a great way to say "serve already-compressed files from disk (if present) and send all other requests upstream," so we're more-or-less stuck with the architecture described above. Maybe a future version of Caddy's proxy directive will support an ext subdirective and this will become an easier problem to solve: we'd be able to send only requests for e.g. .js, .css, .txt, .html, and .php upstream.

Meanwhile, more critically, static assets are often large enough to push dynamic content out of cache, forcing expensive re-renders and largely defeating the purpose of this architecture. If you have a lot (i.e. more than 100 MB) of static assets, it may be worth partitioning your Varnish backend storage into multiple "stevedores" to prevent this from happening. To do so you'll need to modify your Varnish service unit file (e.g. by creating /etc/systemd/system/varnish.service.d/override.conf) to give your malloc backend storage a name (e.g. "dynamic") and declare a new, named file storage backend (e.g. "static"). Here's what mine looks like:

[Service]
Type=Simple
ExecStart=
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F \
  -a localhost:6081 -T localhost:6082 \
  -f /etc/varnish/default.vcl -S /etc/varnish/secret \
  -s dynamic=malloc,256m -s static=file,/var/lib/varnish,2048m

Don't forget to uncomment the relevant lines in the VCL provided in this gist. Register the updated service unit with systemctl daemon-reload and restart Varnish with systemctl restart varnish. Now your static assets will be cached on disk, which should be a performance wash, and precious in-memory cache space will be reserved for dynamic content.

Alternatively, you can simply tell Varnish not to cache such files and pass or pipe them through to Caddy. In my benchmarking, I've found the solution documented above to be significantly (up to 2X) faster than this strategy.

HOWTO

Follow these steps to configure your server.

Firewall

Set up your firewall, minimally:

# ufw disable
# ufw reset
# ufw limit OpenSSH
# ufw allow to any port 80,443 proto tcp
# ufw enable

Filesystem

Make a directory for your WordPress installation (e.g. /var/www/example.com/wordpress) and log files (e.g. /var/www/example.com/logs). Your Caddyfile can go in this directory structure (e.g. /var/www/example.com/Caddyfile) or you can build a Caddyfile "repo" somewhere in /etc (e.g. /etc/caddy) and go nuts with the import directive if you have multiple sites.

Follow the normal process to migrate your WordPress site to its new home directory and restore the database. These steps are well-documented elsewhere and so are not covered here. (Don't install Apache, NGINX, PHP, or any other services; we'll cover that below!)

Packages

Minimally, install php7.0-fpm and varnish from apt. Copy default.vcl from this gist into /etc/varnish. It should work as-is in this basic configuration, and is essentially a blank slate for future customization using the expressive Varnish Configuration Language.

Be sure to install the appropriate PHP extension(s) for your database; for example, php7.0-mysql for MySQL/MariaDB. I found I needed to install php7.0-curl and php7.0-mbstring as well. YMMV. Don't forget to reload the php7.0-fpm service whenever you add/remove extensions.

Start and enable the php7.0-fpm and varnish services. Optionally stop and disable the varnishlog and varnishncsa services.

Caddy

Download and install Caddy with the ratelimit, filter, and ipfilter addons.

You'll need to write a service unit file and install it to /etc/systemd/system; one generated by antoiner77.caddy v2.0.1 is included in this gist for reference. (If you're already familiar with Ansible, this galaxy role is great for downloading and installing Caddy on your host.)

You'll need to decide how Caddy will acquire its SSL certificate. If you don't already have an SSL certificate for the site and can afford a few minutes of downtime, the easiest method by far is to simply point the domain name at the new server, wait for the TTL to expire, and let Caddy's Automatic HTTPS feature apply the http-01 and/or tls-sni-01 challenge(s) at startup. Be warned that you might get temporarily blocked by Let's Encrypt if something isn't configured correctly and it looks like you're spamming their systems.

Copy the Caddyfile from this gist to wherever you've decided to keep it and customize it for your site:

  • Set the canonical domain name for each stanza.
  • Set the email address for your Let's Encrypt account (Caddy will create the account for you).
  • Decide whether you want to support older TLS 1.0 clients or not.
  • Update directory paths to reflect your WordPress installation and log directories.
  • Choose log rotation strategies for access.log and error.log.
  • Update the ipfilter block with your allowed IP address range(s).
  • If new user signup is open to the public, remove /wp-login.php from the ipfilter directive.
  • If you need XML-RPC, remove or comment out the statements dealing with X-Pingback and xmlrpc.php.
  • Adjust the filter rule to correctly rewrite your non-canonical URLs to the canonical URL. For example, rewrite http://example.org to https://www.example.com.
  • Adjust the caching policy for (truly) static assets.

Start and enable the caddy service when all's said and done.

WordPress

Most WordPress optimization guides suggest unsetting cache control headers such as Cache-Control, ETag, Last-Modified, Expires, etc. Instead, we're going to set them and let Varnish serve posts, pages, and attachments as slowly-changing dynamic content. Yes, you read that correctly: your WordPress site is now a glorified static site generator!

  • Change your WordPress and Site Addresses to https:// in wp-config.php, or by using the wp-cli tool.
  • Apply the wp-config.php snippets provided in this gist.
  • Install and activate the Add Headers and Cache Control plugins.
  • After the site is up-and-running, you can (and should!) browse it with the error console open to see which remote assets (e.g. analytics tracking scripts) are still being loaded via http://, and take the appropriate action(s) to update those links to https://. If you have a larger site, there are more efficient methods outside the scope of this guide.
  • Force your adminstrator users to change their passwords over HTTPS!

If you have a lot of dynamic fragments and/or pages on your site, for example a shopping cart or PHP-based discussion forum, caching may not work correctly out of the box. Fortunately, there are ways to vary the cache based on the logged-in user's session cookie or other factors. (TODOC.) If you'd rather not deal with this part of the puzzle, simply deactivate the Add Headers and Cache Control plugins.

Notes

  • The ports and Varnish instance can be shared between Caddyfiles. Caddy will select vhosts using the Host header, and Varnish will vary its cache by the Host header. (You may not want to re-use the Varnish instance if you have some gnarly, site-specific VCL, but that's another question entirely.)
  • Optionally update/override the ExecStart statement in varnish.service to bind to localhost, i.e. -a localhost:6081.
  • Don't forget to hit up Google and Bing Webmaster Tools to change the preferred URL to https://, upload a new site map, etc.
  • I've also had good luck with the HTTP Headers plugin, to add Strict-Transport-Security, X-Frame-Options, etc.

TODO

Open Questions

  • Is there a way to safely apply a global ratelimit?
  • Will Varnish 5.0 allow us to use HTTP/2 all the way down to FastCGI?
  • What more can we do to secure a typical WordPress installation, particularly .php files in wp-includes and wp-content?

FAQ

Why is this a weird, rambly gist?
It started out as a few code snippets and grew into a huge thing. I think it should probably be a blog post, a repo, an Ansible role, etc. I'll develop it further if there's interest.
[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
[Service]
Restart=on-failure
; User and group the process will run as.
User=www-data
Group=www-data
; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy
; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
ExecReload=/bin/kill -USR1 $MAINPID
; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=64
; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
; This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy
; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
www.example.com {
tls hostmaster@example.com {
# Android 4 and IE 8-10 only support TLS 1.0.
# Change tls1.0 to tls1.1 (or remove this block entirely) for better privacy.
protocols tls1.0 tls1.2
}
log / /var/www/example.com/logs/access.log "{combined}" {
rotate {
size 100 # rotate after 100 MB
age 14 # keep log files for 14 days
keep 10 # keep at most 10 log files
}
}
ipfilter /wp-admin /wp-login.php {
rule allow
ip 1:2:3:4::/56
strict
}
ipfilter /wp-admin/admin-ajax.php {
rule block # disallow nobody === allow everybody
}
ratelimit /wp-login.php 5 7 minute
# Protect secrets against misconfiguration.
internal /wp-config.php
# Disable XML-RPC (assuming you're not using it).
internal /xmlrpc.php
proxy / localhost:6081 {
transparent
}
}
www.example.com:2020 {
bind localhost
tls off
errors {
log /var/www/example.com/logs/error.log {
size 100 # rotate after 100 MB
age 14 # keep log files for 14 days
keep 10 # keep at most 10 log files
}
}
root /var/www/example.com/wordpress
rewrite {
if {path} not_match ^/wp-admin
to {path} {path}/ /index.php?_url={uri}
}
# Be sure to set fastcgi.logging=0 in php.ini to work around
# https://github.com/echocat/caddy-filter/issues/4
fastcgi / /run/php/php7.0-fpm.sock php
filter rule {
content_type (?:text|javascript)
search_pattern https?://(?:www\.)?example\.(?:com|net|org)
replacement https://www.example.com
}
# Set caching directives for static content. Caddy will automatically add
# Last-Modified and ETag headers for files on disk. Neat!
header /wp-content Cache-Control "max-age=2592000, s-maxage=86400"
header /wp-includes Cache-Control "max-age=2592000, s-maxage=86400"
# XML-RPC is disabled, so delete the header that points to it.
header / -X-Pingback
# Let Varnish re-count the bytes because `filter' does the wrong thing. :c
header / -Content-Length
}

2017/03/17

  • Update Varnish service unit files to play nice with Varnish 5.x.

2017/02/02

  • Add more gzip-able content-types.
  • Add note about alternative to Varnish partitioning.

2017/01/30

  • Move internal/rewrite directives from terminal to initial stanza.
  • Add section discussing the pros and cons of caching static assets.

2017/01/25

  • Reflect Caddy 0.9.5 and updated plugins
  • Merge :2020 and :2021 proxy stanzas (huzzah!)

2017/01/10

  • Add note about options for adding the filter to wp-config.php.
  • Improve the rewrite rule slightly.
  • Oops, s-max-age should be s-maxage!

2017/01/09

  • Add note/link for HTTP Headers plugin.
  • Varnish and Caddy port sharing tested and confirmed working!
  • Raise the ratelimit on /wp-login.php based on real-world experience.

2017/01/07

  • Add /wp-admin to ipfilter and /wp-config.php to internal.
  • Remove /wp-admin from ratelimit.
  • Move ratelimit to initial stanza and change parameters.
  • Remove now-unnecessary realip directive.

2017/01/06

  • Initial publication.
vcl 4.0;
backend default {
.host = "localhost";
.port = "2020";
}
sub vcl_backend_response {
if (beresp.http.content-type ~ "text|javascript|json|svg+xml|icon|font" && beresp.http.content-type !~ "woff") {
set beresp.do_gzip = true;
}
/*
* Uncomment the following lines only if you've partitioned your Varnish
* backend storage into multiple stevedores as described above.
*
* This takes advantage of the fact that PHP sets an X-Powered-By header
* on its responses. If you've set `expose_php = Off' in your php.ini,
* you'll need to find some other criteria to differentiate dynamic from
* static requests. I recommend leaving it on and removing the header in
* this conditional (the commented line) if it worries you.
*
* If this is working properly, varnishstat will show separate SMA.dynamic
* and SMF.static key groups that change over time as requests are served.
*
if (beresp.http.x-powered-by) {
set beresp.storage_hint = "dynamic";
//unset beresp.http.x-powered-by;
}
else {
set beresp.storage_hint = "static";
}
*/
}
// Insert *before* requiring wp-settings.php
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && 'https' === $_SERVER['HTTP_X_FORWARDED_PROTO']) {
$_SERVER['HTTPS'] = 'on';
}
// Insert *after* requiring wp-settings.php
// Alternatively, you can add this to your child theme's functions.php, or into its own (mu) plugin.
add_filter('addh_options', 'my_addh_options');
function my_addh_options($options) {
// We only want the last-modified header, so disable these other ones.
$options['add_etag_header'] = false;
$options['add_expires_header'] = false;
$options['add_cache_control_header'] = false;
return $options;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment