Code Club Dev: Automating Project Builds

gistfile1.md

Code Club Dev: Automating Project Builds

Disclaimer: I don’t work for Code Club; I’m a volunteer. Also: I don’t blog.

Rationale

I’ve been doing a fair bit of work on the Code Club lesson formatter lately. That’s the tool used to turn source markdown, resources and json into a static website of Code Club projects (UK; World).

left: the source markdown; right: the output

Currently, the lesson formatter partially automates the process of generating the projects site, but there’s still quite a lot of human intervention required.

Now… Volunteers and staff are making updates and tweaks to these projects pretty regularly. 24 updates were made to the scratch curriculum in the last month, from 6 contributors:

Pointless code sample

So to stay up-to-date with the source, projects sites need to be built and deployed almost daily. If the source and sites get out of sync, it’s pretty confusing for volunteers looking to contribute fixes (since they try to fix a source that’s actually already correct, but not deployed.)

In practice:

• the bits that the lesson formatter generates reliably (e.g. HTML for projects; HTML club leaders’ notes) get deployed regularly (though even this takes time and is a bit faffy).
• Other stuff that needs to be either generated manually (PDFs for projects and PDF club leaders’ notes) get updated less often. For most non-UK projects, these simply aren’t generated.
• As a volunteer, I don’t want to spend time manually creating PDFs – I wouldn’t even know how. So it’s generally staff that spend time doing this stuff.

The mission

• Auto-generate all UK and World PDFs
• Nightly (or post-receive) build & deploy when stuff changes
• Auto-test the output, to ensure crazy stuff isn’t happening

Auto-generating PDFs

The groundwork on this was done by @tef, @gahjelle and @jonic. Thanks to them, the initial code was already in place to get phantom.js and wkhtmltopdf (two webkit-based tools that can convert HTML to PDF) to generate PDFs from lesson HTML. The output (on the right) still needed styling, though:

left: the manually generated PDF; right: output of the first iteration PDF generator

I won’t go into loads of detail about this because it’s mega boring. But I did set out some rules:

• don’t require any modifications to the lesson markdown
• don’t modify the HTML version of the lesson at all. Not even extra print styles or unused javascript
• get everything working in both wkhtmltopdf and phantom.js

The first rule is tricky because bits like this:

…need to be rendered from: # Step 1: Felix follows the mouse pointer { .activity }

This light blue text before the colon can’t be done in CSS without extra markup. Doing this with pandoc (the markdown to HTML converter) breaks the second rule. Spoiler: javascript!

That’s enough about that! There’s an example auto-generated PDF here or you can view a generated projects site here. There are still some issues here. If you spot others, please add them!

Testing

The generated projects sites are tested using LinkChecker to ensure nothing 404s – a useful first step. This is running on Travis CI (thanks to @immunda for his help on this!) Here’s an actual badge for the current CodeClubUK-Projects:

Nightly builds

I’ve created an autobuild wrapper that runs various git commands and then submits a pull request. I got this running on heroku (and learned all about custom buildpacks in the process.)

Of course, these pull requests need to come from someone! Meet @codeclubrobot:

codeclubrobot’s github profile

Here’s one of his friendly pull requests.

• Note that Travis is set to run on pull requests, so we’re alerted if codeclubrobot is malfunctioning!
• Since codeclubrobot has his own fork that he commits to, a live preview of his changes is served by github pages and can be viewed. He helpfully sends a link to this in his pull request.

The autobuild script is deliberately dumb – it pulls the latest version of the project repository, then deletes everything and does a full rebuild. This makes committing the right stuff much easier, and means it cleans up as it goes. It’s slightly smarter with zip files, where it attempts to update the existing zip.

Next steps

The crucial next step is to get this building Code Club World -style PDFs (here’s the relevant ticket). This is just a question of writing some more Sass. Any volunteers? Tweet me!

I was wondering about the last two rules. Are they really necessary?
I think that print css could be very useful... If there were print css in HTML version of projects, people could just print it from the web, no need to download PDF. But even if they need PDF its creation by phantomjs or wkhtmltopdf actually uses print styles.
And the last rule - phantomjs and wkhtmltopdf. Could you explain why there is a need to have two tools which do the same thing?
I'm just wondering and trying to make it easier. :)

If there were print css in HTML version of projects, people could just print it from the web, no need to download PDF

It’s a good point. Originally, it was so that my code could be merged with minimal impact on existing functionality. However, I think you’re right: it might well be better to allow for printing direct from HTML :)

its creation by phantomjs or wkhtmltopdf actually uses print styles.

You’re right – they’re both currently set to use print styles. wkhtmltopdf doesn’t by default.

Could you explain why there is a need to have two tools which do the same thing?

It was just to test which would work better. For instance, my phantom.js code has a dodgy hack to wait until the js has finished, which wkhtmltopdf doesn’t need. Neither allows for variable sized headers. Anyway – I’m now more inclined to use phantomjs. At some point we could strip out wkhtmltopdf, but I think it’s fine for now.