Skip to content

Instantly share code, notes, and snippets.


Ross Reedstrom reedstrm

  • Connexions (Rice University)
  • Houston, TX
View GitHub Profile
reedstrm /
Last active Jun 15, 2017
Notes on what gets baked when, with what recipe

Baking and Recipes

Baking has become a critical step for publishing - however, it is not planned for it to cause publishing to fail. This is partly due to how long baking may take - much like PDF generation, it can take several hours, if there are external parts, like exercises, that need to be fetched and incorporated into the pages.

This slightly odd situtation (is baking a publish blocker, or not?) comes about partly due to derived copies - publishing a new version of content impacts not only the immediate book, but any book that shares pages with it. It was decided that failure of related books to bake is not a publication blocker, so by extension, it is not for any book. However, we want to be careful to not break existing books by publishing new content, but we cannot test every book that contains a given page.

Selection of Recipe and Fallback

So, we will implement two ways to mitigate this potential problem. First, will we keep track of when and with what recipe a given book successfully ba

View preserving composite page

When publishing a new minor version of an existing book (i.e. the structure of the book does not change, but the content of one or more pages does), the new book will need to be cooked. This will generate new versions of all the composite pages, that is, pages that are defined by the cooking recipe, not by the raw content, such as end-of-chapter exercises, end-of-book index or glossary, etc. Ideally, these new pages should retain their uuid, and only get a new version number. This is tricky, since the pages do not exist in the raw content that is sent to be cooked, so the rulesets cannot set thier ids. However, the rulesets do set a locally unique class on each of these pages. This, combined with which chapter a given composite page is in, is enough to identify that particular page across minor versions. So a combination of chapter-identifier and composite-page class can be used as a key to lookup uuids from the soon-to-be-one-version-old book tree, and assign them to the parallel page of the newly-cooked

reedstrm /
Last active Nov 16, 2015
Using session_exec to initialize plpythonu virtualenv inside postgresql

It is possible using the session_exec extension to postgresql to run a single function at session creation time

CREATE FUNCTION workon(venv text) RETURNS void
    LANGUAGE plpythonu
    AS $_$
    import os
    import sys
View deep_nested_books.txt
WITH RECURSIVE t(node, title, path,value, depth, corder) AS (
SELECT nodeid, title, ARRAY[nodeid], documentid, 1, ARRAY[childorder]
FROM trees tr, latest_modules m
WHERE tr.documentid = m.module_ident AND
tr.parent_id IS NULL
SELECT c1.nodeid, c1.title, t.path || ARRAY[c1.nodeid], c1.documentid, t.depth+1, t.corder || ARRAY[c1.childorder] /* Recursion */
FROM trees c1 JOIN t ON (c1.parent_id = t.node)
WHERE not nodeid = any (t.path)
reedstrm /
Last active Dec 14, 2018
URL object hashes (shortening URL proposal)


We the developers of OpenStax/Connexions, in order to form a more readable URL, establish user harmony, ensure URL persistance, and support all the use cases, do suggest the following id hashing strategy for content objectIds.


The canonical form for an id is a uuid-v4, in the 8n-4n-4n-4n-12n format, e.g. 6f0881fc-4d30-43e1-9a3b-52a210ab5980 Python code for generating these:

  >>> import uuid
  >>> u=uuid.uuid4()
reedstrm / xml cleanup
Last active Aug 29, 2015
Cleaning up our namespace mess
View xml cleanup
mjs="""{"publishers": [{"website": null, "hasAccepted": true, "surname": "Reedstrom", "firstname": "Ross", "title": "Dr", "emails": ["", ""], "id": "reedstrm", "requester": "reedstrm", "assignmentDate": "2015-02-10T14:57:51.543583-06:00", "fullname": "Ross Reedstrom", "type": "cnx-id", "email": ""}], "version": "draft", "derivedFrom": null, "abstract": "<div xmlns=\"\" xmlns:bib=\"\" xmlns:data=\"\" xmlns:epub=\"\" xmlns:rdf=\"\" xmlns:dc=\"\" xmlns:lrmi=\"\" class=\"description\" itemprop=\"description\" data-type=\"description\">\n <div xmlns=\"\" xmlns:bib=\"\" xmlns:data=\"\" xmlns:epub=\"\" xmlns:rdf=\"
reedstrm / Generating SQL update
Last active Nov 15, 2018
Generating SQL update scripts
View Generating SQL update

A reasonable tool to generate SQL diffs for our databases is Pyrseas. Basically this tool works with existing databases, to generate a YAML representation that is more easily diffed, and can be used to auto-generate an SQL update file.

Pyrseas is a python module, and can be installed into your dev virtualenv w/ pip:

pip install pyrseas

This gives you two commands: dbtoyaml and yamltodb

To generate an sql update, you'll need a YAML representation of the new state, and the database to be updated. Let's assume you've

reedstrm / gist:e95db628d794a93d4f8f
Last active Aug 29, 2015
Code Versioning for OpenStax CNX
View gist:e95db628d794a93d4f8f

Recommendation for code versioning for OpenStax (née Connexions) code.

  1. use git tags, of the form: v[numeric-version]
  2. we tag complete (or at least, stable) features, for roll out to demo, qa, sprint and production a. exception: dev will roll from HEAD of master b. tags will be on the master branch, post merge (if other patchfix tags are needed, devops will tag on staging/production) c. Tags are done by developers, primarily, and include "in development" extra version info (see below)
    1. git describe at minimum
    2. bump versions as needed
    3. devops will finalize tags (and versions) at rollout
reedstrm / gist:9395601
Last active Aug 29, 2015
The new workspace: what the heck is it?
View gist:9395601

#Possible models of the new workspace

  • Everything I could possibly edit (everything I have rights to edit - problematic for prolific authors, org accounts, and managers)
  • Everything I could possibly edit that currently has a draft in flight
  • Everything I have expressed an interest in editing (permissions + another bit)

Does new content have a diff. set of conditions?

  • I don't thing so: adding an new editor (acl change) is the same for either, and is not a role request
  • N.B. We do not display to the public in any place who is allowed to edit a given piece of content.