mmulich/2016-apr-release.md

## 2016-apr-release.md

      
    Raw
  

              2016-apr-release.md
            
          
    April 2016 CNX release

Two stage upgrade... First stage introduces a database change that will allow archive and zope to remain up at the same time. The second stage introduces the new features.
~~The first set of migration steps did take ~3hrs on cte-dev.~~ (Fixed in Remove warning info from the migration)
TODO Post a summary version of these changes to blog.openstax.org (O... I guess, I'll need to create blog.openstax.org first. 😉 )
Deployment

Stage One

Notes for cnx-deploy

These notes have been commented out. View and edit the raw document, if you'd like to see them.


Preparation


install db-migrator in the virtual-environment that contains cnx-publishing.


dbmigrator --config=/opt/demo/cnx-publishing/development.ini init --version 0


release Products.RhaptosModuleStorage @ HEAD


Update code


cnx-epub @ HEAD


cnx-easybake @ HEAD


cnx-archive @ e3fa7d038607894c281433f9f3828b7597a3128b


cnx-publishing @ HEAD


Products.RhaptosModuleStorage as egg


Run migrations

This migration runs to a middle of the road solution,
where things will continue working while we transition to the next stage
dbmigrator --config <publishing-config-ini> --context cnx-archive migrate --version 20160101000000
dbmigrator --config <publishing-config-ini> --context cnx-archive migrate --version 20160128110515
Note, we purposefully leave --context cnx-publishing out of the command.
We will wait until stage two to introduce those migrations.
Restart services


restart cnx-archive
restart cnx-publishing
restart zope front-ends

Restarting these will use the changes and transition us
to a point where we can continue the database upgrade
while keeping the services up.
Stage Two

Notes for cnx-deploy

These notes have been commented out. View and edit the raw document, if you'd like to see them.


Update code


cnx-archive @ HEAD

Run migrations

This migration runs away from the middle of the road solution.
This is the final set of migrations.
dbmigrator --config <publishing-config-ini> --context cnx-publishing --context cnx-archive migrate
Restart services


restart cnx-archive

Restarting these will bring everything up-to-date.
Testing

Are things working between the stages?


cnx-archive https://archive.cnx.org/extras => 200 Ok
cnx-publishing https://archive.cnx.org/a/ => 302 Found
zope https://legacy.cnx.org/content => 200 Ok

Hit a few notable book pages on archive and legacy.
No features where added during the stage one deploy;
so nothing out of the ordinary should be happening.
Is everything up?


cnx-archive https://archive.cnx.org/extras => 200 Ok
cnx-publishing https://archive.cnx.org/a/ => 302 Found
zope https://legacy.cnx.org/content => 200 Ok

Can you view pages?

Hit a few notable book pages on archive and legacy.
Do a search or two on both archive and legacy.
Do a little dance!
Are the new features functional?

Since nothing is using a ruleset (CTE styles) yet,
the feature cannot be tested.
What can be tested, is that no existing functionality has broken.
The following should not block the release,
because we won't be doing any collation in production just yet.
In fact, I'm leaving the instructions intentionally vague.
We will come back to them after the release is made.
Next step, is to talk to the textbook team.
Have them supply devops with a ruleset that we can inject into archive.
After the ruleset is in archive, manually invoke the collation.
This will enable you to test the archive routes deliver collated content.
:bug: I suspect webview will not be able to render composite-pages without modifications.
Ignoring


setting up api-keys for cnx-publishing (not used ATM anyhow)

Changelog

cnx-epub


🐛 fix

9a65856 - 🐛 Fix the import for the optional collation feature (#66) (5 minutes ago)


✨ collation integration

156f2c9 - Merge pull request #64 from Connexions/collated-integration (5 hours ago)
e83ada7 - Use Connexion's fork of cssselect2 in .travis.yml (5 hours ago)
73807f5 - Use the document title rather than the TOC title (6 hours ago)
e3a1f0b - Add test that uses a real ruleset (2 days ago)
841175c - Integrate cnx-easybake into collation process (2 days ago)
e24f09f - Include cnx-easybake in travis config (2 days ago)
403098f - 🎉 Add optional dependency for collation (8 days ago)


✨ collation

245fefb - Merge pull request #61 from Connexions/collate (12 days ago)
7f2db4b - Add reconstitution test case (the same as adapt single html test case) (13 days ago)
059ddba - Call adapt_single_html from cnxepub.collation.reconstitute (13 days ago)
430e96d - Add composite document to adapt_single_html (13 days ago)
e7fec0c - Add a generalized flatten_to function (13 days ago)
3681370 - ✨ Add functions for collation (13 days ago)
880dc46 - Factor methods a reusable base testcase (13 days ago)


🐛 fix

6ae578e - Merge pull request #62 from Connexions/html-fixes (13 days ago)
3549b72 - Change single html formatter to output html (no self closing tags) (13 days ago)
1a71ffe - Add a composite document to the SingleHTMLFormatter test case (13 days ago)
bfbc3d2 - Add composite document to SingleHTMLFormatter (13 days ago)
a8f3a2b - Add a CompositeDocument model (13 days ago)


🐛 fix

42005e2 - Merge pull request #63 from Connexions/coverage-fix (13 days ago)
d9ce1cc - Ignore coverage report and development eggs (13 days ago)
dadebdf - Configure coverage tool (13 days ago)


🐛 fix

d6069f1 - Merge pull request #60 from Connexions/package-data-fix (3 weeks ago)
1803389 - 🐛 Fix package data to include styles.css (3 weeks ago)


✨ reconstitution

e772a95 - Merge pull request #59 from Connexions/reconstitution (6 weeks ago)
99afc86 - Add functionality to adapt single html to a Binder (6 weeks ago)
63f2915 - Move formatters to cnxepub/formatters.py (6 weeks ago)
7575137 - Display translucent binder title within the chapter / unit div (6 weeks ago)
ee98a8b - Add tests for SingleHTMLFormatter (6 weeks ago)
3be008a - Remove unused archive_href from SingleHTMLFormatter (6 weeks ago)


🐛 fix

2f93ba8 - Merge pull request #58 from Connexions/html-parsers-changes (6 weeks ago)
352f1b2 - Add DocumentSummaryFormatter to make sure summary is xml (6 weeks ago)
a2add09 - Change html metadata description parser to not return the div (6 weeks ago)
3b74ab6 - Change html metadata parser to use relative xpath (6 weeks ago)


🐛 fix

8752e20 - Merge pull request #57 from Connexions/fix-tests (6 weeks ago)
28774e5 - Fix tests because mimetypes.guess_extension returns ".xht" sometimes (7 weeks ago)


✨ Resource listing in metadata

7fb2d7b - Merge pull request #56 from Connexions/resources-in-html (7 weeks ago)
0eeea6d - List resources in metadata in epub html only if there are resources (7 weeks ago)


✨ metadata

797ce2e - Merge pull request #55 from Connexions/single-html (7 weeks ago)
718e30b - Update SingleHTMLFormatter to output metadata (7 weeks ago)


🐛 fix

a6c8513 - Merge pull request #54 from Connexions/single-html (7 weeks ago)
b7859dd - Add some tests for cnx-epub-single_html script (7 weeks ago)
e9bf26e - Add script for generating a single HTML file from an epub (7 weeks ago)


🐛 fix

d5234f1 - Merge pull request #53 from Connexions/fixes (7 weeks ago)
88dd5d3 - (origin/fixes, fixes) 🐛 Fix a typo in the variable name (7 weeks ago)
f81d05d - Add a logging line for parse failure (7 weeks ago)


🐛 fix

fd26bec - Merge pull request #52 from Connexions/escape-metadata (7 weeks ago)
814825a - Escape metadata when generating HTML for epub (7 weeks ago)


✨ Single-html file

f98cce3 - Merge pull request #51 from Connexions/html-formatter (7 weeks ago)
165d0dd - Add tests for adapters.HTMLFormatter (7 weeks ago)
452d0e7 - Create HTMLFormatter for generating html in epub for models (7 weeks ago)
29edede - Implement Document.html in models as DocumentContentFormatter in adapters instead (7 weeks ago)


✅ Test data updated

6b1c42d - Merge pull request #50 from Connexions/test-data (7 weeks ago)
4c55463 - Change cnx-archive-uri in test data from uuid to ident hashes (8 weeks ago)
76fc4ed - Add cnx-archive-uri to test book to allow parsed binder to have an id (8 weeks ago)


✨ Resources added to Binder model

07a1d64 - Merge pull request #49 from Connexions/binder-resources (8 weeks ago)
b104f7c - Parse and attach binder resources from EPUB (8 weeks ago)
c61d90f - Output list of resources in metadata in EPUB html (8 weeks ago)
9763916 - Add resources to the binder model (8 weeks ago)


PEP 8

49ca725 -  Merge pull request #48 from Connexions/pep8-compliance (6 months ago)
d4571f0 - Fix PEP8 issues in tests (except for line length) (6 months ago)
35b1649 - Run pep8 on travis (6 months ago)
4d0c235 - Fix pep8 issues in code excluding tests (6 months ago)


cnx-archive


✨ view updates for collated content

42a80d2 - Merge pull request #424 from Connexions/collated-content (25 hours ago)
afd3554 - Change composite page to use the collated_file_associations table (5 days ago)
4a560b2 - Add mimetype for composite modules (6 days ago)
cce1e99 - Return collated content for pages when context is given (8 days ago)
aa51bc3 - Add test for getting content for collated collections / books (9 days ago)
eb99618 - Implement getting contents of composite pages / modules (9 days ago)
14a922d - Add a collated page and book to test data (9 days ago)
8156cec - Add optional argument as_collated to database.get_tree (9 days ago)


✨ database migrations to allow for collation

310dfec - Merge pull request #423 from Connexions/collate (9 days ago)
6370063 - Exclude migrations when running pep8 (9 days ago)
9c1dddd - ✨ Add database migration for collation schema (9 days ago)
0904f59 - ✨ Add a collated files association table (9 days ago)
8dc7e2d - ✨ Add is_collated column to trees (9 days ago)
c032e3f - Fix tree select to use explicit columns rather than wildcard (12 days ago)
06ff0ed - Update test data with collation schemata changes (12 days ago)


🐛 fix

7dad406 - Merge pull request #421 from Connexions/fix-epub-modeling (12 days ago)
ef6f5a2 - Save subcollection title in TranslucentBinder metadata (12 days ago)


🐛 fix

0270bf5 - Merge pull request #419 from Connexions/update-travis (3 weeks ago)
390faaf - Remove outdated comment from .travis.yml (4 weeks ago)


✨ remove plpydbapi 🎊

1fb24d2 - Merge pull request #399 from Connexions/remove-plpydbapi (5 weeks ago)
08e6041 - Remove dependency plpydbapi (5 weeks ago)
1992ac0 - Rewrite transforms without plpydbapi (5 weeks ago)
bac7983 - Rewrite module and collection republish triggers without plpydbapi (5 weeks ago)


🐛 fix

a0ede20 - Merge pull request #417 from Connexions/keyword-cleanup (5 weeks ago)
140d509 - Fix to remove XML from keywords (5 weeks ago)


🐛 fix

e4ff3dc - Merge pull request #416 from Connexions/unique-files (5 weeks ago)
93e5eaf - 🐛 Fix transform file producer code to check before insert (5 weeks ago)
dc0de21 - Fix tests to use unique file entries (5 weeks ago)
d8e2ab3 - 🐛 Fix duplicate files issue by setting sha1 column unique (5 weeks ago)
ee19e1d - 🐛 Dedupe files in test/example data (5 weeks ago)
add3d71 - 🐛 Dedupe files table and set sha1 column to unique (5 weeks ago)
2fab870 - Have travis fail fast on pep8 errors (5 weeks ago)


✅ test data update

0b0d63f - Merge pull request #414 from Connexions/update-data (6 weeks ago)
ce78578 - Remove top-level tree entries for modules (6 weeks ago)
3c69b01 - Update example data latest modules to have the same data as modules (6 weeks ago)
03a47ec - Update example data modules to include submitter, submitlog and stateid (6 weeks ago)


🐛 fix

1430009 - Merge pull request #415 from Connexions/remove-epub (7 weeks ago)
56055c3 - Remove the epub route (7 weeks ago)


🐛 fix

a227275 - Merge pull request #413 from Connexions/close-zip (7 weeks ago)
a7b66df - 🐛 Fix to close the epub export zipfile (7 weeks ago)


✨ script to produce an internal epub

90b4f25 - Merge pull request #409 from Connexions/internal-epub (7 weeks ago)
e983dbb - Add export_epub sql files to package manifest (7 weeks ago)
41bc9d1 - ✨ Add route and view to download an internal epub (7 weeks ago)
cc5e41c - Add function to create an internal EPUB from an ident-hash (7 weeks ago)
62d5121 - ✨ Add a commandline interface for exporting an internal EPUB (7 weeks ago)
448d5ae - Add a factory function for calling factories based on type (7 weeks ago)
3a18129 - Add function to get the content type from the database (7 weeks ago)
72973d2 - Add archive to cnxepub model factory for Binder (7 weeks ago)
a13a3f7 - Add a function to get a collection tree from the database (7 weeks ago)
c7f5dd5 - Add archive to cnxepub model factories for Document and Resource (7 weeks ago)
5e75ad7 - Add function to lookup a module's files (7 weeks ago)
54d0984 - Add function to get a file entry (7 weeks ago)
6808f73 - Add function to get file info (7 weeks ago)
fc16016 - Add function to get the content from the database (7 weeks ago)
4b0e302 - Add a function to get metadata in a cnx-epub formatted way (7 weeks ago)
5b2c6af - Add function to get id and version from ident-hash (7 weeks ago)
59aed23 - ✨ Add export_epub script package (7 weeks ago)
1089a4b - Ignore generated html coverage (7 weeks ago)


🐛 fix

e3fa7d0 - Merge pull request #411 from Connexions/separate-migrations (8 weeks ago)
57d5c04 - Move the file trigger update to a separate migration (8 weeks ago)


🐛 fix

33fceaa - Merge pull request #410 from Connexions/migration-fixup (8 weeks ago)
d58fedc - 🐛 Fix SQL syntax error in migrations (8 weeks ago)


🐛 fix

a69c69a - Merge pull request #408 from Connexions/files-triggers (8 weeks ago)
adf9901 - limit affected rows for second migration (8 weeks ago)
a07e6b6 - improve triggers on files table (8 weeks ago)


🐛 fix

4b7484d - Merge pull request #407 from Connexions/inject-resource (8 weeks ago)
7e4525b - Fix inject resource script for modules (8 weeks ago)


✨ script to inject resource files

c106067 - Merge pull request #404 from Connexions/inject-resource (9 weeks ago)
d298c6a - Add tests for inject_resource script (9 weeks ago)
78d0322 - 💄 Move script tests into a subpackage (9 weeks ago)
bea4bff - ⚡ Only run coverage over the cnxarchive package (9 weeks ago)
30ecb06 - Add callable to get the testing config location (9 weeks ago)
f5bcdeb - ✨ Add a script to inject a resource into archive (9 weeks ago)
f733915 - 💄 Move the parser logic into the script's main function (9 weeks ago)
8f1715b - 💄 Replace relative imports for absolute (9 weeks ago)


✨ move the mimetype data

96801e6 - Merge pull request #403 from Connexions/mimetype-shuffle (9 weeks ago)
832aa14 - Update tests to be compatible with the mimetype column move to media_type (9 weeks ago)
e6c7be5 - Update the SQL statements that use the media_type column (9 weeks ago)
c872b1b - Adjust transforms to use media_type from the files table (9 weeks ago)
f67d248 - Amend the process for updating the example data (9 weeks ago)
b0b08b9 - Move mimetype from module_files to files and update example data (9 weeks ago)
15d3692 - Remove mimetype from the module_files table (9 weeks ago)
0e98d0f - Move mimetype column from module_files to files (9 weeks ago)
075d902 - ✨ Add a migrations directory for use with db-migrator (9 weeks ago)


🐛 fix

42988bf - Merge pull request #405 from Connexions/ident-hash-syntax-error (9 weeks ago)
fa5caa6 - Add __str__ for the Ident Hash exception classes (9 weeks ago)
e088202 - Raise ident hash syntax error if ident hash ends with @ (9 weeks ago)


🎨 refactoring

1549c01 - Merge pull request #397 from Connexions/refactor-routes (3 months ago)


✨ listed-resources & 🎨 refactoring

ed7e2f2 - Merge pull request #402 from Connexions/listed-resources (3 months ago)
a3d772b - ✨ Add a list of resources to the content response. (3 months ago)
fe868b8 - Add IdentHashError base exception class for all ident hash exceptions (3 months ago)
4a5ab6a - Remove return_type from split_ident_hash (3 months ago)
ccfcbeb - Replace redirect to canonical with exception views (3 months ago)
f54b896 - Remove unnecessary line "len(res)" from views in_book_search (3 months ago)
2e201f2 - Remove some unnecessary try-except blocks from views.py (3 months ago)
09a8cea - Move view helper functions to the top of the file (3 months ago)
4d7007c - Simplify is_latest helper function in views (3 months ago)
b1d96bd - Remove duplicate test from ViewsTestCase (3 months ago)
791188a - Get route name/args and params from request in redirect to canonical (3 months ago)
c47650f - Change redirect_to_canonical to use matched route from request (3 months ago)
5f79ef3 - Refactor get uuid from shortid code and get latest version code (3 months ago)


cnx-publishing


🐛 fix

2e250f1 - Merge pull request #122 from Connexions/unique-files (5 weeks ago)
86a6a90 - 🐛 Fix for unique files table entries (5 weeks ago)
e5b878d - Have travis fail fast on pep8 errors (5 weeks ago)


✅ refactor

d9fe70a - Merge pull request #121 from Connexions/update-template-path (6 weeks ago)
a2928a5 - 🐛 Avoid redirect loop if user is logged in but can't access /a/ (6 weeks ago)
77113e1 - 🐛 Update MANIFEST.in to include views templates directory (6 weeks ago)
ad9a6c2 - 🐛 Update template paths in views after templates are moved (6 weeks ago)


✅ refactor

f1f65db - Merge pull request #120 from Connexions/rawr (6 weeks ago)
dfec563 - Remove direct use of psycopg2 in view tests (6 weeks ago)
0bbf340 - 🎨 Reorg view tests (6 weeks ago)
6a2f666 - 🎨 Reorg the views by usage context (6 weeks ago)
674ce0a - 🎨 Move all __all__ declarations to EOF (6 weeks ago)
8c31eed - Refactor utility tests (6 weeks ago)
b27f93f - Ignore coverage and testing build files (6 weeks ago)


🐛 fix

b2adffc - Merge pull request #119 from Connexions/epub-summary-parser (6 weeks ago)
a8da9bc - Use cnxepub.DocumentSummaryFormatter on document before publish (6 weeks ago)
49516a5 - Merge pull request #118 from Connexions/html-formatter (7 weeks ago)
d41bbc5 - Use cnxepub.DocumentContentFormatter instead of cnxepub.Document.html (7 weeks ago)


✨ move media-type data

4efab55 - Merge pull request #117 from Connexions/mimetype-shuffle (9 weeks ago)
f1ca310 - 🐛 Include migrations directory in the package manifest (10 weeks ago)


✨ put api-keys in the database

fa81082 - Merge pull request #114 from Connexions/api-keys-in-db (10 weeks ago)
a21db38 - Add admin UI for listing API keys (2 months ago)
dad07e5 - ✨ Add Beaker for function caching (2 months ago)
df67d6a - Connect directly to the database from within the lookup function. (2 months ago)
08b4a26 - 🐦 Add migration to create the api_keys table. (2 months ago)
354d966 - Move api-key configuration into the database (2 months ago)
7a876f8 - Address the move from module_files.mimetype to files.media_type (3 months ago)


✅ refactor

2fc8928 - Merge pull request #116 from Connexions/conn-str-fix (3 months ago)
0b61e72 - Add the database host and port to the testing configuration (3 months ago)


✅ refactor

ab0013e - Merge pull request #113 from Connexions/split-ident-hash (3 months ago)
043373a - Update split_ident_hash to catch missing version and short id exceptions (3 months ago)


✅ update tests

d69f417 - Merge pull request #115 from Connexions/doctests (3 months ago)
c542b83 - Make sure unittest fails if doctest has failures (3 months ago)


✅ refactor

a585077 - Merge pull request #112 from Connexions/move-testing-ini (4 months ago)
fccc363 - Move testing.ini to cnxpublishing/tests/ (4 months ago)
5b3bc23 - Remove environment variable TESTING_CONFIG on travis (use default) (4 months ago)


🐛 fix

45dc4f3 - Merge pull request #111 from Connexions/fix-tests (4 months ago)
cc8d992 - Change .travis.yml to install packages using virtualenv python (4 months ago)
32f3ff2 - Change uuid sql to use text match in case the argument is not a uuid (4 months ago)
acea40c - Update tests following short id changes in cnx-archive (4 months ago)


PEP 8

01758dc - Merge pull request #110 from Connexions/pep8-compliance (6 months ago)
63e4109 - Fix PEP8 issues in tests except line length (6 months ago)
a82ca4d - Run pep8 on travis (6 months ago)
6734edb - Fix PEP8 issues in code excluding tests (6 months ago)
9522e9f - bump version to match tag (8 months ago)


🐛 fix

eb407a2 - Merge pull request #107 from Connexions/fix-travis (9 months ago)
cfa1dcd - Pin mock==1.0.1 for tests in setup.py (9 months ago)