Skip to content

Instantly share code, notes, and snippets.

David Read davidread

Block or report user

Report or block davidread

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@davidread
davidread / nosetests
Created Jul 12, 2019
Running nosetests locally
View nosetests
(default) vagrant@ubuntu-xenial:/vagrant/src/ckan$ nosetests --nologcapture --with-pylons=test-core.ini -v ckan ckanext
/usr/lib/ckan/default/local/lib/python2.7/site-packages/webassets/loaders.py:162: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
obj = self.yaml.load(f) or {}
No handlers could be found for logger "ckan.lib.uploader"
ckan.tests.config.test_environment.TestDisplayTimezone.test_missing_timezone ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_missing_siteurl ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_missing_schema ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_removes_backslash ... ok
ckan.tests.config.test_environment.TestSiteUrlMandatory.test_siteurl_wrong_schema ... ok
CKAN_SQLALCHEMY_URL in the env takes precedence over CKAN_DB ... ok
View reset_ckan.sh
# Resets a CKAN install e.g. after you switch branches, particularly downgrading.
# Destructive! Designed for developers, not production installs.
# Run this script from the ckan repo directory.
# wipe all the databases
sudo -u postgres dropdb ckan_default
sudo -u postgres createdb -O ckan_default ckan_default -E utf-8
sudo -u postgres dropdb ckan_test
sudo -u postgres createdb -O ckan_default ckan_test -E utf-8
sudo -u postgres dropdb datastore_default
@davidread
davidread / migrate_test.md
Created Feb 16, 2019
Migration performance tests
View migrate_test.md

I ran the migrate_package_activity.py (https://github.com/ckan/ckan/pull/3972) on a real existing CKAN database to get a sense of the time taken and the change in database size

Timing: Migration took 49s for 236 datasets. If you scale that linearly to 900k datasets (europeandataportal.eu) then you're talking 1 hour. There's likely to be more activities per dataset though, so it would be more like several hours. Seems reasonable to want to do the migration while the site is up.

Size: size of the activity table went up about 20%, because every revision we now store resources, extras and tags on each package. However the overall size came down when we deleted the content of the activity_detail table (which is no longer needed), because that had a quirk that when you add a tag, it stored the dictized package as well in that table - this seems like a mistake.

Test Details

Test database properties:

View gist:98a883005586db33b325c1bcb69f5be7
"working tree" = "repository"
"untracked files" - diff
"changes"
---- "add" --->
<--- "rm" ----
"Staged" = "Changes to be committed" = "index"
Similar: "indexed" "tracked files" "snapshot"
View gist:518650c795244b411e1fc817cc3c89aa
$ nosetests --ckan --with-pylons=test-core.ini ckan/tests/lib/test_helpers.py:TestBuildNavMain -x
No handlers could be found for logger "ckan.lib.uploader"
E
======================================================================
ERROR: ckan.tests.lib.test_helpers.TestBuildNavMain.test_dataset_navigation_legacy_routes
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/vagrant/src/ckan/ckan/tests/lib/test_helpers.py", line 606, in test_dataset_navigation_legacy_routes
View validator.diff
diff --git a/ckan/lib/navl/validators.py b/ckan/lib/navl/validators.py
index 172da1531..b041d478e 100644
--- a/ckan/lib/navl/validators.py
+++ b/ckan/lib/navl/validators.py
@@ -4,7 +4,7 @@ from six import text_type
import ckan.lib.navl.dictization_functions as df
-from ckan.common import _, json
+from ckan.common import _, json, config
View gist:a6b47f2ebada39ed21fe66016702805b
Lots of stack traces and error messages for CKAN, followed by a comment about how I fixed it (that time).
NB don't email me privately for help - ask on StackOverflow #ckan instead.
David
File "/vagrant/src/ckan/ckan/templates/home/snippets/search.html", line 1, in top-level template code
{% set tags = h.get_facet_items_dict('tags', limit=3) %}
File "/vagrant/src/ckan/ckan/lib/helpers.py", line 949, in get_facet_items_dict
if not c.search_facets or \
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
@davidread
davidread / config.yaml
Last active Apr 25, 2018
zero-to-jupyterhub auth0 example config
View config.yaml
auth:
type: custom
custom:
className: "oauthenticator.auth0.Auth0OAuthenticator"
config:
client_id: "3maZCyDXXXXXXXXXIDbz0W8A"
client_secret: "aMSobNMr0OkPjXXXXXXXXX7MdZkAcpL"
oauth_callback_url: "http://a172XXXX77d46cd-18XXXXX48.eu-west-1.elb.amazonaws.com/hub/oauth_callback"
scope:
- "openid"
@davidread
davidread / container-start.sh
Created Feb 20, 2018
Jupyter container start
View container-start.sh
#!/usr/bin/env bash
set -ex
# NB_USER=$USER
# NB_UID=$USER_UID
# USER_UID=1001
# GROUP=staff
# # Add the $USER user
# useradd -g $GROUP -u $USER_UID -d /home/$USER $USER
@davidread
davidread / Dockerfile
Created Feb 20, 2018
Jupyter dockerfile
View Dockerfile
FROM jupyter/datascience-notebook
# Install Anaconda
#RUN conda install anaconda
# Install Jupyter Dashboard
#RUN pip install jupyter_dashboards
#RUN jupyter dashboards quick-setup --sys-prefix
#RUN jupyter nbextension enable jupyter_dashboards --py --sys-prefix
You can’t perform that action at this time.