jennyd

# Prerequisites:
# - ruby 2.6.1 (or another relatively recent version is probably fine too) and bundler installed
# - postgres 9.3+ running (I've used 9.6 and 10.7 and they were both fine)
# - if you're creating more than 100 mappings at once, the app processes that in the background using Sidekiq with Redis, but the app runs fine without Redis otherwise

git clone git@github.com:alphagov/transition.git
cd transition

bundle install
bundle exec rake db:setup

jennyd

Keybase proof

I hereby claim:

• I am jennyd on github.
• I am jennyd (https://keybase.io/jennyd) on keybase.
• I have a public key ASBRMANVJPqvoLReJoQDHUeB4f3xaAkhHISxv8hMxZVrpwo

To claim this, I am signing this object:

jennyd

#!/usr/bin/env python

import csv
import urlparse
from collections import Counter


'''Analyse data from govuk-delivery's database to find out which URLs exist as
subscription topics and which query params they use. This will help us work out
whether all Whitehall content is already tagged to enough things in the links

jennyd

from performanceplatform.client import DataSet


query_parameters = {
    'group_by': 'pagePath',
    'period': 'day',
    'start_at': '2016-07-18T00:00:00Z',
    'end_at': '2016-07-20T00:00:00Z',
    'collect': 'uniquePageviews:sum',
    'filter_by': 'pagePath:/pay-council-tax/lambeth'

jennyd

Keybase proof

I hereby claim:

• I am jennyd on github.
• I am jennyd (https://keybase.io/jennyd) on keybase.
• I have a public key whose fingerprint is DF1A 9E71 8F42 0F6B 64F2 B011 302C 8726 3CBA 8CAD

To claim this, I am signing this object:

jennyd

# with performanceplatform-client==0.8.2
from performanceplatform.client import DataSet


dataset = DataSet.from_group_and_type('https://www.performance.service.gov.uk/data',
                                      'govuk-info',
                                      'page-contacts')

query_parameters = {
    'start_at': '2015-03-12T00:00:00Z',

jennyd

import falcon
import logging

logger = logging.getLogger(__name__)
logger.addHandler(logging.FileHandler('test.log'))
logger.setLevel(logging.INFO)

class ResponseLoggerMiddleware(object):
    def process_response(self, req, resp):
        logger.info('{0} {1} {2}'.format(req.method, req.relative_uri, resp.status[:3]))

jennyd

#!/opt/mawk/bin/mawk -f

##
# Skip lines with invalid UTF8 byte sequences.
# Adapted from http://unix.stackexchange.com/questions/6516/filtering-invalid-utf8 by @rgarner
#
$0 !~ /^(([\x00-\x7F])|([\xC2-\xDF][\x80-\xBF])|((([\xE0][\xA0-\xBF])|([\xED][\x80-\x9F])|([\xE1-\xEC\xEE-\xEF][\x80-\xBF]))([\x80-\xBF]))|((([\xF0][\x90-\xBF])|([\xF1-\xF3][\x80-\xBF])|([\xF4][\x80-\x8F]))([\x80-\xBF][\x80-\xBF])))*$/ {
 next
}

jennyd

time bundle exec rake import:all:hits

With all hits imported already and no updates to counts to make:
    - master:
        real    0m43.212s
        user    0m8.517s
        sys 0m1.288s
    - hits_import_update_count:
        real    0m47.302s
        user    0m8.029s