kevindew

---
root:
  type: selection
  options:
    - news
    - guidance
    - transparency
    - policy
    - document_collection
    - corporate_information

kevindew

require "bundler/inline"

# To run this you need to be connected to the Publsihing API via a port forward
# I set this up by adding myself as a user in: https://signon.integration.publishing.service.gov.uk/api_users
# then storing the bearer token in ~/.publishing-api-token
#
# Then I logged onto the VPN and port forwarded Publishing API to 9000 with:
# ssh $(ssh integration "govuk_node_list --single-node -c publishing_api").integration -CNL 9000:127.0.0.1:3093
#
# and ran: PUBLISHING_API_BEARER_TOKEN=$(cat ~/.publishing-api-token) ruby import-publishing-api.rb

kevindew

Debugging Athena JSON errors on Fastly Logs

Argh the dreaded HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: Unterminated string at 227 [character 228 line 1]

To work out what's going on:

Download the files of the day in question

aws s3 sync s3://govuk-production-fastly-logs/govuk_www/year=2019/month=04/date=25 ./ --profile=govuk-production

kevindew

class Images::Create
  include Interactor

  def initialize(params:, user:)
    super
  end

  def call
    find_and_lock_edition do
      check_issues(context.params[:image])

kevindew

#!/bin/bash
set -e

git checkout master
git pull --ff-only
git checkout dependabot/bundler/govuk_app_config-1.3.2
sed -i '' 's/"govuk_app_config"/"govuk_app_config\/govuk_unicorn"/g' config/unicorn.rb
git add config/unicorn.rb
git commit -m "Update config/unicorn for govuk-app-config 1.3.2" -m "In accordance with the upgrade instructions for 1.3.2 release of govuk_app_config. Note: this commit was done by an automated script."
git push

kevindew

How it started

We seem to be seeing the 504s begin at 16:54:27 which seems to be the start of the snowball.

None of the content changes leading up to that seem particularly remarkable for the amount of subscription contents they generated:

irb(main):086:0> pp SubscriptionContent.where(content_change_id: 21300..21335, digest_run_subscriber_id: nil).joins(:content_change).group(:content_change_id, "content_changes.content_id", "content_changes.created_at").order("content_changes.created_at").count
{[21300, "25000916-b720-4734-ada9-5e132cfb531f", Thu, 08 Feb 2018 16:19:57 UTC +00:00]=>128,
 [21301, "c28088ef-056f-49f3-932f-8d52e2fc6094", Thu, 08 Feb 2018 16:21:52 UTC +00:00]=>128,

kevindew

(import "shared/default_format.jsonnet") + {
  document_type: "service_signin",
  definitions: {
    details: {
      type: "object",
      additionalProperties: false,
      required: [
        "choose_sign_in",
        "create_new_account",
      ],

kevindew

# This file is auto-generated from the current state of the database. Instead
# of editing this file, please use the migrations feature of Active Record to
# incrementally modify your database, and then regenerate this schema definition.
#
# Note that this schema.rb definition is the authoritative source for your
# database schema. If you need to create the application database on another
# system, you should be using db:schema:load, not running all the migrations
# from scratch. The latter is a flawed and unsustainable approach (the more migrations
# you'll amass, the slower it'll run and the greater likelihood for issues).
#

kevindew

How can we make Expanded Links faster in Publishing API

Introduction

Link Expansion is used to produce a graph of links between content. An example can be seen by looking at the "links" section of https://www.gov.uk/api/content/world/all

You can access the expanded links on demand through the /v2/expanded-links/{content-id} endpoint.

Most graphs of expanded links are small which produce no performance problems - however there are some that represent a large taxonomoy - which are large and slow to load. Eg /world/all (91b8ef20-74e7-4552-880c-50e6d73c2ff9) which has over 2000 items in it's graph.

kevindew

<div class="diff-table-container" data-reactid="41">
  <div class="diff-table-content" data-reactid="42">
    <table class="diff-table" data-reactid="43">
      <tbody data-reactid="44">
        <tr class="diff-row">
          <td class="diff-line-number" data-content="1."></td>
          <td class="diff-line start"><span class="diff-chunk diff-chunk-equal">title: Pay Stamp Duty Land Tax</span></td>
          <td class="diff-line-number" data-content="1."></td>
          <td class="diff-line start"><span class="diff-chunk diff-chunk-equal">title: Pay Stamp Duty Land Tax</span></td>
        </tr>