Skip to content

Instantly share code, notes, and snippets.

View floehopper's full-sized avatar

James Mead floehopper

72 followers · 40 following
View GitHub Profile
@floehopper
floehopper / assets-origin-nginx.log
Last active January 23, 2018 16:20
Debugging X-Forwarded-Host for asset requests v2
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Host: assets-origin.dev.gov.uk"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Connection: keep-alive"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safar
i/537.36"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Upgrade-Insecure-Requests: 1"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Accept-Encoding: gzip, deflate"
2018/01/23 16:17:58 [debug] 18102#0: *15 http header: "Accept-Language: en-GB,en-US;q=0.9,en;q=0.8"
GET /media/5a67581d759b7424f4e81ba1/README.md?jm345 HTTP/1.0
@floehopper
floehopper / asset-manager-nginx.log
Last active January 23, 2018 16:15
Debugging X-Forwarded-Host for asset requests
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "Host: asset-manager.dev.gov.uk"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "X-Real-IP: 127.0.0.1"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "X-Forwarded-Server: static.dev.gov.uk"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "X-Forwarded-For: 127.0.0.1"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "X-Forwarded-Host: static.dev.gov.uk"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "Connection: close"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari
/537.36"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "Upgrade-Insecure-Requests: 1"
2018/01/23 16:11:22 [debug] 17761#0: *5 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
@floehopper
floehopper / 2018-01-23-gds-updates.md
Last active January 23, 2018 10:40

Achievements

  • Migrated more non-attachment assets from Whitehall to Asset Manager
  • Started serving the migrated Whitehall non-attachment assets from Asset Manager

Ongoing and next

  • Continuing to migrate remaining non-attachment assets from Whitehall to Asset Manager and removing them from NFS
  • Continuing to switch these Whitehall assets over to be served from Asset Manager
  • Spiking on supporting attachment assets from Whitehall in Asset Manager (authorisation for drafts)
@floehopper
floehopper / whitehall-attachment-problem.md
Last active January 19, 2018 11:56

Backend/Admin

jamesmead@ec2-integration-blue-whitehall_backend-ip-10-1-6-139:~$ curl -s -v "https://whitehall-admin.integration.govuk-internal.digital/government_data/file/672533/netLec7.pdf" >/dev/null
* Hostname was NOT found in DNS cache
*   Trying 10.1.6.248...
* Connected to whitehall-admin.integration.govuk-internal.digital (10.1.6.248) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
@floehopper
floehopper / 2018-01-16-gds-updates.md
Created January 16, 2018 10:39

Achievements

  • Deleted all Asset Manager assets from NFS - they are now stored on S3
  • Started serving more of the migrated Whitehall assets from Asset Manager
  • Fixed problem with migrating assets with non-ascii characters in filenames
  • Improved performance of Whitehall-related actions in Asset Manager
  • Helped diagnose problems with assets relating to the move of the integration environment to AWS

Ongoing and next

@floehopper
floehopper / GDS_SSO_STRATEGY.log
Last active January 15, 2018 16:35
GDS SSO make_oauth_work_in_dev
vagrant@development:/var/apps/whitehall$ sudo sh -c "echo 'real' > /etc/govuk/whitehall/env.d/GDS_SSO_STRATEGY"
vagrant@development:/var/apps/whitehall$ sudo cat /etc/govuk/whitehall/env.d/GDS_SSO_STRATEGY
real
@floehopper
floehopper / explain-with-index.rb
Created January 15, 2018 14:47
Asset Manager query without/with index
{"queryPlanner"=>
{"plannerVersion"=>1,
"namespace"=>"govuk_assets_development.assets",
"indexFilterSet"=>false,
"parsedQuery"=>
{"$and"=>
[{"_type"=>{"$eq"=>"WhitehallAsset"}},
{"deleted_at"=>{"$eq"=>nil}},
{"legacy_url_path"=>{"$eq"=>"foo"}}]},
"winningPlan"=>
@floehopper
floehopper / links.md
Created January 14, 2018 16:21
Linux Namespaces
@floehopper
floehopper / publishing-api-development.log
Created January 11, 2018 12:33
Publishing API error
Started PUT "/v2/content/665bdfa0-a8a9-4588-9030-f2fb338f9878" for 127.0.0.1 at 2018-01-11 12:21:30 +0000
Processing by V2::ContentItemsController#put_content as JSON
Parameters: {"title"=>"Testing attachment access", "locale"=>"en", "publishing_app"=>"whitehall", "redirects"=>[], "update_type"=>"major", "description"=>"Testing attachment access - Summary", "details"=>{"body"=>"<div class=\"govspeak\"><p>Testing attachment access - Body</p>\n</div>", "change_history"=>[], "documents"=>[], "emphasised_organisations"=>["300ee6db-9ed5-4d74-8f06-889663fd4a3d"], "political"=>false, "government"=>nil, "tags"=>{"browse_pages"=>[], "policies"=>[], "topics"=>[]}, "first_public_at"=>"2018-01-11T12:21:29.000+00:00"}, "document_type"=>"policy_paper", "public_updated_at"=>"2018-01-11T12:21:29.000+00:00", "rendering_app"=>"government-frontend", "schema_name"=>"publication", "links"=>{"topics"=>[], "parent"=>[], "organisations"=>["300ee6db-9ed5-4d74-8f06-889663fd4a3d"], "world_locations"=>[], "policy_areas"=>["3f2c03fa-00
@floehopper
floehopper / govuk-development-vm.md
Last active February 28, 2018 15:46
Setup GOV.UK development VM with local SSO, Whitehall & its draft stack

Data replication

The idea here is to load the routes for the router and draft-router. Life's too short to work out how to create the routes manually!

  • cd /var/govuk/govuk-puppet/development-vm/replication
  • Apply replicate-data-local.diff patch to skip all dbs on mongo-1.backend & api-mongo-1.api
  • Run the following command to import data for router & draft-router: ./replicate-data-local.sh -s -d backups/YYYY-MM-DD -p -q -e -t -i 'collections_publisher contacts release search_admin tariff_admin tariff_temporal'

Authenticating proxy