Skip to content

Instantly share code, notes, and snippets.

@practicalparticipation
Last active August 29, 2015 14:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save practicalparticipation/ca48ed2ffc0dee3dc872 to your computer and use it in GitHub Desktop.
Save practicalparticipation/ca48ed2ffc0dee3dc872 to your computer and use it in GitHub Desktop.
From Open Contracting Data Spec 0.3 to Trade Mapper Data

This GIST runs through the steps taken to generate a CSV file that will work with the TradeMapper tool to show flows of contract spending.

(1) Checkout the draft Open Contracting Data from Development Gateway

https://github.com/devgateway/ca-app-ocds-export/

(2) Get a copy of jq command line library for working with JSON

http://stedolan.github.io/jq/

(3) Run the rather long command below which parses out a CSV line for every buyer, buyer country, supplier country and contract value in the file (and adds the csv header)

echo "OCID,Buyer,BuyerCountry,SupplierCountry,Value,Currency" > map.csv; jq -r '.releases[] | {ocid:.ocid,buyerName:.buyer.id.name,buyerCountry:.buyer.address["country-name"], awards:.awards[]} | {ocid:.ocid,buyerName:.buyerName,buyerCountry:.buyerCountry,awardCountry:.awards.suppliers[0].address["country-name"],awardValue:.awards.awardValue.amount,awardCurrency:.awards.awardValue.currency} | [.ocid,.buyerName,.buyerCountry,.awardCountry,.awardValue,.awardCurrency] | @csv' EUAwards-2006-1.json >> map.csv

(This runs just against one small file from the set: could be adapted to run against all of them). To deal with the nesting of the data, and as CSV is only generated from arrays, I've had to pipe it all through a couple of JQ processes)

(4) Cross-map country names to ISO codes

I did this in Open Refine using a lookup list with Country names in one column (called Country, and ISO2 codes in another called ISO2), loaded into Refine as 'ISOLookup', and then 'Edit Column> Add column based on this column' on the country names with the formula

cell.cross("ISOLookup","Country").cells["ISO2"].value[0]

but you could use your tool of choice. From my quick test with the first dataset, all the country names mapped to ISO codes. However, this is only for 35 countries. No guarantee that the source data uses same names as all the ISOLookup fields, so this would require checking when processing against the rest of that data.

(5) Upload the CSV somewhere and call Trade Mapper

Add the CSV address onto the end of the URL below:

http://trademapper.aptivate.org/?csvtype=ocds&loadcsv=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment