Skip to content

Instantly share code, notes, and snippets.

View pwalsh's full-sized avatar
🤘

Paul Walsh pwalsh

🤘
View GitHub Profile
@pwalsh
pwalsh / gist:cd764c058c975661715924e0315074ca
Created May 9, 2022 09:23 — forked from Miserlou/gist:11500b2345d3fe850c92
1000 Largest US Cities By Population
Largest 1000 Cities in America
2013 popuation data - Biggest US Cities By Population
rank,city,state,population,2000-2013 growth
1,New York,New York,8405837,4.8%
2,Los Angeles,California,3884307,4.8%
3,Chicago,Illinois,2718782,-6.1%
4,Houston,Texas,2195914,11.0%
5,Philadelphia,Pennsylvania,1553165,2.6%

incorrect validation

A resource descriptor with fields with format set to None validates with validate, but fails with dump_to_sql.

{'name': 'srm_services', 'path': 'res_1.csv', 'profile': 'tabular-data-resource', 'schema': {'fields': [{'name': 'id', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'source', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'name', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'payment_required', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'urls', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'description', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'details', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'status', 'type': 'string', 'format': None, 'constraints': {}}, {'name': 'payment_details', 'type': 'string', 'format': None, 'constraints': {}}], 'missingValues': ['']}}

Solution: set format to "default"

@pwalsh
pwalsh / viderum-developer.md
Created December 27, 2018 18:31
Developer at Viderum

Developer

Overview

We are looking for Developers to join our growing team. Viderum implements a range of data management solutions for government and the enterprise using open source technologies, including many that our team creates and maintains in house.

Developers at Viderum work as part of a delivery team under technical supervision of a Senior Developer. They communicate about their work with technical and non-technical stakeholders on their team, and therefore need to be proficient in spoken and written English. Our Developers scope and define work with our Senior Developers - depending on level of experience - and are responsible for implementation of a range of data management solutions.

Most of the work we do at Viderum is in Python, and some experience implementing web applications in Python or Node.js is required. As we do a lot of work with data, experience with data wrangling, data engineering, and/or data science is desirable, as is some familiarity with SQL or other database technolog

@pwalsh
pwalsh / viderum-senior-developer.md
Last active January 5, 2019 20:44
Senior Developer at Viderum

Senior Developer

Overview

We are looking for Senior Developers to join our growing team. Viderum implements a range of data management solutions for government and the enterprise using open source technologies, including many that our team creates and maintains in house.

Senior Developers at Viderum have experience working with teams of developers in a leading role. They communicate regularly with technical and non-technical stakeholders about the team’s work, and therefore need excellent spoken and written English. Our Senior Developers scope and define work for other developers on their delivery teams, as well as lead on technical implementation.

Most of the work we do at Viderum is in Python, and experience designing and implementing web applications in Python is a must. As we do a lot of work with data, experience with data wrangling, data engineering, and/or data science is highly desirable, as is a strong working knowledge of SQL and other database technologies. We also do frontend development

@pwalsh
pwalsh / keybase.md
Last active October 22, 2017 06:32
Keybase proof.

Keybase proof

I hereby claim:

  • I am pwalsh on github.
  • I am pwalsh (https://keybase.io/pwalsh) on keybase.
  • I have a public key ASCzOJE9TdT0SeKkhN1zw4Jj0vGEOfOPzQGYdF8I02HrYgo

To claim this, I am signing this object:

@pwalsh
pwalsh / datasets_with_missing_resources.csv
Created January 12, 2016 10:01
Missing data on DataHub.io
title url created
EU Budget https://datahub.io/dataset/eu-budget 2009-04-10 18:27:52.860908
EU - Financial Transparency System https://datahub.io/dataset/eu-fts 2009-04-21 01:40:13.634086
Country and Regional Analyses (CRA) - UK Government Finances https://datahub.io/dataset/ukgov-finances-cra 2009-12-18 18:36:41.139595
Country and Regional Analyses (CRA) - UK Government Finances https://datahub.io/dataset/ukgov-finance-cra 2010-02-10 19:35:33.612496
South Africa Development Co-operation Information System https://datahub.io/dataset/south-africa-development-cooperation-system 2010-03-11 18:10:04.119287
Israel State Budget 2009-2010 https://datahub.io/dataset/israel-state-budget 2010-07-29 09:11:01.520248
AGROVOC https://datahub.io/dataset/agrovoc-skos 2010-09-07 08:57:40.040957
IATI Registry https://datahub.io/dataset/iati-registry 2011-04-25 08:23:25.841919
Aquiosquedais https://datahub.io/dataset/aquiosquedais 2011-06-05 22:25:14.005077
@pwalsh
pwalsh / uk-gov-spend-publishing-data-fields.md
Last active August 29, 2015 14:16
A table of the data fields for spend publishing in the UK government.

This table is taken from this public document.

Column Field name What is required Reason for inclusion Additional information Inclusion status
1 Department family Name of department or parent department. To identify spend to each deaprtment area. Consistent naming conventions must be used. Departments need to issue naming conventions. Mandatory inclusion
2 Entity Name of individual entity To identify spend of individual bodies within the departmental family Where a core department is completing the return, the entry in the department “family field” should be repeated here. Consistent naming conventions must be used. Mandatory inclusion
3 Date The payment date as recorded in department's purchase or general ledger. To identify the date that the
@pwalsh
pwalsh / jack.conf
Last active November 20, 2022 14:45
Raspberry Pi Headless Setup
; Supervisor configuration to manage a jack server
[program:jackd]
command=/usr/bin/jackd -r -t2000 -ddummy -r44100 -p1024
user=pi
redirect_stderr=true
autostart=true
autorestart=true