Skip to content

Instantly share code, notes, and snippets.

Rory digitalWestie

View GitHub Profile
@digitalWestie
digitalWestie / scraper.js
Last active Apr 26, 2020
A Gofundme scraper that takes links from the first tab in a Google Spreadsheet, and fills in details on the second tab.
View scraper.js
const cheerio = require('cheerio'),
puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const { GoogleSpreadsheet } = require('google-spreadsheet');
const doc = new GoogleSpreadsheet('your-google-spreadsheet-id');
const fs = require('fs');
async function getRows(sheetIndex) {
try {
View field_name_example.html
<html lang="en" class="">
<head>
<style class="cp-pen-styles">li { display: inline-block;
list-style:none; margin-bottom: 0.6rem; margin-top: 0.6rem;
margin-right: 1rem;
}
li input { margin-right: 0.3rem; }
ul { margin: 0px; padding: 0px; }
fieldset { width:33vw; margin: 1rem; }</style></head><body>
<fieldset class="group">
View bach-queries.md

Bach's children

SELECT ?child
WHERE
{
  wd:Q1339 wdt:P40 ?child.
}
View cork_charities.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 6 should actually have 13 columns, instead of 2. in line 5.
Benefacts Id,Registered Name,Subsector Code,Subsector Name,Registered Address,County,Eircode, CRO,CRA,CHY,AHB,DES, BenefactsUrl
5960958,A SENSE OF CORK MID- SUMMER ARTS FESTIVAL COMPANY LIMITED BY GUARANTEE,1.1,Arts,"Civic Trust House, 50 Pope's Quay, Cork",CORK,T23 R6XC,290323,20075514,19295,,,https://benefacts.ie/org/edfc103e-9c03-42ea-a195-37c9bae0ed8a?src=open
2909215,ABBEYSTREWRY N S,3.2,Primary education,"SKIBBEREEN, CO CORK",CORK,P81 X386,,20112837,,,14433K,https://benefacts.ie/org/6809b5aa-8b34-43a8-983e-798fda7a14f3?src=open
7226046,"ABODE, DOORWAY TO LIFE COMPANY LIMITED BY GUARANTEE",6.2,Job creation,"KILBRACK GROVE, SKEHARD ROAD, BLACKROCK, CORK",CORK,,104718,20013776,6876,Registered,,https://benefacts.ie/org/9113b1fd-e5c5-438d-8308-3094746dc80b?src=open
1859209,ACTIVE CONNECTIONS COMPANY LIMITED BY GUARANTEE,5.2,Family support services,"BALLINLOUGH, KILLEAGH, CO. CORK",CORK,,563671,20201356,,,,https://benefacts.ie/org/0f7021db-2ef8-47a9-9541-8fce7a45de37?src=open
2287798,ADORERS OF THE SACRED HEA
@digitalWestie
digitalWestie / anon_examples.csv
Last active May 1, 2019
anonymisation examples =(ROUNDDOWN((E2-1)/5,0)*5)+1&" - "&((ROUNDDOWN((E2-1)/5,0)+1)*5)
View anon_examples.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 35 columns, instead of 8. in line 4.
Age,Attrition,BusinessTravel,DailyRate,Department,DistanceFromHome,Education,EducationField,EmployeeCount,EmployeeNumber,EnvironmentSatisfaction,Gender,HourlyRate,JobInvolvement,JobLevel,JobRole,JobSatisfaction,MaritalStatus,MonthlyIncome,MonthlyRate,NumCompaniesWorked,Over18,OverTime,PercentSalaryHike,PerformanceRating,RelationshipSatisfaction,StandardHours,StockOptionLevel,TotalWorkingYears,TrainingTimesLastYear,WorkLifeBalance,YearsAtCompany,YearsInCurrentRole,YearsSinceLastPromotion,YearsWithCurrManager
41,Yes,Travel_Rarely,1102,Sales,1,2,Life Sciences,1,1,2,Female,94,3,2,Sales Executive,4,Single,5993,19479,8,Y,Yes,11,3,1,80,0,8,0,1,6,4,0,5
49,No,Travel_Frequently,279,Research & Development,8,1,Life Sciences,1,2,3,Male,61,2,2,Research Scientist,2,Married,5130,24907,1,Y,No,23,4,4,80,1,10,3,3,10,7,1,7
37,Yes,Travel_Rarely,1373,Research & Development,2,2,Other,1,4,4,Male,92,2,1,Laboratory Technician,3,Single,2090,2396,6,Y,Yes,15,3,2,80,0,7,3,3,0,0,0,0
33,No,Travel_Frequently,1392,Research & Development,3,4,L
View a-intro.md

Intro to data anon

You'll need to use Jupyter notebooks to run / evaluate snippets of code here.

Go to: https://jupyter.org/try and click 'Classic notebook'. It'll take a moment to load but eventually you should see a welcome message.

When that's open we can clear out welcome text and start with a clean slate by clicking on File -> New Notebook -> Python 3

You can start by copying over blocks of code and pasting them into the cells. Once copied in you can run them by pressing holding shift and pressing enter.

View UT_IrelandOpenDataLandscape_Visualisation.csv
Theme Count By hundred Popular percentage Popular
Environment (2750) 2750 275 10 2
Society (2231) 2231 223 10 2
Economy (1192) 1192 119 5 1
Health (834) 834 83 10 2
Government (522) 522 52 10 2
Transport (398) 398 39 30 6
Housing (355) 355 36 10 2
Science (295) 295 30 0 0
Education and Sport (180) 180 18 0 0
View electoral_divisions_internet_unemployment.csv
CODE ED_NAME COUNTY NUTS_II NUTS_III Planning Region Perc_Persons_Aged_15_And_Over_By_Principal_Economic_Status_Unemployed_2011 Perc_Of_Households_With_Internet_Access_No_Access_To_Internet_2011
1 001 Carlow Urban Carlow SE South-East Southern 16.4 35.18
2 002 Graigue Urban Carlow SE South-East Southern 16.74 38.49
3 003 Clonmore Carlow SE South-East Southern 10.37 36.17
4 004 Hacketstown Carlow SE South-East Southern 18.43 36.81
5 005 Haroldstown Carlow SE South-East Southern 9.69 36.96
6 006 Kineagh Carlow SE South-East Southern 11.97 34.71
7 007 Rahill Carlow SE South-East Southern 13.56 23.75
8 008 Rathvilly Carlow SE South-East Southern 19.46 36.42
9 009 Tiknock Carlow SE South-East Southern 13.47 32.17
View Make Model Data 2016-edited.csv
We can't make this file beautiful and searchable because it's too large.
Label,VehicleMake,VehicleModel,YearOfBirth,Total,PASS,PASS %,FAIL,FAIL %,Vehicle and Safety Equipment,Vehicle and Safety Equipment %,Lighting and Electrical,Lighting and Electrical %,Steering and Suspension,Steering and Suspension %,Braking Equipment,Braking Equipment %,Wheels and Tyres,Wheels and Tyres %,"Engine, Noise and Exhaust","Engine, Noise and Exhaust %",Chassis and Body,Chassis and Body %,Side Slip Test,Side Slip Test %,Suspension Test,Suspension Test %,Light test,Light test %,Brake Test,Brake Test %,Emmissions,Emmissions %,OTHER,OTHER %,Incompletable,Incompletable %
ALFA ROMEO 145 1996,ALFA ROMEO,145,1996,1,1,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ALFA ROMEO 145 1997,ALFA ROMEO,145,1997,1,1,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ALFA ROMEO 145 1998,ALFA ROMEO,145,1998,4,2,50,2,50,1,25,2,50,2,50,1,25,0,0,0,0,1,25,2,50,0,0,0,0,1,25,0,0,0,0,0,0
You can’t perform that action at this time.