Skip to content

Instantly share code, notes, and snippets.

View digitalWestie's full-sized avatar

Rory digitalWestie

33 followers · 45 following
View GitHub Profile
@digitalWestie
digitalWestie / scraper.js
Last active April 26, 2020 21:59
A Gofundme scraper that takes links from the first tab in a Google Spreadsheet, and fills in details on the second tab.
const cheerio = require('cheerio'),
puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const { GoogleSpreadsheet } = require('google-spreadsheet');
const doc = new GoogleSpreadsheet('your-google-spreadsheet-id');
const fs = require('fs');
async function getRows(sheetIndex) {
try {
@digitalWestie
digitalWestie / field_name_example.html
Created September 9, 2019 09:59
<html lang="en" class="">
<head>
<style class="cp-pen-styles">li { display: inline-block;
list-style:none; margin-bottom: 0.6rem; margin-top: 0.6rem;
margin-right: 1rem;
}
li input { margin-right: 0.3rem; }
ul { margin: 0px; padding: 0px; }
fieldset { width:33vw; margin: 1rem; }</style></head><body>
<fieldset class="group">
@digitalWestie
digitalWestie / bach-queries.md
Created May 2, 2019 10:21
Wikidata Queries

Bach's children

SELECT ?child
WHERE
{
  wd:Q1339 wdt:P40 ?child.
}
@digitalWestie
digitalWestie / cork_charities.csv
Created May 2, 2019 08:38
JSON-LD Example with http://demo.ld-r.org
We can make this file beautiful and searchable if this error is corrected: It looks like row 6 should actually have 13 columns, instead of 2. in line 5.
Benefacts Id,Registered Name,Subsector Code,Subsector Name,Registered Address,County,Eircode, CRO,CRA,CHY,AHB,DES, BenefactsUrl
5960958,A SENSE OF CORK MID- SUMMER ARTS FESTIVAL COMPANY LIMITED BY GUARANTEE,1.1,Arts,"Civic Trust House, 50 Pope's Quay, Cork",CORK,T23 R6XC,290323,20075514,19295,,,https://benefacts.ie/org/edfc103e-9c03-42ea-a195-37c9bae0ed8a?src=open
2909215,ABBEYSTREWRY N S,3.2,Primary education,"SKIBBEREEN, CO CORK",CORK,P81 X386,,20112837,,,14433K,https://benefacts.ie/org/6809b5aa-8b34-43a8-983e-798fda7a14f3?src=open
7226046,"ABODE, DOORWAY TO LIFE COMPANY LIMITED BY GUARANTEE",6.2,Job creation,"KILBRACK GROVE, SKEHARD ROAD, BLACKROCK, CORK",CORK,,104718,20013776,6876,Registered,,https://benefacts.ie/org/9113b1fd-e5c5-438d-8308-3094746dc80b?src=open
1859209,ACTIVE CONNECTIONS COMPANY LIMITED BY GUARANTEE,5.2,Family support services,"BALLINLOUGH, KILLEAGH, CO. CORK",CORK,,563671,20201356,,,,https://benefacts.ie/org/0f7021db-2ef8-47a9-9541-8fce7a45de37?src=open
2287798,ADORERS OF THE SACRED HEA
@digitalWestie
digitalWestie / anon_examples.csv
Last active May 1, 2019 10:52
anonymisation examples =(ROUNDDOWN((E2-1)/5,0)*5)+1&" - "&((ROUNDDOWN((E2-1)/5,0)+1)*5)
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 35 columns, instead of 8. in line 4.
Age,Attrition,BusinessTravel,DailyRate,Department,DistanceFromHome,Education,EducationField,EmployeeCount,EmployeeNumber,EnvironmentSatisfaction,Gender,HourlyRate,JobInvolvement,JobLevel,JobRole,JobSatisfaction,MaritalStatus,MonthlyIncome,MonthlyRate,NumCompaniesWorked,Over18,OverTime,PercentSalaryHike,PerformanceRating,RelationshipSatisfaction,StandardHours,StockOptionLevel,TotalWorkingYears,TrainingTimesLastYear,WorkLifeBalance,YearsAtCompany,YearsInCurrentRole,YearsSinceLastPromotion,YearsWithCurrManager
41,Yes,Travel_Rarely,1102,Sales,1,2,Life Sciences,1,1,2,Female,94,3,2,Sales Executive,4,Single,5993,19479,8,Y,Yes,11,3,1,80,0,8,0,1,6,4,0,5
49,No,Travel_Frequently,279,Research & Development,8,1,Life Sciences,1,2,3,Male,61,2,2,Research Scientist,2,Married,5130,24907,1,Y,No,23,4,4,80,1,10,3,3,10,7,1,7
37,Yes,Travel_Rarely,1373,Research & Development,2,2,Other,1,4,4,Male,92,2,1,Laboratory Technician,3,Single,2090,2396,6,Y,Yes,15,3,2,80,0,7,3,3,0,0,0,0
33,No,Travel_Frequently,1392,Research & Development,3,4,L
@digitalWestie
digitalWestie / a-intro.md
Last active February 26, 2020 15:13
Anonymisation example with titanic dataset (see: https://raw.githubusercontent.com/Anshumank399/Titanic-Disaster/master/train.csv)

Intro to data anon

You'll need to use Jupyter notebooks to run / evaluate snippets of code here.

Go to: https://jupyter.org/try and click 'Classic notebook'. It'll take a moment to load but eventually you should see a welcome message.

When that's open we can clear out welcome text and start with a clean slate by clicking on File -> New Notebook -> Python 3

You can start by copying over blocks of code and pasting them into the cells. Once copied in you can run them by pressing holding shift and pressing enter.

@digitalWestie
digitalWestie / UT_IrelandOpenDataLandscape_Visualisation.csv
Created April 30, 2019 13:58
Theme Count By hundred Popular percentage Popular
Environment (2750) 2750 275 10 2
Society (2231) 2231 223 10 2
Economy (1192) 1192 119 5 1
Health (834) 834 83 10 2
Government (522) 522 52 10 2
Transport (398) 398 39 30 6
Housing (355) 355 36 10 2
Science (295) 295 30 0 0
Education and Sport (180) 180 18 0 0
@digitalWestie
digitalWestie / electoral_divisions_internet_unemployment.csv
Created April 30, 2019 12:05
internet & unemployment csv
CODE ED_NAME COUNTY NUTS_II NUTS_III Planning Region Perc_Persons_Aged_15_And_Over_By_Principal_Economic_Status_Unemployed_2011 Perc_Of_Households_With_Internet_Access_No_Access_To_Internet_2011
1 001 Carlow Urban Carlow SE South-East Southern 16.4 35.18
2 002 Graigue Urban Carlow SE South-East Southern 16.74 38.49
3 003 Clonmore Carlow SE South-East Southern 10.37 36.17
4 004 Hacketstown Carlow SE South-East Southern 18.43 36.81
5 005 Haroldstown Carlow SE South-East Southern 9.69 36.96
6 006 Kineagh Carlow SE South-East Southern 11.97 34.71
7 007 Rahill Carlow SE South-East Southern 13.56 23.75
8 008 Rathvilly Carlow SE South-East Southern 19.46 36.42
9 009 Tiknock Carlow SE South-East Southern 13.47 32.17
@digitalWestie
digitalWestie / theme_11_small_areas.csv
Created April 30, 2019 11:08
theme 11
We can't make this file beautiful and searchable because it's too large.
Datacode,SA_NAME,GEOGID,Electoral Division CSO Code,Electoral Division Name,County,NUTS_III,NUTS_II,Planning Region,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_On_Foot_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Bicycle_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Bus_Minibus_Coach_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Train_Dart_Luas_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Motorcycle_Scooter_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Car_Driver_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Car_Passenger_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Van_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Other_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Soft_Modes_Comb_2011,Population_Aged_5_Over_By_Means_Of_Travel_To_Work_School_College_Public_Transport_Comb_2
@digitalWestie
digitalWestie / Make Model Data 2016-edited.csv
Last active April 30, 2019 09:05
We can't make this file beautiful and searchable because it's too large.
Label,VehicleMake,VehicleModel,YearOfBirth,Total,PASS,PASS %,FAIL,FAIL %,Vehicle and Safety Equipment,Vehicle and Safety Equipment %,Lighting and Electrical,Lighting and Electrical %,Steering and Suspension,Steering and Suspension %,Braking Equipment,Braking Equipment %,Wheels and Tyres,Wheels and Tyres %,"Engine, Noise and Exhaust","Engine, Noise and Exhaust %",Chassis and Body,Chassis and Body %,Side Slip Test,Side Slip Test %,Suspension Test,Suspension Test %,Light test,Light test %,Brake Test,Brake Test %,Emmissions,Emmissions %,OTHER,OTHER %,Incompletable,Incompletable %
ALFA ROMEO 145 1996,ALFA ROMEO,145,1996,1,1,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ALFA ROMEO 145 1997,ALFA ROMEO,145,1997,1,1,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ALFA ROMEO 145 1998,ALFA ROMEO,145,1998,4,2,50,2,50,1,25,2,50,2,50,1,25,0,0,0,0,1,25,2,50,0,0,0,0,1,25,0,0,0,0,0,0
ALFA ROMEO 145 1999,ALFA ROMEO,145,1999,3,0,0,3,100,0,0,0,0,0,0,0,0,1,33.3,0,0,2,66.7,0,0,0,0,0,0,0,0,2,66.7,0