Chris Kenwright thekensta
thekensta
/ window_function_comparison.ipynb
Created
December 23, 2015 11:35
Window Function Comparison
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
thekensta
/ spark_s3a_instructions.sh
Created
October 14, 2015 20:27
Set up Apache Spark 1.5+ with Hadoop 2.6+ s3a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # For a local environment | |
| # Install hadoop and apache-spark via homebrew | |
| # Apache Spark conf file | |
| # libexec/conf/spark-defaults.conf | |
| # Make the AWS jars available to Spark | |
| spark.executor.extraClassPath /usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/hadoop-aws-2.7.1.jar | |
| spark.driver.extraClassPath /usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/aws-java-sdk-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/hadoop-aws-2.7.1.jar | |
| # Add file |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* | |
| # https://cloud.google.com/bigquery/audit-logs#sample_audit_queries_in_bigquery | |
| select | |
| protopayload_auditlog.authenticationInfo.principalEmail as user_email, | |
| protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobConfiguration.query.query, | |
| -- protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobStatus.state, | |
| -- protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobStatus.error.code, | |
| -- protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobStatus.error.message, | |
| sum(protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobStatistics.totalBilledBytes) billedBytes, |
thekensta
/ price_elasticity.ipynb
Created
April 9, 2018 18:08
Price Elasticity from Analytics Made Skeazy
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
thekensta
/ bq_sales_yoy.sql
Created
December 21, 2017 14:35
Year on Year Day of Week Sales in Big Query using GENERATE_DATE_ARRAY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- | |
| -- Week Daily for non-Orthogonal Date dimensions e.g. day of week | |
| -- | |
| with Calendar as ( | |
| select | |
| this_year, | |
| date_add(this_year, interval - 364 day) as last_year | |
| from | |
| unnest(GENERATE_DATE_ARRAY('2017-01-01', '2017-12-31')) as this_year | |
| ), |
thekensta
/ fun_with_time.bash
Created
December 6, 2017 21:26
Load and Extract DATETIMES and TIMESTAMPS in Big Query
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| DATASET="<YOUR DATASET HERE>" | |
| TABLE="fun_with_time" | |
| SCHEMA_FILE="/tmp/ts-schema.json" | |
| DATA_FILE="/tmp/ts-data.ndjson" | |
| SQL_FILE="/tmp/query.sql" | |
| BUCKET="<YOUR BUCKET HERE>" | |
| cat <<EOM >${SCHEMA_FILE} |
thekensta
/ gigaclear_scrape.py
Last active
November 7, 2017 08:43
Fetch raw data for Gigaclear to see where their network is live
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Script to scrape Gigaclear status for each postcode (in this case in GL* areas) | |
| Writes a Json file with lat-lon, postcode and response for display and | |
| later reference | |
| """ | |
| import requests | |
| import csv | |
| import io | |
| import time |
thekensta
/ gigaclear_map.ipynb
Last active
November 7, 2017 08:37
Map of Postcodes Activated by Gigaclear
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| with X as ( | |
| -- select STRUCT<x int64, y string>(1, 'a') as R | |
| select * | |
| -- from unnest(split("I am the bad man, I am the bad man", " ")) as w with offset as c | |
| from | |
| unnest([ | |
| -- Will be named and explictly typed | |
| -- STRUCT<x int64, y string>(1, 'a'), | |
| -- STRUCT<x int64, y string>(2, 'b'), | |
| -- STRUCT<x int64, y string>(3, 'c') |
NewerOlder