Skip to content

Instantly share code, notes, and snippets.

View pascalwhoop's full-sized avatar

Pascal Bro pascalwhoop

46 followers · 30 following
  • Zoom
View GitHub Profile
@pascalwhoop
pascalwhoop / bootstrap.sh
Last active January 16, 2023 13:00
bootstrap script for machine learning env
#!/bin/bash
# install core requirements
NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
sudo apt install -y tmux htop bmon fzf wget curl
# get brew working
curl https://gist.githubusercontent.com/pascalwhoop/4d01bb179132118c565b3db56c4ce68e/raw/fa0ab508cac760efaa399442047256a36578c62e/tmux.conf > ~/.tmux.conf
# fish shell
@pascalwhoop
pascalwhoop / slack_message_table_schema.json
Created July 16, 2020 20:43
{
"fields": [
{
"name": "index",
"type": "integer"
},
{
"name": "type",
"type": "string"
},
@pascalwhoop
pascalwhoop / dataflow_job.tf
Created July 8, 2020 20:06
sample pipeline for data cleaning
import apache_beam as beam
import logging
import json
from apache_beam.io import ReadFromText
from apache_beam.io import BigQuerySource
from apache_beam.io import BigQuerySink
from apache_beam.io import WriteToText
from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
from apache_beam.io.gcp.internal.clients import bigquery
@pascalwhoop
pascalwhoop / dataflow_job.tf
Created April 30, 2020 16:46
locals {
foo_directory = "../../../../services/foo/target"
jobName = "company-${var.environment}-datalayer-foo-app-v1"
region = "europe-west1"
//parameters for the java jar
dataflow_parameters = {
runner = "DataflowRunner"
jobName = local.jobName
update = data.external.running_job.result.name == local.jobName ? true : false
project = var.project
@pascalwhoop
pascalwhoop / fix.sh
Created April 26, 2020 20:46
gcloud rsync -r gs://our-bucket /tmp/bucket
cd 0-bootstrap && vim backend.tf #comment out backend
terraform init #approve copying state to local
gcloud rm -r gs://our-bucket #delete all data
terraform apply #force recreates bucket in different region
gcloud rsync -r /tmp/bucket gs://our-bucket #bring state files back
gcloud rm -r gs://our-bucket/terraform/state/bootstrap #clean old state
vim backend.tf #uncomment the backend again
terraform init #copy state back to bucket
@pascalwhoop
pascalwhoop / PR-bootstrap.yaml
Created April 26, 2020 20:18
gcp foundation github actions
name: 'Bootstrap Terraform'
on:
- pull_request
env:
tf_version: 'latest'
tf_working_dir: '0-bootstrap'
GOOGLE_CREDENTIALS: ${{secrets.GOOGLE_CREDENTIALS}}
jobs:
terraform:
name: 'Terraform'
@pascalwhoop
pascalwhoop / populations.csv
Created March 5, 2020 22:09
We can make this file beautiful and searchable if this error is corrected: Unclosed quoted field in line 3.
"Country Name","Country Code","Indicator Name","Indicator Code","1960","1961","1962","1963","1964","1965","1966","1967","1968","1969","1970","1971","1972","1973","1974","1975","1976","1977","1978","1979","1980","1981","1982","1983","1984","1985","1986","1987","1988","1989","1990","1991","1992","1993","1994","1995","1996","1997","1998","1999","2000","2001","2002","2003","2004","2005","2006","2007","2008","2009","2010","2011","2012","2013","2014","2015","2016","2017","2018","2019",
"Aruba","ABW","Population, total","SP.POP.TOTL","54211","55438","56225","56695","57032","57360","57715","58055","58386","58726","59063","59440","59840","60243","60528","60657","60586","60366","60103","59980","60096","60567","61345","62201","62836","63026","62644","61833","61079","61032","62149","64622","68235","72504","76700","80324","83200","85451","87277","89005","90853","92898","94992","97017","98737","100031","100834","101222","101358","101455","101669","102046","102560","103159","103774","104341","104872","105366","105845","",
"
@pascalwhoop
pascalwhoop / occurences.txt
Last active February 28, 2020 15:57
0 161769
1 16771
2 1397
3 315
4 167
5 123
6 75
7 41
8 46
9 33
@pascalwhoop
pascalwhoop / state.json
Created February 17, 2020 08:54
"20200217:0.hist": ".........................................................................................................................................................................................................................................................OIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII",
"20200217:0.last": "248",
"20200217:0.ooi": "247",
"20200217:0.ooo": "1",
"20200217:0.total": "497",
"20200217:1.hist": "...................................................................................................................................................................................................................................................................................................................................................................................
@pascalwhoop
pascalwhoop / both_runs.log
Last active January 14, 2020 10:02
[pascalwhoop@pascalwhoop-xps example]$ cd a
[pascalwhoop@pascalwhoop-xps a]$ terraform init
Initializing modules...
Initializing the backend...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands