Skip to content

Instantly share code, notes, and snippets.

View pascalwhoop's full-sized avatar

Pascal Bro pascalwhoop

46 followers · 33 following
  • Zoom
View GitHub Profile
@pascalwhoop
pascalwhoop / add_users_in_bulk.py
Created June 24, 2024 11:41
Bulk addition of people to specific teams in an org, invites them to org if not yet part of it.
#!/usr/bin/python
# requires typer, tqdm, requests, gh cli
import typer
from pathlib import Path
from tqdm import tqdm
import subprocess
app = typer.Typer()
GITHUB_API_URL = "https://api.github.com"
@pascalwhoop
pascalwhoop / bootstrap.sh
Last active January 16, 2023 13:00
bootstrap script for machine learning env
#!/bin/bash
# install core requirements
NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
sudo apt install -y tmux htop bmon fzf wget curl
# get brew working
curl https://gist.githubusercontent.com/pascalwhoop/4d01bb179132118c565b3db56c4ce68e/raw/fa0ab508cac760efaa399442047256a36578c62e/tmux.conf > ~/.tmux.conf
# fish shell
@pascalwhoop
pascalwhoop / slack_message_table_schema.json
Created July 16, 2020 20:43
{
"fields": [
{
"name": "index",
"type": "integer"
},
{
"name": "type",
"type": "string"
},
@pascalwhoop
pascalwhoop / dataflow_job.tf
Created July 8, 2020 20:06
sample pipeline for data cleaning
import apache_beam as beam
import logging
import json
from apache_beam.io import ReadFromText
from apache_beam.io import BigQuerySource
from apache_beam.io import BigQuerySink
from apache_beam.io import WriteToText
from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
from apache_beam.io.gcp.internal.clients import bigquery
@pascalwhoop
pascalwhoop / dataflow_job.tf
Created April 30, 2020 16:46
locals {
foo_directory = "../../../../services/foo/target"
jobName = "company-${var.environment}-datalayer-foo-app-v1"
region = "europe-west1"
//parameters for the java jar
dataflow_parameters = {
runner = "DataflowRunner"
jobName = local.jobName
update = data.external.running_job.result.name == local.jobName ? true : false
project = var.project
@pascalwhoop
pascalwhoop / fix.sh
Created April 26, 2020 20:46
gcloud rsync -r gs://our-bucket /tmp/bucket
cd 0-bootstrap && vim backend.tf #comment out backend
terraform init #approve copying state to local
gcloud rm -r gs://our-bucket #delete all data
terraform apply #force recreates bucket in different region
gcloud rsync -r /tmp/bucket gs://our-bucket #bring state files back
gcloud rm -r gs://our-bucket/terraform/state/bootstrap #clean old state
vim backend.tf #uncomment the backend again
terraform init #copy state back to bucket
@pascalwhoop
pascalwhoop / PR-bootstrap.yaml
Created April 26, 2020 20:18
gcp foundation github actions
name: 'Bootstrap Terraform'
on:
- pull_request
env:
tf_version: 'latest'
tf_working_dir: '0-bootstrap'
GOOGLE_CREDENTIALS: ${{secrets.GOOGLE_CREDENTIALS}}
jobs:
terraform:
name: 'Terraform'
@pascalwhoop
pascalwhoop / populations.csv
Created March 5, 2020 22:09
We can make this file beautiful and searchable if this error is corrected: Unclosed quoted field in line 3.
"Country Name","Country Code","Indicator Name","Indicator Code","1960","1961","1962","1963","1964","1965","1966","1967","1968","1969","1970","1971","1972","1973","1974","1975","1976","1977","1978","1979","1980","1981","1982","1983","1984","1985","1986","1987","1988","1989","1990","1991","1992","1993","1994","1995","1996","1997","1998","1999","2000","2001","2002","2003","2004","2005","2006","2007","2008","2009","2010","2011","2012","2013","2014","2015","2016","2017","2018","2019",
"Aruba","ABW","Population, total","SP.POP.TOTL","54211","55438","56225","56695","57032","57360","57715","58055","58386","58726","59063","59440","59840","60243","60528","60657","60586","60366","60103","59980","60096","60567","61345","62201","62836","63026","62644","61833","61079","61032","62149","64622","68235","72504","76700","80324","83200","85451","87277","89005","90853","92898","94992","97017","98737","100031","100834","101222","101358","101455","101669","102046","102560","103159","103774","104341","104872","105366","105845","",
"
@pascalwhoop
pascalwhoop / occurences.txt
Last active February 28, 2020 15:57
0 161769
1 16771
2 1397
3 315
4 167
5 123
6 75
7 41
8 46
9 33
@pascalwhoop
pascalwhoop / state.json
Created February 17, 2020 08:54
"20200217:0.hist": ".........................................................................................................................................................................................................................................................OIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII",
"20200217:0.last": "248",
"20200217:0.ooi": "247",
"20200217:0.ooo": "1",
"20200217:0.total": "497",
"20200217:1.hist": "...................................................................................................................................................................................................................................................................................................................................................................................