Skip to content

Instantly share code, notes, and snippets.

@pascalwhoop
pascalwhoop / cleanup.sh
Created November 20, 2019 14:08
Cleans the Kubernetes API of all pods in a namespace (in our case airflow)
NAMESPACE=airflow
#edit the > 3 < in the jq statement to correspond to the number of days you want to keep pods around for
kubectl delete pod -n $NAMESPACE $(kubectl get pods -n $NAMESPACE -o json | jq -r '.items[] |
select(.status.phase != "Running") |
select(last(.status.containerStatuses)[].state | has("terminated")) |
select((last(.status.containerStatuses)[].state.terminated.finishedAt | fromdate) < (now - 60*60*24*3)) |
.metadata.name')
@pascalwhoop
pascalwhoop / clone_all_github.sh
Created October 28, 2019 20:28
simple script using git and jq to clone all repos (below a decent threshold of 200)
url=https://api.github.com/orgs/$1/repos?per_page=200
all=$(curl $url | jq -r ".[].full_name)
for r in $all
do
git clone "https://github.com/$r.git"
done
mport unittest
import apache_beam as beam
import cleanup_pipeline as cp
ec = cp.ElementCleanup()
sample_in = [
{
"battery_status": "12",
"bluetooth_status": "on",
def run():
with beam.Pipeline(options=options) as p:
raw_values = (
p
| "ReadTable" >> beam.io.Read(source)
| "cleanup" >> beam.ParDo(ElementCleanup())
| "writeTable" >> beam.io.Write(target)
)
# pipeline
# parDo for all values in PCollection: process
import apache_beam as beam
import json
from apache_beam.io import ReadFromText
from apache_beam.io import BigQuerySource
from apache_beam.io import BigQuerySink
from apache_beam.io import WriteToText
from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
from apache_beam.io.gcp.internal.clients import bigquery
from apache_beam.options.pipeline_options import PipelineOptions
image:
name: hashicorp/terraform:0.12.8
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
before_script:
## Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | ssh-add -

Keybase proof

I hereby claim:

To claim this, I am signing this object:

@pascalwhoop
pascalwhoop / custom_style_youtube_distract_free.css
Created January 21, 2019 15:02
A quick gist that removes all the noise from youtube
.ytd-browse{
display: none;
}
.ytd-watch-next-secondary-results-renderer{
display:none;
}
.ytd-comments{
display:none;
}
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.