Skip to content

Instantly share code, notes, and snippets.

View hsm207's full-sized avatar

Shukri hsm207

View GitHub Profile
@hsm207
hsm207 / bert_pretraining_share.ipynb
Created October 1, 2019 23:50
BERT_pretraining_share.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@hsm207
hsm207 / 01_create_bundle.ipynb
Created November 14, 2020 08:08
Zipline futures bug
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@hsm207
hsm207 / multicondition.ipynb
Created May 7, 2020 16:16
multi condition column in pandas
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@hsm207
hsm207 / cleanup.sh
Created April 28, 2020 21:39
Cleanup after playing with EKS and kubeflow
#!/bin/bash
# delete the jupyter deployment
kubectl delete -f /tmp/deploy.yaml
# delete the FSx filesystems
kubectl delete -f /tmp/claim.yaml
kubectl delete -f /tmp/claim-kf.yaml
kubectl delete -f /tmp/storage-class.yaml
kubectl delete -k github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=master
@hsm207
hsm207 / fsx_pvclaim_kf.yaml
Created April 28, 2020 20:45
FSx on Kubeflow
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fsx-claim-kf
namespace: anonymous
spec:
accessModes:
- ReadWriteMany
storageClassName: fsx-sc
resources:
@hsm207
hsm207 / install_kubeflow.sh
Last active April 28, 2020 20:37
Script to install kubeflow on EKS
#!/bin/bash
# based on https://www.kubeflow.org/docs/aws/deploy/install-kubeflow/
# install the the latest kfctl
LATEST_KFCTL=`curl -s https://api.github.com/repos/kubeflow/kfctl/releases |
jq ".[0].assets | map(select(.name | contains(\"linux\"))) | .[0].browser_download_url" |
tr -d '"'`
echo "Downloading kfctl from" $LATEST_KFCTL
@hsm207
hsm207 / jupyter_with_fsx.yaml
Created April 28, 2020 19:07
Deploy jupyter container with fsx mounted
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: jupyter
name: jupyter
spec:
replicas: 1
selector:
@hsm207
hsm207 / modify_deploy.yaml
Created April 28, 2020 19:00
Modify a deployment from a template
kubectl create deployment --image=jupyter/tensorflow-notebook:latest jupyter --dry-run -o yaml > /tmp/deploy.yaml
code-insiders /tmp/deploy.yaml
kubectl apply -f /tmp/deploy.yaml
@hsm207
hsm207 / setup_storage.sh
Created April 28, 2020 18:33
Setup dynamic FSx lustre on EKS
#!/bin/bash
# from https://aws.amazon.com/blogs/opensource/using-fsx-lustre-csi-driver-amazon-eks/
# and
# from https://github.com/kubernetes-sigs/aws-fsx-csi-driver
pushd /tmp
# create an IAM policy to allow FSx use
cat >policy.json <<EOF
{
@hsm207
hsm207 / deploy_jupyter_fast.sh
Created April 28, 2020 17:28
Launch jupyter notebook in k8s
kubectl create deployment --image=jupyter/tensorflow-notebook:latest jupyter