Skip to content

Instantly share code, notes, and snippets.

View martin-g's full-sized avatar
🏠
Working from home

Martin Grigorov martin-g

🏠
Working from home
246 followers · 53 following
View GitHub Profile
@martin-g
martin-g / duckdb-top-x86_64-recipes.sql
Created March 21, 2024 09:59
CREATE OR REPLACE TABLE stats AS
SELECT * FROM read_csv('./packages.tsv',
delim = '\t',
header = true,
columns = {
'package': 'VARCHAR',
'total': 'INTEGER'
})
WHERE package NOT LIKE 'bioconductor-%'
AND package NOT LIKE 'perl-%'
@martin-g
martin-g / collect-non-aarch64-recipes.sh
Created March 21, 2024 09:57
Generates a tab separated values file with all Bioconda recipe names and their aarch64/noarch status
#!/usr/bin/env bash
# set -x
RECIPES_PATH=${1:-recipes}
AARCH64_STATUS_FILE="/home/martin/git/bioconda/bioconda-stats/package-downloads/anaconda.org/bioconda/aarch64-status.tsv"
rm -f ${AARCH64_STATUS_FILE}
echo -e "package\thas-aarch64\tnoarch" > ${AARCH64_STATUS_FILE}
@martin-g
martin-g / Rust Optimization.md
Created July 21, 2022 13:54 — forked from jFransham/Rust Optimization.md
@martin-g
martin-g / my-podgroup-template.yaml
Created March 25, 2022 07:54
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
spec:
minMember: 1
minResources:
cpu: "5"
memory: "7Gi"
priorityClassName: medium
queue: my-queue
@martin-g
martin-g / run-spark-pi-3.3-SNAPSHOT-on-k8s-volcano.bash
Created March 24, 2022 12:44
Bash shell script to run Spark Pi example on Volcano using PodGroup template
#!/usr/bin/env bash
K8S_NAMESPACE="spark-on-k8s"
K8S_SERVICE_ACCOUNT_NAME="spark-account-name"
K8S_CLUSTER_ROLE="spark-on-k8s-cluster-role"
CURRENT_CONTEXT=(kubectl config view -o jsonpath="{.current-context}")
K8S_MASTER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$CURRENT_CONTEXT\")].cluster.server}")
@martin-g
martin-g / my-queue.yaml
Last active March 25, 2022 07:53
Volcano Queue. See https://volcano.sh/en/docs/queue/
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
name: my-queue
spec:
weight: 1
capability:
cpu: "6"
memory: "8096Mi"
@martin-g
martin-g / priority-classes.yaml
Created March 24, 2022 12:31
A list of PriorityClass-es to be created by an admin. See https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high
value: 100
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: medium
@martin-g
martin-g / run-spark-pi-3.3-SNAPSHOT-on-k8s.bash
Created March 18, 2022 10:19
#!/usr/bin/env bash
DRIVER_CORES=1
DRIVER_MEM=2048m
EXECUTOR_INSTANCES=3
EXECUTOR_CORES=2
EXECUTOR_MEM=2048m
SPARK_HOME="/path/to/spark-3.3.0-SNAPSHOT-bin-with-volcano" # CHANGE 1
PATH="$JAVA_8_HOME/bin:$SPARK_HOME/bin:$PATH"
@martin-g
martin-g / create-docker-image.bash
Created March 18, 2022 09:10
#!/usr/bin/env bash
BASE_OS="openeuler/openeuler:20.03-lts-sp3"
SPARK_VERSION="3.3.0-SNAPSHOT"
SCALA_VERSION="scala_2.12"
DOCKERIMAGETAG="11-jre-slim"
cd $SPARK_HOME
./bin/docker-image-tool.sh \
-t ${SPARK_VERSION}-${SCALA_VERSION}-${DOCKERIMAGETAG} \
@martin-g
martin-g / successful-logs-spark-3.2.1-on-k8s.txt
Created March 15, 2022 13:04
22/03/15 13:01:05 INFO LoggingPodStatusWatcherImpl: State changed, new state:
pod name: spark-on-k8s-app-b336777f8daa93b6-driver
namespace: spark-on-k8s
labels: spark-app-selector -> spark-9e850f59cfa54f77bec1a2f30f4753a7, spark-role -> driver
pod uid: 48088478-9896-4019-b263-47866214cbb9
creation time: 2022-03-15T13:00:41Z
service account name: spark-account-name
volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-slmsc
node name: minikube
start time: 2022-03-15T13:00:41Z