Skip to content

Instantly share code, notes, and snippets.

@tahashmi
tahashmi / arrow_flight_dremio.py
Last active February 9, 2023 20:36 — forked from koolay/arrow_flight_dremio.py
demo of arrow-flight+dremio+vaex
from collections import namedtuple
import vaex
import time
import orjson
import os
import psutil
from pyarrow import flight
import pyarrow as pa
@tahashmi
tahashmi / install.def
Created December 11, 2022 22:34
bio tools
BootStrap: debootstrap
OSVersion: bionic
MirrorURL: http://us.archive.ubuntu.com/ubuntu/
%runscript
echo "This is what happens when you run the container..."
%post
@tahashmi
tahashmi / log.md
Last active June 3, 2021 22:12
BWA multi-threaded vs. multiple-instances (with single thread each) performance comparison

Download reference genome:

wget ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz 
gunzip GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz 
mv GCA_000001405.15_GRCh38_no_alt_analysis_set.fna GRCh38.fa

Index reference genome:

bwa index GRCh38.fa
@tahashmi
tahashmi / vcfmerge.sh
Last active May 31, 2021 02:41
Merge VCFs
#!/bin/bash
#Usage: ./vcfmerge.sh {PATH FOR VCFs} a/b/c/
PICARDPATH=/usr/local/bin
STR=""
for f in ${1}*.dv.vcf.gz
do
if [[ $f != *.dv_merged.vcf.gz ]]
@tahashmi
tahashmi / Singularity
Created August 29, 2020 05:38
Singulariy file for scziDesk
BootStrap: debootstrap
OSVersion: trusty
MirrorURL: http://us.archive.ubuntu.com/ubuntu/
%runscript
echo "This is what happens when you run the container..."
%post
echo "Hello from inside the container"
@tahashmi
tahashmi / sparksingularity.sh
Created August 6, 2020 14:16
Setting up a sample Spark cluster based on Singularity container on Cartesius Surfsara Server
#!/bin/bash
#SBATCH -N 5
#SBATCH -t 1:10:00
#SBATCH --ntasks-per-node 1
#SBATCH --cpus-per-task 16
#SBATCH --output=sparksingu.out
#SBATCH --mem=50G
############################
#For custom configurations python3.6 and Java path
@tahashmi
tahashmi / Singularity
Created August 6, 2020 14:09
Creating Singularity container with Apache Arrow and Spark
BootStrap: debootstrap
OSVersion: bionic
MirrorURL: http://us.archive.ubuntu.com/ubuntu/
%runscript
echo "This is what happens when you run the container..."
#/home/tahmad/bulk/images/adam.sh
%post
@tahashmi
tahashmi / master.sh
Last active August 6, 2020 14:17
Setting up a sample Spark cluster based on Singularity container on Cartesius Surfsara Server (To run SparkGA2)
#!/bin/bash
#SBATCH -N 5
#SBATCH -t 1:10:00
#SBATCH --ntasks-per-node 4
#SBATCH --cpus-per-task 4
#SBATCH --output=master6.out
#SBATCH --mem=50G
############################
module load pre2019