Convert fastq to Parquet with zstd compression via duckdb

duckdb-to-parquet.sh

#!/bin/bash

echo "converting FASTQ to tab-delimited text format, one read per line..."
dsh-bio fastq-to-text -i seqkit-benchmark-data/dataset_C.fq -o seqkit-benchmark-data/dataset_C.txt

echo "dataset_C.txt:"
head -n 2 seqkit-benchmark-data/dataset_C.txt

echo "CREATE TABLE reads(description VARCHAR, sequence VARCHAR, quality VARCHAR);" > convert.sql
echo "COPY reads FROM 'seqkit-benchmark-data/dataset_C.txt' (AUTO_DETECT TRUE);" >> convert.sql
echo "COPY reads TO 'dataset_C-zstd.parquet' (FORMAT 'PARQUET', CODEC 'ZSTD');" >> convert.sql

echo "converting text format to Parquet with zstd compression via duckdb..."
duckdb dataset_C.duckdb < convert.sql

echo "file sizes:"
du -h dataset*

$ ./duckdb-to-parquet.sh
converting FASTQ to tab-delimited text format, one read per line...
dataset_C.txt:
K00137:236:H7NLVBBXX:6:1126:29721:23241 1:N:0	TGGTAGGGAGTTGAGTAGCATGGGTATAGTATAGTGTCATGATGCCAGATTTTAAAAAAAATACTGGAGACAGTCAGCTTATTTATCAGAAAGGTTTATT	```eeiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiieiiiiiiiiiiiiiiiiiiiieiiiii
K00137:236:H7NLVBBXX:6:1117:32410:45906 1:N:0	NCATTCATTATCTCAGCACCGGCATCACGCACGCGGTCTACATAACGGCCCGGCTCGGCCACCATCATGTGGACATCCAGAGGTTTTTCGGCAATGGTGC	B``eeiiiiiiiiiiiieiiiiiiiieiiiiiiiiiiiiiiiiiiiiiiiieii`i`eii[eiiiieVeeieiii[`ei``e[L``eiiiii`i`i`[ei
converting text format to Parquet with zstd compression via duckdb...
100% ▕████████████████████████████████████████████████████████████▏
100% ▕████████████████████████████████████████████████████████████▏
file sizes:
465M	dataset_C-zstd.parquet
608M	dataset_C.duckdb