Skip to content

Instantly share code, notes, and snippets.

@mehd-io

mehd-io/csv_to_parquet.sh

Created March 21, 2023 16:23
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mehd-io/7978fb606845f07a2142fb0644e9d818 to your computer and use it in GitHub Desktop.
Save mehd-io/7978fb606845f07a2142fb0644e9d818 to your computer and use it in GitHub Desktop.
Download ZIP
Convert CSV to Parquet using DuckDB CLI
Raw
csv_to_parquet.sh
#!/bin/bash
# You can put this in your .bashrc or .zshrc
function csv_to_parquet() {
file_path="$1"
duckdb -c "COPY (SELECT * FROM read_csv_auto('$file_path')) TO '${file_path%.*}.parquet' (FORMAT PARQUET);"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment