Skip to content

Instantly share code, notes, and snippets.

@MartyLake
Created September 19, 2017 12:23
Show Gist options
  • Save MartyLake/4006d3928c09e9ec7a49b949e86185fe to your computer and use it in GitHub Desktop.
Save MartyLake/4006d3928c09e9ec7a49b949e86185fe to your computer and use it in GitHub Desktop.
filters a big file quickly with bash
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
input_file="in.csv"
output_file="out.csv"
NUMOFLINES=$(wc -l < ${input_file})
it=0
if [ -f ${output_file} ]
then
rm ${output_file}
fi
while IFS= read line
do
it=$((it+1))
# display $line or do somthing with $line
filtered=$(echo "$line" | cut -d '"' -f2)
echo "$filtered" >> ${output_file}
(( it % 500 == 0 )) && echo "Iteration $it/$NUMOFLINES"
done <"$input_file"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment