| Thing I want to do | Command |
|---|---|
| overwrite file | > |
| append to file | >> |
| search in file | grep '??' file.csv |
| search in file (case insensitive) | grep 'LLC' -i file.csv |
| search in file (show line numbers) | grep ',llc,' -n file.csv |
| remove header | tail -n +2 file.csv > new_file.csv |
| remove header | sed 1d file.csv > new_file.csv |
| add header | { head -1 with_header.csv; cat headerless.csv;} > new_file.csv |
| delete line with pattern | sed '/pattern/d' file.csv > new_file.csv |
| delete line with pattern inline | sed -i.bak '/pattern/d' file.csv > new_file.csv |
| delete line number | sed -e '100d' file.csv |
| print line number | sed -n 100p file.csv |
| print range of line numbers | sed -n 98,102p file.csv |
| cut a set of columns from a csv with a pipe delimiter | `cat file.csv |
| clean a csv file | csvclean file.csv |
| change a delimter (comma to colon) | sed 's/,/:/g' file.csv > out.csv |
| change a tsv to a csv | `cat file.tsv |
| split a file by 1000 lines | split -l 1000 file.csv |
| find similar lines between two files | fgrep -x -f file1.csv file2.csv |
| find difference between two files | grep -Fxvf file1 file2 |
| unzip a chunk of a gz file | `gunzip -c input.gz |
| delete files that are empty | `find -type f -size 0 -print0 |
| loop over files | for f in *.csv; do cat $f; done |
| fetch remote file | scp remote:/home/ubuntu/stuff.txt ~/Downloads/thing.txt |
-
csvkit - A suite of utilities for converting to and working with CSV
-
jq - jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data
-
awscli - CLI for working with AWS.
-
[Explain Shell] (http://explainshell.com/)