#bash tricks

bash.md

Output intermediate results to terminal using tee

TLDR; tee /dev/tty

Use the tee /dev/tty command in between pipes. This outputs the current STDIn to the terminal. This helps you to confirm the result in between pipes.

$ echo "hello" | sed -e 's/o//g' | tee /dev/tty | sed -e 's/hell/heaven/g' hell heaven

Corrupt a gzip file

TLDR; dd if=/dev/urandom bs=1024 count=1 of=bad.tar.gz

For testing purposes it can be useful to purposefully corrupt a GZIP. Use the command below and replace "of=bad.tar.gz" with "of=location_of_gzip"

Count size and convert to MB

TLDR; awk '{ total += $column-number }; END {printf("%.2f MB\n",total/1024^2)}'

By summing the file sizes and converting the bytes to MB by using this command (choose the right column for the size):

awk '{ total += $column-number }; END {printf("%.2f MB\n",total/1024^2)}'

Trim whitespaces at the beginning/end of the line

TLDR; cat file1 | xargs -I% bash -C "echo % >> cleaned_file1".

It can be hard to see if there are whitespace characters in your text files, you could do wc -l filename and see if that matches the actual number of characters.

Find common/different lines in filenames

TLDR; comm -1 -2 <(sort file1) <(sort file2)

Note that you have to sort both files and they lines have to match.

comm -1 -2 <(sort file1) <(sort file2) - display common lines in files

comm -2 -3 <(sort file1) <(sort file2) - display lines only in file 1

comm -1 -3 <(sort file1) <(sort file2) - display lines only in file 2

Display size and basename

TLDR; awk '{printf("%s ",$1); system("basename " $2)}' file.txt

e.g. 12893472 file.txt

Grep match on multiple words

TLDR; awk '{print $1"*.*"$2) | xargs -I % grep % source

If you want to grep a combination of words from an input, e.g. size and filename you can pipe the combination to awk and combine the arguments with "." so that grep can regex match for multiple keywords in a source.

find file size and basename for comparison

TLDR; find dir "regex" -exec du -s {} \; awk '{printf("%s ",$1); system("basename " $2)}

When using find to search for local files, these do not include sizes and have to full name. In order to compare files between different folders, we need both the size and the basename.

finding all lines in a sorted list after specific name

TLDR; grep -i -A 1000 "pattern" <(input_file) | grep -v "pattern"

It will find the next 1000 lines, so increase this accordingly (there is no max).

Outputting interactive programs to a file

TLDR; script -q -c "command" output

For some commands, like "top" or other interactive programs, it is not possible to just export/dump the results to a file/stdout with command > output. This is because the result is "in motion".