reverse-i-search: A command line history archive (because my memory just aint that good)

Amazing-CLI-Archive.md

My personal reverse-i-search: A command line history lookup (because my memory just aint that good)

bash.md

Bash

Files

Rename a group of files with one command

for file in *.png; do mv "$file" "${file%.png}_512x512.png"; done

Bash has an extensive set of variable expansion options. The one used here, '%', removes the smallest matching suffix from the value of the variable. The pattern is a glob pattern, so ${file%.} would also work. The '%%' operator removes the largest matching suffix, and is interchangeable in the example above. As the pattern is fixed, ${file%%.}.html would turn a.b.htm into a.html though.

Delete files matching a pattern recursively

Them .DS_Store files

You know those .DS_Store files that someone commited to a source tree instead of the .gitignore? (That annoys the $#!+ out of me). This will delete all that it can find.

find . -type f -name .DS_Store -exec rm {} +

Delete files older than X days, minutes etc

Want to delete old log files after 7 days (-ctime +7) or perhaps some csv files uploaded to your ftp after 30 minutes (-cmin +30)? Well it is find to the rescue again.

find /home/sftp -type f -name "*.csv" -ctime +2 -exec rm -f {} \;

More info on find and time here: https://www.gnu.org/software/findutils/manual/html_mono/find.html#Time

Sed, Awk & Xargs

Want to kill a bunch of processes you dont like?

Kill those pesky Python Celery process that just seems to go off on their own and never come back. Do it by using grep to get out what you are looking for from the ps command then grab the 2nd 'column' of values using awk and pipe it to the kill command using xargs which will run it for each value that was piped to it.

ps aux --sort=start_time | grep 'celery worker' | grep 'celery' | awk '{print $2}' | xargs kill -TERM

Delete all git branches on remote which have been merged

git branch -r --merged | grep -v master | sed 's/origin\///' | xargs -n 1 git push --delete origin

Nginx (Apache format) log files

top 20 URLs from the last 5000 hits

tail -5000 access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20

top 20 URLS excluding POST data from the last 5000 hits

tail -5000 access.log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20

top 20 IPs from the last 5000 hits

tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20

top 20 URLs requested from a certain ip from the last 5000 hits

IP=1.2.3.4
tail -5000 access.log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20

top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits

IP=1.2.3.4
tail -5000 access.log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20

sum of data (in MB) transferred

cat access.log | awk '{sum+=$10} END {print sum/1048576}'

top 20 referrers from the last 5000 hits

tail -5000 access.log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
tail -5000 access.log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20

list of bad referers

cat access.log | awk '$11 !~ /google|bing|yahoo|yandex|mywebsite.com/' | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000

list user agents

cat access.log | awk '{print $14}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000

exiftool.md

Exiftool

Using Exiftool on the command line cheat sheet

Install on a Mac with brew install exiftool

Shift timestamps

Shift from one point in time to another

This adds 1 hour to all images in a directory - like say you forgot to adjust your camera for daylight savings or you went on a holiday to another timezone...

exiftool -AllDates+=1 dir

The AllDates flag shifts the values of DateTimeOriginal, CreateDate and ModifyDate.

exiftool -AllDates+=1:30 -if '$make eq "Canon"' dir

Shifts forward by 1 hour and 30 minutes for all Canon images in a directory.

Shifting by timezone

Change timezone of all timestamps to Australian Eastern Standard Time

More info on timeshifts here: https://sno.phy.queensu.ca/~phil/exiftool/Shift.html

Geo Coordinates

Add GPS Position

Add latitude and longitude coordinates (for somewhere in Sydney) to all images in a directory

exiftool -XMP:GPSLongitude=151.271305 -XMP:GPSLatitude=-33.891035 -GPSLongitudeRef=East -GPSLatitudeRef=South .

image-magik.md

Image Magick

Command line cheat sheet

Install on a Mac with brew install imagemagick

Resize one file by a percentage

convert airline.png -resize 50%  airline_half.png

Generate new thumbnail files with new suffix in filename

convert orig-*.png -thumbnail 20x20 -set filename:fname '%t_20x20' +adjoin '%[filename:fname].png'

networking.md

Networking

Forgot how to find listening ports again?

netstat -an | grep -i "listen"

List all listening ports and their processes IDs.

lsof -iTCP -sTCP:LISTEN

This will return every TCP connection with the status LISTEN. This reveals all the open TCP ports on your box. It also lists the processes associated with those open ports. This is a significant upgrade over netstat, which lists PIDs at most.

sudo lsof -i -u^$(whoami)

Returns all connections not owned by the currently logged-in user. The caret ^ is used for negation. Anything matching the text after the caret will be removed from the results.

lsof -nP -iTCP@localhost:513

Lists all the TCP connections with the hostname localhost and the port 513. It will also run lsof without connecting names to IP addresses and ports, making the command run noticeably faster.