Skip to content

Instantly share code, notes, and snippets.

@sergiitk

sergiitk/shell-tools.md

Last active April 4, 2018 20:45
Show Gist options
  • Save sergiitk/2493fc4dff059e5037c4 to your computer and use it in GitHub Desktop.
Save sergiitk/2493fc4dff059e5037c4 to your computer and use it in GitHub Desktop.
Download ZIP
Useful shell tools
Raw
shell-tools.md

Useful shell tools (BSD)

Strings

  • l | tail +2 — Skip first line of stdout
  • l | head -n-1 — Skip last line of stdout (GNU version only)
  • echo "test" | head -c-1 — Remove last newline
  • comm -12 <(echo 1; echo 2) <(echo 1; echo 2; echo 3) — Shows intersection (here 1 2).
  • cat template.txt | envsubst — substitutes environment variables in file
  • echo 123abc | od -c -t d1 — Show decimal dump
  • echo 123abc | xxd — Show a hexdump (or do the reverse)
  • echo 123abc | hexdump -C — Show canonical hexdump
  • echo $(tput setaf 1)Text — Text in red using tput
  • p *.txt | xargs -n1 -I{} sh -c 'iconv --from-code=CP1251 --to-code=UTF-8 {} > ../fixed/{}' — Convert bunch of txt files to UTF-8.
  • jot -r 100 — print sequential or random data: 100 random numbers
  • rs — Column from nl separated data

Date

  • gdate --date "Jan 01 2031 -1 sec" — Fri Dec 31 23:59:59 1999
  • gdate -u --date "Jan 01 2000" +%s — Unix timestamp in UTC at midnight, Jan 1 2000
Xpath
  • Slower (23.17s) — xpath PC_COM_DK "/articles/article[1]"
  • Faster (0.31s), all inline — xmllint --xpath "/articles/article[1]" PC_COM_DK
  • Select attribute, newline each — xmlstarlet sel -t -m "/articles/article/package" -m "@relationtype" -v . -n PC_COM_DK
  • Select nodes, newline each — xmlstarlet sel -t -m "/articles/article/package" -c . -n PC_COM_DK

Files and directories

Tools

  • rmdir -p /tmp/test — try to remove /tmp/test directory; if successful, try to remove /tmp; continue until an error.
  • lsof ~/.Trash/*(.) — Show what apps are blocking files in the trash
  • time dd if=/dev/zero bs=1024k of=$TMPDIR/tstfile count=1024 — Bench write speed (B/s)
  • time dd if=$TMPDIR/tstfile bs=1024k of=/dev/null count=1024 — Bench read spead (B/s)

Size

  • du -hd0 src — display disk usage for one folder, guess the unit
  • du -hs src — the same, but in 1048576-byte (1-Mbyte) blocks.
  • du -hd0 .*(/) | gsort -h -r — Size sorted list of all dot directories
  • _ gdu --si --max-depth=1 ~o/ — GNU: Calculate in powers of 1000 each folder size in /opt/local
  • _ gdu --si --max-depth=1 ~o/ | gsort -h -r — GNU: The same, but sorted my size

FS links

  • ln -s [real] [link-name] — Create link
  • ln -nvfs [real] [link] — Edit link

Descriptiors

  • man fd — The files /dev/fd/0 through /dev/fd/# refer to file descriptors which can be accessed through the file system.
  • Standard streams: in the C programming language, the standard input, output, and error streams are attached to the existing Unix file descriptors 0, 1 and 2 respectively.
  • print -u2 Hello — Print to fd2 (stderr). ZSH, KSH only

lsof

  • _ lsof | grep Xcode.SDK.iPhoneSimulator.5.1-5.1.dmg — Show who has opened the file

Archives

  • unzip ~/Downloads/wordpress-3.6-ru_RU.zip -d /tmp - Unzip to specific dir
  • zip -e sergii.zip sergii.keychain — Zip with password

Processes

Tools

  • man 3 signal — Signals documentation
  • ps -x -o etime,nice,command | tail -n +2 | sort -k1n — Show latest processes
  • nice -n20 mysqldump — Set lowest pririty to the process
  • nohup jconsole — invoke a utility immune to hangups, redirect out to nohup.out
  • kill — Terminate or signal a process
    • man 3 signal — Signails numbers, codes, default actions with explanation
    • kill -l — Signails list
  • { time ( ./psnr-old.sh) } 2>&1 | tee out-psnr-old.log — Time a script, print it stdout+stderr and save them to a log file

Jobs

  • jobs — List user's job
  • fg — Switch current job to the foreground
  • bg — Switch current job to the background
  • disown — Detach current job from your terminal
  • wait %1 && say 'It is done!' — Run command after current job is finished

Users

  • who — Display who is logged in
  • w — Display who is logged in and what they are doing
  • last -30 root — Display last 30 logins of root
  • last | grep -Pv "^($USER|reboot|shutdown) " - Show non-user and non-system logins
  • ac -p — Display accumulated connect time (h) for all logins to the standard output
  • ac -pd root — Display root's connect times in 24 hour chunks

Networking

tcpdump

  • _ tcpdump -qpi en1 tcp and src host 192.168.1.10 — Sniff all outgoing traffic
  • _ tcpdump -s0 -A -q -npi lo0 tcp and host 127.0.0.1 and port 143 — Sniff local imap connections. Look for AUTHENTICATE.PLAIN and decode base64 :)
  • _ tcpdump -s0 -A -q -npi eth0 tcp and host 172.20.0.36 and dst host not sergiis-mbp.jysk.com and port not ssh
Arguments

DNS

  • dig +short google.com — Show host IPs list
  • nslookup -query=mx google.com — Show domain MX records
  • host -v google.com — Shows all DNS records

curl

  • curl https://google.com 2> /dev/null | head — Suppress newline when piping curl
  • curl -L — Follow Location header
  • curl -O -J — Save out to file provided in "Content-Disposition" header

wget

  • echo ca-certificate=/opt/local/etc/openssl/cert.pem >> ~/.wgetrc — Set custom root certificates file
  • echo content-disposition=on >> ~/.wgetrc — Set honoring file name from content disposition header
  • wget -O out.html, wget --output-document=out.html — Write result to out.html
  • wget -N, wget --timestamping — Override existing file if newer provided
  • wget -P $TMPDIR, wget --directory-prefix $TMPDIR — Save file to temp dir
  • wget --content-disposition — Support for "Content-Disposition" headers
  • wget -e robots=off — The only way to not download robots.txt file itself
  • wget -r --no-host-directories --no-parent --cut-dirs=2 http://nginx.org/en/docs/ — Recursive download site directory, do not follow links upper, keep directory structure, but only from level 2
  • wget -r --no-host-directories --convert-links --adjust-extension http://hyperpolyglot.org — Recursive download site root, convert links in html and adjust extension to index html files which represent "directory"
  • wget --page-requisites --adjust-extension --no-directories -e robots=off https://sites.google.com/site/gson/gson-user-guide — Download one page with styles
  • wget -r --no-host-directories --convert-links --adjust-extension --level=1 --span-hosts --domains="hyperpolyglot.org,hyperpolyglot.wdfiles.com,static.wikidot.com" --reject=gif -e robots=off http://hyperpolyglot.org --directory-prefix=hyperpolyglot.org

rsync

  • /usr/bin/keychain -q /home/deploy/.ssh/id_rsa; source /home/deploy/.keychain/emeadkhovapp15-sh;
  • ssh 10.238.81.13 find -P ~/proddump/ -type f -name '*.sql.gz' -mtime -30 -printf '%f\\n' | rsync -rt -e ssh --delete-excluded --delete-after --include-from=- --exclude='*' 10.238.81.13:proddump/ /home/deploy/backups/jytlwmp0/

mailx

  • Press h — print out active message headers

Manuals

man

  • man -w zsh — Shows zsh man page file path
  • man -aW zsh — Shows all avaliable zsh man page files
  • man /usr/share/man/man1/zsh.1 — Shows exact man file
  • man -d man — Debug information
  • groff -Thtml -man /usr/share/man/man1/zsh.1 > zsh.html — Convert not compressed man page to html
  • groffer --www --www-viewer cat $(man -w zsh) — Find and display any man page as html
  • nroff -man proxy-config.1 — Print man file using nroff
  • apropos zsh, man -k zsh — Search the whatis database for zsh

info

  • info -w emacs — Shows print physical location of Info file

Miscellaneous

  • Insert control character — ctrl+v, ctrl+[char], example echo ctrl+v, ctrl+g rings a bell
  • Watch + awk escape — watch "l | awk '{print \$5}'"
  • Replace spaces with sed, column with custom delimiter — grep -PHni 'catch[^\(]*\([ ]*PDOException' **/*.(inc|php|module) | sed -E 's/:[[:space:]]+/@/g' | column -ts@
  • Pretty print JSON, tab 4 :( — pbpaste | python -mjson.tool | pbcopy

Links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment