Skip to content

Instantly share code, notes, and snippets.

@tomaslibal

tomaslibal/command-line.md

Last active June 19, 2018 08:13
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tomaslibal/d8f19c74964dab5a89e0 to your computer and use it in GitHub Desktop.
Save tomaslibal/d8f19c74964dab5a89e0 to your computer and use it in GitHub Desktop.
Download ZIP
Collection of commands for CLI (mostly Bash), and settings of my environment (.dot files in my home directory).
Raw
command-line.md

Command Line snippets, info and settings

IO redirection

There are three file descriptors open by default:

  • 0 for stdin
  • 1 for stdout
  • 2 for stderr

Per-line settings (temporar redirection)

# stdout to a file
ls /usr 1>out.txt

# stderr to a file
ls /usr 2>out.txt

# both stdout and stderr to a file
ls /usr &>out.txt

Truncate a file. The colon is optional (in BASH) but in some shells required.

: > truncate.me

Prompt for delete confirmation for each file, i.e. the interactive mode of rm, store in .bashrc:

alias rm='rm -i'

Verify integrity of files

Sha256 with Openssl

cat $filename | openssl dgst -sha256
openssl dgst -sha256 <filename>

Less secure md5

md5 $filename

Encrypt/decrypt files with openssl

Encrypt (you'll be asked to choose a password)

# -a will be base64 encoded so it can be pasted as ASCII
openssl aes-256-cbc -a -salt -in <file.zip> -out <output.enc>

Decrypt

# -d for decrypt
openssl aes-256-cbc -d -a -in <input.enc> -out <origianlfile>

Cmd line arithmetics

Round a floating point number $f to $n decimal points.

printf "%.*f" $n $f

Use bc for arithmetics. Add -l (--mathlib) for the standard math library to be included.

bc -l <<< "42 / 12"

Sum a series of numbers.

awk '{i+=$1} END {print i}' data_series.txt

AWK uses dynamic variables and i is zero by default, see AWK's man page

Uninitialized variables have the numeric value 0

Reading files

Read file by line and split the line of whitespace into columns

while read -r col1 col2 ...
do
    echo $col1
done < $FILENAME;

Bash functions

Basic function with two arguments and the invocation of the said function.

function foo {
    FIRST_ARG=$1
    SECOND_ARG=$2
}

foo "abc" "xyz"

Check if script file run as root

#!/bin/bash

if [ "$EUID" -ne 0 ]; then
    echo "Please run as root"
    exit 1
fi

Bash exit on error or an unset variable

# e: exit on error
# u: exit on unset variable
set -eu

Processes

# Print file descriptors in use by a process
ls -l /proc/${PID}/fd
# Print open network connections of a process, and how to close a selected one below
# lsof -n -i 4 -a -p ${PID}
COMMAND   PID FD   TYPE   DEVICE SIZE/OFF NODE NAME
vlc     26133  3u  IPv4 21652774      0t0  UDP 192.168.1.1:46833->192.168.1.2:2001
vlc     26133  4u  IPv4 21652775      0t0  UDP 192.168.1.1:46834->192.168.1.2:2002
vlc     26133  5u  IPv4 21652776      0t0  UDP 192.168.1.1:48852->192.168.1.3:2001
vlc     26133  6u  IPv4 21652777      0t0  UDP 192.168.1.1:48853->192.168.1.3:2002
# Issuing the command 'close(3)' will close the file descript 3u (3=number, u= unsigned?) if done as below using gdb:
# echo -e "call close(3)\nquit" > gdb_commands
# gdb -p 26133 --batch -x gdb_commands

Process states

> man ps|grep -A 20 'output specifiers' 
       Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a
       process:

               D    uninterruptible sleep (usually IO)
               R    running or runnable (on run queue)
               S    interruptible sleep (waiting for an event to complete)
               T    stopped, either by a job control signal or because it is being traced
               W    paging (not valid since the 2.6.xx kernel)
               X    dead (should never be seen)
               Z    defunct ("zombie") process, terminated but not reaped by its parent

       For BSD formats and when the stat keyword is used, additional characters may be displayed:

               <    high-priority (not nice to other users)
               N    low-priority (nice to other users)
               L    has pages locked into memory (for real-time and custom IO)
               s    is a session leader
               l    is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
               +    is in the foreground process group

Search

# Search in files, get the line number reference:
grep -Hn '8080' server.js | cut -f1,2
# -iname = ignore case mask
# -name = case sensitive mask
# -type f: file, d: directory
find /search/under/here -iname *.js -type f

sed stream editor

regex search, similar to grep

sed '/yourregex/!d' foo.log

replace string

sed 's/matchthis/replacewiththis/g' bar.txt

Replace multiple strings with the same string:

sed 's/one|two|three/four/g' bar.txt

Replaces any one, two, or three with four

Use sed as a filter in find

find ./src -type f -exec sed 's/\t/    /g'

tr translate characters

Change CRLF file to LF

tr -d '\r' < input.txt > output.txt

JSON in cmd line

I use the jq program

cat package.json | jq .author.name
"Foo Bar"

SSH

Change permissions to your key and .ssh directory:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 600 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/id_rsa.pub

From http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html/Translation_Quick_Start_Guide/sect-Translation_Quick_Start_Guide-Accounts_and_Subscriptions-Making_a_SSH_Key.html

Display file permission in octal format:

unix:     stat -c '%a' ~/.ssh/id_rsa.pub

sshd

sshd config files are usually in /etc/ssh/sshd_config, restart the daemon after changing the config by sudo service ssh restart

client config for ssh is usually in /etc/ssh/ssh_config. to use ssh key login to from hosts user1@A to user1@B, add A's public key to B's authorized keys file, which is configured in sshd_config and is usually /home/user1/.ssh/authorized_keys. The authorized keys file must be readable only by root and no one else (chmod 600 ~/.ssh/authorized_keys).

Escape characters

While connected to a host through ssh, the default escape character "~" can be used with some of these commands (more available):

  • ~? to display the list of escape characters
  • ~. to disconnect

Git

Auto rebase on pull in ~/.gitconfig

git config --global pull.rebase true

Pretty format git logs

# show only filenames of a commit
git show --pretty="format:" --name-only bd61ad98
git show --pretty="format:" --name-only HEAD~5

Reverting commits

# revert
git help revert
git revert --no-commit REF
git revert             HEAD~3..HEAD  # last three commits

GCC

# Compile a C program that uses MySQL connection
gcc -o output_file $(mysql_config --cflags) source_file.c $(mysql_config --libs)

PROMPT_COMMAND

This environmental variable gets executed in bash every time the prompt receives a command.

if [[ "$TERM" == screen* ]]; then
  do_something_with_screen() {
  }
  PROMPT_COMMAND="do_something_with_screen; $PROMPT_COMMAND"
fi

Networking

Scan address and port using netcat

nc -uvz <address> <port>
  • u for UDP, v for verbose, z for scan only, without sending data

Inspect which ports are listening

sudo netstat -tanp

Restart network manager on system resume

#!/bin/sh
 
# place this in /usr/lib/pm-utils/sleep.d/A_restart-network-manager
# and sudo chmod +x /usr/lib/pm-utils/sleep.d/A_restart-network-manager (give it the same permissions as the other files in the folder have)
 
case "$1" in
    resume)
        service network-manager restart
esac

Restart the network manager.

# restart network manager
sudo service network-manager restart

inspect traffic with ngrep

# see all packets with source or destination 127.0.0.1
$ sudo ngrep host 127.0.0.1

tcpdump on a specific host source + ignore tcp (ipv4) handshake flags

sudo tcpdump -A -n 'src host 127.0.0.1 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

network manager cli nmcli tool

$ nmcli general status
$ nmcli con up id <connectionid> # start connection
$ nmcli con down id <connectionid> # stop connection

kernel ip routing

$ sudo route -n

Add a static route for packets with destination 192.168.1.1 to go through a gateway 4.5.6.7

$ sudo route add -host 192.168.1.1 gw 4.5.6.7

make a request + measure time

Measures the total time, dns lookup, connection time, and time to start transfer

curl -w '\n %{http_code},%{time_total},%{time_namelookup},%{time_connect},%{time_starttransfer} \n' https://example.com

Screen

My ~/.screenrc:

# An alternative hardstatus to display a bar at the bottom listing the
# windownames and highlighting the current windowname in blue. (This is only
# enabled if there is no hardstatus setting for your terminal)
hardstatus on
hardstatus alwayslastline
hardstatus string "%{.bW}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H %{..Y} %m/%d %C%a "
defscrollback 5000

Miscellaneous

# This polls the 'stat' command on a given file and when the output
# of that command differs to the output from the previous reading, 
# it executes the specified commands. 
while true    
do
    # In this case the command is 'stat -t %s ...' that outputs
    # the timestamp in seconds of the file's last change.
    ATIME=`stat -t %s ${FILE_TO_WATCH}`
 
    # when the file is changed, the code branch in the if statement
    # will be executed
    if [[ "$ATIME" != "$LTIME" ]]
    then    
       echo "${FILE_TO_WATCH} has changed"
       bash ${CMDS_TO_EXECUTE}
       LTIME=$ATIME
   fi
   sleep 5
done

Use watch to check the CPU load. NB. Older versions of watch did not support color output.

# Watch the average CPU load 
watch -n 1 "(uptime | awk '{ print \$8 \$9 \$10 }')"

Create a new dir and change into it right away:

mkdir newdir && cd $_

Calendar

cal

See man page for options. Linux's cal can show more than one month but I couldn't find this option in MacOS sierra

BASH environment

To have the settings from .bashrc in a login session of BASH, add the following to ~/.bash_profile:

if [ -f ~/.bashrc ]; then 
    source ~/.bashrc 
fi

My aliases

alias dir='ls -la'
alias ll='ls -alF'

alias gia='git add'
alias giaa='git add *'
alias gis='git status'
alias gil='git log'
alias gid='git diff'
alias gic='git commit -m'
alias branches='git branch -a'

Set color output in BASH (?)

export PS1="\[\033[36m\]\u\[\033[m\]@\[\033[32m\]\h:\[\033[33;1m\]\w\[\033[m\]\$ "
export CLICOLOR=1
export LSCOLORS=ExFxBxDxCxegedabagacad

Git config

Contents of ~/.gitconfig:

[user]
	email = tomas.libal@unrulymedia.com
	name = Tomas Libal
[color]
	ui = true
[alias]
	today = log --since=\"6am\" --author=\"Tomas Libal\"
[push]
	default = simple

Spacemacs

Instead of configuring ~/.vimrc I use Spacemacs, which is essentially an Emacs configuration directory ~/.emacs.d stored in a Git repository:

git clone --recursive https://github.com/syl20bnr/spacemacs ~/.emacs.d

Lynx CLI web browser

Set a custom home page by setting the WWW_HOME env. variable, e.g. in ~/.bashrc:

WWW_HOME='https://duckduckgo.com'
export WWW_HOME

Spider all links from a website

lynx -listonly -image_links -dump ${URL}

Installation on macos with brew (enable HTTPS)

Sometimes brew install lynx will install it without the HTTPS support. To enable it, run brew edit lynx, and add the following line to the ./configure: "--enable-gnutls-compat",. Then uninstall lynx and install it again (this will force the installation to use the updated recipe).

New terminal session

Welcome message

Print the message of the day on opening new terminal session by adding cat /etc/motd to ~/.bashrc

List detached screen sessions

Add this to ~/.bashrc

echo "Existing screen sessions"
screen -ls

Crontab

List crontab for a given user crontab -l or edit it crontab -e.

Cron runs as its own user with no environment variables like PATH. You can set these variables in the Crontab file before the cron lines. Eg.:

SHELL=/bin/bash
PATH=/user/local/bin

...

Use crontab to run a target on reboot:

> crontab -e
# run this on startup:
@reboot /path/to/exe

The time and date fields and their allowed values:

    minute		0-59
    hour		0-23
    day of month	1-31
    month		1-12 (or names)
    day of week		0-7 (0 and 7 are Sunday, or names)

Add a job that runs every day at 05:10am:

> crontab -e
# minute hour day-of-month month day-of-week
# asterisk=every minute/hour/day/month
10 05 * * * /path/to/exe

Users

Add a user to a group

sudo usermod -aG $group $user

If the group does not yet exist, create it with sudo groupadd <groupname>

Add a user

  • -e sets expiry on the user account
  • -G group membership (group has to already exist)
  • -p password (encrypted with crypt)
  • -M do not create the home directory
  • -m create the home directory if it does not exist
useradd <username> -e <YYYY-MM-DD> -G <group1,group2,...>

Set user password

passwd <username>

Switch between users as a super-user

su - <username>

See groups

# groups

See user's groups

# groups <username>

Last logins, and failed logins

List failed logins (btmp is a binary file so it need to be read with the last utility)

last -f /var/log/btmp

Logins

last
# or
lastlog

Additional user accounting utilities

GNU acct https://www.gnu.org/software/acct/

Install it

sudo yum install psacct

Provides

  • ac -d summarises login times
  • lastcomm <username> last commands per user
  • lastcomm <command_name> last invocations
  • sa summarises previous commands (all commands, all users)

RHEL 6+ auditing: http://people.redhat.com/sgrubb/audit/

sudo yum install audit

Audit reports

aureport # system wide report
aureport -au # authentication report

Permanent audit rules are at /etc/audit/audit.rules

# log every read and write to sshd_config
# in /etc/audit/rules.d/audit.rules
-w /etc/ssh/sshd_config -p warx -k sshd_config

Search failed login attempts

ausearch --message USER_LOGIN --success no --interpret

Keystroke logging

In both /etc/pam.d/system-auth and /etc/pam.d/password-auth add

session required pam_tty_audit.so disable=* enable=<username>

(does not record keystrokes when is in password entry mode, although this can be further enabled with log_passwd appended to the line). Report is generated by aureport --tty

X server

In RHEL derivatives this might be a service gdm status or service lightdm status. When stopping it, you might be presented with a text tty. If not, a shortcut <Ctrl><Alt>+<Fn> where n=1-6 switches between text terminals.

X server forwarding

ssh -Y user1@host firefox

tmux

Default bidning is same as in screen which is Ctrl+b

C-b % vertical split
C-b " horizontal split
C-b <up> move to the pane above
C-b <down> 
C-b <left>
C-b <right>
C-b : respawn-pane -k # this one respawns the pane if stuck in some process

docker

When docker attach <container> doesn't work then the following also logs you into the container's term

docker exec -it <container> bash

csv files

remove the header

tr -n +2 file.csv > no_header_file.csv

select n-th column

# this selects 1st, 3rd, and 7th columns from data.csv
cut -d, -f1,3,7 data.csv > output.csv

files

run a command on each file in the directory tree. Notice the {} placeholder which will replace the file's path there.

search <dir> -type f -exec <command> {} <arg1> ... <argN> \;

E.g. shred every file inside the folder tree

search <dir> -type f -exec gshred {} --remove \;

GPG (Gnu Privacy Guard)

This is a GNU implementation of OpenPGP

yum -y install gnupg

Public/Private Key operations:

# first time use you need to generate your key
gpg --gen-key
# check the public key
gpg --keyid-format long --with-fingerprint SomePublicKey.asc
# import the public key
gpg --import SomePublicKey.asc
# you'll need to sign this key, so first find it in your list of keys
gpg --list-keys
# find the two lines that correspond to the imported key:
# pub 3082A/<name-of-the-key> 2019-01-01 uid
#           Name of the Person/Company
# accept the key by issuing:
gpg --sign-key <name-of-the-key>
# now you can verify a message/file
gpg --verify Something.zip.asc Something.zip

Encrypt/Decrypt

gpg --output someFile.gpg --sign --encrypt --recipient someone@example.com someFile.txt
# shorthand (outputs <original-file>.gpg
gpg -se -r someone@example.org someFile.txt

gpg --output someFile.txt --decrypt someFile.gpg

Symmetric key encrypt

gpg --output doc.gpg --symmetric doc
Enter passphrase:

Decryption using the --symmetric scheme is done using the same --decrypt command. You will be prompted for the passphrase.

Exporting your keys

# -a option is for ASCII. Default is binary
gpg --export-secret-keys -a UID > secret.asc
gpg --export -a UID > public.asc

Generate a revokation certificate

gpg --gen-revoke UID

In my version of GnuPG this generates the ascii version of the revocation certificate.

Signing:

gpg --output <document>.sig --sign <document>
gpg -u 0xDIFFKEY --output <document>.sig --sign <document>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment