shell_scripts_and_one_liners.md

shell_scripts_and_one_liners.md

Shell scripts and one liners

Snippets, tips and tricks in no particular order. FOR ADVANCED USERS ONLY. Some of these one liners may be dangerous since they might delete files in loops. Please pay attention and make backups. :-)

Extracting a column from a text file

Extract the 10th column while skipping the first line:

awk -F ' ' 'NR!=1{print $10}' filename.txt

Remove space in file names

replace space with _ in file names using SED:

for oldname in *; do newname=`echo $oldname | sed -e 's/ /_/g'`; mv "$oldname" "$newname";done

PDF manipulation

Improve the quality of scanned books

Run ubuntu docker

docker run -it --rm -v ${PWD}:/source --entrypoint /bin/bash ubuntu
apt-get update && apt-get install -y imagemagick poppler-utils
pdfimages -png input.pdf output_directory
for i in *.png; do convert $i -channel RGB -negate $i.png;done

Then collect the pages that contain the text and create a PDF out of them.

Cropping the PDF

apt install texlive-extra-utils 
pdfcrop -margins '-110 0 0 -300' p1.pdf

Margins are left, top, right and bottom.

Gray scale PDF

gs  -sOutputFile=output.pdf  -sDEVICE=pdfwrite  -sColorConversionStrategy=Gray  -dProcessColorModel=/DeviceGray  -dCompatibilityLevel=1.4  -dNOPAUSE  -dBATCH  -dAutoRotatePages=/None color.pdf

the "/" before None is essential

taken from this post.

Convert PDF to PNG

taken from this post.

pdftoppm input.pdf outputname -png -f {page} -rx 300 -ry 300 <-singlefile>

convert -density 150 input.pdf -quality 90 output.png
convert -density 150 input.pdf[page] -quality 90 output.png

OCR on PDF

using OCRmyPDF:

docker run -it --rm  -v ${PWD}:/source jbarlow83/ocrmypdf -l deu+fra+eng /source/Desktop/input.pdf /source/Desktop/output.pdf

External drive does not show the full capacity

Taken from this link. Basically you have to convert from MBR to GPT. By running Windows "Disk Management" either through control panel --> Aminstrative tools --> Disk managemnt OR you run "mmc" and add do File --> Add/Remove Snap-in --> Disk Management

Assuming you have a backup of your data, then you should remove all volumes. Then right-click on the drive letter and click "convert to GPT". After that create a new volume using exFAT and full capacity.

Under Mac:

First find your external drive:

diskutil list

Then:

port install gptfdisk
gpt show -l /dev/disk2
gpt create -f /dev/disk2
gpt add -t windows /dev/disk2

Then format your hard disk using disk utility to exFAT.

making sure x forwarding works

just type one of the x commands:

xmessage -center hello!

wonderful xargs

use xargs to pipe list to command. like in this case download all gists:

gist -l | awk {'print $1'} | xargs -n1 git clone

PDF to SVG

pdf2svg ../b.pdf asd%04d.svg all && svgcleaner.py *.svg && pdfunite *.pdf b_removed.pdf

find empty directories

find . -type d -empty -exec rmdir {} \;
find . -type f -empty

group of files

find the group of files using awk:

find . -type f -exec ls -la {} \; | awk '{ print $3 }' > ~/usr.txt

Make a local copy of LDAP

Sometimes it is needed to make a local copy of the LDAP directory for offline use. Thunderbird offers this option, but it somehow does not work for me. So here is a workaround.cSave this code and call it proc.py:

import csv, base64, sys
with open (sys.argv[1]) as file:
    b = csv.reader(file)
    for line in b:
        givenname = line[0]
        sn = line[1]
        mail = line[2]
        try:
            givenname = base64.decodebytes(givenname.encode('ascii')).decode('utf-8')
        except:
            pass
        try:
            sn = base64.decodebytes(sn.encode('ascii')).decode('utf-8')
        except:
            pass
        print(','.join((givenname, sn, mail)))

Then download the converter in this gist. Then step by step:

ldapsearch -h SITE -p PORT -D "cn=USERNAME,dc=SITE,dc=COM" "objectClass=*" > longlist.txt
cat longlist.ldif | ./ldif-to-csv.sh givenname sn mail > new_longlist.csv
python proc.py new_longlist.csv > new_longlist_convert.csv

Then open Thunderbird cardbook. Create a new addressbook, then import everything.

Calculate pi using bc

to 500 digits:

 time echo "scale=500; 4*a(1)" | bc -l -q

Copy mp3 name as track ID

Using the wonderful python library mutagen:

for i in *.mp3; do o=`basename $i`;mid3v2 -t "$o" "$i"; done

Perform a command in every directory

For example perform git status:

find . -maxdepth 1 -type d -exec bash -c "cd '{}' && git status" \;

Check file encoding

file -I file.ext

PDF tricks

remove restrictions:

qpdf --decrypt infile.pdf outfile.pdf

convert to txt:

gs -sDEVICE=txtwrite -o outfile.txt infile.pdf

Tweet this button

Add a tweet this button to existing html code:

 <a href="https://twitter.com/intent/tweet?text=Some text here&url=some_url&hashtags=some_hashtag&via=some_user"
 target="_blank"><img src="https://g.twimg.com/dev/documentation/image/Twitter_logo_blue_16.png"
 alt="Tweet this page" /></a>

Extract twitter pic link

In the web browser, click on the tweet. Then view source in browser, search for pic.twitter. You will find something like:

\u003epic.twitter.com\/JAxO3ltfb8\u003c

In the client right click on the tweet, then click "open in browser". remove the escape characters before. You can also get the tweet page using curl.

###Convert Jalali to Gregorian date Python oneliner:

python -c 'from khayyam import JalaliDate;print(JalaliDate(1374,8,5).todate())'

Simple Python server

Quick HTTP server:

python -m http.server --bind 127.0.0.1

with CGI support:

python3 -m http.server --bind localhost --cgi 8000

Gist in command line

You can use GitHUB Gist from command line, either to create new or to update existing ones:

sudo -H gem install gist

For update, load your gist. You will see the gist ID in the URL. Remember it / copy paste somewhere. Then go to the raw version of your gist and download it as a markdown file. Edit it, then use the command:

gist -u GIST_ID FILE_NAME

for update.

rsync examples

Usage:

rsync --verbose --archive --recursive --checksum --prune-empty-dirs --stats --human-readable --safe-links source-path user@server:/destination-path/

• -v, --verbose verbose
• -a, --archive archive mode (preserves permissions, links etc...)
• -r, --recursive recursive
• -c, --checksum skip based on checksum, not mod-time & size
• n or --dry-run trial run, no changes
• --safe-links
• --stats show statistics
• --human-readable
• z, --compress compress file data during the transfer
• -m, --prune-empty-dirs
• -e, --rsh run command on remote shell
• --delete delete extraneous files from dest dirs

Use the last uption for making exact mirros.

Disable IPV6 in Debian

add these in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Check hard disks

df
more /etc/mtab
ls /sys/block/* | grep block | grep sd | wc -l

word frequency

word frequency thanks to this post

tr '[A-Z]' '[a-z]' < file.txt | tr -cd '[A-Za-z0-9_ \012]' | tr -s '[ ]' '\012' | sort | uniq -c | sort -nr

tar archives

make archive:

tar czvf blah.tar.gz *.txt

list the CRC checksum:

gzip -lv blah.tar.gz

list files inside (also for test):

tar -tvf blah.tar.gz

add one file to it:

gunzip blah.tar.gz
tar -zf blah.tar.gz new_file
gzip blah.tar.gz

get only java files out of archive:

tar zxvf blah.tar.gz --wildcards --no-anchored '*.java'

test tar.gz

gunzip -tv blah.tar.gz

Compare line count

 if [[ $(cat file2.ext | wc -l) -eq $(cat file2.ext | wc -l) ]];then echo 'Equal';else echo 'Not equal';fi

Cut video with ffmpeg

Type:

ffmpeg -ss 00:00:30 -i inputfile -t 00:00:05 -vcodec copy -acodec copy outputfile

Determine host name

host SERVER_ADDRESS | ip2host

Make animated GIF

convert -delay 10 -loop 0 *.png animation.gif

Convert DAT to MPG

for i in *.DAT; do o=`basename $i .DAT`;vcdgear -dat2mpg $i ~/$o.mpg; done

Updating owncloud via FTP

• Make backup of everything.
• Delete all folders except config and data
• unpack the owncloud tarball on pc
• edit the .htaccess file from tarball to add a # in front of the line # Options -Indexes
• upload everything to the server via FTP and take care of the permissions

Apple Mail attachment problem

To change the default behaviour of the attachments in the mail program:

defaults write com.apple.mail DisableInlineAttachmentViewing -bool yes

to set back to the default behavior:

defaults write com.apple.mail DisableInlineAttachmentViewing -bool false

Insert a line at the beginning of a file

Thanks to this post it is easy to insert a line at the begining of file.

#!/bin/bash
# script insertline
# insert a line at the beginning of an ascii file
TXT="#temperature(C)    0.0"
(echo "$TXT"; cat $1) > $1.new
mv $1{.new,}

Finally use it with the following command:

# find ./ -iname input.file.txt -exec ./insertline {} \;

Permutation of strings

After a long day of work, you type everything else than "make" or other commands that you use very often. In order to help yourself, just put the permutation in the BASH profile like this:

ruby -e '"make".chars.permutation.map(&:join).each {|str| puts "alias "+str.to_s+"=\"make\""}' | cat >> .bash_profile

Change dates in EXIF Tool

find ./ -iname "*.jpg" -exec exiftool "-AllDates=2013-08-27 12:00:00" {} \;

OwnCloud UI doesn't delete files?

Just check your control panel and find the configuration section and set the following PHP value to off

magic_quotes_gpc Off

File with random content

Define block size and count:

dd if=/dev/urandom of=ran.dom bs=8 count=1

Mathematica fonts error over remote X session

• goto http://support.wolfram.com/

• search for MathFonts and download tgz file to your local computer and unpack it.

• under debian create folder Mathematica in:

    /usr/share/fonts/Mathematica
  

• under Mac OS X, assuming Macports installation create the folder in:

    /opt/local/share/fonts/Mathematica
  

• then copy BDF and Type1 folders there.

• then use "sux" (or sudo with graphics)

    xset fp+ /usr/share/fonts/Mathematica/BDF
    xset fp+ /usr/share/fonts/Mathematica/Type1
  

  or

    xset fp+ /opt/local/share/fonts/Mathematica/BDF
    xset fp+ /opt/local/share/fonts/Mathematica/Type1
  

under Mac OS X.

• Then ssh -X to the remote computer where Mathematica is installed.
• run mathematica.

OSX: compile cern ROOT

I was building svn root under mac, compilation was successful but the linker had a problem:

ld: in /opt/local/lib, can't map file, errno=22

After searching, it seemed it is a common problem. Then I opened the Makefile.config and searched for /opt/local/lib one of them FTGLLIBDIR was without -L. After fixing it compiled fine.

OSX: Poster with Keynote

Presentation using Keynote:

Paper size: A0 Paper mm: 841 × 1189 inch: 33.1 × 46.8 Pixel: 2383.2 x 3369.6 for Keynote every inch is 72 DPI. change to percent notation first corner distance: 1.5% from top 2.5% from left

Uninstall an application under OSX

Warning! the following code could DAMAGE your mac os x installation if improperly used. So be careful and use it only if you know what you do.

lsbom -f -l -s -pf Archive.bom | awk '{print "rm -rvf " $1}' > remover

then make remover executable. The rest you guess yourself.

extract data from the .C ROOT file

sed 's|[^(]*(\([^)]*\)).*|\1|' 103379210_c_projection_1001.C > newfile.txt

convert and concat AVI --> MP4

First convert many AVI files into MP4:

for i in *.AVI; do o=`basename $i .AVI`;ffmpeg -i $i -c:v libx264 -crf 19 -preset slow -c:a aac -b:a 192k -ac 2 $o.mp4; done

$ cat mylist.txt
file '/path/to/file1'
file '/path/to/file2'
file '/path/to/file3'

Then use:

ffmpeg -f concat -i mylist.txt -c copy output.mp4

Process multiple files in a for loop in bash

here is one example (note the quotation marks are needed for files that have spaces in their names)

time for i in *.mp4;do ffmpeg -i "$i" "`basename "$i" .mp4`".mp3;done

one example here:

time for i in *.DAT;do vcdgear -dat2mpg  $i ~/Desktop/`basename $i .DAT`.mpg;done

another example:

time for i in *.dv;do ffmpeg -i $i -sameq -deterlace `basename $i .dv`.avi;done

here a multiliner:

#!/bin/bash
#
# This script searches all the subdirectories for Flash Videos (e.g. YouTube)
# and extracts the MP3 Audio out of them.and saves them as an MP3 file near the original FLV.
#

echo "Searching in current and all subdirectories for Flash Video (*.flv) files ...";
LIST=`find ./ -iname '*.[fF][lL][vV]' | sed -e 's/\ /IHopeThisExpressionNeverOccursInAFileName/g'`;
#it is 3:45 A.M. Sorry for this awful technique!
for d in $LIST; do
    file=`echo $d | sed -e 's/IHopeThisExpressionNeverOccursInAFileName/\ /g'`;
    ffmpeg -sameq -i "$file" "${file%*.flv}.mp3";
done

another:

#!/bin/bash
for i in *.flv; do ffmpeg -i $i -f mp3 -vn -acodec copy `basename $i .flv`.mp3;done

an exmaple with mencoder:

mencoder MVI_3192.AVI -ovc xvid -oac mp3lame -xvidencopts bitrate=-10000 -o new.avi

or lame:

#!/bin/bash
for i in *.AVI; do mencoder $i -ovc xvid -oac mp3lame -xvidencopts bitrate=-10000 -o `basename $i _small.avi`.avi;done

CP without overwrite

Sometimes a long copy process is interrupted. In order to continue one must avoid overwrite existing files. Use / at the end to take the content, not the whole directory.

cp -vrn /source/src1/ /dest/dest1

SED delete first line from file

sed '1d' file.txt

AWK print selected columns

Delete second to last columns of a data file. Change numbers for other ranges:

awk '{for(i=2;i

after %s you can choose another delimiter. Taken from:

SED search and replace

sed again :-) be careful of the overwrite option -i should not be used on large files, when disc is exhausted could cause data loss.

sed -i "" 's/;/ /g' input.file

This means search for semicolon in all lines of the input.file, replace it with one space character and do it for all other occurrences in that line (option g).

Command line Email using MUTT

Before starting, setup your .muttrc file:

echo 'set from="real name <adress@mail>"' >> .muttrc

then

echo 'Body message' | mutt -s 'Subject here' -a file.one file.two file.three -c cc@cc.net -b bcc@bcc.net -- adr1@mail adr2@mail ...

SED Example

remove lines that start with "SOMETHING".

sed -e '/^SOMETHING/d' inputfile > outputfile

Converting PS to JPG

Use this one liner:

for i in *.ps; do o=`basename $i .ps`;convert -verbose $i $o.jpg; done

Converting AVI to MPEG

Use ffmpeg like:

ffmpeg -i movie.AVI -qscale 0 movie.mpg

Xorg Errors

To see what problems xorg might have had:

cat /var/log/Xorg.0.log | grep EE

Formatting SD Card or USB Stick

Make sure the device is not mounted. Use 'mkfs.vfat' which is the same as 'mkdosfs':

mkdosfs -I /dev/sdd

XRANDR

Configuration for xrandr: With 'Xrandr' you can dynamically change the resolution of the display:

xrandr --output VGA --off --output LVDS --mode 1280x800

or with:

xrandr --output VGA --mode 1024x768 --output LVDS --off

DCOP

Some programs use DCOP command structure, for example:

dcop amarok player play

LDD

with 'ldd' you can see the dynamically linked libraries in memory e.g.:

ldd qana
ps auxwww| grep qana
cat /proc/"PROCESSNUMMER"/maps

Converting FLV to MP3

Using the following script you can convert FLV Flash Videos to MP3. Also many thanks to .

#!/bin/bash
#
# This script searches all the subdirectories for Flash Videos (e.g. YouTube)
# and extracts the MP3 Audio out of them.and saves them as an MP3 file near the original FLV.
#

echo "Searching in current and all subdirectories for Flash Video (*.flv) files ...";
LIST=`find ./ -iname '*.[fF][lL][vV]' | sed -e 's/\ /IHopeThisExpressionNeverOccursInAFileName/g'`;
#it is 3:45 A.M. Sorry for this awful technique!
for d in $LIST; do
    file=`echo $d | sed -e 's/IHopeThisExpressionNeverOccursInAFileName/\ /g'`;
    ffmpeg -i "$file" -y -acodec copy "${file%*.flv}.mp3";
done

Recover Crashed Terminal

Some programs crash the font on the terminal console. Do as follows:

c-v c-o

Simple shell script

This is a simple bash script which traverses the directories and does the specified command on the file.

#!/bin/bash
echo $@
echo $1
echo $2

dummy=`find $1 -iname *.$2`
# dummy=`find $@ -iname '*html'`
# for d in *.html; do
for d in $dummy; do
echo $d;
echo ${d%*.$2};
# echo ${d#*.$2};
done

EMACS: compile lisp code

emacs -batch -q -f batch-byte-compile file.el

Conver Flash Video to MP3

To extract the MP3 from the Flash Video try this:

ffmpeg -i input_file.flv -f mp3 -vn -acodec copy output_file.mp3

This will extract an MP3 which is compatible with K3B.

Convert windows-1256 to utf-8

You need iconv installed. It is under /usr/ports/converters/iconv in FreeBSD. Then you type:

iconv -f windows-1256 -t utf-8 old.html > new.html

This also works for XML or other documents. For a list of supported encodings on your platform type:

iconv -l

For bigger documents (> 1GB) it may be better to use siconv which is also available in the ports collection /usr/ports/converters/siconv in the FreeBSD distribution. You can use it like this

cat old.html | siconv windows-1256 utf-8 > new.html

This works like a real charm :)

xterm with UTF support

xterm -class UXTerm -u8 -fg white -bg black -fn -misc-fixed-medium-r-normal--18-120-100-100-c-90-iso1064

Display with SU-Xterm

sometimes when you write 'xterm -s' and start a root xterm, you may not have the display parameter set. so do:

root# xhost +localhost
root# export DISPLAY=":0.0"

mounting an iso file:

Mounting:

vnconfig -c /dev/vn0 image.iso
mount -t cd9660 /dev/vn0c /mnt/whatever

Unmounting:

umount /mnt/whatever
vnconfig -u /dev/vn0

find and simultaneous chmod of directories

An example:

user$ find ./Personal-Docs/ -type d -exec chmod 755 {} \;
user$ find ./Personal-Docs/ -type f -exec chmod 644 {} \;
user$ find /usr/ports/comms/ -type f -name pkg-descr -exec cat {} \; | more
user$ find ./* -name '*html' -type f -exec sed -i "" -e 's/July 2004/January 2005/g' {} \;
user$ find ./* -type f -exec sed -i "" -e 's/.$//' {} \;
user$ find ./ -type f -iname  '*.core' -exec rm -v {} \;

cvs update:

cvsup -L 2 -g /etc/ports-supfile
cd /usr/ports
make fetchindex
portsdb -u

Running a jar file:

sudo java -noverify -cp rfid.jar de.tudarmstadt.ito.rfid.reader.ReaderManager -c ./

sub partitions under FBSD 4.9

a : 327680
b :      1
c:      82
d:      83
         4
mknod ad0s4a c 116 327680 root:operator
mount -o rw -f /dev/ad0s4a /mnt

XMODMap

setxkbmap -layout bg -variant phonetic -option "grp:alt_shift_toggle" -compat "g
roup_led"

setxkbmap -layout gb -option "grp:alt_shift_toggle" -compat "group_led"

A neat PostScript rendering of a manual page

use ``-t'' switch
of the man(1) utility: ``man -t ''.  For example:

        man -t grep > grep.ps   # Save the PostScript version to a file
or
        man -t printf | lp      # Send the PostScript directly to printer

Remove all those ^M characters from a DOS file

col -bx  newfile

add user to the specified group.

sudo usermod -a -G groupName userName

Diff and Patch

diff -bur fontutils.c.orig fontutils.c > blah.diff
patch -p0

Burning Data CDs under FreeBSD

best direct burning method:

mkisofs -rU -V Pharmakobotanik  /mnt/xara/Pharmako/ | burncd -emv -s 2 -f /dev/acd0c data  - fixate

one can let the -r away.

mkisofs -v -U -V label -o /usr/temp/file.iso /usr/dir
burncd -emv -s 2 -f /dev/acd0 data /usr/temp/file.iso fixate

you can add -J to add juliett file extensions for WDose.

Copying data CDs directly

assuming that you have two CD-ROM drives on your computer, and the writer is '/dev/acd0' you can copy CDs like this:

dd if=/dev/acd1 bs=2048 | burncd -ev -s 24 -f /dev/acd0 data - fixate

Burning AudioCD from MP3s under FreeBSD:

Here is how to do it:

mpg123 -w - anysong.mp3 > newsong.wav
sox -t wav -r 44100 -s -w -c 2 track.wav track.raw
burncd -f /dev/acd0c -s 8 audio *.raw fixate

Plotting in GNUPLOT

plot "data2" using 1:2

Proxy

Setting up proxy in the shell is easy:

export FTP_PROXY="http://proxy.tu-darmstadt.de:80"
export HTTP_PROXY="http://proxy.tu-darmstadt.de:80"

make master boot record writable

sysctl -a | grep geom

so that you know the name. then:

sysctl kern.geom.debugflags=16
boot0cfg

Now you ask why on earth should you make master boot record writable??!!

SourceForge CVS Check out

To check out a module name just type the directory name without path.

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pcb login
cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pcb co -P pcb

Press ENTER for password.

Umlaut characters in Emacs

to print german umlaut in Emacs:

[c-x 8 "]

then u,o or a

Install path

You want to know where your port is installed? Acquire the complete port name with 'pkg_info' then use:

less /var/db/pkg/port_name_comes_here/+CONTENTS

Xdefaults changes

If you want the changes in .Xdefaults to take effect without a restart, then run:

xrdb -merge .Xdefaults

Some commands

strace follows the system calls. tail -f follows the last parts of a file as it is being written on. xrander -s changes the resolution of the screen sudo nmap -sS scans the ports of an DUT computer (like an old AIX computer!)

See who is using the sound card!

lsof -w $( find /dev -group audio )

Using Timidity

Type:

timidity -Ov1S *.mid

If you want .wav files instead, change the v to a w.

Misc commands

Here are some more commands.

du -hd1|sort -n
truncate -s 1k inputfile
xterm -e su -
ln -s ../lib-pcb packages
TERM=xterm-color
export TERM
oder:
TERM=cons25;    export TERM
export PACKAGEROOT="ftp://ftp2.de.freebsd.org"
remove lines with tab or space:
sed -e '/^[[:blank:]]*$/d' ./praktikum.txt > prak3.txt
add " - " to the beginning ot each line
sed -e 's/.*/ - &/' ./prak6 > prak7
remove commas from a matlab data output:
sed -e 's/\,/ /g' 15.txt > 16.txt
to add compile flags before ./configure:
export CFLAGS="$CFLAGS -I/usr/X11R6/include"