Useful Linux Commands

usefull-linux-commands.md

Screen Commands

1. Create a screen : screen -S SCREEN_NAME
2. Detach (getting out of the screen) : Press ctrl+A & ctrl+D respectively
3. Reattach (getting in the screen) : screen -r SCREEN_NAME/ID
4. List of screens with process id : screen -ls
5. Kill a screen with process id : screen -X -S [PID] kill

Kill processes with known keywords

• for PID in `ps ax | grep 'grakn' | awk ' { print $1;}'`; do 
      kill -9 $PID; 
  done

Kill processes bound with a port

• sudo kill -9 $(sudo netstat -anp | awk '/ LISTEN / {if($4 ~ ":8080$") { gsub("/.*","",$7); print $7; exit } }')

  or

  sudo kill -9 $(sudo lsof -ti tcp:8080)

Find details of a process id

1. $ ps -a PID
2. $ ps -p PID -o comm=

VIM editor essentials

1. Open a file: $ vim FILE_NAME
2. Edit: press i
3. Save: press esc, then type :wqa and press enter
4. Exit without saving any change: pres esc, then type :q! and press enter
5. Undo last change: press esc, press u respectively

SCP commands

1. Copy to remote server:

   scp -r FILE_OR_FOLDER_PATH USER@HOSTNAME_OR_IP:COPY_TO_DIRECTORY

2. With password in single command:

   sshpass -p PASSWORD scp -r FILE_OR_FOLDER_PATH USER@HOSTNAME_OR_IP:COPY_TO_DIRECTORY

   Note: Install sshpass using the following command.

   sudo apt-get install sshpass

Count number of files

• ls | wc -l

Add a prefix/suffix to all files in a direcotry

• for file in *.txt; do mv "$file" "en_${file}"; done

Regex for not including a word

• Let's say the word is git

  ^(?:(?!git).)*$

Convert ASCII to UNICODE

• sudo apt install uni2ascii
  ascii2uni -a U file_ascii.json > file_unicode_char.json

JSON pretty print

• sudo apt install python-demjson
  jsonlint-py -f file.json > file_pp.json

Port forwarding:

1. From world:

   sudo iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-ports 8080

2. From own machine:

   sudo iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080

Bash command for string replacement

• first="My name is Sayef"
  replace="isn't"
  edited="${first/is/$replace}"
  

  Note: For all occurances use double slash (//)

Change all file names using string replace

• Replace spaces ( ) with underscore (_) from all jpg file name in a directory

  for file in *.jpg; do 
      newfile="${file/ /_}"; 
      mv "${file}" "${newfile}"; 
  done

Add new user and password in ubuntu

• adduser sayef
  usermod -aG sudo sayef
  su - sayef
  passwd sayef

Convert image to PDF using ImageMagick command:

• Sort according to file creation time

  convert $(find . -type f -print0 -name \*.png | xargs -0 stat -c "%y|%n" | sort | awk -F'|' '{print $2}') output.pdf

Rename files with numbers based on file creation time

1. Filter file extension

   from=1;
   extension=jpg;
   for i in $(find . -type f -print0 -name \*.$extension | xargs -0 stat -c "%y|%n" | sort | awk -F'|' '{print $2}'); do 
     mv -i -- "$i" $(printf "%04d.$extension" "$from"); 
     let from=from+1; 
   done

2. Regardless of file extension (but keep original extension)

   from=1;
   for i in $(find . -type f -print0 | xargs -0 stat -c "%y|%n" | sort | awk -F'|' '{print $2}'); do 
     mv -i -- "$i" $(printf "%04d."${i##*.} "$from"); 
     let from=from+1; 
   done

Randomly move validation data

• Given an input directory where each sub-directory is a unique class and contains files to represent that class

  $1=INPUT_DIR
  $2=OUTPUT_DIR
  $3=PERCENTAGE
  
  count=1
  total=$(ls -d $1/* | wc -l)
  for dir in "$1"/*; do
          if [ -d "$dir" ]; then
                  echo -ne ">> Processing $count/$total ...\r"
                  n=$[$[$(ls "$dir"/* | wc -l) * $3] / 100]
                  if [ "$n" -ne "0" ]; then
                          files=$(ls "$dir"/* | shuf -n $n)
                          dir=$2/$(basename $dir)
                          mkdir -p $dir
                          mv $files $dir
                  fi
                  count=$[$count+1]
          fi
  done

Copy files from subfolders to a flat directory

• Rename if already same named file exits

  find source_dir/ -type f -exec cp --backup=numbered -- {} target/ \;
  find source_dir/ -name ".jpg" -exec cp --backup=numbered -- {} target/ \;

SCP alternative for progress status

```
rsync -r --info=progress2 -e 'ssh -p XX' source/folder user@host:~/
```

Copy (or move) files from remote to local that have not been modified/created in last N minutes

```
#!/bin/bash

TARGET=/mnt/d/output/
HOST=user@host
PORT=xxxx
SOURCE=/root/output/

# -a                    : archive
# -h                    : human readable
# -r                    : recursive
# -v                    : verbose
# --no-perms            : ignore keeping original permissions
# --no-owner            : ignore keeping original owner
# --no-group            : ignore keeping original group
# --no-times            : ignore keeping original creation/modification time
# --info:progress2      : 
# --info=name0          : progress information in one line (instead for all files individually)
# --no-inc-recursive    : ignore incremental file lookup
# --update              : update if file already exists
# --remove-source-files : remove source files after syncing
# --files-from          : define files at once

rsync \
-ahrv \
--no-perms \
--no-owner \
--no-group \
--no-times \
--info=progress2 \
--info=name0 \
--no-inc-recursive \
--update \
--remove-source-files \
--files-from=<(ssh -q -p $PORT $HOST "find $SOURCE -name *.txt -type f -mmin +30 -exec realpath --relative-to=$SOURCE '{}' \;") \
-e "ssh -q -p $PORT" \
$HOST:$SOURCE \
$TARGET

```

Move files with similar prefix to a directory

• Assuming prefixes are 000, 001 up to 120.

  SOURCE=/mnt/d/flat/
  TARGET=/mnt/d/dirs/
  
  for PREFIX in $(seq -f "%03g" 0 120)
  do
    mkdir -p ${TARGET}/${PREFIX}
    find $SOURCE -maxdepth 1 -name $PREFIX-*.txt -type f -exec mv {} $TARGET/$PREFIX \;
  done
  

Rename files by replacing a pattern

for file in *.png; do mv "$file" "${file/_h.png/_half.png}"; done