MeenachiSundaram/Helpful_Unix_Commands.md

## Helpful_Unix_Commands.md

      
    Raw
  

              Helpful_Unix_Commands.md
            
          
    Combining multiple Files

Method 1

    head -n1 file1.txt > combined.txt
    for fname in *.txt
    do
        tail -n+3 $fname >> combined.txt
    done

Method 2

    awk '
        FNR==1 && NR!=1 { while (/^HEADING/) getline; }
        1 {print}
    ' file*.txt >all.txt

The first line of the awk script matches the first line of a file (FNR==1) except if it's also the first line across all files (NR==1). When these conditions are met, the expression while (/^HEADING/) getline; is executed, which causes awk to keep reading another line (skipping the current one) as long as the current one matches the regexp ^
. The second line of the awk script prints everything except for the lines that were previously skipped.
Creating Dummy file with specific size

Method 1

You can create a 10MB gzip file like this:
    head -c 10M /dev/urandom | gzip -1 >10m.gz

This uses urandom to get a high-entropy stream of bytes: since this is incompressible, the gzipped version will be about the same size as the input.
Method 2

File created using this method will be highly compressed during compression.
To create 100M size, enter:
    fallocate -l 100M file.out

The -l option specifies the length of the allocation, in bytes. Suffixes of k, m, g, t, p, e may be specified to denote KiB, MiB, GiB, etc.
Method 3

dd command syntax
The basic syntax is:
    dd if=/path/to/input of=/path/to/output [options]

OR
    dd if=/dev/zero of=/path/to/output.img [options]

OR
    dd if=/dev/zero of=YOUR-IMAGE-FILE-NAME-HERE bs=1 count=0 seek=Size-HERE

To create 1MB file (1024kb), enter:
    $ dd if=/dev/zero of=test.img bs=1024 count=0 seek=1024

You will get an empty files (also known as "sparse file") of arbitrary size using above syntax.
To create 100MB file , enter:
    $ dd if=/dev/zero of=test.img bs=1024 count=0 seek=$[1024*100]
    $ ls -lh test.img

To create 1GB, file:
    $ dd if=/dev/zero of=1g.img bs=1 count=0 seek=1G

Useful find Commands

Find files older than x days

    find /path/to/ -type f -mtime +7 -name '*.gz' -execdir rm -- '{}' \;

Explanation:

find: the unix command for finding files/directories/links and etc.
/path/to/: the directory to start your search in.
-type f: only find files.
-name '*.gz': list files that ends with .gz.
-mtime +7: only consider the ones with modification time older than 7 days.
-execdir ... \;: for each such result found, do the following command in ....
rm -- '{}': remove the file; the {} part is where the find result gets substituted into from the previous part. -- means end of command parameters avoid prompting error for those files starting with hyphen.

Find files and sum the total result

    find /path/to/ -type f -mtime +7 -name '*.gz' -exec du -ch {} + | grep total$

du Commands

Folder size and sort

    du -h --max-depth=1 | sort -hr

which will give you the size of all sub-folders (level 1). The output will be sorted (largest folder on top).
While Loop - print increasing two digits number

    i=1; \
    while [ $i -le 20 ]; \
      do \
        echo $i; \
        printf -v i "%02d" $i; \
        echo $i; \
        i=$[10#$i+1]; \
      done


i=$[10#$i+1] and  i=$[${i#0}+1] can be used since this will not work i=$[$i+1]
Octal issue - https://stackoverflow.com/questions/12821715/convert-string-into-integer-in-bash-script/12821845#12821845

Find file by date and group into folders by year and month

    #!/bin/bash
    BASE_DIR=/var/tmp/testfolder

    ## Find those files that are older than a month
    find "$BASE_DIR" -maxdepth 1 -mtime +30 -type f -name "*" | 
     while IFS= read -r file; do
        ## Get the file's modification year
        year="$(date -d "$(stat -c %y "$file")" +%Y)"
        ## Get the file's modification month
        month="$(date -d "$(stat -c %y "$file")" +%b)"

        ## Create the directories if they don't exist. The -p flag
        ## makes 'mkdir' create the parent directories as needed so
        ## you don't need to create $year explicitly.
        [[ ! -d "$BASE_DIR/$year/$month" ]] && mkdir -p "$BASE_DIR/$year/$month"; 

        ## Move the file
        mv "$file" "$BASE_DIR/$year/$month"
    done

find for a string and sed to replace

    find . -name "*.py" -exec sed -i "s/foo/bar/g" {} +

To know more on using + instead of \; - https://askubuntu.com/a/339019
Remove prefix from multiple files

Bash
for file in prefix*; do mv "$file" "${file#prefix}"; done;

The for loop iterates over all files with the prefix. The do removes from all those files iterated over the prefix.
Here is an example to remove "bla_" form the following files:
bla_1.txt
bla_2.txt
bla_3.txt
blub.txt

Command
for file in bla_*; do mv "$file" "${file#bla_}";done;

Result in file system:
1.txt
2.txt
3.txt
blub.txt

Rename multiple files

cwd=$(pwd)
echo ${cwd}
for i in `ls`
do
echo $i
cd ${cwd}/$i
x=$(echo $i | cut -d '.' -f 1)
echo $x
# add prefix
rename "s/^/${x}./" *
echo ""
sleep 5

cd ${cwd}
done

remove space in files and replace with underscore

rename 's/\s+/_/g' *