Skip to content

Instantly share code, notes, and snippets.

@SantoshSrinivas79

SantoshSrinivas79/Using the command line on a terminal like a pro.md

Last active December 13, 2020 02:32
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save SantoshSrinivas79/63c4062dc0ce322599df to your computer and use it in GitHub Desktop.
Save SantoshSrinivas79/63c4062dc0ce322599df to your computer and use it in GitHub Desktop.
Download ZIP
Using the command line on a terminal like a pro
Raw
Using the command line on a terminal like a pro.md

Using the command line on a terminal like a pro

The terminal is extremely powerful and it is now my topmost choice in coding. Here are some useful commands for common tasks that will hopefully help appreciate the power of the command line!

  • List files sorted by time
    • ls -lt
  • List of files sorted by size
    • ls -alS
  • List of files from s3 ... only name:
    • s4cmd ls s3://tempbin/ | grep zerod_archive_ | awk '{print $NF}' | head
  • List of files from s3 AND download them in one command:
    • for i in s4cmd ls s3://tempbin | grep zerod_archive_ | awk '{print $NF}' ; do s3cmd get $i ; done
  • List files above a certain size:
  • List files in directory and upload all to s3 using s3cmd
  • List only files in directory and count them
    • ls -p | grep -v / | wc -l
  • List ls -l only certain columns
    • first and last column
      • ls -l *.sh | sed 's/|/ /' | awk '{print $1, $9}'
    • only last column
      • ls -l *.sh | sed 's/|/ /' | awk '{print $9}'
  • Create a folder using a pattern. Pretty cool!
    • mkdir -p /usr/local/etc/nginx/sites-{enabled,available}
  • List files and pipe to copy
  • Look at fish shell history 
for i in `ls -al | grep zerod_archive_ | awk '{print $NF}'` ; do s3cmd put $i s3://tempbin/zerod_final/ ; done

  • Find a file in bash

  • Find all zip files in a directory and delete them

    • find . -name '*.zip' | xargs rm

  • Find a directory or folder in bash with a certain name

    • find . -name oknpjjbmpnndlpmnhmekjpocelpnlfdi -type d

  • Finding files on a Mac OSX with a fuzzy file name

    • mdfind 'kMDItemDisplayName == "*flickr*.js"c'

  • Finding files with a -ve grep

    • mdfind 'kMDItemDisplayName == "*sequelize*.js"c' | grep -v node_modules

  • List only directories

    • ls -d */

  • Find size of directories

    • du -sh *

  • Find disk utilization

    • df -h

  • Download zip file with curl command

    • curl -LOk
    • curl -LOk -X 'GET' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0' -H 'Referer: https://nseindia.com/products/content/equities/equities/archieve_eq.htm' -H 'Upgrade-Insecure-Requests: 1' 'https://nseindia.com/content/historical/EQUITIES/2000/MAR/cm7MAR2000bhav.csv.zip'

  • Count number of lines in a file wc -l mytextfile

  • Count number of files in a directory

    cd directory_to_count
ls | wc -l

  • Find or search text in files in recursively from the command line

  • Find lines in a text file that contain a certain word

grep -w GAL Model_GFD-Top-5-Stocks-Prev-Day.Rout

grep -i GAL Model_GFD-Top-5-Stocks-Prev-Day.Rout

  • Rename all files in a directory

    • for f in * ; do mv "$f" "$f.mp3" ; done
    • This needs bash ... doesn't seem to work in fish
    • Run the below in bash:
    for FILE in *; do
# Strip of any extension, and save to FILE2
FILE2=${FILE%\.*}

# If it's the same afterward, then there was no extension - rename it
if [ "$FILE" = "$FILE2" ]; then
mv "$FILE" "$FILE.mp3"
fi
done

  • Zip an entire directory

    • zip -r tobwis.zip tobwis/

  • Moving files in a folder in parallel using rsync

msrsync -p 20 --progress  /Users/santosh/Dropbox/ /Volumes/Seagate-Slim/mac-air-2015/Dropbox/

msrsync -p10 --rsync "-a --remove-source-files  --progress" /Users/santosh/Dropbox/ /Volumes/Seagate-Slim/mac-air-2015/Dropbox/
cd ~

rm /Users/santosh/transfer.log
rm /Users/santosh/result.log


# build the dry-run list
rsync -avzm --stats --safe-links --ignore-existing --dry-run \
    --human-readable /Users/santosh/Dropbox/ /Volumes/Seagate-Slim/mac-air-2015/Dropbox/ > /Users/santosh/transfer.log

cd /Users/santosh/Dropbox/

# transfer them!
cat /Users/santosh/transfer.log | \
    parallel --will-cite -j 20 rsync -avzm --progress --relative \
      --stats --safe-links --remove-source-files --ignore-existing \
      --human-readable {} /Volumes/Seagate-Slim/mac-air-2015/Dropbox/ > result.log
# credit: https://unix.stackexchange.com/questions/189878/parallelise-rsync-using-gnu-parallel
# build the dry-run list
rsync -avzm --stats --safe-links --ignore-existing --dry-run \
    --human-readable /Users/santosh/ /Volumes/Backup\ Plus/mac-backup/ > /Users/santosh/transfer.log

# transfer them!
cat /Users/santosh/transfer.log | \
    parallel --will-cite -j 20 rsync -avzm --relative \
      --stats --safe-links --ignore-existing \
      --human-readable {} /Volumes/Backup\ Plus/mac-backup/ > result.log
./msrsync -p 20 --progress  /Users/santosh/ /Volumes/Backup\ Plus/mac-backup/
for pid in $(ps axo pid=,stat= | awk '$2~/^Z/ { print $1 }') ; do
    echo $pid;
    sudo kill $pid; # do something interesting here
done

Changing the editor to edit a file

Sometimes, you want to use a different editor while editing a file. So here is the trick from my fish shell.

The below script will change my editor to vim and allow me to edit crontab.

bash
export EDITOR=vim
crontab -e

Networking Related

  • Unzip into a folder
    • unzip /path/to/file.zip -d temp_for_zip_extract

R-Project

  • Run R script in terminal using

    • R CMD BATCH --no-save --no-restore "Read-NSE_CM_EOD_Bhavcopy.R"
    • R CMD BATCH --no-save --no-restore "Analysis-EOD-Data-Setup.R"

  • One R Tip A Day: Running R Programs on clusters

  • Create shortcut to ripgrep:

    • ln -s "/Users/santosh/tools/ripgrep/rg" /usr/local/bin/rg

s3cmd

  • Put a folder recursively

    • s3cmd put -r LM s3://tempbin/LM/
      • Need to start to "BODYPUMP 100 XXX"
    • BETTER: s3cmd sync -v LM --skip-existing s3://tempbin/LM/
    • Try this first: s3cmd sync --dry-run --skip-existing --delete-removed -v s3://tempbin/LM/ LM/

  • In DO folder run:

mv *.mp4 LM/
s3cmd put -r LM s3://tempbin/LM/
rm LM/*.mp4
  • Playing with s3cmd output to get only the filenames
#!/bin/bash

# Blog at https://www.santoshsrinivas.com/text-manipulation-on-the-command-line/

awk '{$1=""; $2=""; $3=""; print}' lm-1.txt > lm-1-files.txt
awk -F"s3://tempbin/LM/LM/" '{ print $2}' lm-1-files.txt > lm-2-files.txt

sshfs

I'm thinking of using sshfs to sync my remote ubuntu machine disk a local mounted folder on my mac.

To replicate the folder:

sshfs ghost@3.85.118.126://home/ghost/Dropbox/pandora/My-Projects/repos/dalalbeat/python/ ~/ec2_db/python

If I choose to rsync

Sources

  • Conquering the command line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment