ggrrll/useful_tips.md

## useful_tips.md

      
    Raw
  

              useful_tips.md
            
          
    Bash / Unix admin

https://www.cheatsheet.wtf/Nano/


grep with logic or grep -e patter1 -e pattern1


file stats: stat [file] and debugfs


remove lines matching pattern from file


how-to-go-to-the-previous-working-directory-in-terminal cd ~-


bash: how-to-check-how-long-a-process-has-been-running/


find process: pgrep -f [pattern] -- midn -f!; or ps aux | grep [pattern]


variables with nested shell scripts https://stackoverflow.com/questions/9772036/pass-all-variables-from-one-shell-script-to-another


n. of files in all subdir: https://unix.stackexchange.com/questions/4105/how-do-i-count-all-the-files-recursively-through-directories


find files and list info e.g. with csv: find . -name *csv  -exec ls -lsh {} \;


line endings:

to check them: https://stackoverflow.com/a/3570574/5533078
to control how git handles them:

https://stackoverflow.com/a/5834094/5533078
https://stackoverflow.com/a/39461324/5533078


checksums:

crc32 -t d4 -f yourfile (d4 is for decimal format)


hexadecimal conversion $((16#$hexNum)) or in your bashrc


	hex2dec(){

        	hexnum="$1" 
	        echo $((16#$hexnum))
	}


diff --color=always -y --suppress-common-lines tips: https://phoenixnap.com/kb/diff-color


diff for directories

tar -cf - somedir | md5sum
 find somedir -type f -exec md5sum {} \; | sort -k 2 | md5sum
diff -qr dir1 dir2


update directories -- mind tha cp only looks at timestamp -> https://superuser.com/a/434101/687355


(re)directing stdout / stderr to files:

https://linuxize.com/post/bash-redirect-stderr-stdout/
https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/


run shell command on remote host -- detach from ssh session:

https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
https://stackoverflow.com/questions/4549489/can-i-change-the-name-of-nohup-out


detach jobs:

Press Ctrl+Z to pause the currently running process.
Type bg and press Enter
disown jobID (list with jobs -l or ps aux)


reown / reattach: https://github.com/nelhage/reptyr


compare two files -- SO question

cmp --silent $old $new || echo "files are different"
use a checksum


compares files contained in two folders -> https://gist.github.com/ggrrll/7ab0107182c30e9154dd3e7d6e48b62a


ssh connection timeout

https://www.tecmint.com/increase-ssh-connection-timeout/ & https://man.openbsd.org/ssh_config
https://stackoverflow.com/questions/4936807/how-to-set-ssh-timeout


copy files from server, that need  sudo rights > https://askubuntu.com/a/872537/641871


pipeline to open package location in sublime :)
sublime $(pip show [your_package] | grep Location | cut -d ' ' -f 2)


update-a-file-in-hdfs


get FQDN server


working on remote with sublime :) > https://github.com/randy3k/RemoteSubl


touchpad behaviour (xinput)

how_to_invert_buttons_below_the_touchpad/
change scrolling speed


close background ssh tunnel (ssh -f) : https://unix.stackexchange.com/a/164656/143800


fast (de)compression of archives: https://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression , in particular https://github.com/facebook/zstd


replace string in filenames: find . -name '*jpg' -exec bash -c ' mv $0 ${0/\old/new}' {} \; 


replace whitespaces in filenames: https://vitux.com/how-to-replace-spaces-in-filenames-with-underscores-on-the-linux-shell/


fuzzy search in bash history https://github.com/junegunn/fzf


colorized  ls https://askubuntu.com/a/973569/641871


networks


get host's IP dig +short myip.opendns.com @resolver1.opendns.com

useful bashrc / bash_profile

# some more ls aliases
alias ll='ls -larth -F'
alias la='ls -Al'
alias l='ls -CF'
alias nano='nano -L -N -c'
#alias less='less -N'
alias diff='diff --suppress-common-lines --color=always -y'

if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto -n'
    alias fgrep='fgrep --color=auto -n'
    alias egrep='egrep --color=auto -n'
fi

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoredups

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=
HISTFILESIZE=


GPU monitornig


lspci | grep VGA					
pip install gpustat (only NVIDIA)
apt-get install nvtop

file names


detox -r [dir] https://stackoverflow.com/a/14266025/5533078

partitions and disks


fix filesystem corruption (one approach) https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Mount-mount-dev-mapper-vg00-lvol1-on-mnt-StorageMedia-failed-Structure-needs-cleaning-error-when-trying-to-mount-storage-in-Flame.html
sudo lsblk -a also allows to see which are mounted

disks sizes


average file size! ls -l | gawk '{sum += $5; n++;} END {print sum/n;}'


list files by size find . -type f -printf "%s\t%p\n" | sort -n


du -hsc * | sort -hr https://askubuntu.com/questions/1224/how-do-i-determine-the-total-size-of-a-directory-folder-from-the-command-line


df -h or ncdu for usage %


tree -h


count files by type! :) find . -type f | sed -n 's/..*\.//p' | sort | uniq -c
(credis to https://www.2daygeek.com/how-to-count-files-by-extension-in-linux/ )


count files in each sub-dir : du --inodes


cleaning up space


journalctl https://unix.stackexchange.com/questions/130786/can-i-remove-files-in-var-log-journal-and-var-cache-abrt-di-usr

network monitoring


https://askubuntu.com/questions/257263/how-to-display-network-traffic-in-the-terminal

desktop env


check des manager wmctrl -m

users  & groups

proper permissions  -- chmod

home: 750
.ssh/authorized_keys: 644
private ssh key: 400

create new user

sudo useradd -m username
sudo passwd username
usermod -aG sudo username

check groups and members


groups [user]


members [group-name]  or  grep 'group-name-here' /etc/group


cat /etc/group | cut -d: -f1 -- list all groups


find /home -group test -- files owned by group 'test'


check open ports

from https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/

sudo lsof -i -P -n | grep LISTEN  --  less /etc/services
netstat -tulpn | grep LISTE

Security


tools  for scanning repo for credentials: https://geekflare.com/github-credentials-scanner/

Ubuntu / Mint


change config of progressive apps (browser apps), edit files in ~.local/share/applications

Python

parallelization


joblib: https://joblib.readthedocs.io/en/latest/
RAY:

https://betterprogramming.pub/python-use-ray-to-run-functions-in-parallel-871f7a6678c4
https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1


code analysis


UML diagram tools: see discussion and comment on GH and this good review;    see also this description of UML diagram conventions

pynsource -- nice , but needs pro
pylint: FOSS, not interactive (webp cannot be generated...to double check)
epydoc : does not work with python3


(sub) plotting


https://engineeringfordatascience.com/posts/matplotlib_subplots/
seaborn:

xticks rotation in facegrid https://stackoverflow.com/a/72099702/5533078


debugging


✅ ipdb has autocompletion, loads variables.. : ipdb3 my_script.py
reload(my_package) -- credits to this SO psot
pip update selected packages (bash pipe): e.g. pip freeze  | grep jupy |  cut -d'=' -f1 | cut -d":" -f2 | xargs pip install -U

miscellanea


file open modes


extract numbers from strings (e.g. filenames) https://stackoverflow.com/a/4703409/5533078


get variable name :)!

in list: https://stackoverflow.com/a/52038959/5533078
simple variable: https://stackoverflow.com/a/58451182/5533078


VS code


list of python symbols https://code.visualstudio.com/docs/editor/intellisense

Jupyter notebook

layout


decrease side space -- mind the different css code for notebook 7 https://stackoverflow.com/a/76778615/5533078
-- see also jupyter/notebook#7152

%load_ext autoreload

this is problematic...many things tried (none worked):

%autoreload 1 + %aimport tiqcovid
%autoreload 2
removing the module from subfolder > put in external folder
sys.path.append('path_to_package')

debugging


%debug -- here is an intro


from IPython.core.debugger import  set_trace  -> better to use pdb.set_trace()


here are a description of commands + GUI

in iPython, better to use %autoreload (see the doc)
code profiling https://jakevdp.github.io/PythonDataScienceHandbook/01.07-timing-and-profiling.html

plotting


matplotlib backends https://stackoverflow.com/a/36021869/5533078
save plotly graphs inside nb, after export https://community.plotly.com/t/exporting-jupyterlab-notebook-as-html-plotly-loses-interactiveness/15310

saving outputs

save cell to file :)

%%writefile example.txt
from interactive prompt


first capture the output:

%%capture my_var
my_command

like explained in the 2nd answer here

to save do:

%%store my_var.stdout >> my_file.txt
from script in terminal

mind the buffering! -.-
https://unix.stackexchange.com/questions/182537/write-python-stdout-to-file-immediately
print all attributes of python object

for attr in dir(obj):
    print "obj.%s = %s" % (attr, getattr(obj, attr))

shut down all notebooks servers kept open


list all servers with jupyter notebook list
try: jupyter notebook stop PORTID
else: deleted the related file in /Library/Jupyter/runtime
find the PIDs related to each port with ps aux | grep PORTID
kill the process with  kill -9 PID

find kernel version

from sys import executable
notebook on remote server

on local host
ssh -L localhost:8888:localhost:8889 username@your_remote_host_name (-N -f flags are not needed)
and on remote:
jupyter notebook --no-browser --port=8889
refs:
https://amber-md.github.io/pytraj/latest/tutorials/remote_jupyter_notebook
https://towardsdatascience.com/running-jupyter-notebooks-on-remote-servers-603fbcc256b3
presentation

https://voila.readthedocs.io/en/stable/using.html is a nice tool to make nb nore suitable for presentations
tricks:

show code cells with --strip_sources=False

python packaging


nice tutorial and collection of resources > https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html


https://medium.com/38th-street-studios/creating-your-first-python-package-181c5e31f3f8


generate static html


https://www.fullstackpython.com/static-site-generator.html
https://stackoverflow.com/questions/6748559/generating-html-documents-in-python
jinjia2 https://towardsdatascience.com/using-python-to-create-static-web-pages-the-easy-way-6eb16c997571

GitHub

actions


mind to use lowercase letters only in yml! -> https://github.com/orgs/community/discussions/69617

Docker

manage-docker-as-a-non-root-user
general good practises

https://docs.docker.com/develop/develop-images/instructions/
https://docs.docker.com/develop/develop-images/guidelines/
tips


add a label to existing image https://stackoverflow.com/a/54598281/5533078
tool to visualize containers & images relations https://github.com/justone/dockviz

docker on github


github working-with-the-container-registry

some useful commands


images

docker image ls
docker image rm [image_ID]
docker build --network=host -t [full_tab_incl_webhost]  -f  [docker_file]  (build an image to be set on a remote host)
docker push [full_tag]
docker inspect [image_ID]
 docker run -it  [full_tag] (to run a container on that image)
docker exec -it <mycontainer> bash to log into a container shell


containers

docker ps -a
docker container rm [cont_ID]
docker restart <container_id>
where to find container logs:  docker inspect --format='{{.LogPath}}' containername  https://stackoverflow.com/a/41147654/5533078


Kubernetes


kubectl get pods -A
kubectl delete pod  [pod_ID] -n [namespace]
kubectl describe pod  [pod_ID] -n [namespace]
kubectl exec -it   [pod_ID] -n [namespace] engine -- bash

Git

intro tutorials

nice cheatsheet https://www.quora.com/What-is-the-best-Git-cheat-sheet
https://learngitbranching.js.org/.

https://gitimmersion.com/.

https://githowto.com/.
long list: https://medium.com/javarevisited/11-best-online-places-to-learn-git-for-beginners-in-2021-6dc2b7c6ef48
tips


store safely credentials (use cache)


working on shared folder:

https://serverfault.com/questions/26954/how-do-i-share-a-git-repository-with-multiple-users-on-a-machine
https://git-scm.com/docs/git-init#Documentation/git-init.txt---sharedfalsetrueumaskgroupallworldeverybodyltpermgt
if the repo already exists: git config core.sharedRepository true, after setting up the dedicated group


pull file from another branch https://www.freecodecamp.org/news/git-checkout-file-from-another-branch/


verbose log git log flags: -p , --stat, --patch-with-stat 


remove file from last commit (reset + amend) https://stackoverflow.com/a/42450337/5533078


git add -u to stage only tracked files


make private fork of public repo:  https://stackoverflow.com/questions/10065526/github-how-to-make-a-fork-of-public-repository-private


checkout new branch: use new command git switch [branch] https://stackoverflow.com/a/9537923/5533078


mind possiblity to udpate-index: https://stackoverflow.com/a/20241145/5533078


careful when working with ignored files on different branches!

https://stackoverflow.com/questions/30643119/git-checkout-recover-lost-files
https://stackoverflow.com/questions/1836742/using-git-how-do-i-ignore-a-file-in-one-branch-but-have-it-committed-in-another


to avoid messing up after fetching (merge conflicts), check the diff first!:

git diff origin/master..


modify past commit message: https://help.github.com/en/github/committing-changes-to-your-project/changing-a-commit-message


git diff in same branch > git diff  --no-index [file1] [file2]


git diff specific commit: git diff sha~ sha https://stackoverflow.com/a/17563740/5533078


commit description : you need 2 new lines to be correctly shown in github desktop (see also this SO discussion)


for notebooks comparison:   nbdiff-web -M [branch1] [branch2] [notebook_name]

(git diff across branches works only after commit!)


apply .gitignore: git rm -r --cached . & git add .


untrack files (without necessary having them in gitignore) git rm --cached [file]


'going back'

mind! revert VS reset VS checkout
https://www.w3docs.com/snippets/git/how-to-reset-or-revert-a-file-to-a-specific-revision.html
merge


pulling  with rebase

https://www.atlassian.com/git/tutorials/merging-vs-rebasing
https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/about-merge-methods-on-github


for selective merge :

git checkout --patch B f
or try also
git merge --no-ff --no-commit branchX


notebook merge (follow also instructino from github) (e.g. branch branch_b into branch_a):

open a PR!
git checkout branch_b
git merge branch_a   > these creates/show conflicts
nbdime mergetool


git LFS


nice intro tutorial https://sabicalija.github.io/git-lfs-intro/


nice reading on pros and cons https://stackoverflow.com/questions/75946411/how-does-git-lfs-track-and-store-binary-data-more-efficiently-than-git


in case export does not work for reverting, I wrote a script git-lfs/git-lfs#5463 (comment)


cons:

if you convert to file (lfs pull) and back to pointer (see tutorial) it generates a file change, in principle to be commited


Android

flash a ROM with fastboot

do these steps, in this order, before restoring your stuff...

OEM + UBS from dev options
check adb devices
adb reboot bootloader
fastboot oem unlock
sh flash_your_ROM.sh
fastboot oem lock

source: https://forum.xda-developers.com/mi-a2/how-to/guide-official-stock-android-10-t4033835
adb sideload

TWRP issue

I got Failed to mount '/system'  error while trying to wipe

channging TWRP v. did not work (as suggested here https://android.stackexchange.com/a/214375/210859)
solution: force again ext4 partition https://www.youtube.com/watch?v=ysBfcasLnGM

OTA installation

https://www.androidsage.com/2017/11/10/how-to-install-ota-updates/
https://fossbytes.com/install-ota-updates-using-android-recovery-and-adb-sideload/
https://www.theandroidsoul.com/how-to-install-ota-updates-manually-using-recovery-and-adb-sideload/
SQL

remove annoying crtl + C:
use --sigint-ignore
credtits https://dba.stackexchange.com/questions/20566/mysql-console-ctrlc-driving-me-nuts
Regex

easy regex generators
https://www.autoregex.xyz/

https://simple-regex.com/

https://regex-generator.olafneumann.org
R

clean code

(thanks to this SO post )

packages:

install.packages("formatR")
install.packages("styler")  -- styler doc and this tutorial
https://github.com/jimhester/lintr


Latex

tools to prettify latex code:

install latest texstudiousing these commands

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null

then
brew cask install texstudio


https://latexindentpl.readthedocs.io/en/latest/sec-how-to-use.html
to see nice diff, use --> --word-diff