Skip to content

Instantly share code, notes, and snippets.

@leminhtr

leminhtr/linux_machine_learning_setup_from_scratch.md

Last active June 4, 2024 10:22
Show Gist options
  • Save leminhtr/21039afc18642a0d2ff4e826f9626a7a to your computer and use it in GitHub Desktop.
Save leminhtr/21039afc18642a0d2ff4e826f9626a7a to your computer and use it in GitHub Desktop.
Download ZIP
Linux python setup for machine learning from scratch
Raw
linux_machine_learning_setup_from_scratch.md

LINUX ENVIRONMENT SETUP FROM SCRATCH (~40mn)

I. Linux environment

Setup folder structure

cd && mkdir -p dev/repositories &&
cd && mkdir Downloads &&
cd && mkdir Shares;
cd && mkdir -p dev/binaries;

Add alias to ~/.zshrc

alias reload-terminal='exec zsh'

alias grep='grep  --color=auto'
alias cat="batcat -ppn" #replace bat by cat
alias ls="colorls --sd"
# alias ls='ls --color=auto'
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias cd..='cd ..'
alias ..='cd ..'
alias ...='cd ../../'
alias ....='cd ../../../'
alias .....='cd ../../../../'
alias cdrepo='cd $HOME/dev/repositories'

alias cdl='cd $1 & ls'
alias ll='ls -l'
alias la='ls -A'
alias l='ls'
alias lla='ls -lA'
alias finst='sudo apt-fast install -y'
alias fupdate='sudo apt-fast update -y'
alias fdistupgrade='sudo apt-fast dist-upgrade -y'

## get top process eating memory
alias psmem='ps auxf | sort -nr -k 4'
alias psmem3='ps auxf | sort -nr -k 4 | head -3'
alias psmem10='ps auxf | sort -nr -k 4 | head -10'
## get top process eating cpu ##
alias pscpu='ps auxf | sort -nr -k 3'
alias pscpu3='ps auxf | sort -nr -k 3 | head -3'
alias pscpu10='ps auxf | sort -nr -k 3 | head -10'
alias meminfo='free -m -l -t'
alias cpuinfo='lscpu'

alias untar='tar -xvzf'


# pretty ps aux | grep
psgrep() { ps up $(pgrep -f $@) 2>&-; }

alias pip='python -m pip'
alias pipinstall='python -m pip install'
alias pip_upgrade='python -m pip install --upgrade pip'
alias pip_update='python -m pip install --upgrade'

export BAT_THEME="Monokai Extended Origin"


function tensorboard_open ()
{
        tensorboard --port 8888 --logdir $1
}

install apt-fast: https://doc.ubuntu-fr.org/apt-fast

sudo apt-add-repository ppa:apt-fast/stable -y && sudo apt-get update && sudo apt-get -y install apt-fast;
1. Maximum number of connections = 32
2. Suppress apt-fast confirmation = yes

Install zsh, git, curl, ...:

sudo apt-fast install zsh git curl bat autojump make htop cifs-utils -y

Install LaTex

sudo apt-fast install texlive-full texlive-latex-extra texlive-fonts-recommended dvipng cm-super -y

Setup oh-my-zsh terminal & friends

oh-my-zsh

  1. Change default terminal to zsh: ➡ Add this to the top of ~/.profile: nano ~/.profile
if [ "$SHELL" != "/usr/bin/zsh" ]
then
     export SHELL="/usr/bin/zsh"
     exec zsh -l    # -l: login shell again
fi
  • Log out/login again
  1. Install oh-my-zsh
sh -c "$(wget https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)";
  1. Install powerlevel10k:
git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ~/powerlevel10k
echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >>~/.zshrc
  1. Configure style using: p10k configure if not done.

Install fzf for fuzzy typing

cd ~/Downloads && \
git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf && \
~/.fzf/install

Install zplug to manage/auto-update zsh plugins

curl -sL --proto-redir -all,https https://raw.githubusercontent.com/zplug/installer/master/installer.zsh | zsh

Install zsh completions

cd ~/Downloads && \
git clone https://github.com/zsh-users/zsh-completions ~/.oh-my-zsh/plugins/zsh-completions

Add this to .zshrc after source $ZSH/oh-my-zsh.sh to install plugins:

# zplug
source ~/.zplug/init.zsh

# Make sure to use double quotes
zplug 'zplug/zplug', hook-build:'zplug --self-manage'
zplug "zsh-users/zsh-history-substring-search"
zplug "plugins/git",   from:oh-my-zsh
zplug "plugins/dircycle",   from:oh-my-zsh
zplug "plugins/extract",   from:oh-my-zsh
zplug "plugins/command-not-found",   from:oh-my-zsh
zplug "plugins/colored-man-pages",   from:oh-my-zsh
zplug "plugins/common-aliases",   from:oh-my-zsh
zplug "plugins/autojump",   from:oh-my-zsh
zplug "plugins/colored-man-pages", from:oh-my-zsh
zplug "gretzky/auto-color-ls", as:command
zplug romkatv/powerlevel10k, as:theme, depth:1
zplug "MichaelAquilina/zsh-auto-notify"
zplug "marlonrichert/zsh-autocomplete"
zplug "zsh-users/zsh-completions"
zplug "mtxr/zsh-change-case"
zplug "zpm-zsh/clipboard" # echo hello | pbcopy, pbpaste | grep hi, clip
zplug "supercrabtree/k"
zplug "zsh-users/zsh-autosuggestions"
zplug "zsh-users/zsh-syntax-highlighting"
# zplug "g-plane/zsh-yarn-autocompletions", hook-build:"./zplug.zsh", defer:2
# setopt  autocd autopushd \ pushdignoredups
source ~/.zplug/repos/marlonrichert/zsh-autocomplete/zsh-autocomplete.plugin.zsh

# echo "include ~/.nano/syntax/sh.nanorc
# include ~/.nano/syntax/autoconf.nanorc
# include ~/.nano/syntax/c.nanorc
# include ~/.nano/syntax/default.nanorc
# include ~/.nano/syntax/makefile.nanorc 
# include ~/.nano/syntax/json.nanorc  
# include ~/.nano/syntax/html.nanorc
# include ~/.nano/syntax/man.nanorc 
# include ~/.nano/syntax/sh.nanorc
# include ~/.nano/syntax/python.nanorc
# include ~/.nano/syntax/tex.nanorc
# include ~/.nano/syntax/xml.nanorc
# include ~/.nano/syntax/cmake.nanorc" >> ~/.nanorc;

# Install plugins if there are plugins that have not been installed
if ! zplug check --verbose; then
    printf "Install? [y/N]: "
    if read -q; then
        echo; zplug install
    fi
fi

Add this to .zshrc below "bindkey -e"

bindkey "^[[1;5C" forward-word
bindkey "^[[1;5D" backward-word
  • Re-login again to install zplug plugins

Install nanorc : nano w/ syntax highlightning

cd ~/dev/repositories && \
git clone https://github.com/scopatz/nanorc.git && \
cd nanorc && \
make install

Install colorls with ruby

  1. Install dependencies
sudo apt-fast install git curl autoconf bison build-essential \
    libssl-dev libyaml-dev libreadline6-dev zlib1g-dev \
    libncurses5-dev libffi-dev libgdbm6 libgdbm-dev libdb-dev yank direnv autojump -y
  1. Install ruby using rvenv
curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash - && \
sudo apt-fast install ruby-full rubygems -y
  1. Add rbenv to .zshrc
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(rbenv init -)"' >> ~/.zshrc
  1. Install colorls
sudo gem install colorls
echo "source $(dirname $(gem which colorls))/tab_complete.sh" >> .zshrc;
echo 'PATH=$PATH:$(ruby -e 'puts Gem.bindir')' >> ~/.zshrc

To mount a cifs drive :

FOLDER = myfolder
USER = myuser
cd && mkdir Shares/$FOLDER
sudo mount -t cifs //server_name/path/to/remote/$FOLDER ~/Shares/$FOLDER -o username=$USER,dir_mode=0755,file_mode=0755 # Mount windows drive

Other tools

jj: json parser

cd ~/dev/binaries &&
wget https://github.com/tidwall/jj/releases/download/v1.9.2/jj-1.9.2-linux-amd64.tar.gz
tar -xvf jj-1.9.2-linux-amd64.tar.gz
echo 'export PATH="/binaries/jj-1.9.2-linux-amd64/:$PATH"' >> ~/.zshrc

II. PYTHON MACHINE LEARNING SETUP

See this tutorial to make a clean python Deep Learning GPU setup with TensorFlow 2.X.X & PyTorch 1.X

III. Misc

  1. Generate a ssh key:
ssh-keygen
  1. Sublime-text setup: Install these packages:
  • git git blame github notifications git blame
  • diff tab
  • markdown extended markdown preview
  • tabnine
  • rainbow brackets rainbow csv bracketflasher
  • jsonReindent
  • terminal
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment