Notes on SSH - SSH is so confusing!!!

0_ssh_notes.md

SSH Notes

I'm using the convention here that user settings are written in all caps (like USERNAME and SERVER).

Note: Variable SERVER is either a url or an IP.

SSH Client (Connecting to an SSH Server)

• The basic command is ssh USERNAME@SERVER.
• The default SSH Server Port is 22, but a custom port can be specified like this: ssh USERNAME@SERVER -p PORT.
• Running Command immediately after connecting: ssh -t USERNAME@SERVER tmux attach runs tmux attach command in the server. Note that -t command allocates a pseudo-terminal (which is required by tmux). Non-interactive command like ls doesn't need this, so -t can be omitted.

If the SSH command gives error Corrupted mac on input, then you can specify the correct MACs (Message Authentication Codes) manually like this: ssh USERNAME@SERVER -m hmac-sha2-512. Here a single MAC hmac-sha2-512 is specified, but multiple MACs can be set using comma seperated values. For example: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

Authentication with Public & Private Keys

The RSA Private Key is stored by default in file ~/.ssh/id_rsa. But many people also store it in PEM files - these have the same file format, just the file extension is .pem.

The location of this Private Key file is specified as: ssh USERNAME@SERVER -i PATH_TO_PEM_FILE.

Note: If you get the error Permissions for 'private-key' are too open, then do the following and try again:

• Linux / Mac: chmod 600 PATH_TO_PEM_FILE - This makes the current user the file owner with Read & Write permissions, and no permissions for any other non-root user.
• Windows: Use the script in this StackOverflow answer to disable inheritance, make yourself owner and remove all permissions from other users. Make sure to replace the value of Key variable in the script with the path to your PEM file.

Authentication with Plain-Text Password

This is a less secure method than using Public & Private Keys, but is still used by some servers. There are 2 options:

• Connect to server with command ssh USERNAME@SERVER. This should show a password prompt. Enter your password here.
• Install sshpass and use it like this: sshpass -p PASSWORD ssh USERNAME@SERVER -p PORT.

SSH Configuration File

Open ~/.ssh/config (create it if it doesn't exist) and enter the following in it:

Host USER_GIVEN_CONNECTION_NAME
  User USERNAME
  IdentityFile PATH_TO_PEM_FILE
  HostName SERVER
  Port PORT
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
  
  # 1 minute Keep-Alive to avoid unwanted disconnect
  ServerAliveInterval 60

Here, only User and HostName are required fields, rest are all optional. Now we can connect to the SSH Server simply with: ssh USER_GIVEN_CONNECTION_NAME. This is equivalent to entering all the details in command itself like this:

ssh -o ServerAliveInterval=60 USERNAME@SERVER -p PORT -i PATH_TO_PEM_FILE -m hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

In above example ServerAliveInterval setting was set to 60. This means that if SSH Connection is left idle for 60 seconds, then an empty packet will be sent to the SSH server to prevent SSH connection from disconnecting.

Decrypting PEM / Identity File

Sometimes, an encrypted PEM file may be provided for SSH login. So its passphrase will have to be entered every time when connecting. This can be inconvinient - see here for how to decrypt PEM file using ssh-keygen -p so that passphrase doesn't need to be entered!

Note: While doing ssh-keygen -p, if UNPROTECTED PRIVATE KEY FILE error shows, make the PEM file secure by following this answer.

Downloading / Uploading files

This can be done using scp command:

• Download: scp USERNAME@SERVER:/server/path/to/file /local/path/where/file/to/be/downloaded
• Upload: scp /local/path/where/file/to/be/uploaded USERNAME@SERVER:/server/path/to/file

1_jupyter_remote_server.md

How to connect to a Jupyter Notebook / Lab running on a Server via Gateway

WARNING: This jupyter setup on SSH server might have security vulnerabilities - need to check more thoroughly!

This guide is written in the context of the following setup:

              username                     username
Local Machine ------------------> Gateway ----------> Server
              PEM `IdentityFile`           password

Note: Gateway and Jump Host mean the same thing here - it's just a normal server, that is the only machine allowed to access a server. So when we want to connect to the server, we first have to connect to the gateway, and then connect to the actual server from the gateway.

Credit: Chat GPT was a massive help in setting this up!

In your PC / laptop:

• Run command ssh -V. This should show the version of your SSH client. If you get error Command Not Found, install SSH on your laptop.
• Put the following in ~/.ssh/config file:

# We will set a 1 minute keep alive to keep the connection
# active if there is no activity to avoid unwanted disconnects
Host *
  ServerAliveInterval 60

Host jump-host
  User GATEWAY-USERNAME-HERE
  IdentityFile GATEWAY-PEM-FILE-PATH-HERE
  HostName GATEWAY-IP-HERE
  Port GATEWAY-PORT-HERE
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

# Here name `ecomm` is used for the server, but any name can be used
Host ecomm
  HostName SERVER-IP
  Port SERVER-PORT
  User SERVER-USERNAME-HERE 
  MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
  ProxyCommand ssh -W %h:%p jump-host

• Modify permissions of ~/.ssh folder and ~/.ssh/config file as shown here.

In Ecomm Server

Connect to server using terminal (ssh ecomm) or GUI tools like MobaXTerm. You will be prompted for password - enter your server password here (in this case, Ecomm Server's password). Now install & run either Jupyter Notebook or Jupyter Lab:

• pip3 install jupyter (Jupyter Notebook) OR pip3 install jupyterlab (Jupyter Lab)
• jupyter notebook --no-browser --port=8888 (Jupyter Notebook) OR jupyter-lab --no-browser --port=8888 (Jupyter Lab) This will a Jupyter Server on port 8888. A connection url (of the form http://localhost:8888/?token=SOME_TOKEN_HERE) should be printed. Save this connection url somewhere - it will be used later.

Note: There's nothing special about port 8888 - you can use any other free port also.

In your PC / laptop

• Open a new terminal and enter the command ssh -N -L 8888:localhost:8888 ecomm. First a password prompt shows - enter your Ecomm Server's password here. This binds port 8888 in your laptop to port 8888 on the server (which is the port where Jupyter server was started).
• Now open the connection url (which was copied from output of jupyter notebook in Ecomm Server). This will open a Jupyter Notebook / Lab interface in your browser.

Opening Jupyter Notebook in VS Code (optional)

• Open a Jupyter Notebook in VS Code, and run command Select Jupyter Kernel.
• Select Existing Kernel, and then paste the connection url (obtained above from jupyter output in server).
• Now your code in Jupyter Notebook runs directly in the server 🙂

Jupyter Notebook / Lab Config (optional)

In Ecomm Server:

• Run jupyter notebook --generate-config (or jupyter lab --generate-config - both do almost the same thing).
• Open config file path printed above (should be ~/.jupyter/jupyter_notebook_config.py) and edit these settings:

# Jupyter URL prefix - here I've used the server name, but any prefix can be used.
# http://localhost:8888/?token={TOKEN_HERE} (old) -> http://localhost:8888/ecomm/?token={TOKEN_HERE} (new)
# I find this helpful when I am using multiple Jupyter servers at the same time
# (since I can easily understand which server I'm running on from the url only)
c.NotebookApp.base_url = '/ecomm/' 

c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888         # or whatever port you want       

• Now run jupyter notebook (or jupyter lab). Notice that --no-browser and --port=8888 options were not provided, as these were already specified in config. Forward this port and open Jupyter url in your browser (as mentioned above).

Note: If you're using Jupyter Lab, for forward portability, replace NotebookApp in above config with ServerApp.

VS Code Remote Connection (optional)

We can tell VS Code to connect to a Remote SSH server in a new window. This will allow us to easily open files & folders, as well as run commands in terminal in the remote machine.

Password-less authentication (optional)

Entering password every time is quite tedious. To avoid this do the following:

• In laptop: (This step should be done once only)

  • Run command ssh-keygen -t rsa.
  • Open public key file ~/id_rsa.pub - this should have a single line starting with *ssh-rsa *. Note: ~ means your home folder - on Windows, this is C:/Users/YOUR-USERNAME-HERE.

• In Ecomm Server:

  • Create folder ~/.ssh if it doesn't exist.
  • Add the contents of id_rsa.pub (from laptop) on a new line in ~/.ssh/authorized_keys.
  • Set correct file & folder permissions:

  chmod u=rwx,g=,o= ~/.ssh
  chmod u=rw,g=,o= ~/.ssh/authorized_keys

  Note: Your home folder should only be readable by you and nobody else. This should be the case by default, but if not, you can correct permissions for your home folder using chmod go-w ~.

Next time when you login to the server using ssh ecomm, you should connect directly without any password prompt.