Skip to content

Instantly share code, notes, and snippets.

@krisrak

krisrak/idc.md

Last active September 28, 2023 16:12
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save krisrak/fb72aa69f3e11cb20773fba3d04f5e98 to your computer and use it in GitHub Desktop.
Save krisrak/fb72aa69f3e11cb20773fba3d04f5e98 to your computer and use it in GitHub Desktop.
Download ZIP
This document shows how to setup and access Intel Developer Cloud via SSH and JupyterLab
Raw
idc.md

Intel Developer Cloud Beta

This document shows how to setup and access Intel Developer Cloud via SSH and JupyterLab.

(Updated 2023-09-26)

Topics

Login to Intel Developer Cloud

Intel® Max Series GPU (PVC) on 4th Gen Intel® Xeon® processors - 1100 series (4x) (Batch Processing/Scheduled access)
  • The first time, you will be presented with an option to input SSH public key

Adding SSH Key

  • You can add your existing key or create new key and add.
  • The SSH key should be RSA-4096 or ED25519

Create a new SSH Key and Add public key to Intel Developer Cloud

  • On your local terminal, create new SSH key:
    • ssh-keygen -t rsa -b 4096 -f ~/.ssh/idc
  • cat ~/.ssh/idc.pub
  • copy the key and add in the Intel Developer Cloud page and save.
  • the page will now show your user id and connection info, which looks like: ssh uXXXXXX@idcbetabatch.eglb.intel.com

SSH to Intel Developer Cloud

  • Option 1:
    • Connect using the connection info:
    • ssh uXXXXXX@idcbetabatch.eglb.intel.com -i ~/.ssh/idc
  • Option 2:
    • Connect using the config (next section shows how to create config):
    • ssh idc

This is should log you into the Intel Developer Cloud head node, the prompt will look like this:

uXXXXXX@idc-beta-batch-head-node:~$

Create a SSH config to make it easy to SSH

  • On your local terminal, edit SSH config file:
    • vi ~/.ssh/config
  • copy the following in the .ssh/config file:
Host idc
Hostname idcbetabatch.eglb.intel.com
User uXXXXXX
IdentityFile ~/.ssh/idc
#ProxyCommand /usr/bin/nc -x xxx.xxxxxxx.com:XXXX %h %p
ServerAliveInterval 60
ServerAliveCountMax 10
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
  • Edit the User to your user id
  • Edit the IdentityFile to your path to key that was added
  • Uncomment and update ProxyCommand if you need proxy to connect
  • save the config file

Troubleshooting SSH connection issues

  • make sure ssh public key added is of type RSA-4096 or ED25519
  • to add a new key, click on profile icon and update SSH key
  • if key was updated, delete the instance from View Instances tab and then Launch instance again
  • connect using ssh uXXXXXX@idcbetabatch.eglb.intel.com -i ~/.ssh/idc, edit your username and your path to private key

Commands to try:

  • SSH to Intel Developer Cloud
    • From you terminal: ssh idc
    • You should be logged in to the head node, with prompt similar to this: uXXXXXX@idc-beta-batch-head-node:~$
  • Check available GPU nodes: sinfo
    • From your head node: sinfo
    • This will display all available GPU nodes and will show status as available or idle
  • Run command on GPU node from head node: srun <command>
    • From head node try the following command:
    • srun ls
    • srun hostname
    • srun script.sh
      • (script.sh can anything you want to execute on GPU node)
  • Run command on GPU node by connecting interactively
    • From head node try the following command:
    • srun --pty bash
    • This will log you on to a node with GPU and you can run any command directly
  • Run command on specific GPU node by connecting interactively
    • From head node try the following command:
    • srun --nodelist=idc-beta-batch-pvc-node-03 --pty bash
    • This will log you on to node idc-beta-batch-pvc-node-03 and you can run any command directly

Launch Jupyter-Lab on Intel Developer Cloud for HPC

  • SSH to Intel Developer Cloud: ssh idc
  • log into GPU node: srun --pty bash
  • run the following commands to launch JupyterLab
conda init bash
source ~/.bashrc
conda activate pytorch_xpu
jupyter-lab --ip $(hostname -i)
  • you will see a URL similar to this displayed:
http://10.10.10.10:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
http://127.0.0.1:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
  • make a note of the IP and PORT : 10.10.10.10 and 8888
  • open another terminal to create a reverse tunnel by typing command ssh idc -L PORT:IP:PORT:

ssh uXXXXXX@idcbetabatch.eglb.intel.com -i ~/.ssh/idc -L 8888:10.10.10.10:8888

OR

ssh idc -L 8888:10.10.10.10:8888

  • Copy and paste your jupyter lab loopback address URL to a browser window: http://127.0.0.1:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
  • This should launch jupyterlab in browser

Launch Jupyter-Lab on Intel Developer Cloud for AI

  • SSH to Intel Developer Cloud: ssh idc
  • log into GPU node: srun --pty bash
  • run the following commands to launch JupyterLab
conda init bash
source ~/.bashrc
conda activate base
conda config --add channels intel
conda create -n idp intelpython3_full
conda activate idp
conda install -c conda-forge jupyterlab
pip3 install pandas
pip3 install matplotlib
jupyter-lab --ip $(hostname -i)
  • you will see a URL similar to this displayed:
http://10.10.10.10:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
http://127.0.0.1:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
  • make a note of the IP and PORT : 10.10.10.10 and 8888
  • open another terminal to create a reverse tunnel by typing command ssh idc -L PORT:IP:PORT:

ssh uXXXXXX@idcbetabatch.eglb.intel.com -i ~/.ssh/idc -L 8888:10.10.10.10:8888

OR

ssh idc -L 8888:10.10.10.10:8888

  • Copy and paste your jupyter lab loopback address URL to a browser window: http://127.0.0.1:8888/lab?token=19ace0257312e8014115917ebf6d2195b0f16163c166ed76
  • This should launch jupyterlab in browser
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment