coin8086

Distributed Model Training in PyTorch DDP

Prerequisites

Ensure GPU device is ready

Check GPU by lspci

Ensure enough free disk space

coin8086

PowerShell vs Bash

Key different points

• Bash commands output strings while PS commands output objects!

Examples

The following standard aliases in PowerShell are used:

coin8086

WSL Settings and Issue Fixes

Enable systemd

1. Get the latest version WSL from the Windows Store

2. Ensure /etc/wsl.conf has

[boot]

coin8086

cgroup and systemd

cgroup

"The control groups, abbreviated as cgroups in this guide, are a Linux kernel feature that allows you to allocate resources — such as CPU time, system memory, network bandwidth, or combinations of these resources — among hierarchically ordered groups of processes running on a system."

Check the cgroups by systemd-cgls.

Resource controller

coin8086

Setup Service Fabric Development Environment on Linux

The official document at https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-get-started-linux is problematic (ESPECIALLY, DO NOT follow the instruction in that document to install Docker!). Here're the correct steps.

Prerequisites

1. Check if your Linux distro and version are supported at https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-versions#supported-linux-versions-and-support-end-date
2. Setup Docker as instructed by https://docs.docker.com/engine/install/ubuntu

Install Service Fabric SDK

coin8086

Linux Linkers and Loaders

Tools

• ldconfig
• ld.so
• ld
• ldd

ldconfig

coin8086

Proxy

Proxy vs VPN

• Proxy is not VPN.
• An application could provide an option to use a proxy, by a command line parameter, or an environment variable. Or it could provide no option for proxy at all!
• VPN is an easy way to "proxy" the whole system without any application specific settings.

A proxy could use SOCKS or HTTP protocol.

coin8086

SSH Port Forwarding

Refer to https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding

Local Port Forwarding

man:

-L [bind_address:]port:remote_socket

coin8086

Autoscale in CycleCloud

What is CycleCloud

CycleCloud is a product that

• makes it easy to create and manage HPC computer clusters
• is open to integrate with various HPC schedulers, like PBS, Slurm, HPC Pack, etc.
• is deployed by user

coin8086

PBS in Brief

Core Concepts

• resource, chunk, vnode and host
• job
  • job array
• queue
• server