Skip to content

Instantly share code, notes, and snippets.

@sono-bfio

sono-bfio/boost_cluster_update.md

Last active August 27, 2016 14:01
Show Gist options
  • Save sono-bfio/c5ee4ea2af1484967d8e1155c4583e52 to your computer and use it in GitHub Desktop.
Save sono-bfio/c5ee4ea2af1484967d8e1155c4583e52 to your computer and use it in GitHub Desktop.
Download ZIP
Updating a boost cluster
Raw
boost_cluster_update.md

Upgrading the GPU nodes

1. Launch Bitfusion Boost cluster
2. Log into the client node
$ ssh -l ubuntu -i path/to/your_key.pem {IP}
3. Get the list of connected hosts
$ cat /etc/bitfusionio/adaptor.conf

# Example output below
10.0.3.45
10.0.7.117
4. For each node SSH to it and carry out the upgrade

You will need to copy the SSH key you used to connect to the client node to each of the GPU nodes show in /etc/bitfusionio/adaptor.conf

####### 4a. Stop the boost servers

$ sudo service bfboost-opencl stop
$ sudo service bfboost-cuda-server stop

####### 4b. Remove the bitfusionio config dir

$sudo /bin/rm -rf /etc/bitfusionio

####### 4c. Update apt and get the latest boost version

$ sudo apt-get update
$ sudo apt-get -y install bfboost

####### 4c. Verify you have version 0.1.0+1562

$ sudo dpkg --list | grep bfboost
ii  bfboost     0.1.0+1562        amd64       Bitfusionio Boost

######## 4e. Start the service back up

$ sudo service bfboost start

######## 4e. Verify process are running:

$ ps -auxwf | grep -A3 bfboost

####### 4g. Exit back to the head or client node and upgrade it as well

$ exit

Upgrading the the Client Node

1. Stop the boost services
 $ sudo service bfboost-opencl stop
 $ sudo service bfboost-cuda-server stop
2. Remove the bitfusio config dir
 sudo /bin/rm -rf /etc/bitfusion*
3. Update apt and get the latest boost version
 $ sudo apt-get update
 $ sudo apt-get install -y bfboost

####### 4. Verify you have version 0.1.0+1562

 $ sudo dpkg --list | grep bfboost
 ii  bfboost     0.1.0+1562        amd64       Bitfusionio Boost

####### 5. Start Boost

 $ sudo service bfboost start

Verify with Device Query

$ bfboost client  /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery

You should see something similar too:

BFBoost (c) bitfusion.io 2016 licensed to AWS Generic BBE8amtXcntmtJ4hNrRW-1713/0, expires on Sat, 06 Jul 2019 03:51:43 UTC
[/usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery]
/usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 2 CUDA Capable device(s)

Device 0: "GRID K520"
  CUDA Driver Version / Runtime Version          7.5 / 7.5
  CUDA Capability Major/Minor version number:    3.0
  Total amount of global memory:                 4096 MBytes (4294770688 bytes)
  ( 8) Multiprocessors, (192) CUDA Cores/MP:     1536 CUDA Cores
  GPU Max Clock rate:                            797 MHz (0.80 GHz)
  Memory Clock rate:                             2500 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 3
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

Device 1: "GRID K520"
  CUDA Driver Version / Runtime Version          7.5 / 7.5
  CUDA Capability Major/Minor version number:    3.0
  Total amount of global memory:                 4096 MBytes (4294770688 bytes)
  ( 8) Multiprocessors, (192) CUDA Cores/MP:     1536 CUDA Cores
  GPU Max Clock rate:                            797 MHz (0.80 GHz)
  Memory Clock rate:                             2500 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 3
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from GRID K520 (GPU0) -> GRID K520 (GPU1) : No
> Peer access from GRID K520 (GPU1) -> GRID K520 (GPU0) : No

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 2, Device0 = GRID K520, Device1 = GRID K520
Result = PASS
{
  "privateIp" : "10.0.7.117",
  "devpayProductCodes" : null,
  "availabilityZone" : "us-east-1a",
  "version" : "2010-08-31",
  "region" : "us-east-1",
  "accountId" : "863665633681",
  "instanceId" : "i-8f6ad51f",
  "billingProducts" : null,
  "instanceType" : "g2.2xlarge",
  "pendingTime" : "2016-07-06T03:16:13Z",
  "imageId" : "ami-9772b1fa",
  "architecture" : "x86_64",
  "kernelId" : null,
  "ramdiskId" : null
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment