CycleCloud

Autoscale in CycleCloud.md

Autoscale in CycleCloud

What is CycleCloud

CycleCloud is a product that

• makes it easy to create and manage HPC computer clusters
• is open to integrate with various HPC schedulers, like PBS, Slurm, HPC Pack, etc.
• is deployed by user

See more at

• https://docs.microsoft.com/en-us/azure/cyclecloud/overview
• https://docs.microsoft.com/en-us/azure/cyclecloud/qs-install-marketplace

The problems CycleCloud is trying to resolve are

• Cluster provisioning by Cluster Template
• Cluster autoscaling by autoscale lib

Autoscale Lib

The autoscale lib is a python library (hpc.autoscale), which depends on CycleCloud Python Client (cyclecloud.client).

Overview

NOTE in the diagram:

• The "Autoscale Lib" is not a standalone process, but a part of the "Autoscale Routine" process. Here they're separated to show the work flow between them.
• The "Autoscale Routine" and "Scheduler" are two standalone processes.

Example

https://github.com/Azure/cyclecloud-scalelib/tree/master/example-celery

The autoscale routine runs periodically to decide whether to scale up or down a cluster by:

1. Collect host and job information from a scheduler (Celery in the example).
2. Add hosts and jobs to a Demand Calculator (demandcalculator from hpc.autoscale.job), calculate demand
3. Launch new hosts, and/or delete existing hosts as recommended by the Demand Calculator

Scheduler Integration

• PBS Pro (using the Demand Calculator from autoscale lib)
• HPC Pack (using the autoscale lib but not the Demand Calculator)
• Slurm (using CycleCloud Client directly, without the autoscale lib)

NOTE: Whether an integration uses autoscale lib or not, the main work flow remains the same as above overview. The point is: the autoscale routine decides what to do, and eventually makes it by calling CycleCloud REST API.

Create a Custom Application in CycleCloud.md

Create a Custom Application in CycleCloud

Prerequisites:

• A CycleCloud Instance
• CycleCloud CLI

Steps:

1. Create a new project by cyclecloud project init
2. Build & upload the project to CycleCloud storage by cyclecloud project upload
3. Import the project template into CycleCloud by cyclecloud import_cluster -t.

After you import the new template, you can create an instance of it either in UI or by CLI.

Also refer to:

• https://github.com/coin8086/cyclecloud-scalelib/blob/example/example-celery/README.md
• https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/projects
• https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/create-cluster