gathering documentation thoughts for AWS batch use at the Center

aws_batch.md

Using AWS Batch at the Center

Documentation

Outline

• Obtaining credentials
• Getting permissions to use Batch
• Choosing a Compute Environment (for scicomp - how to create a new one, and document the default ones)
• (Choosing a job queue?)
• Creating a Docker image (using github/docker hub to auto-build images), including easybuild & /app (scicomp only) tagging of containers
• Creating a Job Definition
• Creating a Job
• Monitoring your job
• Looking at job logs
• Troubleshooting
• Tips and tricks:
  • AWS tab completion
  • always use --sse with S3 transfers
  • filtering output with jq
  • Adding scratch space; ramdisk and other methods
• Examples:
  • simple hello example
  • more complex example

Wrapper script

Features

• Simplifies operation.
  • User does not have to think about queues (?)
  • Automatically tags all created resources w/user's hutchnet id etc.

Operations:

• Create job definition - options for scratch space, # cpus, etc.
• Submit job - options for cmd line/paramemters
• View job status - show all user's jobs rgdls of status
• Show job output - simple text output, optionally filter stdout/stderr
• Terminate job (?)