AWS batch

aws_batch_commands.md

AWS-CLI way

First create a skeleton to edit

aws batch submit-job --generate-cli-skeleton > job.json

Edit the script as needed e.g. for the fetch-and-run job the final file FetchRunClijob.json looks like this

{
    "jobName": "fetchRunBatchCli",
    "jobQueue": "first-run-job-queue",
    "jobDefinition": "fetch_and_run_trial2:2",
    "containerOverrides": {
        "vcpus": 1,
        "memory": 500,
        "command": [
            "myjob.sh",
            "60"
        ],
        "environment": [
            {
                "name": "BATCH_FILE_TYPE",
                "value": "script"
            },
            {
                "name": "BATCH_FILE_S3_URL",
                "value": "s3://nsarodetest/myjob.sh"
            }
        ]
    }
}

Submit the job

aws batch submit-job --cli-input-json file://FetchRunClijob.json

Check status of the job

Detailed

aws batch describe-jobs --jobs JOBID

Just for status i.e. one of SUBMITTED, PENDING, RUNNABLE, STARTING, RUNNING, FAILED, SUCCEEDED

NOTE: jq (JSON parser) needs to be installed for this to work

aws batch describe-jobs --jobs JOBID | jq -r '.jobs[0].status'