A log of getting start with AWS Batch.

getting-start-with-AWS-Batch.md

AWS Batch

• Ref. https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html

The goal of this document is submitting a job to AWS Batch and confirming the result in CloudWatch Logs.

Since I've worked on ap-northeast-1 region, The following examples includes this region name.

Setting Up with AWS Batch

• Ref. https://docs.aws.amazon.com/batch/latest/userguide/get-set-up-for-aws-batch.html

You need to create some AWS resources as the document explained.

• Create IAM Roles for your Compute Environments and Container Instances
• Create a Key Pair
• Create a Virtual Private Cloud
• Create a Security Group

The details of VPC configurations

When I set up everything and submitted a job to AWS Batch, the submitted job stuck in runnable.

• Ref. https://docs.aws.amazon.com/batch/latest/userguide/troubleshooting.html#job_stuck_in_runnable
• Ref. https://forums.aws.amazon.com/thread.jspa?threadID=247401

The problems was a wrong configuration of MapPublicIpOnLaunch. It have to be true as shown below.

Managed compute environments launch Amazon ECS container instances into the VPC and subnets that you specify when you create the compute environment. Amazon ECS container instances need external network access to communicate with the Amazon ECS service endpoint. If your container instances do not have public IP addresses (because the subnets you've chosen do not provide them by default), then they must use network address translation (NAT) to provide this access. Ref. https://docs.aws.amazon.com/batch/latest/userguide/compute_environments.html#managed_compute_environments

See also https://docs.aws.amazon.com/batch/latest/userguide/create-public-private-vpc.html

Select Enable auto-assign public IPv4 address and choose Save, Close.

If you create VPC and Public Subnet from VPC with a Single Public Subnet in Start VPC Wizard, the creaated public subnet has MapPublicIpOnLaunch as false by default. So you have to enable it.

$ aws ec2 describe-subnets
{
    "Subnets": [
        {
            "AvailabilityZone": "ap-northeast-1a",
            "Tags": [
                {
                    "Value": "Public subnet",
                    "Key": "Name"
                }
            ],
            "AvailableIpAddressCount": 251,
            "DefaultForAz": false,
            "Ipv6CidrBlockAssociationSet": [],
            "VpcId": "[your_vpc_id]",
            "State": "available",
            "MapPublicIpOnLaunch": true,
            "SubnetId": "[your_subnet_id]",
            "CidrBlock": "10.0.0.0/24",
            "AssignIpv6AddressOnCreation": false
        }
    ]
}

Another problem I had was EnableDnsHostnames: false. It have to be true as well.

$ aws ec2 describe-vpc-attribute --vpc-id [your_vpc_id] --attribute enableDnsSupport
{
    "VpcId": "[your_vpc_id]",
    "EnableDnsSupport": {
        "Value": true
    }
}
$ aws ec2 describe-vpc-attribute --vpc-id [your_vpc_id] --attribute enableDnsHostnames
{
    "VpcId": "[your_vpc_id]",
    "EnableDnsHostnames": {
        "Value": true
    }
}

Getting Started with AWS Batch

• Ref. https://docs.aws.amazon.com/batch/latest/userguide/Batch_GetStarted.html

Create compute environment

$ aws batch create-compute-environment --generate-cli-skeleton
$ aws batch create-compute-environment --cli-input-json file://compute-environments.json
$ aws batch describe-compute-environments
$ cat compute-environments.json
{
    "computeEnvironmentName": "on-demand-compute-environment",
    "type": "MANAGED",
    "state": "ENABLED",
    "computeResources": {
        "type": "EC2",
        "minvCpus": 0,
        "maxvCpus": 1,
        "desiredvCpus": 0,
        "imageId": "",
        "instanceTypes": [
            "m3.medium"
        ],
        "subnets": [
            "[your_subnet_id]"
        ],
        "securityGroupIds": [
            "[your_security_group_id]"
        ],
        "ec2KeyPair": "[your_key_pair_name]",
        "instanceRole": "arn:aws:iam::[aws_account_id]:instance-profile/ecsInstanceRole",
        "tags": {
            "Name": "ex-aws-batch"
        },
        "bidPercentage": 0,
        "spotIamFleetRole": ""
    },
    "serviceRole": "arn:aws:iam::[aws_account_id]:role/service-role/AWSBatchServiceRole"
}

Create job queue

$ aws batch create-job-queue --generate-cli-skeleton
$ aws batch create-job-queue --cli-input-json file://job-queue.json
$ aws batch describe-job-queues
$ cat job-queue.json
{
    "jobQueueName": "ex-aws-batch",
    "state": "ENABLED",
    "priority": 1,
    "computeEnvironmentOrder": [
        {
            "order": 1, "computeEnvironment": "arn:aws:batch:ap-northeast-1:[aws_account_id]:compute-environment/on-demand-compute-environment"
        }
    ]
}

Register job definition

$ aws batch register-job-definition --generate-cli-skeleton
$ aws batch register-job-definition --cli-input-json file://job-definition.json
$ aws batch describe-job-definitions
$ cat job-definition.json
{
    "jobDefinitionName": "my-test-job",
    "type": "container",
    "parameters": {
        "Name": "alice"
    },
    "containerProperties": {
        "image": "busybox",
        "vcpus": 1,
        "memory": 500,
        "command": [
            "echo",
            "Ref::name"
        ],
        "jobRoleArn": "",
        "volumes": [
        ],
        "environment": [],
        "mountPoints": [],
        "readonlyRootFilesystem": true,
        "privileged": true,
        "ulimits": [],
        "user": ""
    },
    "retryStrategy": {
        "attempts": 1
    }
}

Submit a job

$ aws batch submit-job \
  --job-name test \
  --job-queue arn:aws:batch:ap-northeast-1:[aws_account_id]:job-queue/ex-aws-batch \
  --job-definition arn:aws:batch:ap-northeast-1:[aws_account_id]:job-definition/my-test-job:1 \
  --parameters name=bob

Confirm the result

• Ref. https://github.com/jorgebastida/awslogs After submitting a job to AWS Batch, you have to be able to see a log in jobDefinitionName/default/ecs_task_id log stream which belongs to /aws/batch/job.

$ awslogs get /aws/batch/job ALL --watch
/aws/batch/job my-test-job/default/dbee4196-0d1c-4d0b-bc13-73642c759b9f bob