Setup user and project level tags in AWS ParallelCluster

cost-explorer-tags.md

AWS ParallelCluster Cost Explorer Tags

In a previous gist we discussed using cost explorer to see how much a cluster costs at the instance type and cluster level.

This gist describes how to get cost at the Project and User level.

Setup

1. Create a Policy

1. Create a new IAM policy called pclusterTagsAndBudget

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteTags",
                "ec2:DescribeTags",
                "ec2:CreateTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "budgets:ViewBudget"
            ],
            "Resource": "arn:aws:budgets::*:budget/*"
        }
    ]
}

2. Upload Files

1. Next we need to modify some of the files, first open up the file post_install.sh and modify the line at the top:

bucket="<bucket>"

Change that to the name of your bucket

2. Next we need to upload all the files to our S3 bucket:

aws s3 cp budget.json s3://<bucket>
aws s3 cp notifications-with-subscribers.json s3://<bucket>
aws s3 cp sbatch s3://<bucket>
aws s3 cp projects_list.conf s3://<bucket>
aws s3 cp post_install.sh s3://<bucket>

3. Create Cluster

1. Create a cluster with the following minimal config, make sure to substitute <bucket> for the name of your bucket you just uploaded those files too:

Image:
  Os: alinux2
HeadNode:
  InstanceType: t2.micro
  Networking:
    SubnetId: <subnet-id>
  Ssh:
    KeyName: <key>
  CustomActions:
    OnNodeConfigured:
      Script: s3://<bucket>/post_install.sh
  Iam:
    S3Access:
      - BucketName: <bucket>
        EnableWriteAccess: False
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::<account_id>:policy/pclusterTagsAndBudget
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: queue1
      Networking:
        SubnetIds:
          - <subnet-id>
      ComputeResources:
        - Name: compute-resource1
          InstanceType: t2.micro
          MinCount: 0
          MaxCount: 10
      CustomActions:
        OnNodeConfigured:
          Script: s3://<bucket>/post_install.sh
        AdditionalIamPolicies:
          - Policy: arn:aws:iam::<account_id>:policy/pclusterTagsAndBudget
      Iam:
        S3Access:
          - BucketName: <bucket>
            EnableWriteAccess: False
Tags:
  - Key: aws-parallelcluster-username
    Value: NA
  - Key: aws-parallelcluster-jobid
    Value: NA
  - Key: aws-parallelcluster-project
    Value: NA

4. Activate Tags in Cost Explorer

1. Activate the aws-parallelcluster-* tags in the Billing Dashboard > Cost Allocation Tags
2. Wait 24 hours for the tags to be activated 📆 🥱

5. Test job submission

You'll now be able to submit jobs and have those jobs get the following tags:

Tag	Description
aws-parallelcluster-username	user who submitted the job
aws-parallelcluster-project	project name specified in --comment <project-name>
aws-parallelcluster-jobid	the id of the submitted job
aws-parallelcluster-jobname	the name of the submitted job
aws-parallelcluster-partition	The Slurm partition these jobs were submitted too.
aws-parallelcluster-account	(Optional) the account associated with the user, needs to be configured in slurm.

To submit a job, login to the cluster as a user

sbatch --comment ProjectA submit.sh

Wait until that job starts running, then take a look at the EC2 Instance and you'll see:

Reports

Now that you have the tags setup appropriately, you can generate reports in cost explorer.

1. Go to AWS Cost Explorer Console
2. Filter by "Tag" name where tag name is from the above list

Job Notifications

Budgets