Created
August 22, 2023 17:14
-
-
Save tsibley/ad8bf3c8d2fad924195b5bbee01823fb to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml | |
index 38bec9b..4bf6219 100644 | |
--- a/.github/workflows/ci.yaml | |
+++ b/.github/workflows/ci.yaml | |
@@ -65,5 +65,6 @@ jobs: | |
runs-on: ubuntu-latest | |
steps: | |
- uses: actions/checkout@v3 | |
+ - run: make check | |
- run: ./devel/check-readme | |
- uses: ./actions/shellcheck | |
diff --git a/.github/workflows/pathogen-repo-build.yaml b/.github/workflows/pathogen-repo-build.yaml | |
index 729a925..8aa26df 100644 | |
--- a/.github/workflows/pathogen-repo-build.yaml | |
+++ b/.github/workflows/pathogen-repo-build.yaml | |
@@ -1,8 +1,10 @@ | |
+# DO NOT EDIT - GENERATED | |
+ | |
# This workflow is intended to be called by workflows in our various pathogen | |
# build repos. See workflow-templates/pathogen-repo-builds.yaml (a "starter" | |
# workflow) in this repo for an example of what the caller workflow looks like. | |
+ | |
name: Pathogen repo build | |
- | |
defaults: | |
run: | |
# This is the same as GitHub Action's `bash` keyword as of 20 June 2023: | |
@@ -11,66 +13,51 @@ defaults: | |
# Completely spelling it out here so that GitHub can't change it out from under us | |
# and we don't have to refer to the docs to know the expected behavior. | |
shell: bash --noprofile --norc -eo pipefail {0} | |
- | |
on: | |
workflow_call: | |
inputs: | |
repo: | |
description: >- | |
- Repository name with owner (e.g. nextstrain/zika). Defaults to the | |
- repository of the caller workflow. | |
+ Repository name with owner (e.g. nextstrain/zika). Defaults to the repository of the caller workflow. | |
type: string | |
default: ${{ github.repository }} | |
required: false | |
- | |
runtime: | |
description: >- | |
- Nextstrain runtime under which to run the build. | |
- Currently only supports docker, conda, and aws-batch. | |
- Defaults to "docker". | |
+ Nextstrain runtime under which to run the build. Currently only supports docker, conda, and aws-batch. Defaults to "docker". | |
+ | |
+ The aws-batch runtime requires AWS credentials. These may come directly from secrets or indirectly from assuming a role via GitHub Actions' OIDC provider. | |
+ | |
+ The following secrets are used if present: | |
- The aws-batch runtime requires the secrets: | |
+ - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY | |
- - AWS_ACCESS_KEY_ID | |
- - AWS_SECRET_ACCESS_KEY | |
+ They must be defined in the repo's Actions secrets and passed to this workflow with `secrets: inherit`. | |
- They must be defined in the repo's Actions secrets and passed to this | |
- workflow with `secrets: inherit`. | |
+ If no secrets are present, the GitHubActionsRoleNextstrainBatchJobs role is assumed. | |
type: string | |
default: docker | |
required: false | |
- | |
run: | |
description: >- | |
- The full `nextstrain build` command to run for the build. | |
- Defaults to `nextstrain build .` | |
+ The full `nextstrain build` command to run for the build. Defaults to `nextstrain build .` | |
- Use the runtime input to select the runtime for the build instead of | |
- the runtime selection options to ensure that the runtime is properly | |
- set up within the GitHub Action job. | |
+ Use the runtime input to select the runtime for the build instead of the runtime selection options to ensure that the runtime is properly set up within the GitHub Action job. | |
- The pathogen repo is cloned to the top level of the working directory | |
- of the GitHub Action, so use `.` to point to the pathogen repo directory. | |
+ The pathogen repo is cloned to the top level of the working directory of the GitHub Action, so use `.` to point to the pathogen repo directory. | |
- If your build runs longer than the 6 hour limit for GitHub Action jobs, | |
- consider using the `--detach` flag for the aws-batch runtime. | |
+ If your build runs longer than the 6 hour limit for GitHub Action jobs, consider using the `--detach` flag for the aws-batch runtime. | |
- All environment variables provided via the env input and all secrets | |
- provided via `secrets: inherit` can be passed to the build runtime | |
- via the `--env` option. | |
+ All environment variables provided via the env input and all secrets provided via `secrets: inherit` can be passed to the build runtime via the `--env` option. | |
type: string | |
default: nextstrain build . | |
required: false | |
- | |
env: | |
description: >- | |
- Environment variables to set for this reusable workflow since | |
- environment variables in the caller workflow are not propagated to | |
- reusable workflows. This is expected to be a string containing YAML. | |
+ Environment variables to set for this reusable workflow since environment variables in the caller workflow are not propagated to reusable workflows. This is expected to be a string containing YAML. | |
+ | |
+ This is easily produced, for example, by pretending you're writing normal nested YAML within a literal multi-line block scalar (introduced by "|"): | |
- This is easily produced, for example, by pretending | |
- you're writing normal nested YAML within a literal multi-line block | |
- scalar (introduced by "|"): | |
with: | |
env: | | |
@@ -81,29 +68,22 @@ on: | |
not | |
yaml | |
- Do not use for secrets! Instead, pass them via GitHub Action's | |
- dedicated secrets mechanism. | |
+ Do not use for secrets! Instead, pass them via GitHub Action's dedicated secrets mechanism. | |
type: string | |
default: "" | |
required: false | |
- | |
artifact-name: | |
description: >- | |
Name to use for the build output artifact uploaded at end of the workflow. | |
- If you're invoking this workflow multiple times from the same calling | |
- workflow, you should set this. Otherwise, the default "build-outputs" | |
- is probably fine. | |
+ If you're invoking this workflow multiple times from the same calling workflow, you should set this. Otherwise, the default "build-outputs" is probably fine. | |
type: string | |
default: build-outputs | |
required: false | |
- | |
artifact-paths: | |
description: >- | |
- List of paths to include in the build output artifact uploaded | |
- at the end of the workflow, as a string following the format of the | |
- `paths` input of the `actions/upload-artifact` action. | |
- For example: | |
+ List of paths to include in the build output artifact uploaded at the end of the workflow, as a string following the format of the `paths` input of the `actions/upload-artifact` action. For example: | |
+ | |
with: | |
artifact-paths: | | |
@@ -113,6 +93,7 @@ on: | |
The default paths included in the artifact are: | |
+ | |
build.log | |
auspice/ | |
results/ | |
@@ -120,34 +101,116 @@ on: | |
logs/ | |
.snakemake/log/ | |
- The "build.log" contains log messages from the `nextstrain build` command. | |
- The other paths are common output paths for Nextstrain builds. | |
- If a path does not exist in your build, then the action will still | |
- succeed and will print out a warning for the non-existent file(s). | |
- Use an exclude pattern for any of the default paths that you would like to | |
- exclude from the artifact (e.g. !build.log). | |
+ The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log). | |
+ | |
+ This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs. | |
+ type: string | |
+ required: false | |
+ workflow_dispatch: | |
+ inputs: | |
+ runtime: | |
+ description: >- | |
+ Nextstrain runtime under which to run the build. Currently only supports docker, conda, and aws-batch. Defaults to "docker". | |
+ | |
+ The aws-batch runtime requires AWS credentials. These may come directly from secrets or indirectly from assuming a role via GitHub Actions' OIDC provider. | |
+ | |
+ The following secrets are used if present: | |
+ | |
+ - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY | |
+ | |
+ They must be defined in the repo's Actions secrets and passed to this workflow with `secrets: inherit`. | |
+ | |
+ If no secrets are present, the GitHubActionsRoleNextstrainBatchJobs role is assumed. | |
+ type: string | |
+ default: docker | |
+ required: false | |
+ run: | |
+ description: >- | |
+ The full `nextstrain build` command to run for the build. Defaults to `nextstrain build .` | |
+ | |
+ Use the runtime input to select the runtime for the build instead of the runtime selection options to ensure that the runtime is properly set up within the GitHub Action job. | |
+ | |
+ The pathogen repo is cloned to the top level of the working directory of the GitHub Action, so use `.` to point to the pathogen repo directory. | |
+ | |
+ If your build runs longer than the 6 hour limit for GitHub Action jobs, consider using the `--detach` flag for the aws-batch runtime. | |
+ | |
+ All environment variables provided via the env input and all secrets provided via `secrets: inherit` can be passed to the build runtime via the `--env` option. | |
+ type: string | |
+ default: nextstrain build . | |
+ required: false | |
+ env: | |
+ description: >- | |
+ Environment variables to set for this reusable workflow since environment variables in the caller workflow are not propagated to reusable workflows. This is expected to be a string containing YAML. | |
+ | |
+ This is easily produced, for example, by pretending you're writing normal nested YAML within a literal multi-line block scalar (introduced by "|"): | |
+ | |
+ | |
+ with: | |
+ env: | | |
+ FOO: bar | |
+ I_CANT_BELIEVE: "it's not YAML" | |
+ would_you_believe: | | |
+ it's | |
+ not | |
+ yaml | |
+ | |
+ Do not use for secrets! Instead, pass them via GitHub Action's dedicated secrets mechanism. | |
+ type: string | |
+ default: "" | |
+ required: false | |
+ artifact-name: | |
+ description: >- | |
+ Name to use for the build output artifact uploaded at end of the workflow. | |
- This is not supported for builds on AWS Batch because the workflow | |
- detaches from the build. Please use the `nextstrain build` command | |
- locally to reattach to AWS Batch builds to download outputs. | |
+ If you're invoking this workflow multiple times from the same calling workflow, you should set this. Otherwise, the default "build-outputs" is probably fine. | |
type: string | |
+ default: build-outputs | |
required: false | |
+ artifact-paths: | |
+ description: >- | |
+ List of paths to include in the build output artifact uploaded at the end of the workflow, as a string following the format of the `paths` input of the `actions/upload-artifact` action. For example: | |
+ | |
+ with: | |
+ artifact-paths: | | |
+ results/ | |
+ auspice/ | |
+ logs/ | |
+ | |
+ The default paths included in the artifact are: | |
+ | |
+ | |
+ build.log | |
+ auspice/ | |
+ results/ | |
+ benchmarks/ | |
+ logs/ | |
+ .snakemake/log/ | |
+ | |
+ The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log). | |
+ | |
+ This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs. | |
+ type: string | |
+ required: false | |
+ repo: | |
+ description: >- | |
+ Repository name with owner (e.g. nextstrain/zika). | |
+ type: string | |
+ default: "" | |
+ required: true | |
env: | |
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github | |
- | |
+permissions: | |
+ id-token: write | |
jobs: | |
workflow-context: | |
runs-on: ubuntu-latest | |
- permissions: | |
- id-token: write | |
steps: | |
- id: workflow-context | |
uses: nextstrain/.github/actions/workflow-context@master | |
outputs: | |
repository: ${{ steps.workflow-context.outputs.repository }} | |
sha: ${{ steps.workflow-context.outputs.sha }} | |
- | |
run-build: | |
needs: workflow-context | |
runs-on: ubuntu-latest | |
@@ -156,27 +219,15 @@ jobs: | |
uses: actions/checkout@v3 | |
with: | |
repository: ${{ inputs.repo }} | |
- | |
- # Need to run this after the build repo is cloned so that cloning the | |
- # build repo does not overwrite the .git dir and remove the extra support files | |
- # that we need from nextstrain/.github repo | |
- - name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ - # Need to run this after the build repo is cloned so that cloning the | |
+ # build repo does not overwrite the .git dir and remove the extra support files | |
+ # that we need from nextstrain/.github repo | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
uses: actions/checkout@v3 | |
with: | |
repository: ${{ needs.workflow-context.outputs.repository }} | |
ref: ${{ needs.workflow-context.outputs.sha }} | |
path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
- | |
- - name: Setup runtime ${{ inputs.runtime }} | |
- uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
- with: | |
- cli-version: ">=7.1.0" | |
- runtime: ${{ inputs.runtime }} | |
- # Required to set up aws-batch as default runtime | |
- env: | |
- AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
- AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
- | |
- if: inputs.env | |
name: Set environment variables | |
env: | |
@@ -184,9 +235,7 @@ jobs: | |
run: > | |
# shellcheck disable=SC2154 | |
- echo "$env" | |
- | "$NEXTSTRAIN_GITHUB_DIR"/bin/yaml-to-envvars | |
- | tee -a "$GITHUB_ENV" | |
+ echo "$env" | "$NEXTSTRAIN_GITHUB_DIR"/bin/yaml-to-envvars | tee -a "$GITHUB_ENV" | |
- name: Set secrets as environment variables | |
env: | |
@@ -194,11 +243,21 @@ jobs: | |
run: > | |
# shellcheck disable=SC2154 | |
- echo "$secrets" | |
- | jq 'del(.github_token)' | |
- | "$NEXTSTRAIN_GITHUB_DIR"/bin/json-to-envvars | |
- | tee -a "$GITHUB_ENV" | |
+ echo "$secrets" | jq 'del(.github_token)' | "$NEXTSTRAIN_GITHUB_DIR"/bin/json-to-envvars | tee -a "$GITHUB_ENV" | |
+ - if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ - name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
- name: Run build via ${{ inputs.runtime }} | |
env: | |
NEXTSTRAIN_BUILD_COMMAND: ${{ inputs.run }} | |
@@ -207,19 +266,16 @@ jobs: | |
set -x | |
eval "$NEXTSTRAIN_BUILD_COMMAND" |& tee build.log | |
- | |
- if: ${{ inputs.runtime == 'aws-batch' }} | |
name: Get AWS Batch job id | |
id: aws-batch | |
run: | | |
echo "AWS_BATCH_JOB_ID=$(tail -n1 build.log | sed -nE 's/.+attach ([-a-f0-9]+).+/\1/p')" >> "$GITHUB_ENV" | |
- | |
- if: env.AWS_BATCH_JOB_ID | |
name: Generate AWS Batch summary | |
run: | | |
"$NEXTSTRAIN_GITHUB_DIR"/bin/interpolate-env < "$NEXTSTRAIN_GITHUB_DIR"/text-templates/attach-aws-batch.md \ | |
> "$GITHUB_STEP_SUMMARY" | |
- | |
- if: always() | |
uses: actions/upload-artifact@v3 | |
with: | |
@@ -233,3 +289,251 @@ jobs: | |
logs/ | |
.snakemake/log/ | |
${{ inputs.artifact-paths }} | |
+ outputs: | |
+ AWS_BATCH_JOB_ID: ${{ env.AWS_BATCH_JOB_ID }} | |
+ # Wait for up to 6 hours (the GitHub Actions job timeout) for the AWS Batch | |
+ # job to finish. | |
+ wait-1: | |
+ # XXX FIXME: drop this | |
+ timeout-minutes: 1 | |
+ needs: [run-build, workflow-context] | |
+ if: needs.run-build.outputs.AWS_BATCH_JOB_ID | |
+ # XXX FIXME: We should use a self-hosted runner to avoid eating up our | |
+ # org-wide concurrency quota on GitHub's runners. For example, for about | |
+ # $100/yr (committed to, but not paid upfront) we could self-host a single, | |
+ # always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM. | |
+ # We could also use other auto-scaling solutions and likely reduce the cost | |
+ # further while increasing capacity. Then we'd select for that with | |
+ # runs-on: labels and/or groups, e.g. with the default labels: | |
+ # | |
+ # runs-on: [self-hosted, linux, ARM64] | |
+ # | |
+ # -trs, 4 August 2023 | |
+ runs-on: ubuntu-latest | |
+ steps: | |
+ # Uses needs.workflow-context.outputs | |
+ - # Need to run this after the build repo is cloned so that cloning the | |
+ # build repo does not overwrite the .git dir and remove the extra support files | |
+ # that we need from nextstrain/.github repo | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ uses: actions/checkout@v3 | |
+ with: | |
+ repository: ${{ needs.workflow-context.outputs.repository }} | |
+ ref: ${{ needs.workflow-context.outputs.sha }} | |
+ path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
+ - if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ - name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
+ # XXX TODO: This will reiterate the full logs emitted so far on every | |
+ # wait-N job, which is annoying, but ah well. Avoiding that is more work | |
+ # than I care to spend right now for too little gain. It's maybe more | |
+ # desirable this way anyhow: the full logs of the Batch job will be | |
+ # captured in GitHub Actions logs by the last successful wait-N job. | |
+ # -trs, 4 August 2023 | |
+ - name: Attach to AWS Batch job | |
+ id: wait | |
+ env: | |
+ AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }} | |
+ run: | | |
+ nextstrain build \ | |
+ --aws-batch \ | |
+ --attach "$AWS_BATCH_JOB_ID" \ | |
+ --no-download \ | |
+ . | |
+ outputs: | |
+ conclusion: ${{ steps.wait.conclusion }} | |
+ continue-on-error: true | |
+ # Wait for up to another 6 hours (hours 6–12) if the preceding wait-N job | |
+ # timed out while attached to the AWS Batch job. | |
+ wait-2: | |
+ # XXX FIXME: drop this | |
+ timeout-minutes: 1 | |
+ # XXX FIXME: We should use a self-hosted runner to avoid eating up our | |
+ # org-wide concurrency quota on GitHub's runners. For example, for about | |
+ # $100/yr (committed to, but not paid upfront) we could self-host a single, | |
+ # always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM. | |
+ # We could also use other auto-scaling solutions and likely reduce the cost | |
+ # further while increasing capacity. Then we'd select for that with | |
+ # runs-on: labels and/or groups, e.g. with the default labels: | |
+ # | |
+ # runs-on: [self-hosted, linux, ARM64] | |
+ # | |
+ # -trs, 4 August 2023 | |
+ runs-on: ubuntu-latest | |
+ steps: | |
+ # Uses needs.workflow-context.outputs | |
+ - # Need to run this after the build repo is cloned so that cloning the | |
+ # build repo does not overwrite the .git dir and remove the extra support files | |
+ # that we need from nextstrain/.github repo | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ uses: actions/checkout@v3 | |
+ with: | |
+ repository: ${{ needs.workflow-context.outputs.repository }} | |
+ ref: ${{ needs.workflow-context.outputs.sha }} | |
+ path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
+ - if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ - name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
+ # XXX TODO: This will reiterate the full logs emitted so far on every | |
+ # wait-N job, which is annoying, but ah well. Avoiding that is more work | |
+ # than I care to spend right now for too little gain. It's maybe more | |
+ # desirable this way anyhow: the full logs of the Batch job will be | |
+ # captured in GitHub Actions logs by the last successful wait-N job. | |
+ # -trs, 4 August 2023 | |
+ - name: Attach to AWS Batch job | |
+ id: wait | |
+ env: | |
+ AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }} | |
+ run: | | |
+ nextstrain build \ | |
+ --aws-batch \ | |
+ --attach "$AWS_BATCH_JOB_ID" \ | |
+ --no-download \ | |
+ . | |
+ outputs: | |
+ conclusion: ${{ steps.wait.conclusion }} | |
+ continue-on-error: true | |
+ needs: [wait-1, run-build, workflow-context] | |
+ if: needs.wait-1.outputs.conclusion == 'cancelled' | |
+ # 12–18 hours | |
+ wait-3: | |
+ # XXX FIXME: drop this | |
+ timeout-minutes: 1 | |
+ # XXX FIXME: We should use a self-hosted runner to avoid eating up our | |
+ # org-wide concurrency quota on GitHub's runners. For example, for about | |
+ # $100/yr (committed to, but not paid upfront) we could self-host a single, | |
+ # always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM. | |
+ # We could also use other auto-scaling solutions and likely reduce the cost | |
+ # further while increasing capacity. Then we'd select for that with | |
+ # runs-on: labels and/or groups, e.g. with the default labels: | |
+ # | |
+ # runs-on: [self-hosted, linux, ARM64] | |
+ # | |
+ # -trs, 4 August 2023 | |
+ runs-on: ubuntu-latest | |
+ steps: | |
+ # Uses needs.workflow-context.outputs | |
+ - # Need to run this after the build repo is cloned so that cloning the | |
+ # build repo does not overwrite the .git dir and remove the extra support files | |
+ # that we need from nextstrain/.github repo | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ uses: actions/checkout@v3 | |
+ with: | |
+ repository: ${{ needs.workflow-context.outputs.repository }} | |
+ ref: ${{ needs.workflow-context.outputs.sha }} | |
+ path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
+ - if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ - name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
+ # XXX TODO: This will reiterate the full logs emitted so far on every | |
+ # wait-N job, which is annoying, but ah well. Avoiding that is more work | |
+ # than I care to spend right now for too little gain. It's maybe more | |
+ # desirable this way anyhow: the full logs of the Batch job will be | |
+ # captured in GitHub Actions logs by the last successful wait-N job. | |
+ # -trs, 4 August 2023 | |
+ - name: Attach to AWS Batch job | |
+ id: wait | |
+ env: | |
+ AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }} | |
+ run: | | |
+ nextstrain build \ | |
+ --aws-batch \ | |
+ --attach "$AWS_BATCH_JOB_ID" \ | |
+ --no-download \ | |
+ . | |
+ outputs: | |
+ conclusion: ${{ steps.wait.conclusion }} | |
+ continue-on-error: true | |
+ needs: [wait-2, run-build, workflow-context] | |
+ if: needs.wait-2.outputs.conclusion == 'cancelled' | |
+ # 18–24 hours | |
+ wait-4: | |
+ # XXX FIXME: drop this | |
+ timeout-minutes: 1 | |
+ # XXX FIXME: We should use a self-hosted runner to avoid eating up our | |
+ # org-wide concurrency quota on GitHub's runners. For example, for about | |
+ # $100/yr (committed to, but not paid upfront) we could self-host a single, | |
+ # always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM. | |
+ # We could also use other auto-scaling solutions and likely reduce the cost | |
+ # further while increasing capacity. Then we'd select for that with | |
+ # runs-on: labels and/or groups, e.g. with the default labels: | |
+ # | |
+ # runs-on: [self-hosted, linux, ARM64] | |
+ # | |
+ # -trs, 4 August 2023 | |
+ runs-on: ubuntu-latest | |
+ steps: | |
+ # Uses needs.workflow-context.outputs | |
+ - # Need to run this after the build repo is cloned so that cloning the | |
+ # build repo does not overwrite the .git dir and remove the extra support files | |
+ # that we need from nextstrain/.github repo | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ uses: actions/checkout@v3 | |
+ with: | |
+ repository: ${{ needs.workflow-context.outputs.repository }} | |
+ ref: ${{ needs.workflow-context.outputs.sha }} | |
+ path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
+ - if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ - name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
+ # XXX TODO: This will reiterate the full logs emitted so far on every | |
+ # wait-N job, which is annoying, but ah well. Avoiding that is more work | |
+ # than I care to spend right now for too little gain. It's maybe more | |
+ # desirable this way anyhow: the full logs of the Batch job will be | |
+ # captured in GitHub Actions logs by the last successful wait-N job. | |
+ # -trs, 4 August 2023 | |
+ - name: Attach to AWS Batch job | |
+ id: wait | |
+ env: | |
+ AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }} | |
+ run: | | |
+ nextstrain build \ | |
+ --aws-batch \ | |
+ --attach "$AWS_BATCH_JOB_ID" \ | |
+ --no-download \ | |
+ . | |
+ outputs: | |
+ conclusion: ${{ steps.wait.conclusion }} | |
+ needs: [wait-3, run-build, workflow-context] | |
+ if: needs.wait-3.outputs.conclusion == 'cancelled' | |
+ continue-on-error: false | |
diff --git a/.github/workflows/pathogen-repo-build.yaml b/.github/workflows/pathogen-repo-build.yaml.in | |
similarity index 66% | |
copy from .github/workflows/pathogen-repo-build.yaml | |
copy to .github/workflows/pathogen-repo-build.yaml.in | |
index 729a925..328b7a4 100644 | |
--- a/.github/workflows/pathogen-repo-build.yaml | |
+++ b/.github/workflows/pathogen-repo-build.yaml.in | |
@@ -14,7 +14,7 @@ defaults: | |
on: | |
workflow_call: | |
- inputs: | |
+ inputs: &inputs | |
repo: | |
description: >- | |
Repository name with owner (e.g. nextstrain/zika). Defaults to the | |
@@ -29,13 +29,20 @@ on: | |
Currently only supports docker, conda, and aws-batch. | |
Defaults to "docker". | |
- The aws-batch runtime requires the secrets: | |
+ The aws-batch runtime requires AWS credentials. These may come | |
+ directly from secrets or indirectly from assuming a role via GitHub | |
+ Actions' OIDC provider. | |
+ | |
+ The following secrets are used if present: | |
- AWS_ACCESS_KEY_ID | |
- AWS_SECRET_ACCESS_KEY | |
They must be defined in the repo's Actions secrets and passed to this | |
workflow with `secrets: inherit`. | |
+ | |
+ If no secrets are present, the GitHubActionsRoleNextstrainBatchJobs | |
+ role is assumed. | |
type: string | |
default: docker | |
required: false | |
@@ -133,14 +140,25 @@ on: | |
type: string | |
required: false | |
+ workflow_dispatch: | |
+ inputs: | |
+ <<: *inputs | |
+ repo: | |
+ description: >- | |
+ Repository name with owner (e.g. nextstrain/zika). | |
+ type: string | |
+ default: "" | |
+ required: true | |
+ | |
env: | |
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github | |
+permissions: | |
+ id-token: write | |
+ | |
jobs: | |
workflow-context: | |
runs-on: ubuntu-latest | |
- permissions: | |
- id-token: write | |
steps: | |
- id: workflow-context | |
uses: nextstrain/.github/actions/workflow-context@master | |
@@ -160,23 +178,14 @@ jobs: | |
# Need to run this after the build repo is cloned so that cloning the | |
# build repo does not overwrite the .git dir and remove the extra support files | |
# that we need from nextstrain/.github repo | |
- - name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
+ - &checkout-workflow-support | |
+ name: Checkout ${{ needs.workflow-context.outputs.repository }} (sha ${{ needs.workflow-context.outputs.sha }}) | |
uses: actions/checkout@v3 | |
with: | |
repository: ${{ needs.workflow-context.outputs.repository }} | |
ref: ${{ needs.workflow-context.outputs.sha }} | |
path: ${{ env.NEXTSTRAIN_GITHUB_DIR }} | |
- - name: Setup runtime ${{ inputs.runtime }} | |
- uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
- with: | |
- cli-version: ">=7.1.0" | |
- runtime: ${{ inputs.runtime }} | |
- # Required to set up aws-batch as default runtime | |
- env: | |
- AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
- AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
- | |
- if: inputs.env | |
name: Set environment variables | |
env: | |
@@ -199,6 +208,23 @@ jobs: | |
| "$NEXTSTRAIN_GITHUB_DIR"/bin/json-to-envvars | |
| tee -a "$GITHUB_ENV" | |
+ - &setup-aws-credentials | |
+ if: inputs.runtime == 'aws-batch' | |
+ uses: aws-actions/configure-aws-credentials@v2 | |
+ with: | |
+ # XXX FIXME | |
+ aws-region: us-east-1 | |
+ aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
+ aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
+ role-to-assume: ${{ secrets.AWS_ACCESS_KEY_ID == '' && 'arn:aws:iam::827581582529:role/GitHubActionsRoleNextstrainBatchJobs' || '' }} | |
+ | |
+ - &setup-runtime | |
+ name: Setup runtime ${{ inputs.runtime }} | |
+ uses: ./.git/nextstrain/.github/actions/setup-nextstrain-cli | |
+ with: | |
+ cli-version: ">=7.1.0" | |
+ runtime: ${{ inputs.runtime }} | |
+ | |
- name: Run build via ${{ inputs.runtime }} | |
env: | |
NEXTSTRAIN_BUILD_COMMAND: ${{ inputs.run }} | |
@@ -233,3 +259,77 @@ jobs: | |
logs/ | |
.snakemake/log/ | |
${{ inputs.artifact-paths }} | |
+ | |
+ outputs: | |
+ AWS_BATCH_JOB_ID: ${{ env.AWS_BATCH_JOB_ID }} | |
+ | |
+ # Wait for up to 6 hours (the GitHub Actions job timeout) for the AWS Batch | |
+ # job to finish. | |
+ wait-1: &wait | |
+ # XXX FIXME: drop this | |
+ timeout-minutes: 1 | |
+ needs: [run-build, workflow-context] | |
+ if: needs.run-build.outputs.AWS_BATCH_JOB_ID | |
+ | |
+ # XXX FIXME: We should use a self-hosted runner to avoid eating up our | |
+ # org-wide concurrency quota on GitHub's runners. For example, for about | |
+ # $100/yr (committed to, but not paid upfront) we could self-host a single, | |
+ # always-on runner on a t4g.small EC2 instance with 2 CPUs and 2GB of RAM. | |
+ # We could also use other auto-scaling solutions and likely reduce the cost | |
+ # further while increasing capacity. Then we'd select for that with | |
+ # runs-on: labels and/or groups, e.g. with the default labels: | |
+ # | |
+ # runs-on: [self-hosted, linux, ARM64] | |
+ # | |
+ # -trs, 4 August 2023 | |
+ runs-on: ubuntu-latest | |
+ | |
+ steps: | |
+ # Uses needs.workflow-context.outputs | |
+ - *checkout-workflow-support | |
+ - *setup-aws-credentials | |
+ - *setup-runtime | |
+ | |
+ # XXX TODO: This will reiterate the full logs emitted so far on every | |
+ # wait-N job, which is annoying, but ah well. Avoiding that is more work | |
+ # than I care to spend right now for too little gain. It's maybe more | |
+ # desirable this way anyhow: the full logs of the Batch job will be | |
+ # captured in GitHub Actions logs by the last successful wait-N job. | |
+ # -trs, 4 August 2023 | |
+ - name: Attach to AWS Batch job | |
+ id: wait | |
+ env: | |
+ AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }} | |
+ run: | | |
+ nextstrain build \ | |
+ --aws-batch \ | |
+ --attach "$AWS_BATCH_JOB_ID" \ | |
+ --no-download \ | |
+ . | |
+ continue-on-error: true | |
+ | |
+ # check here and effectively torpedo the job if we need to instead of using job-level continue-on-error | |
+ - if: steps.wait.outcome == 'failure' | |
+ run: exit 1 | |
+ | |
+ outputs: | |
+ conclusion: ${{ steps.wait.conclusion }} | |
+ | |
+ # Wait for up to another 6 hours (hours 6–12) if the preceding wait-N job | |
+ # timed out while attached to the AWS Batch job. | |
+ wait-2: | |
+ <<: *wait | |
+ needs: [wait-1, run-build, workflow-context] | |
+ if: needs.wait-1.outputs.conclusion == 'cancelled' | |
+ | |
+ # 12–18 hours | |
+ wait-3: | |
+ <<: *wait | |
+ needs: [wait-2, run-build, workflow-context] | |
+ if: needs.wait-2.outputs.conclusion == 'cancelled' | |
+ | |
+ # 18–24 hours | |
+ wait-4: | |
+ <<: *wait | |
+ needs: [wait-3, run-build, workflow-context] | |
+ if: needs.wait-3.outputs.conclusion == 'cancelled' | |
diff --git a/Makefile b/Makefile | |
new file mode 100644 | |
index 0000000..242f420 | |
--- /dev/null | |
+++ b/Makefile | |
@@ -0,0 +1,15 @@ | |
+SHELL := bash -euo pipefail | |
+ | |
+inputs := $(wildcard .github/workflows/*.yaml.in) | |
+outputs := $(inputs:.in=) | |
+ | |
+all: $(outputs) | |
+ | |
+check: all | |
+ git diff --exit-code HEAD | |
+ | |
+.github/workflows/%.yaml: .github/workflows/%.yaml.in PHONY | |
+ ./devel/explode-yaml < $< > $@ | |
+ | |
+.PHONY: PHONY all check | |
+PHONY: | |
diff --git a/README.md b/README.md | |
index 4b0458d..367aba8 100644 | |
--- a/README.md | |
+++ b/README.md | |
@@ -54,7 +54,8 @@ Invoked by other repos. | |
([workflow](.github/workflows/sync-rtd-redirects.yaml)) | |
- Pathogen repo build | |
- ([workflow](.github/workflows/pathogen-repo-build.yaml)) | |
+ ([workflow source](.github/workflows/pathogen-repo-build.yaml.in), | |
+ [workflow compiled](.github/workflows/pathogen-repo-build.yaml)) | |
See also GitHub's [documentation on reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows). | |
@@ -112,3 +113,12 @@ Executable scripts that are used in our workflows. | |
Text templates for messages and summaries in our workflows. | |
- [attach-aws-batch](text-templates/attach-aws-batch.md) | |
+ | |
+ | |
+## Development tools for this repo itself | |
+ | |
+- Linting to ensure the README stays complete | |
+ ([devel/check-readme](devel/check-readme)) | |
+ | |
+- Pre-processing of YAML to satisfy the requirements of GitHub Actions | |
+ ([Makefile](Makefile), [devel/explode-yaml](devel/explode-yaml)) | |
diff --git a/devel/check-readme b/devel/check-readme | |
index 1b83235..9eb6e17 100755 | |
--- a/devel/check-readme | |
+++ b/devel/check-readme | |
@@ -44,7 +44,6 @@ files-to-ignore() { | |
git ls-files \ | |
.gitignore \ | |
README.md \ | |
- devel/check-readme \ | |
'images/*' \ | |
actions/setup-ssh/!(*.yaml|README.md) \ | |
actions/setup-debugger/!(*.yaml|README.md) \ | |
diff --git a/devel/explode-yaml b/devel/explode-yaml | |
new file mode 100755 | |
index 0000000..d460afc | |
--- /dev/null | |
+++ b/devel/explode-yaml | |
@@ -0,0 +1,12 @@ | |
+#!/bin/bash | |
+# Explode YAML with anchors & references to YAML without them, using yq¹, for | |
+# GitHub Action's sake². | |
+# | |
+# ¹ <https://github.com/mikefarah/yq> | |
+# ² <https://github.com/actions/runner/issues/1182> | |
+# | |
+set -euo pipefail | |
+ | |
+docker run --rm --interactive mikefarah/yq ' | |
+ explode(.) head_comment="DO NOT EDIT - GENERATED\n\n" + head_comment | |
+' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment