- Jean is interested in tooling to help clean up his Terra workspace, including deleting intermediate files from
completed workflows, or deleting files that did not successfully complete. Brian, I believe some of your tooling
may be useful here or you could work with Jean to develop your tooling further?
- The terra-notebook-utils package contains tooling to remove workflow intermediate files from your workspace bucket. Code is available now on the master branch, and will be released soon through the pip package.
- From Python:
from terra_notebook_utils import workspace
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
""" | |
Resolve DRS urls with the Broad's "martha" service: | |
https://github.com/broadinstitute/martha | |
Before using, authenticate with `gcloud auth application-default login` | |
""" | |
import os | |
import sys | |
import json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
import io | |
import argparse | |
from contextlib import closing | |
import bgzip | |
parser = argparse.ArgumentParser() | |
parser.add_argument("cloudpath", help="location of file. Can be local or GCP bucket path (e.g. gs://key)") | |
args = parser.parse_args() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# This is an example Terraform script to deploy a GCP service account, create a service account access key, | |
# and store the service account access key in AWS Secretsmanager. | |
# | |
# The service account access key can be rotated by issuing the commands: | |
# 1. `terraform destroy -target=null_resource.gcp-service-account-key` | |
# 2. `terraform apply -target=null_resource.gcp-service-account-key` | |
variable "GCP_SERVICE_ACCOUNT_NAME" { | |
default = "bhannafi-test-service-account" | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
""" | |
Prototyped utilities for working with indexed, unterminated tar archives. | |
See "end-of-archive entry" in the following link for more information about "unterminated": | |
https://www.gnu.org/software/tar/manual/html_node/Standard.html | |
These can be used efficiently with s3 objects to extract individual files, | |
splice archives together, and stream. See examples at end of file. | |
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
import argparse | |
from hca.dss import DSSClient | |
def get_dss_client(deployment): | |
if "prod" == deployment: | |
url = "https://dss.data.humancellatlas.org/v1/swagger.json" | |
else: | |
url = f"https://dss.{deployment}.data.humancellatlas.org/v1/swagger.json" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
import boto3 | |
sf = boto3.client("stepfunctions") | |
failed = sf.list_executions( | |
stateMachineArn="arn:aws:states:us-east-1:861229788715:stateMachine:dss-visitation-dev", | |
statusFilter="FAILED", | |
)['executions'] | |
def events(execution_arn): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
""" | |
This script migrates gs-event-relay AWS IAM user credentials from the AWS Systems Manager | |
parameter store to the Secrets Manager store, and tests that the credentials are valid. | |
It should be placed in $DSS_HOME/scripts before execution. | |
- Brian Hannafious, 29-May, 2018 | |
""" | |
import os | |
import json | |
import boto3 |