xbrianh

#!/usr/bin/env python
"""
This script migrates gs-event-relay AWS IAM user credentials from the AWS Systems Manager
parameter store to the Secrets Manager store, and tests that the credentials are valid.
It should be placed in $DSS_HOME/scripts before execution.
- Brian Hannafious, 29-May, 2018
"""
import os
import json
import boto3

xbrianh

#!/usr/bin/env python
import boto3

sf = boto3.client("stepfunctions")
failed = sf.list_executions(
    stateMachineArn="arn:aws:states:us-east-1:861229788715:stateMachine:dss-visitation-dev",
    statusFilter="FAILED",
)['executions']

def events(execution_arn):

xbrianh

#!/usr/bin/env python
import argparse

from hca.dss import DSSClient

def get_dss_client(deployment):
    if "prod" == deployment:
        url = "https://dss.data.humancellatlas.org/v1/swagger.json"
    else:
        url = f"https://dss.{deployment}.data.humancellatlas.org/v1/swagger.json"

xbrianh

#!/usr/bin/env python
"""
Prototyped utilities for working with indexed, unterminated tar archives.

See "end-of-archive entry" in the following link for more information about "unterminated":
https://www.gnu.org/software/tar/manual/html_node/Standard.html

These can be used efficiently with s3 objects to extract individual files,
splice archives together, and stream. See examples at end of file.
"""

xbrianh

# This is an example Terraform script to deploy a GCP service account, create a service account access key,
# and store the service account access key in AWS Secretsmanager.
#
# The service account access key can be rotated by issuing the commands:
#  1. `terraform destroy -target=null_resource.gcp-service-account-key`
#  2. `terraform apply -target=null_resource.gcp-service-account-key`

variable "GCP_SERVICE_ACCOUNT_NAME" {
  default = "bhannafi-test-service-account"
}

xbrianh

#!/usr/bin/env python
import io
import argparse
from contextlib import closing
        
import bgzip

parser = argparse.ArgumentParser()
parser.add_argument("cloudpath", help="location of file. Can be local or GCP bucket path (e.g. gs://key)")
args = parser.parse_args()

xbrianh

Meeting with Fellow: Jean Monlong

1. Jean is interested in tooling to help clean up his Terra workspace, including deleting intermediate files from completed workflows, or deleting files that did not successfully complete. Brian, I believe some of your tooling may be useful here or you could work with Jean to develop your tooling further?
   • The terra-notebook-utils package contains tooling to remove workflow intermediate files from your workspace bucket. Code is available now on the master branch, and will be released soon through the pip package.
   • From Python:

     from terra_notebook_utils import workspace
     

xbrianh

#!/usr/bin/env python                                                                                    
"""
Resolve DRS urls with the Broad's "martha" service:
https://github.com/broadinstitute/martha

Before using, authenticate with `gcloud auth application-default login`
"""
import os
import sys
import json