Demo for running simple sequence alignments using Bowtie. Lambda step functions are used to execute and monitor AWS Batch job status and the outputs are copied to your bucket in S3.

gistfile1.txt

This is a demo for running simple sequence alignments using Bowtie.  Lambda step functions are used to execute and monitor AWS Batch job status and the outputs are copied to your bucket in S3. 

1)	Build and commit a docker container to the ECR 
a.	Follow the tutorial in this blog post by Dougal Ballantyne : https://aws.amazon.com/blogs/compute/creating-a-simple-fetch-and-run-aws-batch-job/
i.	Added steps :  Modify the Dockerfile so that it includes a line of code for wget:

FROM amazonlinux:latest
	
RUN yum -y install which unzip aws-cli
RUN yum -y install which unzip wget
ADD fetch_and_run.sh /usr/local/bin/fetch_and_run.sh
WORKDIR /tmp
USER nobody
ENTRYPOINT ["/usr/local/bin/fetch_and_run.sh"]

b.	 At the step of the blog tutorial “Submit and Run a Job”  use the following simple bash script which will install and execute Bowtie alignments.

#!/bin/bash
wget -O bowtie-0.12.9-linux-x86_64.zip http://sourceforge.net/projects/bowtie-bio/files/bowtie/0.12.9/bowtie-0.12.9-linux-x86_64.zip/download

unzip bowtie-0.12.9-linux-x86_64.zip

export BOWTIE_INDEXES=bowtie-0.12.9/indexes/

bowtie-0.12.9/bowtie -a -v 2 e_coli --suppress 1,5,6,7 -c ATGCATCATGCGCCAT > AllValidAlignments.txt
bowtie-0.12.9/bowtie -k 3 -v 2 e_coli --suppress 1,5,6,7 -c ATGCATCATGCGCCAT > UpTo3ValidAlignments.txt
bowtie-0.12.9/bowtie -k 6 -v 2 e_coli --suppress 1,5,6,7 -c ATGCATCATGCGCCAT > UpTo6ValidAlignments.txt

aws s3 cp AllValidAlignments.txt s3://mybatchjobs-scripts-mc/AllValidAlignments.txt
aws s3 cp UpTo3ValidAlignments.txt s3://mybatchjobs-scripts-mc/UpTo3ValidAlignments.txt
aws s3 cp UpTo6ValidAlignments.txt s3://mybatchjobs-scripts-mc/UpTo6ValidAlignments.txt


2)	Create Two Lambda Functions for Using ‘Lambda Step Functions with Batch’
In the Console, create a lambda function and use the blueprint for ‘batch-get-job-python27’ and ‘batch-submit-job-python27’.  
	
You don’t have to modify the get job script.  For the ‘submit job function’, modify the script (lines 16-18) to add the specific information for your Bowtie batch job that you set up in step 1. 

    jobName="wgetdockertest"  
    jobQueue="SimpleGenomicsDemo"  
    jobDefinition="arn:aws:batch:us-east-1:508922263819:job-definition/BowtieDemo:5" 


3)	Create a Step Function for executing your job workflow and monitoring the run status (lambda.step.functionBowtie.json)

lambda.step.functionBowtie.json

{
  "Comment": "A simple example that submits a Bowtie Job to AWS Batch",
  "StartAt": "SubmitBowtieJob",
  "States": {
    "SubmitBowtieJob": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:508922263819:function:batchSubmitBowtieJob",
      "Next": "GetBowtieJobStatus"
    },
    "GetBowtieJobStatus": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:508922263819:function:GetBatchBowtieJobStatus",
      "Next": "CheckBowtieJobStatus",
      "InputPath": "$",
      "ResultPath": "$.status"
    },
    "CheckBowtieJobStatus": {
      "Type": "Choice",
      "Choices": [
      	{
          "Variable": "$.status",
          "StringEquals": "FAILED",
          "Next": "GetFinalBowtieJobStatus"
      	},
      	{
          "Variable": "$.status",
          "StringEquals": "SUCCEEDED",
          "Next": "GetFinalBowtieJobStatus"
      	}
      ],
      "Default": "BowtieWait30Seconds"
    },
    "BowtieWait30Seconds": {
      "Type": "Wait",
      "Seconds": 30,
      "Next": "GetBowtieJobStatus"
    },
    "GetFinalBowtieJobStatus": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:508922263819:function:GetBatchBowtieJobStatus",
      "Next": "FinalState",
      "InputPath": "$",
      "ResultPath": "$.status"
    },
    "FinalState": {
      "Type": "Pass",
      "End": true
    }
  }
}