Skip to content

Instantly share code, notes, and snippets.

@radaniba

radaniba/express_pipeline.ipynb

Last active September 9, 2015 22:39
Show Gist options
  • Save radaniba/2b7f2232cc97265f1d7b to your computer and use it in GitHub Desktop.
Save radaniba/2b7f2232cc97265f1d7b to your computer and use it in GitHub Desktop.
Download ZIP
Running pipelines everywhere on any server
Display the source blob
Display the rendered blob
Raw
express_pipeline.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##What is Fabric"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*This is an experimental notebook **you will find some hard coded paths, and that's normal, its a demo tutorial*** \n",
"\n",
">. Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.\n",
"It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.\n",
"\n",
"In this notebook I am going to describe how to use `Fabric` to run piplines that are created by the dev team on genesis. All you need to have is a genesis account.\n",
"\n",
"Using this example you will be able to **run any pipeline on genesis from anywhere**\n",
"\n",
"I stongly suggest that you browse the Fabric webpage for more details\n",
"\n",
"http://www.fabfile.org/\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##What will we be doing ?\n",
"\n",
"- Create remotely a working directory on genesis\n",
"- Rsync files needed by the pipeline factory\n",
"- Sourcing the virtual environment for a specific pipeline\n",
"- Launching the pipline\n",
"\n",
"optionally\n",
"\n",
"- rsyncing back the results into your local machine\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Will you need to know everything ?\n",
"\n",
"No ! All you will need to know is how the pipeline factory generally works, and prepare the files on your local machine.\n",
"**You will be running one single command line to do all the steps described above**\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let's get started ...\n",
"\n",
"First we will create a file called `express_miseq.py` , in this file we will be adding some basic functions needed to get the job done\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importing the libraries\n",
"\n",
"We will need to import fabric api functions as well as some basic libraries\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#!/usr/bin/env python\n",
"\n",
"from __future__ import division\n",
"\n",
"__author__ = \"Rad <aradwen@gmail.com>\"\n",
"__license__ = \"GNU General Public License version 3\"\n",
"__date__ = \"06/30/2015\"\n",
"__version__ = \"0.1\"\n",
"\n",
"try:\n",
" import os\n",
" from fabric import tasks\n",
" from fabric import colors\n",
" from fabric.api import run\n",
" from fabric.api import env, task, hosts, settings\n",
" from fabric.decorators import task\n",
" from fabric.network import disconnect_all\n",
" from fabric.context_managers import cd, prefix\n",
" from fabric.contrib.project import rsync_project\n",
" from argparse import ArgumentParser\n",
" import yaml\n",
" import logging\n",
"except ImportError:\n",
" # Checks the installation of the necessary python modules\n",
" import os\n",
" import sys\n",
"\n",
" print((os.linesep * 2).join(\n",
" [\"An error found importing one module:\", str(sys.exc_info()[1]), \"You need to install it Stopping...\"]))\n",
" sys.exit(-2)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we need to add some **environment variables** to our script in order to let `Fabric` do its magic. We need to tell the script to use : \n",
"\n",
"- the username to connect to the servers \n",
"- the password\n",
"- the gateway to connect to genesis : here we use ssh.bcgsc.ca\n",
"- the host : genesis as we will be running jobs on the cluster\n",
"- the path to your working directory. It doesn't need to exist already since we are going to create that directory anyway\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"env.user = \"raniba\" #put yours here\n",
"env.password = \"xxxxxx\" #put yourrs here\n",
"env.gateway = \"ssh.bcgsc.ca\"\n",
"env.hosts = [\"genesis\"]\n",
"env.path = \"/path/to/your/working/directory/\" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Let's create a function to test our script already **"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def uptime():\n",
" run('uptime')\n",
"\n",
"\n",
"def host_type():\n",
" run('uname -s')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"these two small functions will just print the time on genesis server as well as the host type \n",
"All we need to do is run the script like so :\n",
"\n",
"```\n",
"fab -f express_miseq.py host_type\n",
"\n",
"[genesis] Executing task 'host_type'\n",
"[genesis] run: uname -s\n",
"[genesis] out: Linux\n",
"[genesis] out:\n",
"\n",
"\n",
"Done.\n",
"Disconnecting from ip.addresse.hidden... done.\n",
"Disconnecting from ip.addresse.hidden... done.\n",
"```\n",
"\n",
"And / Or \n",
"\n",
"```\n",
"fab -f express_miseq.py uptime\n",
"\n",
"genesis] Executing task 'uptime'\n",
"[genesis] run: uptime\n",
"[genesis] out: 12:15:09 up 26 days, 19:21, 18 users, load average: 0.56, 0.76, 0.96\n",
"[genesis] out:\n",
"\n",
"\n",
"Done.\n",
"Disconnecting from ip.addresse.hidden... done.\n",
"Disconnecting from ip.addresse.hidden... done.\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's it !! You just connected to the gateway, then to genesis, run command on genesis, and disconnected. **If you are able to do these simple tasks, you'll be able to run jobs on the cluster and here is how we do it** :\n",
"\n",
"We need to create these basic functions, **just 5 functions**"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"## This is for debugging, we need to always check that we are using the right tools\n",
"def which_tool(tool):\n",
" with prefix(source_env()):\n",
" run('which ' + tool)\n",
"\n",
"## THis will create the directory we specified on env.path\n",
"def create_working_directory():\n",
" run('mkdir ' + env.path)\n",
"\n",
"## This will rsync all the files needed by miseq pipeline that are sitting the the same location as our script\n",
"def sync():\n",
" with settings(user=\"raniba\",host_string=\"ip.addresse.hidden\"):\n",
" rsync_project(env.path, os.path.dirname(os.path.realpath(__file__)) + \"/\", delete=False)\n",
"\n",
"## This will source the virtual environment for miseq pipeline\n",
"def source_env():\n",
" pipe_env = \"source /genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/set_env.sh\"\n",
" return pipe_env\n",
"\n",
"## And this will run the pipeline and submit jobs on the cluster\n",
"def run_pipeline():\n",
" with cd(env.path), prefix(source_env()):\n",
" run(\"python /shahlab/dgrewal/cycle007/pipeline-tools/pipeline_runner/pipeline_runner.py --setup miseq.setup --config miseq.yaml --run_id from_beast --samples samples.txt --working_dir $PWD \")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we wrap these function into a main one\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def main():\n",
" print colors.yellow(\"Connected to Genesis\")\n",
" test()\n",
" print colors.red(\"rsyncing pipeline required files\")\n",
" sync()\n",
" print colors.red(\"sourcing the pipeline environment\")\n",
" #source_env()\n",
" print colors.red(\"Checking basic tools used\")\n",
" which_tool(\"python\")\n",
" which_tool(\"bowtie2\")\n",
" which_tool(\"samtools\")\n",
" which_tool(\"java\")\n",
" print colors.red(\"kickstarting the pipeline ..\")\n",
" run_pipeline()\n",
" print colors.green(\"All done .. deconnecting\")\n",
" disconnect_all()\n",
"\n",
" \n",
"if __name__ == '__main__':\n",
" main()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## All done !\n",
"\n",
"Let's run our super express pipeline runner :\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"fab -f express_miseq.py main\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---------\n",
"### The full code"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"#!/usr/bin/env python\n",
"\n",
"from __future__ import division\n",
"\n",
"__author__ = \"Rad <aradwen@gmail.com>\"\n",
"__license__ = \"GNU General Public License version 3\"\n",
"__date__ = \"06/30/2015\"\n",
"__version__ = \"0.1\"\n",
"\n",
"try:\n",
" import os\n",
" from fabric import tasks\n",
" from fabric import colors\n",
" from fabric.api import run\n",
" from fabric.api import env, task, hosts, settings\n",
" from fabric.decorators import task\n",
" from fabric.network import disconnect_all\n",
" from fabric.context_managers import cd, prefix\n",
" from fabric.contrib.project import rsync_project\n",
" from argparse import ArgumentParser\n",
" import yaml\n",
" import logging\n",
"except ImportError:\n",
" # Checks the installation of the necessary python modules\n",
" import os\n",
" import sys\n",
"\n",
" print((os.linesep * 2).join(\n",
" [\"An error found importing one module:\", str(sys.exc_info()[1]), \"You need to install it Stopping...\"]))\n",
" sys.exit(-2)\n",
"\n",
"\n",
"env.user = \"raniba\"\n",
"env.password = \"xxxxxxx\"\n",
"env.gateway = \"x10\"\n",
"env.hosts = [\"genesis\"]\n",
"env.path = \"/genesis/extscratch/shahlab/raniba/test_express3/\"\n",
"env.use_ssh_config = True\n",
"\n",
"logging.basicConfig()\n",
"paramiko_logger = logging.getLogger(\"paramiko.transport\")\n",
"paramiko_logger.disabled = True\n",
"\n",
"\n",
"def test():\n",
" run(\"ls\")\n",
"\n",
"def set_hosts(host):\n",
" env.hosts=[host]\n",
"\n",
"def uptime():\n",
" run('uptime')\n",
"\n",
"\n",
"def host_type():\n",
" run('uname -s')\n",
"\n",
"\n",
"def diskspace():\n",
" run('df')\n",
"\n",
"\n",
"def which_tool(tool):\n",
" with prefix(source_env()):\n",
" run('which ' + tool)\n",
"\n",
"\n",
"def create_working_directory():\n",
" run('mkdir ' + env.path)\n",
"\n",
"\n",
"def sync():\n",
" with settings(user=\"raniba\",host_string=\"ip.addresse.hidden\"):\n",
" rsync_project(env.path, os.path.dirname(os.path.realpath(__file__)) + \"/\", delete=False)\n",
"\n",
"\n",
"def source_env():\n",
" pipe_env = \"source /genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/set_env.sh\"\n",
" return pipe_env\n",
"\n",
"def run_pipeline():\n",
" with cd(env.path), prefix(source_env()):\n",
" run(\"python /shahlab/dgrewal/cycle007/pipeline-tools/pipeline_runner/pipeline_runner.py --setup miseq.setup --config miseq.yaml --run_id from_beast --samples samples.txt --working_dir $PWD \")\n",
"\n",
"\n",
"def main():\n",
" print colors.yellow(\"Connected to Genesis\")\n",
" test()\n",
" print colors.red(\"rsyncing pipeline required files\")\n",
" sync()\n",
" print colors.red(\"sourcing the pipeline environment\")\n",
" #source_env()\n",
" print colors.red(\"Checking basic tools used\")\n",
" which_tool(\"python\")\n",
" which_tool(\"bowtie2\")\n",
" which_tool(\"samtools\")\n",
" which_tool(\"java\")\n",
" print colors.red(\"kickstarting the pipeline ..\")\n",
" run_pipeline()\n",
" print colors.green(\"All done .. deconnecting\")\n",
" disconnect_all()\n",
"\n",
"\n",
"if __name__ == '__main__':\n",
" main()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running th code"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[genesis] Executing task 'main'\n",
"\u001b[33mConnected to Genesis\u001b[0m\n",
"[genesis] run: ls\n",
"[genesis] out: 2015-08-01T00:00:05.243944_quotawatcher.log\n",
"[genesis] out: 2015-08-02T00:00:04.396093_quotawatcher.log\n",
"[genesis] out: 2015-08-03T00:00:03.990846_quotawatcher.log\n",
"[genesis] out: 2015-08-04T00:00:03.891508_quotawatcher.log\n",
"[genesis] out: 2015-08-05T00:00:05.038785_quotawatcher.log\n",
"[genesis] out: 2015-08-05T14:18:05.404240_quotawatcher.log\n",
"[genesis] out: 2015-08-06T00:00:04.620503_quotawatcher.log\n",
"[genesis] out: 2015-08-07T00:00:04.525099_quotawatcher.log\n",
"[genesis] out: 2015-08-07T14:18:03.806468_quotawatcher.log\n",
"[genesis] out: 2015-08-08T00:00:05.770975_quotawatcher.log\n",
"[genesis] out: 2015-08-09T00:00:04.191317_quotawatcher.log\n",
"[genesis] out: 2015-08-10T00:00:04.826070_quotawatcher.log\n",
"[genesis] out: 2015-08-11T00:00:09.455555_quotawatcher.log\n",
"[genesis] out: 2015-08-12T00:00:07.599454_quotawatcher.log\n",
"[genesis] out: 2015-08-12T14:18:04.978526_quotawatcher.log\n",
"[genesis] out: 2015-08-13T00:00:05.225572_quotawatcher.log\n",
"[genesis] out: 2015-08-14T00:00:05.730837_quotawatcher.log\n",
"[genesis] out: 2015-08-14T14:18:06.501024_quotawatcher.log\n",
"[genesis] out: 2015-08-15T00:00:07.633554_quotawatcher.log\n",
"[genesis] out: 2015-08-16T00:00:07.065964_quotawatcher.log\n",
"[genesis] out: 2015-08-17T00:00:08.845804_quotawatcher.log\n",
"[genesis] out: 2015-08-18T00:00:08.273872_quotawatcher.log\n",
"[genesis] out: 2015-08-19T00:00:05.042167_quotawatcher.log\n",
"[genesis] out: 2015-08-19T14:18:06.410454_quotawatcher.log\n",
"[genesis] out: 2015-08-20T00:00:10.681688_quotawatcher.log\n",
"[genesis] out: 2015-08-21T00:00:05.008527_quotawatcher.log\n",
"[genesis] out: 2015-08-21T14:18:04.987991_quotawatcher.log\n",
"[genesis] out: 2015-08-22T00:00:06.228136_quotawatcher.log\n",
"[genesis] out: 2015-08-23T00:00:03.396594_quotawatcher.log\n",
"[genesis] out: 2015-08-24T00:00:03.733816_quotawatcher.log\n",
"[genesis] out: 2015-08-25T00:00:08.553300_quotawatcher.log\n",
"[genesis] out: 2015-08-26T00:00:05.757512_quotawatcher.log\n",
"[genesis] out: 2015-08-27T00:00:06.521502_quotawatcher.log\n",
"[genesis] out: 2015-08-28T00:00:03.703168_quotawatcher.log\n",
"[genesis] out: 2015-08-29T00:00:06.652384_quotawatcher.log\n",
"[genesis] out: 2015-08-30T00:00:04.817086_quotawatcher.log\n",
"[genesis] out: 2015-08-31T00:00:05.606286_quotawatcher.log\n",
"[genesis] out: averagecov.pl\n",
"[genesis] out: coverage_result\n",
"[genesis] out: history.txt\n",
"[genesis] out: igv\n",
"[genesis] out: mm.sh\n",
"[genesis] out: pipelines\n",
"[genesis] out: powerline-shell.py\n",
"[genesis] out: results_singlecell\n",
"[genesis] out: targets\n",
"[genesis] out: test.count\n",
"[genesis] out: test.py\n",
"[genesis] out: test.py~\n",
"[genesis] out: test_unicode.py\n",
"[genesis] out: test_unicode.py~\n",
"[genesis] out: \n",
"\n",
"\u001b[31mrsyncing pipeline required files\u001b[0m\n",
"[ip.addresse.hidden] rsync_project: rsync -pthrvz --rsh='ssh -p 22 ' /Users/raniba/Documents/Dev/express_pipe/ raniba@ip.addresse.hidden:/genesis/extscratch/shahlab/raniba/test_express2/\n",
"[localhost] local: rsync -pthrvz --rsh='ssh -p 22 ' /Users/raniba/Documents/Dev/express_pipe/ raniba@ip.addresse.hidden:/genesis/extscratch/shahlab/raniba/test_express2/\n",
"\u001b[31msourcing the pipeline environment\u001b[0m\n",
"\u001b[31mChecking basic tools used\u001b[0m\n",
"[genesis] run: which python\n",
"[genesis] out: /genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/bin/python\n",
"[genesis] out: \n",
"\n",
"[genesis] run: which bowtie2\n",
"[genesis] out: /genesis/extscratch/shahlab/pipelines/apps/bowtie2-2.0.2/bowtie2\n",
"[genesis] out: \n",
"\n",
"[genesis] run: which samtools\n",
"[genesis] out: /genesis/extscratch/shahlab/pipelines/apps/samtools-1.2/samtools\n",
"[genesis] out: \n",
"\n",
"[genesis] run: which java\n",
"[genesis] out: /genesis/extscratch/shahlab/pipelines/apps/jdk1.7.0_06/bin/java\n",
"[genesis] out: \n",
"\n",
"\u001b[31mkickstarting the pipeline ..\u001b[0m\n",
"[genesis] run: python /shahlab/dgrewal/cycle007/pipeline-tools/pipeline_runner/pipeline_runner.py --setup miseq.setup --config miseq.yaml --run_id from_beast --samples samples.txt --working_dir $PWD \n",
"[genesis] out: 2015-09-08 14:10:10,903 : INFO : --------------------------------\n",
"[genesis] out: 2015-09-08 14:10:10,903 : INFO : PIPELINE RUNNER v2.4.2 LOG\n",
"[genesis] out: 2015-09-08 14:10:10,903 : INFO : --------------------------------\n",
"[genesis] out: 2015-09-08 14:10:10,903 : INFO : -------- COMMAND LINE ARGUMENTS\n",
"[genesis] out: 2015-09-08 14:10:10,904 : INFO : no_init = False\n",
"[genesis] out: 2015-09-08 14:10:10,904 : INFO : verbose = False\n",
"[genesis] out: 2015-09-08 14:10:10,904 : INFO : run_id = from_beast\n",
"[genesis] out: 2015-09-08 14:10:10,904 : INFO : factory_dir = None\n",
"[genesis] out: 2015-09-08 14:10:10,904 : INFO : no_run = False\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : inbox = None\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : message = None\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : qsub_options = None\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : results_dir = None\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : components_dir = None\n",
"[genesis] out: 2015-09-08 14:10:10,905 : INFO : working_dir = /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast\n",
"[genesis] out: 2015-09-08 14:10:10,906 : INFO : samples = /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast/samples.txt\n",
"[genesis] out: 2015-09-08 14:10:10,906 : INFO : config = /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast/miseq.yaml\n",
"[genesis] out: 2015-09-08 14:10:10,906 : INFO : email = None\n",
"[genesis] out: 2015-09-08 14:10:10,906 : INFO : profile = False\n",
"[genesis] out: 2015-09-08 14:10:10,906 : INFO : remote_transfer_host = None\n",
"[genesis] out: 2015-09-08 14:10:10,907 : INFO : python_installation = None\n",
"[genesis] out: 2015-09-08 14:10:10,907 : INFO : num_pipelines = None\n",
"[genesis] out: 2015-09-08 14:10:10,907 : INFO : job_scheduler = drmaa\n",
"[genesis] out: 2015-09-08 14:10:10,907 : INFO : pipeline = miseq\n",
"[genesis] out: 2015-09-08 14:10:10,908 : INFO : drmaa_library_path = None\n",
"[genesis] out: 2015-09-08 14:10:10,908 : INFO : setup = /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast/miseq.setup\n",
"[genesis] out: 2015-09-08 14:10:10,908 : INFO : num_jobs = None\n",
"[genesis] out: 2015-09-08 14:10:11,195 : INFO : -------- IMPORTING SETUP FILE\n",
"[genesis] out: 2015-09-08 14:10:11,195 : INFO : ['__OPTIONS__', 'factory_dir', '/genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/lib/python2.7/site-packages/pipeline_factory/']\n",
"[genesis] out: 2015-09-08 14:10:11,195 : INFO : ['__OPTIONS__', 'components_dir', '/genesis/extscratch/shahlab/pipelines/factory/components/']\n",
"[genesis] out: 2015-09-08 14:10:11,196 : INFO : ['__OPTIONS__', 'drmaa_library_path', '/opt/sge/lib/lx24-amd64/libdrmaa.so']\n",
"[genesis] out: 2015-09-08 14:10:11,196 : INFO : ['__GENERAL__', 'python', '/genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/bin/python']\n",
"[genesis] out: 2015-09-08 14:10:11,196 : INFO : ['__GENERAL__', 'java', '/genesis/extscratch/shahlab/pipelines/apps/jdk1.7.0_06/bin/java']\n",
"[genesis] out: 2015-09-08 14:10:11,197 : INFO : ['__GENERAL__', 'strelka', '/genesis/extscratch/shahlab/softwares/strelka_workflow-1.0.13/bin/configureStrelkaWorkflow.pl']\n",
"[genesis] out: 2015-09-08 14:10:11,197 : INFO : ['__GENERAL__', 'perl', '/usr/bin/perl']\n",
"[genesis] out: 2015-09-08 14:10:11,197 : INFO : ['__GENERAL__', 'samtools', '/genesis/extscratch/shahlab/pipelines/apps/samtools-1.2/samtools']\n",
"[genesis] out: 2015-09-08 14:10:11,198 : INFO : ['__GENERAL__', 'bowtie2', '/genesis/extscratch/shahlab/pipelines/apps/bowtie2-2.0.2/bowtie2']\n",
"[genesis] out: 2015-09-08 14:10:11,198 : INFO : ['__GENERAL__', 'gatk', '/genesis/extscratch/shahlab/pipelines/apps/GenomeAnalysisTK-3.1-1/GenomeAnalysisTK.jar']\n",
"[genesis] out: 2015-09-08 14:10:11,198 : INFO : ['__GENERAL__', 'picard', '/genesis/extscratch/shahlab/pipelines/apps/picard-tools-1.71/AddOrReplaceReadGroups.jar']\n",
"[genesis] out: 2015-09-08 14:10:11,199 : INFO : ['__GENERAL__', 'mutationseq', '/genesis/extscratch/shahlab/pipelines/apps/mutationSeq_4.3.5_python2.7']\n",
"[genesis] out: 2015-09-08 14:10:11,199 : INFO : ['__SHARED__', 'ref_genome', '/genesis/extscratch/shahlab/pipelines/apps/bowtie2-2.0.2/genome/GRCh37-lite.fa_bt2-2.0.2']\n",
"[genesis] out: 2015-09-08 14:10:11,199 : INFO : ['__SHARED__', 'ref_genome_fa', '/genesis/extscratch/shahlab/pipelines/apps/reference/GRCh37-lite.fa']\n",
"[genesis] out: 2015-09-08 14:10:11,200 : INFO : ['__SHARED__', 'strelka_ref', '/genesis/extscratch/shahlab/pipelines/reference/GRCh37-lite.fa']\n",
"[genesis] out: 2015-09-08 14:10:11,200 : INFO : ['__SHARED__', 'dbsnp', '/genesis/extscratch/shahlab/pipelines/reference/common_all_dbSNP138.vcf']\n",
"[genesis] out: 2015-09-08 14:10:11,200 : INFO : ['__SHARED__', 'thousand_genomes', '/genesis/extscratch/shahlab/pipelines/reference/1000G_biallelic.indels.hg19.vcf']\n",
"[genesis] out: 2015-09-08 14:10:11,200 : INFO : ['__SHARED__', 'combined_vcfs', '/genesis/extscratch/shahlab/raniba/Tasks/component_dev/test_zone/ENOCA-83/VOA1976/combined_voa1976n-1.vcf']\n",
"[genesis] out: 2015-09-08 14:10:11,201 : INFO : ['__SHARED__', 'mutation_assessor_db', '/genesis/extscratch/shahlab/pipelines/reference/MA.hg19_v2/']\n",
"[genesis] out: 2015-09-08 14:10:11,201 : INFO : ['__SHARED__', 'cosmic_db', '/genesis/extscratch/shahlab/pipelines/reference/CosmicMutantExport.vcf']\n",
"[genesis] out: 2015-09-08 14:10:11,201 : INFO : ['__SHARED__', 'isSkipDepthFilters', False]\n",
"[genesis] out: 2015-09-08 14:10:11,202 : INFO : ['__SHARED__', 'combination_log', '/genesis/extscratch/shahlab/jzhou/factory/runs/ENOCA-83/combination_logs/voa1976n-1.log']\n",
"[genesis] out: 2015-09-08 14:10:11,202 : INFO : ['__SHARED__', 'amplicons_targets', '/genesis/extscratch/shahlab/raniba/Tasks/component_dev/test_zone/ENOCA-83/amplicon_mergeBed.txt']\n",
"[genesis] out: 2015-09-08 14:10:11,202 : INFO : ['__SHARED__', 'number_of_samples', 2]\n",
"[genesis] out: 2015-09-08 14:10:11,202 : INFO : ['__SHARED__', 'ld_library_path', ['/genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/lib', '/genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/lib64/']]\n",
"[genesis] out: 2015-09-08 14:10:11,232 : INFO : ['__SHARED__', 'normal_fastq_file_1', '/genesis/extscratch/shahlab/jzhou/factory/runs/ENOCA-83/data/VOA1976N-1_S11_L001_R1_001.fastq.gz']\n",
"[genesis] out: 2015-09-08 14:10:11,232 : INFO : ['__SHARED__', 'normal_fastq_file_2', '/genesis/extscratch/shahlab/jzhou/factory/runs/ENOCA-83/data/VOA1976N-1_S11_L001_R2_001.fastq.gz']\n",
"[genesis] out: 2015-09-08 14:10:11,232 : INFO : ['__SHARED__', 'snv_calls', '/genesis/extscratch/shahlab/raniba/Tasks/component_dev/test_zone/ENOCA-83/VOA1976/voa1976N-1.snv_calls']\n",
"[genesis] out: 2015-09-08 14:10:11,233 : INFO : ['__SHARED__', 'snv_calls_ids', '/genesis/extscratch/shahlab/raniba/Tasks/component_dev/test_zone/ENOCA-83/VOA1976/voa1976N-1.snv_calls_id']\n",
"[genesis] out: 2015-09-08 14:10:11,233 : INFO : ['__SHARED__', 'normal_id', 'VOA1976N-1']\n",
"[genesis] out: 2015-09-08 14:10:11,233 : INFO : -------- IMPORTING SAMPLES FILE\n",
"[genesis] out: 2015-09-08 14:10:11,234 : INFO : importing VS12-8909-2\n",
"[genesis] out: 2015-09-08 14:10:11,234 : INFO : importing VS12-8909-1\n",
"[genesis] out: 2015-09-08 14:10:11,407 : INFO : -------- INITIALIZING PIPELINE\n",
"[genesis] out: 2015-09-08 14:10:11,407 : INFO : RUN : python /genesis/extscratch/shahlab/pipelines/virtual_environments/targeted_sequencing_pipeline/lib/python2.7/site-packages/pipeline_factory//factory.py --working_dir /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast init_pipeline -c /genesis/extscratch/shahlab/raniba/test_express2/working_dir_miseq_from_beast/miseq.yaml -p miseq\n",
"[genesis] out: Disconnecting from ip.addresse.hidden... \n",
"done.\n",
"Disconnecting from ip.addresse.hidden... done.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"Stopped.\n"
]
},
{
"ename": "SystemExit",
"evalue": "1",
"output_type": "error",
"traceback": [
"An exception has occurred, use %tb to see the full traceback.\n",
"\u001b[0;31mSystemExit\u001b[0m\u001b[0;31m:\u001b[0m 1\n"
]
}
],
"source": [
"%run fab -f express_miseq.py main"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment