CCI API See cci.ipynb

cci.ipynb

{
 "metadata": {
  "name": "cci"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# CCI Python Package\n",
      "\n",
      "This notebook is to flesh out the API of a python package the CCI should provide to users who want to run large-scale image processing jobs against the VUIIS XNAT.\n",
      "\n",
      "## Goals\n",
      "\n",
      "-    Provide a high-level interface so CCI-developed image processing routines (Freesurfer, DTI QA, fMRI QA, DCM->NII, etc)  can be implemented in extremely short spiders\n",
      "-    Provide medium- and low-level hooks so advanced developers can pick and choose what level they'd like to use the CCI/XNAT infrastructure.\n",
      "\n",
      "These goals accomplish two things:\n",
      "\n",
      "-    Basic users can get spiders up and running extremely quickly.\n",
      "-    Advanced users are not shut out from:\n",
      "    -    using provided infrastructure \n",
      "    -    sending advanced image processing pipelines \"upstream\" to the CCI.\n",
      "\n",
      "First, the basics:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from cci import FMRIQARunner, DTIQARunner, FreesurferRunner\n",
      "\n",
      "# So a spider can include all three?\n",
      "\n",
      "for sub in project.subjects().get():\n",
      "    for exp in s.experiments().get():\n",
      "        for scan in exp.scans.get():\n",
      "            for runner in [FMRIQARunner, DTIQARunner, FreesurferRunner]:\n",
      "                if runner.can_execute(scan):\n",
      "                    runner.execute()\n",
      "                # Each Runner understands what it can process\n",
      "                # only jobs that make sense actually run\n",
      "        \n",
      "# And/or we provide the loop infrastructure\n",
      "from cci import ScanLoop, ExperimentLoop, SubjectLoop\n",
      "# Loops take Runner(s)\n",
      "ScanLoop(project, subject, experiment, [FMRIQARunner, DTIQARunner]) # Must provide ScanRunners\n",
      "ExperimentLoop(project, subject, [ExperimentRunner]) # Hits all experiments\n",
      "SubjectLoop(project, [SubjectRunner]) # Hits all subjects "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Lower-Level API\n",
      "\n",
      "The `cci` package is fundamentally talking to two systems: the ACCRE cluster and VUIIS' XNAT."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from cci import cluster\n",
      "\n",
      "pbs_path = cluster.make_pbs(list_of_commands, account_info)\n",
      "job_id = cluster.submit_pbs(pbs_path)\n",
      "status = cluster.check_status(job_id)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The XNAT API is a little more advanced though. Nominally, we'd like to provide support to:\n",
      "\n",
      "- Easily create accessors without users knowing the folder structure.\n",
      "- Sync a local filesystem directory with XNAT. We'll want to provide both directions so beginnig jobs can pull data and ending jobs can push data.\n",
      "- ???"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from cci import xnat\n",
      "\n",
      "# Creating accessors\n",
      "jh = xnat.JobHandler(my_job.project, my_job.subject, my_job.experiment) # accessors are at the subject or experiment level?\n",
      "my_job.start()\n",
      "\n",
      "# this is really stupid but you get the idea\n",
      "while not my_job.is_finished():\n",
      "    pass\n",
      "\n",
      "if my_job.success():\n",
      "    jh.add_pdf(my_job.get_pdf())\n",
      "    jh.add_snapshots(path_to_local_thumbnail, path_to_local_large_version)\n",
      "    jh.add_directory(local_source, accessor_directory)\n",
      "    jh.add_provenance(my_job.get_provenance())\n",
      "    jh.finish() #???\n",
      "else:\n",
      "    jh.clean_up()\n",
      "\n",
      "# Synchronizing resources from local <--> XNAT\n",
      "rs = xnat.ResourceSyncer(my_job.project, my_job.subject, my_job.experiment) # Maybe lower?\n",
      "success = rs.sync_to_local(local_top_level)\n",
      "success = rs.sync_to_xnat(local_top_level)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}