Skip to content

Instantly share code, notes, and snippets.

@gregcaporaso

gregcaporaso/jgcaporaso-qiime2-scipy2016.ipynb

Last active March 25, 2020 19:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save gregcaporaso/2a7b5f316a111e81910f0f19f7304a33 to your computer and use it in GitHub Desktop.
Save gregcaporaso/2a7b5f316a111e81910f0f19f7304a33 to your computer and use it in GitHub Desktop.
Download ZIP
QIIME 2 demo at SciPy2016
Display the source blob
Display the rendered blob
Raw
jgcaporaso-qiime2-scipy2016.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is not currently set up to work standalone. See [@gregcaporaso](http://github.com/gregcaporaso)'s [slides from SciPy 2016](http://bit.ly/qiime2-scipy2016) for more information. See http://2.qiime.org if you'd like to try QIIME 2. \n",
"\n",
"Installation steps prior to demo:\n",
"```bash\n",
"conda create -n q2-scipy2016 python=3.5 q2-diversity q2cli -c qiime2 -c biocore -c bioconda\n",
"source activate q2-scipy2016\n",
"```\n",
"\n",
"Download example Artifacts (QIIME 1 data files can be imported as QIIME 2 Artifacts, see [Importing data](https://github.com/qiime2/qiime2/wiki/Importing-data)): "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"!rm -rf *.qza *.qzv feature-table-frequency q2-demo-sample-md.tsv\n",
"!curl -sO https://raw.githubusercontent.com/qiime2/q2-types/master/q2_types/tests/data/feature-table-frequency.qza\n",
"!curl -sO https://raw.githubusercontent.com/qiime2/q2-types/master/q2_types/tests/data/phylogeny.qza\n",
"!curl -sO https://dl.dropboxusercontent.com/u/2868868/q2-demo-sample-md.tsv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Artifacts"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from qiime import Artifact"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"<artifact: FeatureTable[Frequency] uuid: 1bacd8ec-65bb-4938-b413-84ec7dea0af2>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"table = Artifact.load('./feature-table-frequency.qza')\n",
"table"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"tree = Artifact.load('./phylogeny.qza')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"658 x 104 <class 'biom.table.Table'> with 5670 nonzero entries (8% dense)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import biom\n",
"biom_table = table.view(biom.Table)\n",
"biom_table"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>New.CleanUp.ReferenceOTU0</th>\n",
" <th>New.CleanUp.ReferenceOTU1</th>\n",
" <th>New.CleanUp.ReferenceOTU3</th>\n",
" <th>New.CleanUp.ReferenceOTU6</th>\n",
" <th>New.CleanUp.ReferenceOTU7</th>\n",
" <th>New.CleanUp.ReferenceOTU8</th>\n",
" <th>New.CleanUp.ReferenceOTU9</th>\n",
" <th>New.CleanUp.ReferenceOTU10</th>\n",
" <th>New.CleanUp.ReferenceOTU12</th>\n",
" <th>New.CleanUp.ReferenceOTU13</th>\n",
" <th>...</th>\n",
" <th>New.CleanUp.ReferenceOTU1502</th>\n",
" <th>New.CleanUp.ReferenceOTU1503</th>\n",
" <th>New.CleanUp.ReferenceOTU1504</th>\n",
" <th>New.CleanUp.ReferenceOTU1507</th>\n",
" <th>New.CleanUp.ReferenceOTU1508</th>\n",
" <th>New.CleanUp.ReferenceOTU1509</th>\n",
" <th>New.CleanUp.ReferenceOTU1510</th>\n",
" <th>New.CleanUp.ReferenceOTU1512</th>\n",
" <th>New.CleanUp.ReferenceOTU1514</th>\n",
" <th>New.CleanUp.ReferenceOTU1516</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>K3.H</th>\n",
" <td>2.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>13.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>K3.Z</th>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>M2.Middle.L</th>\n",
" <td>0.0</td>\n",
" <td>6.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>6.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>K3.A</th>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>K3.R</th>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 658 columns</p>\n",
"</div>"
],
"text/plain": [
" New.CleanUp.ReferenceOTU0 New.CleanUp.ReferenceOTU1 \\\n",
"K3.H 2.0 0.0 \n",
"K3.Z 0.0 1.0 \n",
"M2.Middle.L 0.0 6.0 \n",
"K3.A 0.0 1.0 \n",
"K3.R 0.0 1.0 \n",
"\n",
" New.CleanUp.ReferenceOTU3 New.CleanUp.ReferenceOTU6 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU7 New.CleanUp.ReferenceOTU8 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU9 New.CleanUp.ReferenceOTU10 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU12 New.CleanUp.ReferenceOTU13 \\\n",
"K3.H 13.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" ... New.CleanUp.ReferenceOTU1502 \\\n",
"K3.H ... 0.0 \n",
"K3.Z ... 0.0 \n",
"M2.Middle.L ... 0.0 \n",
"K3.A ... 0.0 \n",
"K3.R ... 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU1503 New.CleanUp.ReferenceOTU1504 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 1.0 \n",
"M2.Middle.L 0.0 6.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU1507 New.CleanUp.ReferenceOTU1508 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU1509 New.CleanUp.ReferenceOTU1510 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 0.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU1512 New.CleanUp.ReferenceOTU1514 \\\n",
"K3.H 0.0 0.0 \n",
"K3.Z 0.0 0.0 \n",
"M2.Middle.L 1.0 0.0 \n",
"K3.A 0.0 0.0 \n",
"K3.R 0.0 0.0 \n",
"\n",
" New.CleanUp.ReferenceOTU1516 \n",
"K3.H 0.0 \n",
"K3.Z 0.0 \n",
"M2.Middle.L 0.0 \n",
"K3.A 0.0 \n",
"K3.R 0.0 \n",
"\n",
"[5 rows x 658 columns]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"table.view(pd.DataFrame)[:5]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Archive: feature-table-frequency.qza\r\n",
" inflating: feature-table-frequency/VERSION \r\n",
" inflating: feature-table-frequency/README.md \r\n",
" inflating: feature-table-frequency/metadata.yaml \r\n",
" inflating: feature-table-frequency/data/feature-table.biom \r\n"
]
}
],
"source": [
"!unzip feature-table-frequency.qza"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Executing actions defined by plugins"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'qiime.plugins.diversity',\n",
" 'qiime.plugins.feature_table',\n",
" 'qiime.plugins.types'}\n"
]
}
],
"source": [
"import qiime.plugins\n",
"import pprint\n",
"pprint.pprint(qiime.plugins.available_plugins())"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import qiime.plugins.feature_table\n",
"import qiime.plugins.diversity"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"table100 = qiime.plugins.feature_table.actions.rarefy(table, counts_per_sample=100)\n",
"dm = qiime.plugins.diversity.actions.beta_phylogenetic(table100, tree, metric='unweighted_unifrac')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(Note: alternatives to rarefaction, such as those discussed in [McMurdie *et al.* (2014)](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003531), are planned for addition as a plugin.)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"<artifact: DistanceMatrix % Properties(['phylogenetic']) uuid: 127b8e69-aa96-4c49-b6b3-0118b6df8c3c>"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Provenance is very basic at this stage, but it is being improved in alpha ([qiime2/qiime2#88](https://github.com/qiime2/qiime2/issues/88))."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"{'artifact_uuids': {'phylogeny': UUID('44b60757-effa-494d-a348-4b8a98f0084e'),\n",
" 'table': UUID('bff917f6-5017-4da3-9126-1be34e90ebd8')},\n",
" 'execution_uuid': UUID('bba2b170-4806-479d-a2b7-5ff52c444e38'),\n",
" 'executor_reference': 'beta_phylogenetic. Details on plugin, version, '\n",
" 'website, etc. will also be included, see '\n",
" 'https://github.com/biocore/qiime2/issues/26',\n",
" 'parameter_references': {'metric': 'unweighted_unifrac'}}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dm.provenance"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"table100.save('table100.qza')\n",
"dm.save('dm.qza')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Types help users use QIIME correctly"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Passing an incorrect type results in a ``TypeError``:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"ename": "TypeError",
"evalue": "Argument to parameter 'table' is not a subtype of FeatureTable[Frequency] % Properties(['uniform-sampling']).",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-14-6f0dc2091517>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mqiime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplugins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdiversity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbeta_phylogenetic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtree\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'unweighted_unifrac'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<decorator-gen-125>\u001b[0m in \u001b[0;36mbeta_phylogenetic\u001b[0;34m(table, phylogeny, metric)\u001b[0m\n",
"\u001b[0;32m/Users/caporaso/miniconda3/envs/q2-scipy2016/lib/python3.5/site-packages/qiime-2.0.1-py3.5.egg/qiime/sdk/method.py\u001b[0m in \u001b[0;36mcallable_wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 256\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 257\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 258\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msignature\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_types\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 259\u001b[0m \u001b[0moutput_types\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msignature\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msolve_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 260\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/Users/caporaso/miniconda3/envs/q2-scipy2016/lib/python3.5/site-packages/qiime-2.0.1-py3.5.egg/qiime/core/type/signature.py\u001b[0m in \u001b[0;36mcheck_types\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 61\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtype_\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m raise TypeError(\"Argument to parameter %r is not a subtype of\"\n\u001b[0;32m---> 63\u001b[0;31m \" %r.\" % (key, type_))\n\u001b[0m\u001b[1;32m 64\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 65\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0msolve_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0minput_types\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mTypeError\u001b[0m: Argument to parameter 'table' is not a subtype of FeatureTable[Frequency] % Properties(['uniform-sampling'])."
]
}
],
"source": [
"qiime.plugins.diversity.actions.beta_phylogenetic(table, tree, 'unweighted_unifrac')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Command line interface (``q2cli``) demo commands\n",
"\n",
"```bash\n",
"qiime \n",
"qiime info\n",
"qiime diversity --help\n",
"qiime diversity beta_phylogenetic --help\n",
"qiime diversity pcoa --help\n",
"qiime diversity pcoa --distance-matrix dm.qza --pcoa pc\n",
"conda install q2-emperor -c qiime2 -c biocore -c bioconda\n",
"qiime info\n",
"```\n",
"\n",
"See [Installing and using QIIME 2](http://2.qiime.org/Installing-and-using-QIIME-2) documentation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# QIIME Studio demo\n",
"\n",
"```\n",
"cd qiime-studio-0.0.1\n",
"npm start\n",
"```\n",
"\n",
"Then, through the UI, change to the directory where the previous demos were run. (During alpha, we will transition so this can be started as a normal desktop application, e.g., with a QIIME icon.)\n",
"\n",
"See [QIIME Studio](http://2.qiime.org/QIIME-studio) documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment