Skip to content

Instantly share code, notes, and snippets.

@dotsdl

dotsdl/pickling-anchors.ipynb

Created January 11, 2017 20:25
Show Gist options
  • Save dotsdl/37fb09f662bc093b782d2bca5b1f6d1b to your computer and use it in GitHub Desktop.
Save dotsdl/37fb09f662bc093b782d2bca5b1f6d1b to your computer and use it in GitHub Desktop.
Download ZIP
A quick gist exploring pickling using anchors in MDAnalysis (and current limitations/problems)
Display the source blob
Display the rendered blob
Raw
pickling-anchors.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We need to test how pickling behaves in MDAnalysis. This notebook tests how it works with the old anchor system. \n",
"\n",
"Full disclosure: I personally dislike this scheme for its use of globals, likelihood for non-deterministic behavior, etc., but here we go."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import MDAnalysis as mda\n",
"from MDAnalysisTests.datafiles import GRO, XTC"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/alter/Library/mdanalysis/MDAnalysis/package/MDAnalysis/topology/guessers.py:54: UserWarning: Failed to guess the mass for the following atom types: M\n",
" \"\".format(', '.join(misses)))\n"
]
}
],
"source": [
"u = mda.Universe(GRO, XTC)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"CA = u.atoms.select_atoms('name CA')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"<AtomGroup with 214 atoms>"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"CA"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alright, so now we'll play with using other processes. Easiest way to do this in a Jupyter notebook is to use `dask.multiprocessing`, since I think the `multiprocessing` module last I checked doesn't play well with the Jupyter kernels this is a quick way to check things that's pretty commonly used by us in production."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using `dask.delayed` lets us build dask graphs from function calls, then we can process these using a multiprocessing scheduler and get the results back."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import dask\n",
"import dask.multiprocessing\n",
"from dask.delayed import delayed"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def get_names(atomgroup):\n",
" return atomgroup.names"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A normal call in this process:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA'], dtype=object)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"get_names(CA)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Building a `dask` graph for the same..."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"names = delayed(get_names)(CA)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAJAAAADwCAIAAABg0Hq1AAAABmJLR0QA/wD/AP+gvaeTAAAgAElE\nQVR4nO2deUATV/7A30xuwSRAQECuiFZtORSlFqJFQF3xWlfqUXWr1Gq3rgu1WmU90K3adm2l+qt2\nPWux2grWC6sgCFoEVBQQAwgRSBQQww2BQJKZvN8fESoh4QhJIDifv+DNm/e+k8/Mmzczb+YhEEJA\nYDqg/R0AQe8ghJkYhDATg9w5SSwW//HHH8YPhUANb29vLperngo7cf369f4Ij0Cdo0ePdraj4QhT\nUVdXx2azjRkfwauwWCyN6cQ5zMQghJkYhDATgxBmYhDCTAxCmIlBCDMxCGEmBiHMxCCEmRiEMBOD\nEGZiEMJMDEKYiUEI6zuw9tR7cw8/hwAAoCz/39yFP9cZbmCT8YUpnx77+kzlYBqqhfGz6t29bBAA\nAJDnZMvGjWcibcsUZQlfvT/JiW3G5vJW/ZDRd5NGF9aUuv/72y3GrtWQKJ9lFtp5vUkGAAD8SaaQ\n6zWSpFoCq+NCA/92FF91qahS8Mu80v/MnPNVZh+33bjClOLfP1v9gwA3aqWGRvrwIRw3bggAAMCG\n7BzG+HE01YKW1C9Dj4unf3FwtRfHzMZnQ+Q6l/u7Qw8V9m3ztY3pqKur67yoI015Zz6b5W5nzmA7\nv700Mr0a77C0uSBm0zzP4UNpDKvRMzace9KqFMd+PGZIe2NBm3WyRqm9cKyuIP5/n783wWnW/0qK\nLm+b62ZtZmY/6ZMYIdaWQ1Z88d/zxg1nUsk0tovPih8yG5UQbyxKOrbl/Xdc/L8peH77uw98nNjm\n1u7v7c9owGvufh/i48xkMF38P/+9/M9YO8UJIYRQmnfyo3eGDx3qPG3zhfs/H4gqxTtF2Hp+iRmi\n8TeljN/Fx2Dz1Q+HoVT/g+2rKu7/ewyZ7B7xsLsfFkIImUymxjEdOgtTVkQvsady3z+ZUy2pyjq+\nmEtGAAAI7d0DQhxCrOTH+W7TIq4J6poqM48vHUmhjv70pgRCrODrtym0vxwTd6EKQgghlrc3wJ5N\nRQB5VFDonl8yXzQ38PcFDCUN+/Bas2rrc74YT7eYeYBf3yopil7pSqbx9hUphIfmOFjSEUB2nR/+\n3S/3yhurs7+fMwyleizauGV/nKBeWp2xe4oZyWFtssqLljixR7u83VbFCOqlDSVJe2Y4+H//rLOw\nl7/Di6Pz/vZjlRJCCPHiyFkrLjSpFigyt71FRq1Xxcnas1adCKIhtKAT3Ww7hFD/wpS1UXMZJO76\nVDmEEEJcdOBdGivo2Mu9qTnxH9xJXz5+eTDgRft8qQjzvV/rei4MQgjldz4fRaLyIotVheIlkTwq\ndcp+IQ4hhPK7W9ysp+xT1SGLW8UhO/wzWQ4hxHJ2epIpE/bkv6y99coHFiTOiitS1b9Y3u4JFNr0\nIxVK7XHWVh2faeYefk+1a+DCwzuOaxUmS1g7LSJLASGEsOnCillt0ULZ1RArlDx6c4aiPa80+j06\noEzY04Nt1ypM13MYQmWxGEBSU6sAAACAWllbIZKHDwowAADA8pNvld7bMpaMIAiCIKSRG9LlUJrP\nL8Z6VQfKsbFCETqDrmp4EMYQBgIVCgwCAABl0h5+ZcpnIyrvnd75d/81v9QoFXIFbFsLUKiUtkjN\nzIcggER+2REAVBoFQEyBQe1xljC4Iyzy987wXrDlp7QyucvHO1c5ooqUMJeXGREEIXPXpyoAALgw\nq8jBazQZAACwwuyKUeMdO/ykf8YBAMAwDABAIoE+oHOnw2zm5p2B5Mu7Iq4WNzaL7x2Ouo1Z+wV6\nUgAAQFldWUOefuRFh8NIkbdngtZBdRpBEATRfJIAAAAgf3o1YrbbpH/Go0Ff7lo0rH1LUBTtYq1X\n0Rqnmf/OqG0BQ4svfRUyZeSowPDfSzXsarIL75uj5DHhCT/NN0MQBEEoE3YlHPCnIgjVa3cujtrY\n2qBQ0tDY3pNXNtTVKxHyMPue/wSd0b2XSHNbvfezaYwHW3m2liPmnVQuPhJ/5D1rBAAAECabiWVc\nuPTMcN1BWH89LDA4ir0r9c7piKU+wxk9dNQB7XEi1gE7Ewvy4/d/zLOqurl30aLIx+i7B0TtHR6I\nCb8LXPxrkyx9U8D6P1SnBdmNdbP25GIQQnnWNjcSeYyPtwUUC57UtxnDRUUinPzmOxP7suG6C5Om\nbg9J8otKfPhCKmsW5yUc/HBc2/Uixe1dX0tJwoZZK/7vhqBWhsnqRWk/hq6PKlUCBAEAKJU9un6E\nMpkcQhzDX+ZWKiEAmEIBAACKtLMxIouZK//m1N7kKFUvTqkaTUzRfkwolUqI422FqEpRKLCu4qyL\n+XJ/AWY2YkbY4ZS85I0e2MP0rGYNESrF2XlWXh4UAADARVlCx/Guf7Z35oEhS1ywu9cSa1RVY/k3\nksvpvh8sHt2TjdeGzsIUGdHRj66HcqltrTpKMnOa+vnV50oAADPo31v82C35Z8Kmj7aiU+gWIwL+\niwfNc0BRFpuNYiU5Oc8Lz/5w4Zmyqxrk5el3ijFMkJb6XA4BUIjv3C3E8JK76WUyCFCODQepvnk2\ntkRSm3/5wMl0iVJaWZ57OSa5MDWtAMOFGellMggA3pCTklmrbOKnP6zHAVBKRal3hTj2JP12aSvU\nHqc8Y/eSj4/eFja0Njc0yiDDa/IEM00h5mTLx40fCgAAoPlhDqX9EgwAAMAQv22Rf7eI+8/mc4JG\nybPEXWEHn078977Vrn279u3cD+lht74hNeIdtnrdqMPaJFU3VlmfefQf/iMt6VQzO8+/bosVqpLx\nitgwn+EWwyZ+dOaJvIvi8bJDAdS2Uqk+/72+l/fnv/4HS3H8edwmf2cW08599uZzjwujgu2ZDn4b\nonYH0duykT13Zpxf3n6XCDFbEn1vp2f7aZQ+N6pOW5wNvx0+ce/KF++Nt2WyHCYs3J1QqtAUpCLn\nP9M+jlNdIMjTN8/acq9Tttbi85uD3rQ2M7MeO2Pdsfu1PekdQwgNcB2mEEV/GhpV0hahEmupf54X\nG+b7fnRTT2Mi6AJtwnrXb2sHy/tueXjT1sfctvUREp1lN8Z9/PymyQzdiiToEToKkxRkFzxL3L1p\nEnl1kPco26GklirB3auxfLulofbEExtDouOva7Hg6O2YUFf+/hU8Vw6TbTt2xr9OPR+3JizIqadX\nhXjubq/2DksnGAtjBtUdff2h4xEGEPMxC7ZHLdiuc8Ukt21Z8m06r/7aQrRfJgYhzMQghJkYhDAT\ngxBmYhDCTAxCmIlBCDMxCGEmBiHMxCCEmRiEMBNj0ApTKrscf2CyaL1bf+nSpSFDhhgzFD1SXl4u\nFou9vLz6OxDdUagGG3VCq7CQkBCDBUOgOwgcdF/VhhC6uLhUVVVVV1ebbiOhjUF4DsvIyHj27FlL\nS8uVK1f6Oxb9MwiF/fLLL1QqlUwmnz59ur9j0T+DrUnEcXzYsGE1NTUAADKZLBaLLS0t+zsofTLY\njrDk5GSVLQAAhPDChQv9G4/eGWzCVO2h6m8I4alTp/o3Hr0zqJpEmUzG4XCampraUxAEKS0tHT58\neD9GpV8G1RF29erV5uYOb5mQSKRz5871VzyGYFAJO3PmDJnc4VYAjuNRUVH9FY8hGDxNokQi4XA4\ncrm88yKBQDBq1Cjjh2QIBs8RduHCBdU7xGpQKJSzZ88aPx4DMXiE/fzzzxpfiVYoFD/99JPRwzEU\ng6RJrKqqsrOzw3GtL1VnZ2ePGzfOmCEZCF1fhhhgJCcnDx8+vF1Yc3OzUqkcOlT1MitAUTQ5OXlw\nCBskR5gaa9asEYlECQkJ/R2I/hk857DXBEKYiUEIMzEIYSYGIczEIISZGIQwE4MQZmIQwkwMQpiJ\nQQgzMQhhJgYhzMQghJkYhDATgxBmYhDCTAxCmIlBCDMxCGEmBiHMxCCEmRiEMBODEGZiEMJMDEKY\niUEIMzEIYSYGIczEIISZGIQwE2MwvNBXXV1d00ZrayuEsKCgoLq6+uzZsyQSiUqlWrXB4XBQ1LT3\nURN7oa+5uTkvLy8nJ4fP5xcVFYlEIqFQ2Nra+moeBEFYLBaEsKmpSe0lWiqV6uzs7OLiMmLECE9P\nT3d3d3d3dxaLZdyN6BMmIKyysjIpKSk9PT0tLe3Ro0c4jpubm7/11lujR4/mcrkuLi4ODg42Njaq\nY4hOp6utLpfLa2pqVEdhaWmpSCQSiURPnjzh8/n19fUIgrz55ps8Hs/X1zcwMNDBwaE/NrEXDFxh\nmZmZly9fjouLy8rKIpFIXl5ePj4+vr6+Xl5eXC5XLy3b06dPc3JyVLvCgwcPWltb3d3dZ86cOW/e\nPF9f3wHaeBpjxtTekJeXt2XLlpEjRwIAXFxcPvnkk8uXLzc1GXzq2paWluvXr3/66aejR48GADg4\nOKxfv/7+/fuGrre3DBRhUqn0p59+4vF4AABnZ+eNGzdmZGT0VzC5ubnbt29XmRs3btyhQ4fq6+v7\nKxg1+l+YWCzesWMHh8OhUqkLFy6MjY3FMKy/g3rJgwcP1qxZY25ubm5uvmbNmsLCwv6OqF+FlZaW\nfvLJJzQazdraeseOHWKxuB+D6YL6+vpvv/3W0dGRTCYvW7asf7X1jzCxWBwaGkqn052dnQ8fPtzS\n0tIvYfQKuVx++vTpsWPHksnkkJAQkUjUL2EYW5hMJtu7dy+TybS3tz948KBMJjNyAH0Ew7BTp06N\nHDmSwWBs3bpVIpEYOQCjCktISFBt6vbt243Q8TMccrl83759bDbb3t4+OjramFUbSVhNTc2KFSsA\nAAsWLHj69KlxKjU0lZWVq1atQhBk7ty5paWlxqnUGMKuXr06bNgwe3v7ixcvGqE6I3Pz5s1Ro0Yx\nmcxTp04ZoTrDCpNKpevWrUMQZPny5QPnUkbvSKXSsLAwBEHef/99Q2+mAYU9ffp04sSJTCbz559/\nNlwtA4fExER7e3snJyeDXvIbSti1a9csLCzGjx9fUlJioCoGIGKxeOrUqQwGw3DNo0GE7d+/H0XR\nZcuWNTc3G6L8gQyGYZs3b0YQJDQ0FMdxvZevZ2EYhv3rX/9CEGTHjh36Ldm0iI6OptPpwcHBUqlU\nvyXrU5hMJgsODmYwGOfOndNjsSZKcnKyhYWFn59fY2OjHovVm7CWlpZZs2YxmcyUlBR9lWnq8Pl8\nW1vbt99+u6amRl9l6kdYS0vLtGnTLC0t+/GZyMBEIBA4OTl5enrqy5kehMlkslmzZllaWmZnZ/e9\ntMGHUCh0cnLy9vZuaGjoe2l9FYZhWHBwMJPJJI6tLhAIBHZ2dlOmTOl7H6SvwtatW8dgMIjzVrfk\n5uZaWloGBwf3sa/fJ2GRkZEIgpw5c6Yvhbw+pKSk0Gi0jRs39qUQ3YVdu3YNRdG9e/f2pfrXjdOn\nTyMIcuLECZ1L0FGYUCi0srJatmyZzhW/toSHh9PpdJ1P+bqMS2xpafHx8UFRNC0tjcFg6Hnc3WAH\nx/GgoCCBQJCZmWllZdXr9XWQvG7dOjab/Vrd1dUvVVVVjo6OCxYs0GHdXguLj48nOhp9JyUlhUQi\nHTt2rLcr9k5YdXX1sGHDli9f3ttqCDqzefNmMzOz4uLiXq3Vu3PYypUrExMT8/PzTeuNj4GJXC6f\nMGGCra1tQkKCxrkFNdNzt0lJSQiCXLhwoXc7EoF27t27RyKRTp482fNVeipMJpONHDkyODhYl7gI\ntBMaGsrhcGpra3uYv6fC9u7dy2Aw+mu46yCmsbHR1tY2LCysh/l7JEwsFrNYrIiIiD4ERqCVI0eO\nUCiUx48f9yRzj4SFhoba29ub9FjdgQyGYR4eHj083XQvrLy8nMFgHDp0qM+BEWjl0qVLCIJkZWV1\nm7P7bv0nn3wSFxdXWFhIo9H62pMl0AKE8O2337azs4uNje0+axe8ePGCRqMdPnxYL/sRQRdcuXIF\nQRA+n991tm6Ebdu2zdraWu9jtQg6o1Qq33rrrZUrV3adrSthUqmUw+G85iMMjcnx48epVGp5eXkX\nebr6skFMTExjY+PatWv12VoTaGf58uUsFuvHH3/sIk9Xwo4dOzZ//nwbGxt9B0agGRqN9ve///3E\niRNKpVJrJm2HXl5eHgAgMTHRAIc+gVby8/MBANevX9eWQesRdubMGWdn54CAAAPsSQRaGTt2rI+P\nz5kzZ7Rl0CosJiZm0aJFA/TzPYOaxYsXX7p0Se2LZ+1o9vHgwYOioqJFixYZMjACzSxatKipqen6\n9esal2oWFhsb6+LiMmHCBEMGRqAZOzs7Hx+fy5cva1yqWdi1a9eCgoJ68RiUQK8EBQXFx8dDTXcN\nNQirqqrKzs4OCgoyfGAEmpk5c2ZFRcWjR486L9IgLCkpiUQi+fv7Gz4wAs14eXnZ2NjcuHGj8yIN\nwlJTU728vMzNzQ0fGIFmEATh8XipqamdF2kQlpaW5uvra/ioCLpCJazzaUxdWHNzM5/P9/HxMVZg\nBJrh8XjV1dXFxcVq6erC+Hw+juNeXl7GCoxAM56eniQSKScnRy1dXdijR4/MzMy4XK6xAiPQDIPB\ncHV15fP5aunqwnJzc93d3Yk7UgMBDw+Pzj17dTFPnjxRfZyYoN954403ioqK1BLVhYlEIhcXFyNF\nRNAlLi4uIpFILbGDMAjhs2fPCGEDBC6XK5FIqqurX03sIKy2tlYqlTo6Oho3MI0onx77+kzlAJ21\nwjg4OTkBAMrKyl5N7CBMJdPa2tqYYWmmKXX/97db+juK/oXD4QAAampqXk3UIEyVrz9Rin//bPUP\nArz7nIMaNptNIpG6ahLr6uoAABYWFnqttzn/lw2zPeyHDrFwmbTsuzs1HceXSAvPbf7rOAcmfQhn\nzF82/lYkg5VX1k5dfFwgl11fPQxF6LN/qu2iYcTrC68f3rRwovPsw8Li2O3z3G3MzYe/s/acqN22\nvOTSlr+Od2DRKHQLru/K/2VJIFBKipOPb13qww34trAidf8KX2eLoTYeCw/cb1TW3jv4oa8LawiL\nG7Dp6vM/Y+0UJwAAgJb8n1b7ODCZLtPDLz44/X+nyrQPnuk9KIpaWFjU1tZ2SH11gEdMTAyCIEql\nUn+DSpQV0Uvsqdz3T+ZUS6qyji/mkhEAAEJ794AQhxAr+XG+27SIa4K6psrM40tHUqijP70pgRAr\n+PptCu0vx8TdRYLl7Q2wZ1MRQB4VFLrnl8wXzQ38fQFDScM+vKb6tKYi54vxdIuZB/j1rZKi6JWu\nZBpvX5FCeGiOgyUdAWTX+eHf/XKvvLE6+/s5w1Cqx6KNW/bHCeql1Rm7p5iRHNYmt6qq0Rwn9miX\nt9uqGEG9tKEkac8MB//vn+n5i5aOjo6RkZGvpnQQdvr0aSqVqs8KlbVRcxkk7vpUOYQQQlx04F0a\nK+hYqWq7mhP/wZ305eOXM63gRft8qQjzvV/rei4MQgjldz4fRaLyIotVheIlkTwqdcp+IQ4hhPK7\nW9ysp+xT1SGLW8UhO/wzWQ4hxHJ2epIpE/bkv6y99coHFiTOiisvxzhjebsnUGjTj1QotcdZW3V8\nppl7+D3VroELD+84rm9hrq6uX3311aspHaZUlMlkVCpVjwc1QKgsFgNIamoVAFAAQK2srRDJ3QcF\n2EcOVIDlJ98qvVcwlrzllTXI+fxibGFvHu2gHBsrFKEz6Krn4whjCAOBTQoMAgAAZdIefiUA8op7\np48c/N+P52uUTLkCtq0FpFRKW6Rm5kMQQCKTXv5PpVEAxBQYBADXEmcJI3CERf7eGd6Faz/fsHYJ\n7+Odq3T8mbRCo9FkMlmHrdV3FWqYzdy8M5B8eVfE1eLGZvG9w1G3MWu/QE8KAAAoqytryNOPvOhw\nGCny9kzo3cScCNLlWAb506sRs90m/TMeDfpy16Jh7RuMomgPR0BojdPMf2fUtoChxZe+CpkyclRg\n+O+lWK8i7x4EUX+9qIMwKpWqUCj0WyXNbfXez6YxHmzl2VqOmHdSufhI/JH3rBEAAECYbCaWceHS\nM8N1B2H99bDA4Cj2rtQ7pyOW+gxn6DJKRXuciHXAzsSC/Pj9H/Osqm7uXbQo8rEegn4FmUym9paX\nujC5XK7fKqWp20OS/KISH76QyprFeQkHPxzHfPmrUdze9bWUJGyYteL/bghqZZisXpT2Y+j6qFIl\nQBAAgFLZo+tmKJPJIcQx/GVupRICgKl2PEXa2RiRxcyVf3Nqa/qAUrXHQoUCgwBTtB8TSqUS4nhb\nIapSFAqsqzjrYr7cX4CZjZgRdjglL3mjB/YwPavPP1gHuhHGYDAghGqNZt9QZERHP7oeyqUiL0FJ\nZk5TP1d1mJlB/97ix27JPxM2fbQVnUK3GBHwXzxongOKsthsFCvJyXleePaHC8+67CvLy9PvFGOY\nIC31uRwCoBDfuVuI4SV308tkEKAcGw5SffNsbImkNv/ygZPpEqW0sjz3ckxyYWpaAYYLM9LLZBAA\nvCEnJbNW2cRPf1iPA6CUilLvCnHsSfrt0laoPU55xu4lHx+9LWxobW5olEGG12Q9DwyUyWTq07C+\n2i7fvn0bAPD8+XP9dXOUDakR77DVz5Sow9ok1TxUyvrMo//wH2lJp5rZef51W6xQlYxXxIb5DLcY\nNvGjM0/kXRSPlx0KaO8lUX3+e30v789//Q+W4vjzuE3+ziymnfvszeceF0YF2zMd/DZE7Q5q/xXI\nnjszzi9vO+oBYrYk+t5Oz/bTKH1uVJ22OBt+O3zi3pUv3htvy2Q5TFi4O6FUob9fDkKlUkkmk3/9\n9ddXEzsIU43Ef/Tokf4qVYiiPw2NKmnbECXWUv88LzbM9/1o4g337lBdMickJLya2KFDpvoanNq9\nkL6A5X23PLxp62NuWzUIic6yG+M+fn7TZOKzfd2huotoaWn5amIHYRwOh0ajlZeX66tKSUF2wbPE\n3ZsmkVcHeY+yHUpqqRLcvRrLt1saak880+4O1X16tbnAO/xsKIo6OjoKhUJ9VWmx4OjtmFBX/v4V\nPFcOk207dsa/Tj0ftyYsyInU/coAAADw3N1e7R2WTjAWxgziO/pCoZDBYKi9UKl+jcrlcp8+faq3\nOhHzMQu2Ry3YrnMBJLdtWfJteovHpFA9/Ve7K6DeMLm6ugoEAiNGRaCVJ0+euLq6qiWqC3N3d1f1\nEo0VFYFWcnJyPD091RLVhXl4eDQ0NJSWlhorKgLNyGSyJ0+euLu7q6VrOMJQFM3OzjZWYASaycvL\nUygU3QtjsVhjx469c+eOsQIj0ExaWhqbzR4zZoxauoarIR6Pl5aWZpSoCLSSlpbG4/E6D8HWIMzX\n1/fBgwfa3mInMA7aXvrSICwwMFAmk6luBBP0C3l5eWVlZdOmTeu8SIMwBweHt956Ky4uzvCBEWgm\nLi6Ow+FMnDix8yLNd/SCgoKuXbtm4KgItBIfHz9jxgyN7xBpFjZv3rzCwkLV56YIjEx1dfUff/wx\nd+5cjUs1C+PxeE5OTtHR0YYMjEAz58+fp1Kpc+bM0bhUszAEQYKDg2NiYgwZGIFmzp07N3v2bK1f\ncdD2uPP+/fsAgLS0NIM/WCV4BZFIhKLoxYsXtWXQ+hhx4sSJ48ePP378uIH2IwKNnDhxwtraevbs\n2doydPXc96OPPoqOjm5oaDBAYAQawHH85MmTISEhFApFW56uhC1btoxEIhEHmdE4f/58RUXF6tWr\nu8jTzUQD69ev/+2330pKSrpwTqAv3nnnHScnp276el2fA4VCIZlMPn36tJ7PrQSduHnzJgDgzp07\nXWfrfiqP5cuXZ2Zm5ubmkkg9HTlDoAMBAQHt2rqiW/MCgYBMJp86dUov+xGBRpKSkgAAt27d6jZn\nj+bADAkJuX37dn5+vp7fHiMAAAAAIZw8ebKZmVlCQkKPcneLUCik0+n79u3r435EoJGzZ8+iKNrD\nudN7OqXi1q1bmUxmRUVFHwIj0IBUKnV2dg4JCelh/p4Kk0gk9vb2q1at0jUwAs1ERESYm5v3/I2h\nXkwLHB0djSDIzZs3dYmLQBN5eXk0Gk3tOwFd07uJt+fPn5+fn5+Tk8NgEC+f9BWlUunn5yeTye7c\nudOLS6Ze7RGlpaUsFuvTTz/t3Y5EoIlvvvmGQqE8fPiwV2v1ThiE8NSpUwiC/P77771dkeBVsrKy\nqFTq119/3dsVe9ckqli6dOmtW7cePnxITC2mG01NTd7e3nZ2djdu3Oj1x1912Dvq6upGjBgxdepU\nhUKv7/S+NixevNja2rq0tFSHdXURBiHMyckZMmTIhg0bdFv9dWbfvn0oiqqmVtEBHYXBtpNZVFSU\nziW8hsTHx5PJZLXPR/UK3YVBCDdt2kShUG7cuNGXQl4fcnNz2Wz2kiVL+vK9vD4Jw3F80aJFFhYW\n3U4XTfDs2TNHR8epU6fKZLK+lKNLL/FVWltbZ86cWVhYmJKSMmrUqL4UNYgRi8V+fn4UCiUlJaWP\n3w/tqzAAgEQimTZt2osXL/744w/ii9ydqa2t9ff3b2lpSUlJsbW17WNpevhaxtChQ+Pi4iwsLPz8\n/Dp/GP81RywW+/v7NzQ03Lhxo++2gL6+l2hpaXnr1i17e/vJkyd3ni7ktaWioiIwMFAikSQnJ6u+\naa4H9HNKhRBC2NTUFBAQYGFhQdzRhxDy+XwnJ6exY8eWl5frsVh9CoMQtra2LlmyhEqlvuZjQBIT\nE1ksFo/Hq6qq0m/JehYGIcRxfMOGDQiChIeHYxim9/IHPpGRkWQyefny5X3swWtE/8JUnDhxgk6n\nT58+Xe+72EBGIpEsXryYRCJ99dVXev2a/J8YShiE8MGDB87Ozo6OjsnJyYarZeCQlZU1ZswYa2tr\ng976MaAwCGF1dfWCBQtQFA0PDzdE+zBAwHH8m2++oVKp/v7+ut2D7zmGFaYiKirK3Nzczc2t23HI\npohAIPD39yeTyTt27DDCOdsYwiCExcXF06ZNQ1E0NDS0vr7eOJUaGqlUGhERQaVSvb29e/ukX2eM\nJAxCqFQqT548yeFwrK2tjxw5YuodyOjoaGdnZ3Nz88jISGNui/GEqaitrZyappoAAAJUSURBVA0L\nC6NQKO7u7pcuXTJQV8qgJCcn+/r6oii6cuVKvX6BvEcYW5iKx48fL1iwAEGQiRMnXrlyxVS03bx5\n09/fHwAwffr0+/fv90sM/SNMRVZW1ty5cxEEefPNN48fP97a2tqPwXQBhmHR0dHe3t4AgKlTp6ak\npPRjMP0pTIVAIAgNDaXT6Ww2e82aNTk5Of0d0Z+UlZV9/fXXLi4uKIrOmTNnIHxTof+FqXj+/Pmu\nXbtUj9N8fX33799fVlbWX8FUVVUdOXIkICAARVFbW9vw8PDi4uL+CkaNgSJMBY7jCQkJH3zwAYvF\nQlGUx+Pt2bMnMzPTOCe53Nzcb7/9NjAwkEwmDxkyZOHChRcvXpTLu5pKxPjo4YmzIZDJZPHx8Zcv\nX46Pj6+oqLCxseHxeJMnT/b19fX09NTXyH65XJ6bm5uenp6Wlnb79u3y8nIOhzN9+vR58+bNnTvX\nzMxML7XolwEqrB0IYU5OTlJSUmpqanp6emVlJYlEcnV19fDweOONN7hcrouLi6OjI4fDUc1Doo2G\nhobKysry8nKRSCQUCgUCAZ/PFwgECoWCzWb7+vryeLzAwEBvb+9eD8U1LgNdmBpFRUUPHz7k8/m5\nubkCgUAkEjU1NakWqeZkZbFYAAA6nY6iqFQqBQBIJJK6ujoMezlRGJ1O53K5I0eOdHNz8/Dw8PDw\nGDNmzACX9ComJqwzVVVV5eXlNTU1VVVVjY2NqpmNW1palEqlqk1jsVhMJtPKysra2tre3l4vAyv6\nEZMX9rphMk0BgQpCmIlBCDMx/h8QQ5ua4DDU4AAAAABJRU5ErkJggg==\n",
"text/plain": [
"<IPython.core.display.Image object>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"names.visualize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"...and executing with a child process:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA', 'CA',\n",
" 'CA', 'CA', 'CA', 'CA', 'CA'], dtype=object)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"names.compute(get=dask.multiprocessing.get)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Okay, so doing something basic works fine. Good."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What if we have two universes that are built from the same files, but they have different topology attributes?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll make two universes, and an `AtomGroup` for each. The universes are generated from the same files:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"u1 = mda.Universe(GRO, XTC)\n",
"u2 = mda.Universe(GRO, XTC)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"CA1 = u1.select_atoms('name CA')\n",
"CA2 = u2.select_atoms('name CA')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's change the atom names of each universe to new values:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"u1.atoms.names = 'lol'"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"u2.atoms.names = 'cats'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, if we get names for each, what do we get?"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol',\n",
" 'lol', 'lol', 'lol', 'lol', 'lol', 'lol', 'lol'], dtype=object)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"get_names(CA1)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats'], dtype=object)"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"get_names(CA2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now what if we do multiprocessing, which requires serializing an `AtomGroup` and the function name, deserializing them in a new process, and executing the function on the `AtomGroup`?"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats'], dtype=object)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"delayed(get_names)(CA1).compute(get=dask.multiprocessing.get)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What's happening here? It looks like getting the names for the first `AtomGroup` yields names you'd expect only to see for the second, since it comes from a different universe. What if we do the same for the second `AtomGroup`?"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats', 'cats',\n",
" 'cats', 'cats', 'cats', 'cats', 'cats', 'cats'], dtype=object)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"delayed(get_names)(CA2).compute(get=dask.multiprocessing.get)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is because the universes for both these `AtomGroup` were defined with the same topology/trajectory, so the only one that got held on to with a global weakref was the second one we defined, and this is the one that got used by both `AtomGroup`s upon being deserialized:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(47681,\n",
" '/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.gro',\n",
" '/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.xtc')"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u1.anchor_name"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(47681,\n",
" '/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.gro',\n",
" '/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.xtc')"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u2.anchor_name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As an aside: no `Universe` objects have to be serialized here, because the processes spawned by `dask.multiprocessing` have access to the memory space of the parent, and therefore to the global store of weakrefs."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Does it work with the distributed scheduler?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given how this scheme works, I don't think this works at all with the distributed scheduler, which can't rely on shared memory between parent and child processes (there is no parent-child relationship; everything must be serialized). I want to test this, though:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from dask.distributed import Client"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cl = Client('127.0.0.1:8786')"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"names_d = delayed(get_names)(CA1)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"L_names_d = cl.compute(names_d)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"RuntimeError('Long error message',\n",
" '(RuntimeError(\"Couldn\\'t find a suitable Universe to unpickle AtomGroup onto with Universe hash \\'(47681, \\'/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.gro\\', \\'/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.xtc\\')\\'. Available hashes: \",), <function _unpickle at 0x7fd1841ab758>, ((47681, \\'/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.gro\\', \\'/home/alter/Library/mdanalysis/MDAnalysis/testsuite/MDAnalysisTests/data/adk_oplsaa.xtc\\'), array([ 4, 21, 45, 64, 83, 102, 121, 128, 137, 152, 159,\\n 169, 176, 198, 205, 219, 236, 246, 263, 283, 302, 319,\\n 334, 356, 377, 384, 402, 417, 434, 453, 464, 478, 485,\\n 497, 514, 533, 557, 567, 577, 593, 615, 626, 633, 644,\\n 659, 678, 685, 707, 724, 734, 756, 768, 787, 804, 816,\\n 826, 833, 855, 874, 890, 904, 916, 931, 950, 966, 985,\\n 995, 1014,')"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"L_names_d.exception()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Yeah, there is no universe to connect the `AtomGroup` to, as one might expect."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iteration through trajectories for e.g. fitting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What happens if we write a function that takes two `AtomGroup`s and iterates through one of their trajectories, when each `AtomGroup` comes from a different Universe made from the same files? This is very often the case when doing, say, fitting to a reference frame."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from MDAnalysis.analysis.align import alignto\n",
"\n",
"def fit_to_reference(ag, reference):\n",
" \n",
" rmsd = list()\n",
" \n",
" for ts in ag.universe.trajectory:\n",
" rmsd.append(alignto(ag, reference)[1])\n",
" \n",
" return rmsd"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"u3 = mda.Universe(GRO, XTC)\n",
"u4 = mda.Universe(GRO, XTC)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"CA3 = u3.select_atoms('name CA')\n",
"CA4 = u4.select_atoms('name CA')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The right answer:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1.2774938328230612e-06,\n",
" 9.81877665004486,\n",
" 8.223452081220534,\n",
" 6.44980413011577,\n",
" 8.28539036019982,\n",
" 19.38629287358165,\n",
" 16.88768126690707,\n",
" 16.88520038716091,\n",
" 18.211024048293112,\n",
" 21.305871377997253]"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fit_to_reference(CA3, CA4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What do we get with `dask.multiprocessing`?"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[1.2774938328230612e-06,\n",
" 1.2231072255363424e-06,\n",
" 1.2231072255363424e-06,\n",
" 9.033245521131802e-07,\n",
" 1.0430693466562854e-06,\n",
" 9.033245521131802e-07,\n",
" 1.0430693466562854e-06,\n",
" 9.757020314584897e-07,\n",
" 1.0430693466562854e-06,\n",
" 8.246187232343996e-07]"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"delayed(fit_to_reference)(CA3, CA4).compute(get=dask.multiprocessing.get)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the wrong answer. It turns out that both `AtomGroup`s get linked into the same universe in the child process, so they both step through the same frames, giving approximately 0 for all rmsds following a fit. This is a big problem."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python2",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment