PyVO: ADASS 2013 Focus Demo: examples of use

PyVOatADASS2013

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Overview"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Most commonly used functions will be available from the top pyvo module"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pyvo as vo"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Building a catalog of images\n",
      "\n",
      "We want to create a catalog of available X-ray images of our favorite source (supernova remnant, Cas A).\n",
      "\n",
      "First we look for archives that have x-ray images:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# find archives with x-ray images\n",
      "archives = vo.regsearch(servicetype='image', waveband='xray') "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`archives` now contains a list of archives with data\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(archives)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "20"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we'll search each archive to find out if they have images of our source.  For that, we need to get its position:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pos = vo.object2pos('Cas A')\n",
      "pos"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "(350.85000000000002, 58.814999999999998)"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As we search, we will write our results to a CSV file.  We need to make sure that be prepared for failures by catching the `DALAccessError`.  This example was run in October the day after the US goverment shutdown and NASA went off-line; consequently, most of the archive queries failed.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# find images and list them in a CSV file\n",
      "with open('cas-a.csv', 'w') as csv:\n",
      "    csv.write(\"Archive short name,Archive title,Image title,RA,Dec,URL\\n\")\n",
      "    \n",
      "    for arch in archives:\n",
      "        print(\"searching %s...\" % arch.shortname)\n",
      "        \n",
      "        try:\n",
      "            matches = arch.search(pos=pos, size=0.25)\n",
      "        except vo.DALAccessError as ex:\n",
      "            print(\"Trouble accesing %s archive (%s)\" % \n",
      "                  (arch.shortname, str(ex)))\n",
      "            continue\n",
      "            \n",
      "        print(\"...found %d images\" % matches.nrecs)\n",
      "        for image in matches:\n",
      "            csv.write(','.join( (arch.shortname, arch.title, image.title,\n",
      "                                 str(image.ra), str(image.dec),\n",
      "                                 image.getdataurl()) )) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "searching ROSAT SIA...\n",
        "WARNING"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W22: dal_query:3:2: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        ": W22: dal_query:3:2: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring [astropy.io.votable.exceptions]\n",
        "WARNING"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:40:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:41:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:46:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:47:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:48:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:66:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:67:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:72:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:73:16: W01: Array uses commas rather than whitespace\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W01: dal_query:74:16: W01: Array uses commas rather than whitespace (suppressing further warnings of this type...)\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        ": W01: dal_query:40:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:41:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:46:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:47:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:48:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:66:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:67:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:72:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:73:16: W01: Array uses commas rather than whitespace [astropy.io.votable.exceptions]\n",
        "WARNING: W01: dal_query:74:16: W01: Array uses commas rather than whitespace (suppressing further warnings of this type...) [astropy.io.votable.exceptions]\n",
        "...found 82 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching NED(images)...\n",
        "Trouble accesing NED(images) archive (E19: E19)"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching SkyView...\n",
        "...found 215 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching HEAVENS @ ISDC...\n",
        "WARNING"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:41:6: W50: Invalid unit string 'bytes'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        ": W50: dal_query:41:6: W50: Invalid unit string 'bytes' [astropy.io.votable.exceptions]\n",
        "...found 4 images\n",
        "searching TGCat SIA...\n",
        "WARNING"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:20:0: W50: Invalid unit string 'degree'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:21:0: W50: Invalid unit string 'degree'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:27:0: W50: Invalid unit string 'pixel'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:28:0: W50: Invalid unit string 'degree/pixel'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "WARNING:astropy:W50: dal_query:29:0: W50: Invalid unit string 'kB'\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        ": W50: dal_query:20:0: W50: Invalid unit string 'degree' [astropy.io.votable.exceptions]\n",
        "WARNING: W50: dal_query:21:0: W50: Invalid unit string 'degree' [astropy.io.votable.exceptions]\n",
        "WARNING: W50: dal_query:27:0: W50: Invalid unit string 'pixel' [astropy.io.votable.exceptions]\n",
        "WARNING: W50: dal_query:28:0: W50: Invalid unit string 'degree/pixel' [astropy.io.votable.exceptions]\n",
        "WARNING: W50: dal_query:29:0: W50: Invalid unit string 'kB' [astropy.io.votable.exceptions]\n",
        "...found 9 images\n",
        "searching RASS.25keV [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching RASSBCK [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching BATSIG [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching GRANAT [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching HEAO1A [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching HRI [1]...\n",
        "...found 5 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching INTEGRALSPI_gc [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching INTGAL [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching PSPC1 [1]...\n",
        "...found 5 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching PSPC2 [1]...\n",
        "...found 5 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching PSPC6 [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching RASSALL [1]...\n",
        "...found 15 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching RXTE [1]...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching CSC...\n",
        "...found 0 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "searching CDA...\n",
        "...found 454 images"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### PyVO Design Features\n",
      "\n",
      "Iteration as a natural, pythonic way to process results\n",
      "\n",
      "  + Dataset discovery:  access tends to be row-based\n",
      "  + Catalog querying:  row- or column-based\n",
      "\n",
      "The interface tries to be self-explanatory\n",
      "\n",
      "  + functions and result objects are documented\n",
      "  + important metadata in a response record are available as properties\n",
      "  + minimize required knowledge of protocols for most common queries\n",
      "  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "archives[0].title"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "'SIA Service for ROSAT Archive'"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "archives[0].publisher"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "'German Astrophysical Virtual Observatory'"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "An archive service is queried via its access URL."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "archives[0].accessurl"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "'http://www.g-vo.org/rosat/SIAP?action=queryImage&siap=siap.service.rosat&'"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_(slide)_"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Querying a catalog"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "usnob = 'http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&'\n",
      "srcs = vo.conesearch(usnob, pos=(45.31, 74.0), radius=0.05)  "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(srcs)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "203"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "srcs.fieldnames()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "[u'id',\n",
        " u'RA',\n",
        " u'DEC',\n",
        " u'sRa',\n",
        " u'sDec',\n",
        " u'sRaEp',\n",
        " u'sDeEp',\n",
        " u'MuRA',\n",
        " u'MuDEC',\n",
        " u'sMuRA',\n",
        " u'sMuDE',\n",
        " u'B1',\n",
        " u'B1_S/G',\n",
        " u'R1',\n",
        " u'R1_S/G',\n",
        " u'B2',\n",
        " u'B2_S/G',\n",
        " u'R2',\n",
        " u'R2_S/G',\n",
        " u'I2',\n",
        " u'I2_S/G',\n",
        " u'Xi',\n",
        " u'Eta',\n",
        " u'Gal_L',\n",
        " u'Gal_B']"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "srcs[0]['sRa']\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "192"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "at = srcs.votable.to_table()\n",
      "b1 = at['B1']\n",
      "b1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "<MaskedColumn name='B1' units='mag' format=u'%.3f' description=None>\n",
        "masked_array(data = [21.09 20.94 21.15 20.89 0.0 0.0 0.0 19.96 19.51 0.0 21.18 0.0 20.58 0.0\n",
        " 19.38 21.16 20.16 19.69 0.0 0.0 0.0 21.14 16.69 0.0 20.79 0.0 20.98 0.0\n",
        " 0.0 0.0 0.0 0.0 0.0 0.0 21.04 20.93 21.02 0.0 0.0 0.0 0.0 0.0 21.11 0.0\n",
        " 0.0 0.0 0.0 0.0 16.58 20.2 0.0 20.83 0.0 0.0 20.32 20.37 19.33 0.0 0.0\n",
        " 21.18 19.15 0.0 0.0 20.38 21.02 21.17 19.36 0.0 0.0 19.45 0.0 0.0 0.0\n",
        " 18.61 20.66 0.0 0.0 20.78 20.95 0.0 0.0 0.0 0.0 0.0 19.2 0.0 0.0 0.0 0.0\n",
        " 21.15 0.0 0.0 0.0 20.9 17.52 0.0 20.08 0.0 20.54 20.61 18.53 0.0 20.15\n",
        " 20.69 0.0 17.88 19.07 21.18 0.0 21.08 18.33 18.01 20.67 20.13 0.0 0.0\n",
        " 21.04 18.68 20.33 20.71 18.9 0.0 0.0 0.0 20.84 0.0 19.26 19.07 0.0 20.98\n",
        " 0.0 0.0 0.0 20.07 0.0 19.87 19.25 19.59 15.81 0.0 0.0 21.18 21.17 20.0 0.0\n",
        " 0.0 19.9 0.0 20.73 21.18 0.0 16.96 18.75 18.54 20.79 21.17 17.59 19.17\n",
        " 21.0 21.08 20.47 20.47 18.54 21.15 17.85 20.65 0.0 0.0 20.9 20.93 21.07\n",
        " 18.36 0.0 20.92 0.0 20.35 20.5 17.73 0.0 20.88 0.0 0.0 0.0 21.05 0.0 19.83\n",
        " 0.0 21.17 0.0 21.04 21.18 0.0 18.11 18.78 0.0 21.18 20.92 0.0 0.0 0.0 0.0\n",
        " 20.3 20.11],\n",
        "             mask = [False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False False\n",
        " False False False False False False False False False False False],\n",
        "       fill_value = 1e+20)\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Creating image cutouts for a list of sources\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# obtain your list of positions from somewhere\n",
      "sourcenames = [\"ngc4258\", \"m101\", \"m51\"]\n",
      "mysources = {}\n",
      "for src in sourcenames:\n",
      "    mysources[src] = vo.object2pos(src) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create an output directory for cutouts\n",
      "import os\n",
      "os.mkdir(\"NVSSimages\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will use a query object to save and re-use query constraints"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# setup a query object for NVSS\n",
      "nvss = \"http://skyview.gsfc.nasa.gov/cgi-bin/vo/sia.pl?survey=nvss&\"\n",
      "query = vo.sia.SIAQuery(nvss)\n",
      "query.size = 0.2    # degrees square\n",
      "query.format = 'image/fits' "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for name, pos in mysources.items():\n",
      "    query.pos = pos\n",
      "    results = query.execute()\n",
      "    for image in results:\n",
      "        image.cachedataset(filename=\"NVSSimages/%s.fits\" % name) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "query.pos = mysources['m101']\n",
      "\n",
      "print query.getqueryurl() "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "http://skyview.gsfc.nasa.gov/cgi-bin/vo/sia.pl?survey=nvss&POS=210.8024292,54.34875&SIZE=0.2,0.2&FORMAT=image%2Ffits\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}