HST Duplication checking .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HST Duplication checking\n",
    "---------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from astropy.table import Table\n",
    "from astropy import coordinates as coords\n",
    "import astropy.units as u\n",
    "import cPickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the PAEC catalog from https://archive.stsci.edu/hst/paec.html.\n",
    "Remove the header and replace with the following:\n",
    "\n",
    "```targname        | ra          | dec       |config  | mode       | aper       |spec         |  wave |time |prop |cy|dataset   |release |\n",
    "```\n",
    "\n",
    "Read it in, add the SkyCoord objects and write it out in a Pickle format so it's *much* faster to re-use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "paec = Table.read('paec_7-present.cat',format='ascii.fixed_width')\n",
    "paec['coords'] = coords.SkyCoord(paec['ra'],paec['dec'],unit=(u.hour, u.deg),frame='icrs')\n",
    "fp = open('paec.p','wb')\n",
    "cPickle.dump(paec,fp,protocol=cPickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The functions below do the duplication checking.** \n",
    "\n",
    " * `gp` is a table of targets that already have a `coords` column that is a `coords.SkyCoord` object\n",
    " * `paec` is the PAEC table\n",
    " * Returns:\n",
    "   * `urls` -- urls to the HST Program status pages. Empty string if there is no match. \n",
    "   * `propids` -- dictionary of matching proposal ids keyed by the index in `gp`; no entry in the dictionary if there is no match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def propurl(id):\n",
    "    base=\"<a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=\"\n",
    "    suffix=\"&observatory=HST\"\n",
    "    str = \"%s%d%s> %d </a>\" % (base,id,suffix,id)  \n",
    "    return str\n",
    "\n",
    "def dup_urls(gp,propids):\n",
    "    urls = [\"\" for i in range(len(gp))]\n",
    "    for i in range(len(gp)):\n",
    "        if i in propids:\n",
    "            for p in propids[i]:\n",
    "                urls[i] += propurl(p)+\" \"\n",
    "            urls[i] = urls[i][:-1]\n",
    "    return urls\n",
    "\n",
    "def duplications(gp,paec):\n",
    "    for g in gp:\n",
    "        idxc, idxcatalog, d2d, d3d = gp['coords'].search_around_sky(paec['coords'],200*u.arcsec)\n",
    "    propids = {}\n",
    "    for id_gp,id_paec in zip(idxcatalog,idxc):\n",
    "        if id_gp not in propids:\n",
    "            propids[id_gp] = [paec['prop'][id_paec]]\n",
    "        else: \n",
    "            propids[id_gp] += [paec['prop'][id_paec]]\n",
    "    for p in propids.keys():\n",
    "        propids[p] = list(set(propids[p]))\n",
    "    urls = dup_urls(gp,propids)\n",
    "    return urls, propids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example usage\n",
    "----\n",
    "First create a table with a couple sources. In reality, you would read in your table, probably using `Table.read`. Create a SkyCoord column for the coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_rows = [('IC10',5.072250,59.303780),\n",
    "             ('Abell209',22.95901,-13.591956)\n",
    "            ]\n",
    "my_catalog = Table(rows=data_rows,names=['name','RA','Dec'])\n",
    "my_catalog['coords']=coords.SkyCoord(my_catalog['RA'],my_catalog['Dec'],\n",
    "                                     unit=(u.deg, u.deg),frame='icrs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the PAEC pickled file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fp = open('paec.p','rb')\n",
    "paec = cPickle.load(fp)\n",
    "fp.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the checking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "urls,propids = duplications(my_catalog,paec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: [14073, 10242, 9683], 1: [8249, 12451]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Format the catalog for the notebook, hacking the URL fields so they are clickable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<Table length=2>\n",
       "<table id=\"table4461948432-349768\" class=\"table table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>name</th><th>RA</th><th>Dec</th><th>coords</th><th>paec</th></tr></thead>\n",
       "<thead><tr><th></th><th></th><th></th><th>deg,deg</th><th></th></tr></thead>\n",
       "<tr><td>IC10</td><td>5.07225</td><td>59.30378</td><td>5.07225,59.30378</td><td><a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=14073&amp;observatory=HST> 14073 </a> <a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=10242&amp;observatory=HST> 10242 </a> <a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=9683&amp;observatory=HST> 9683 </a></td></tr>\n",
       "<tr><td>Abell209</td><td>22.95901</td><td>-13.591956</td><td>22.95901,-13.591956</td><td><a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=8249&amp;observatory=HST> 8249 </a> <a href=http://www.stsci.edu/cgi-bin/get-proposal-info?id=12451&amp;observatory=HST> 12451 </a></td></tr>\n",
       "</table><style>table.dataTable {clear: both; width: auto !important; margin: 0 !important;}\n",
       ".dataTables_info, .dataTables_length, .dataTables_filter, .dataTables_paginate{\n",
       "display: inline-block; margin-right: 1em; }\n",
       ".paginate_button { margin-right: 5px; }\n",
       "</style>\n",
       "<script>\n",
       "require.config({paths: {\n",
       "    datatables: 'https://cdn.datatables.net/1.10.9/js/jquery.dataTables.min'\n",
       "}});\n",
       "require([\"datatables\"], function(){\n",
       "    console.log(\"$('#table4461948432-349768').dataTable()\");\n",
       "    $('#table4461948432-349768').dataTable({\n",
       "        \"order\": [],\n",
       "        \"iDisplayLength\": 50,\n",
       "        \"aLengthMenu\": [[10, 25, 50, 100, 500, 1000, -1], [10, 25, 50, 100, 500, 1000, 'All']],\n",
       "        \"pagingType\": \"full_numbers\"\n",
       "    });\n",
       "});\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_catalog['paec']=urls\n",
    "ipy_html = my_catalog.show_in_notebook()\n",
    "ipy_html.data = ipy_html.data.replace('&lt;','<')\n",
    "ipy_html.data = ipy_html.data.replace('&gt;','>')\n",
    "ipy_html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}