Demitri Muna’s suggestions for HiPS/MOC survey footprints, 30 October 2015

Muna_MOC_2015-10-30.md

Notes on HiPS

HiPS is a useful iteration on top of Healpix, however, it is missing a few things that could make it useful in production pipeline code (and other uses). Of particular note, it has a notable functional overlap with mangle (http://space.mit.edu/~molly/mangle/), a tool for creating polygons on a sphere. Mangle is used extensively in SDSS and DES.

These are the area that I feel HiPS could be improved upon.

1. HiPS should have "memory" of the individual files that comprise the survey footprint. Given an ra/dec position, one should be able to query the MOC and have returned all of the files that contain that position. Take for example SDSS. Nearly every image frame (if not all) overlap at least a little, and in many cases (such as Stripe 82) many times, with other images. A much more useful footprint survey map will contain a) the overlap information and b) the name (or some unique identifier) of the file of each "tile". In fact, ESA Sky requires this information, but it was pointed out to me by Thomas Boch that it is contained in a separate index.

   This form could be optional. Since the MOC is a FITS file, a second table extension could be added to contain this information. This is off the top of my head, but a second table HDU could be added with two columns: "pixel ID" and "file ID" (effectively a relational "join" table), and a third HDU containing the columns "file ID" and "filename". This would effectively (and optimally?) map every HiPS pixel to the files that cover that pixel. Additional optional columns could be added to the file field, such as URL, date, etc.

2. Additional geometry support for round shapes. Most images that come from a CCD are square(ish), and HiPS is optimized for this. However, many footprints can contain circular features, most notably SDSS spectroscopic plates (and I'm sure this isn't an outlier). A curved surface will require a large number of HiPS pixels to describe it at the edges. Having a way to describe a circle with an ra/dec point and a radius would be more concise. I would recommend stealing the best aspects of mangle for this.

3. Catalog support. This is more of a question, but what support is there for catalog objects? What if my "survey" is a list of 500 point sources across the sky? I'd like to describe this in a MOC, but I don't have a list of FITS files as input that describe a footprint. A point source should be a special type of entry in the MOC, perhaps having a specified (or preferably, default) radius, which can be overridden in queries.

4. Polygon generation / visualization. I want to use the MOC to draw the footprint using my own software, be it to overlay on a matplotlib plot, my Nightlight application (C), or anything else. This requires being able to get a list of points describing a polygon from a library that reads the MOC file. The Starlink AST package has fantastic support for polygons and could, for example, convert a list of ra/dec points to graphics coordinates given a WCS. I have code that does this in Python (and I could easily make a C version). Ideally I would like to get the polygon that describes the full footprint as well as one that would draw the individual polygons of the individual images/fields. You should provide library support to make visualization trivial in a way that is agnostic to the drawing device (e.g. don't make it specific to matplotlib, for example).

5. Library support. I strongly recommend that any new proposed standard like this be delivered with functional libraries FROM DAY ONE. In fact, you should implement your own code using these libraries (there is the expression, "eat your own dog food"). At the very least, the libraries should be in Python and C. (You can write one in C and have Python wrap that.) I saw your command line program to create a MOC from individual FITS files. This is great, but it should be calling a library that has the actual funtionality. This makes it far easier for me to embed this functionality into a pipeline or an application. Library support is not something that can come "later"; if you truly want the community to embrace it, it is required immediately. Sure, the API may change or the specification/functionality may change. That's fine; it will! But the community should be a part of that, and that will only happen when you make the tools available for people to use it, poke it, and find problems.

6. GitHub. The command line tools and the libraries (point 5) need to be released (immediately!) on GitHub. You need to provide an easy way for people from the community to submit issues, requests, and code in a public fashion. Astropy's model here is a clear, runaway success and should be followed. Users comments should be public to spur discussion; this is far superior to emails to the "helpdesk". If you provide this, people will help!

7. License. I noticed the license is GPL. I personally feel this is too restrictive. Why is this the case? As an example, my Nightlight program (http://nightlightapp.io) will not be free/open source. No one funded it; it was made in my spare time. Do you not want me to use HiPS (specifically, the libraries/command line tools around it)? Why not? If you truly want it to be a community standard, these tools must be more open than GPL. At the very least, I would urge you to consider LGPL. It's not like I (or anyone else) is going to make a pile of money from implemeting HiPS. You should want it to be widely adopted. What if I embed it into my pipeline code? I may not (will not!!) want my pipeline code to be GPL. Of course, any changes I make/propose to the library (and similar libraries), I will want pushed back to the source.