Measurements take place within the meas_base framework.
meas_base distinguishes between a measurement “algorithm” (e.g. a bit of code which takes some pixels and returns some useful result based upon them) and a measurement “driver”, which takes an image and runs a bunch of algorithms on it.
Typical measurement algorithms include (say) PSF fluxes, the SDSS shape algorithm, etc.
The measurement driver depends on the type of measurement being performed. meas_base provides two: single frame measurement, which is measurement on a single visit, and forced measurement, which takes an image and an external catalog which it uses to fix source centroids and shapes.
Measurement drivers are implemented as tasks, while algorithms are meas_base plugins. They can be combined fairly arbitrarily. Thus, the PSF flux algorithm can be used with single frame measurement or forced measurement, etc.
meas_base provides those two drivers and a bunch of simple algorithms. Other packages can provide more of either, following the same conventions. Most of our meas_extensions packages provide measurement algorithms for use in the meas_base framework (e.g. simpleShape, shapeHSM, etc).
For AP's purposes, the most notable extension package is ip_diffim. It provides a driver suitable for meas_base-style measurement on difference images1, and a bunch of algorithms (e.g. dipole measurement) which are relevant to us.
For the purposes of the current discussion, measurement drivers are fairly academic: we have the ones we need for now, and they mostly “just work”. The interesting stuff all happens in algorithms.
First, understand that algorithms may be referred to by strings which describe both their home package and their name. Thus, base_PsfFlux refers to the PsfFlux algorithm in meas_base.
When configuring a measurement driver (i.e. a task like SingleFrameMeasurementTask) we specify a list of plugins to be run when executing the driver. See, for example, here. These can be over-ridden by the user in task configuration.
When the driver runs, it will return an afw::table. Each row in the table corresponds to a particular source. Each algorithm plugin can add columns to the tables to contain its results. They follow the same naming scheme. Thus, if the base_PsfFlux algorithm is run, you'll see a bunch of fields in the table with names starting base_PsfFlux_ — base_PsfFlux_instFlux, base_PsfFlux_instFluxErr, etc.
(Incidentally, note that this means that the schema of the output table depends on the particular plugins that were enabled at task configuration time. That's why it's not easy to provide a single document that describes pipeline outputs — they vary with user configuration.)
Following from the above, when a new measurement driver (task) is created, it will:
- Create a new schema for its output;
- Hand that to each of the measurement algorithms (plugins) it's planning to run, and ask them to add fields which will be used to store their outputs.
In the limit of poor documentation, we can look at the code for the algorithm to understand what fields it's adding and what they mean.
Note that algorithms can be implemented in C++ or Python, so we need to be able to look at both.
Here's an example in Python: the base_FPPosition plugin. When this plugin is created, it is given a schema argument, which will ultimately be the schema of the measurement driver outputs. It adds three things to the schema: fields to contain its outputs, and two flag fields:
schema.addField(name + "_flag", type="Flag", doc="Set to True for any fatal failure")
schema.addField(name + "_missingDetector_flag", type="Flag", doc="Set to True if detector object is missing")When you look at the output of any measurement driver that has executed base_FPPosition, you should see those fields in the schema. And you can see that the flags are defined and (briefly) documented: if the algorithm fails for any reason, base_FPPosition_flag will be set to True, and if it fails specifically because the detector is missing, base_FPPosition_missingDetector_flag will be set to True.
A bunch of other plugins are defined in Python and follow the same pattern. See, e.g., base_Jacobian (look here), base_PeakCentroid (this one's here), etc.
Of course, plugins which are complex or have to run fast are written in C++. This follows the same principle, except that the code may be a little less familiar looking. Check out the flags defined for aperture or PSF fluxes, for example.
Often we don't care about failures with a particular measurement algorithm, which will be flagged as above, but about whether a source happened to fall on some area of the image which we know is suspect (interpolated, at the edge of the CCD, saturated, whatever).
This information is captured at the pixel level by the image mask planes. A special measurement plugin, base_PixelFlags, can be run to transfer information from the image mask to the source flags. For example, when the base_PixelFlags algorithm is run, it will set a base_PixelFlags_flag_interpolated field to True if any of the pixels in the footprint of the source being measured were marked as INTRP on the image.
Of course, if you're looking at the association database, you're not actually looking at the output of the measurement driver, but at whatever fields have been copied or otherwise manipulated from that driver output by the association code. If you can't see the flags you expect, likely they aren't being propagated properly.
I assume... I didn't actually check this!↩