Some definitions, since precision in this terminology can matter a lot.
- ingest: move/copy/symlink files from outside of a butler repository and
butler.ingest
them to a butler repository addingDimension
data to theregistry
as necessary. - put: write in-memory objects to a butler repository using
butler.put
: this is distinct fromingest
because the butler handles allregistry
updates and the objects have to be in memory, not as on-disk files. - export/import: read data from one butler repository and put it into a different butler repository.
- raw: an un-modified image from a camera (including bias, dark, flat, science, etc.).
- calibrations: data that is used to convert raw science images to processed images (includes master bias/dark/flat images, defect lists, linearity corrections, crosstalk corrections, brighter-fatter kernels).
The only ingest that exists will be lsst.obs.base.RawIngestTask
. Once raw files are ingested into a butler, cp_pipe
is used to generate the calibrations that are used for processing.
If a user does not want to run cp_pipe
to produce calibrations themselves (due to disk space or time limitations), they will export the desired calibrations from an existing butler repository and import them into their own. This is the only supported system by which calibrations can get into a repository without running cp_pipe
.
To help us transition from gen2, we will provide a version of the gen2convert.py
script to get create working gen3 repositories for the existing testdata that have externally-supplied calibration data. This is not a long term solution, but only exists to maintain our existing testdata.
butler.export
is not fully supported currently. According to @JimBosch these exports currently cannot go into a repository in a different directory. There is no ticket for this at present.
This approach does not allow us to ingest externally-supplied calibrations. For existing test datasets (e.g. ap_verify
), the work-around would be to make the existing gen2convert
code work on any gen2 repo (not just HSC), and use that to bootstrap gen3 repos with existing external master calibration data.
This proposal does not support user-tweaked master calibrations. For example, someone notices a chip defect that isn't caught by cp_pipe
and wants to add it to the defect list. cp_pipe
would have to be updated to correctly identify that defect.
It's not clear to me what the current system for dealing with this is. What is the source of truth, the obs_*_data
git repo or some particular on-disk butler somewhere? This proposal sidesteps that issue by having one source of truth: cp_pipe
.
The obs_*_data
packages would become either obsolete and removed, or would have to contain butler.export
ed repository subsets instead of "user-curated text files". If we do want a place where users can download master calibrations supported by LSST, we could use these for that purpose.
Dealing with the Camera Geometry is a fundamental part of this question: linearity, crosstalk, gains, overscan size could each change over time. Potentially anything in the Amplifier object could change over time other than the rawBBox/rawDataBBox.
However, I don't think that decisions about what to do with Camera Geometry affect whether this proposal is workable: we don't currently have the ability to version camera parts of the camera geometry, and updates to those values should come from DM code (e.g. cp_pipe
, jointcal
, fgcm
), which is consistent with this overall proposal (said code will read from a butler, and write back to it).
For prompt products, I would expect that the PP butler repo would have the latest "blessed" versions of the calibrations produced by
cp_pipe
. They could have gotten there via the standardcp_pipe
approach+blessing, or by export/import from some "master" butler repo. This would be done whenever master calibrations are updated (which is probably weekly to monthly-ish).ap_verify
would resemble the operations environment under the above scheme: we never runcp_pipe
as part ofap_pipe
, we just have a set of "blessed" master calibrations in the repo available forap_pipe
to use (either from an earlier run ofcp_pipe
in that repo, or from an export/import from another repo). Having a skeleton gen3 repo with master calibrations in it to export/import from to bootstrap the process forap_verify
seems like the right approach to me: it gives you something that looks like a typicalap_pipe
starting point (a gen3 butler repo with master calibrations in it).