Skip to content

Instantly share code, notes, and snippets.

@SimonKrughoff

SimonKrughoff/obs_package_discussion.md Secret

Last active February 27, 2019 21:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save SimonKrughoff/71a9314a2fc0ce6649d007333439fbbc to your computer and use it in GitHub Desktop.
Save SimonKrughoff/71a9314a2fc0ce6649d007333439fbbc to your computer and use it in GitHub Desktop.
Download ZIP
obs package discussion
Raw
obs_package_discussion.md

Moving forward with obs packages in a Gen3 world

High level summary

Tim J. and Simon K. have been selected to start the process of redefining what we call obs_ packages. We have decided to use obs_decam as the starter package. The tasks are as follows:

  1. Produce tests for yaml camera (yamera?)
  2. Define schema for yamera definitions
  3. Translate current camera definition into yamera
  4. Produce versioned calibration repository of obs_decam
  5. Handle user generated (and third party?) calibration products. Note that DECam has master calibs that are produced by the community pipeline. They won't look like the CPP produced master calibs. How do we deal with that?
  6. Solve the issue that pipelines configuration does not belong in the obs packages. This is something that we can't really start right now. The PipelineTasks first have to know how to run different configs in different contexts (on different cameras).
  7. Produce documentation that describes the design of obs packages at a level that people don't need to cargo cult the exemplar, but can understand well enough to build an obs package from scratch.
  8. Convert other obs packages.

Tasks 1-5 can be started today. Tasks 6-8 either depend on previous tasks or need more design of the Gen3 system to begin.

Notes from the discussion within the session.

Pick a package to start with

obs_subaru is further along in Gen3
obs_lsst has test stands which are very different animals
Maybe we should split up test stand obs packages from on sky (and simulated on sky) data
The risk of choosing obs_subaru is that it will not have the complexities of obs_lsst so you may pain yourself into a corner.
CS: The obs packages encode the way the packages deal with data that come off the instruments. Specifically, how defects are handled e.g. Need some social engineering to make sure people understand that it's ok that their workflow is being interrupted for the common good.
JB: obs_decam has third party calibrations. This could be a problem since they don't look like our datasets.
Decision: Patient 0 will be obs_decam
Frossie: do we keep things backward compatible? We could fork and go all Gen3.
Simon and Tim: It would be nice to keep obs_decam in CI.

Tasks to be done

  1. Not tests for yaml camera. Add tests and schema. Robert has features he'd like added (can we get a list of features for yamera or camera geom out of Robert/Jim). Can be started now.
  2. Subaru has versioned calibs now. Can be started now. Gen 3 requires cameras in the calib repo
  3. User generated calibs can now be added to the repository
  4. Configuration override and code override: single --> double dispatch needs to wait for conversation.
  5. Produce documentation

K-T: Why is the config application hard? JB: You want different configuration for a task depending on what pipeline is running it.

Colin: The most useful end product is text on how to use/build an obs package.
K-T: We want more than a prototype that people cargo cult. We want an understanding of how things work so people can do it right.

When is this done?

FE: Should we do this now?
SK: Needs to be done by November so that we can remove Gen2 right after release v18.
JB: Tim's need on Gen3 is not huge right now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment