Skip to content

Instantly share code, notes, and snippets.

@nevencaplar

nevencaplar/gist:99d8cd4bce0cca243f3ed312614ddc63

Created November 10, 2021 16:06
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save nevencaplar/99d8cd4bce0cca243f3ed312614ddc63 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
(lsst-scipipe) [ncaplar@tiger2-sumire ~]$ reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun ncaplar/PIPE2D-918_ApCorFalse --id visit=67787 -j 12 -c doWriteCalexp=True isr.doFlat=False isr.doDefect=False repair.cosmicray.keepCRs=False photometerLines.doApertureCorrection=False --clobber-config
root INFO: Loading config overrride file '/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2021.45/config/reduceExposure.py'
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
root INFO: Running: /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/bin/reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun ncaplar/PIPE2D-918_ApCorFalse --id visit=67787 -j 12 -c doWriteCalexp=True isr.doFlat=False isr.doDefect=False repair.cosmicray.keepCRs=False photometerLines.doApertureCorrection=False --clobber-config
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/calibRegistry.sqlite3
reduceExposure INFO: Processing [{'visit': 67787, 'dateObs': '2021-09-18', 'site': 'S', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'HEADER_TEST', 'ccd': 1, 'filter': 'r', 'expTime': 15.0, 'dataType': 'COMPARISON', 'taiObs': '2021-09-18T10:57:35.093', 'pfsDesignId': 512368543363980592, 'slitOffset': 0.0}]
reduceExposure INFO: Processing [{'visit': 67787, 'dateObs': '2021-09-18', 'site': 'S', 'category': 'A', 'arm': 'b', 'spectrograph': 1, 'field': 'HEADER_TEST', 'ccd': 0, 'filter': 'b', 'expTime': 15.0, 'dataType': 'COMPARISON', 'taiObs': '2021-09-18T10:57:35.059', 'pfsDesignId': 512368543363980592, 'slitOffset': 0.0}]
reduceExposure.isr INFO: Performing ISR on sensor {'visit': 67787, 'dateObs': '2021-09-18', 'site': 'S', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'HEADER_TEST', 'ccd': 1, 'filter': 'r', 'expTime': 15.0, 'dataType': 'COMPARISON', 'taiObs': '2021-09-18T10:57:35.093', 'pfsDesignId': 512368543363980592, 'slitOffset': 0.0}
reduceExposure.isr INFO: Performing ISR on sensor {'visit': 67787, 'dateObs': '2021-09-18', 'site': 'S', 'category': 'A', 'arm': 'b', 'spectrograph': 1, 'field': 'HEADER_TEST', 'ccd': 0, 'filter': 'b', 'expTime': 15.0, 'dataType': 'COMPARISON', 'taiObs': '2021-09-18T10:57:35.059', 'pfsDesignId': 512368543363980592, 'slitOffset': 0.0}
reduceExposure.isr INFO: Converting exposure to floating point values
reduceExposure.isr INFO: Converting exposure to floating point values
reduceExposure.isr INFO: Assembling CCD from amplifiers
reduceExposure.isr INFO: Assembling CCD from amplifiers
reduceExposure.isr INFO: Applying bias correction.
reduceExposure.isr INFO: Applying bias correction.
reduceExposure.isr INFO: Masking NAN value pixels.
reduceExposure.isr INFO: Widening saturation trails.
reduceExposure.isr INFO: Applying dark correction.
reduceExposure.isr INFO: Masking NAN value pixels.
reduceExposure.isr INFO: Widening saturation trails.
reduceExposure.isr INFO: Interpolating masked pixels.
reduceExposure.isr INFO: Applying dark correction.
reduceExposure.isr INFO: Interpolating masked pixels.
reduceExposure.isr WARN: No rough magnitude zero point set for filter b
reduceExposure.isr INFO: Setting rough magnitude zero point: 30.940228
reduceExposure.isr WARN: No rough magnitude zero point set for filter r
reduceExposure.isr INFO: Setting rough magnitude zero point: 30.940228
reduceExposure.repair INFO: Identified 66 cosmic rays.
reduceExposure.repair INFO: Identified 126 cosmic rays.
reduceExposure.readLineList INFO: Filtered line lists for lamps: Kr
reduceExposure.readLineList INFO: Filtered line lists for lamps: Kr
reduceExposure.centroidLines INFO: Measured 147838 line centroids
reduceExposure.adjustDetectorMap INFO: Final fit: chi2=2054.311434 dof=1468 xRMS=0.062137 yRMS=0.093799 (0.006396 nm) from 746/1393 lines
reduceExposure.adjustDetectorMap INFO: Fit quality from reserved lines: chi2=42368.432554 xRMS=0.061393 yRMS=0.676552 (0.046130 nm) from 155 lines (10.0%)
reduceExposure.adjustDetectorMap INFO: Softening errors by x=0.000000, y=0.082622 pixels (0.005633 nm) to yield chi^2/dof=1
reduceExposure.adjustDetectorMap INFO: Softened fit: chi2=1439.689015 dof=1468 xRMS=0.062135 yRMS=0.108402 (0.007391 nm) from 746 lines
reduceExposure.adjustDetectorMap INFO: Softened fit quality from reserved lines: chi2=21801.981265 xRMS=0.061421 yRMS=0.501269 (0.034178 nm) from 155 lines
reduceExposure FATAL: Failed on dataIds=[{'visit': 67787, 'dateObs': '2021-09-18', 'site': 'S', 'category': 'A', 'arm': 'b', 'spectrograph': 1, 'field': 'HEADER_TEST', 'ccd': 0, 'filter': 'b', 'expTime': 15.0, 'dataType': 'COMPARISON', 'taiObs': '2021-09-18T10:57:35.059', 'pfsDesignId': 512368543363980592, 'slitOffset': 0.0}]: LengthError:
File "src/image/Image.cc", line 68, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_allocateView(const Extent2I&, ndarray::Manager::Ptr&) [with PixelT = double; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<double, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>; ndarray::Manager::Ptr = boost::intrusive_ptr<ndarray::Manager>]
Image dimensions (2147479557 x 4176) too large; int overflow detected. {0}
lsst::pex::exceptions::LengthError: 'Image dimensions (2147479557 x 4176) too large; int overflow detected.'
Traceback (most recent call last):
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
result = self.runTask(task, dataRef, kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/python/pfs/drp/stella/reduceExposure.py", line 251, in runDataRef
calibs = self.getSpectralCalibs(sensorRef, exposure, pfsConfig)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/python/pfs/drp/stella/reduceExposure.py", line 496, in getSpectralCalibs
fiberTraces = fiberProfiles.makeFiberTracesFromDetectorMap(detectorMap) # use new detectorMap
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/python/pfs/drp/stella/fiberProfileSet.py", line 135, in makeFiberTracesFromDetectorMap
return self.makeFiberTraces(detectorMap.bbox.getDimensions(), centers)
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/python/pfs/drp/stella/fiberProfileSet.py", line 155, in makeFiberTraces
traces.add(self[fiberId].makeFiberTrace(dimensions, centers[fiberId], fiberId))
File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2021.45/python/pfs/drp/stella/fiberProfile.py", line 244, in makeFiberTrace
self.profiles, ~self.profiles.mask, centerFunc(rows), self.norm)
lsst.pex.exceptions.wrappers.LengthError:
File "src/image/Image.cc", line 68, in static lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_allocateView(const Extent2I&, ndarray::Manager::Ptr&) [with PixelT = double; lsst::afw::image::ImageBase<PixelT>::_view_t = boost::gil::image_view<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<double, boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t> > >*> > >; lsst::geom::Extent2I = lsst::geom::Extent<int, 2>; ndarray::Manager::Ptr = boost::intrusive_ptr<ndarray::Manager>]
Image dimensions (2147479557 x 4176) too large; int overflow detected. {0}
lsst::pex::exceptions::LengthError: 'Image dimensions (2147479557 x 4176) too large; int overflow detected.'
reduceExposure.centroidLines INFO: Measured 125331 line centroids
reduceExposure.adjustDetectorMap INFO: Final fit: chi2=11422.377910 dof=3844 xRMS=0.032079 yRMS=0.031978 (0.002761 nm) from 1934/11377 lines
reduceExposure.adjustDetectorMap INFO: Fit quality from reserved lines: chi2=20042607.352247 xRMS=0.135327 yRMS=0.878016 (0.075808 nm) from 1264 lines (10.0%)
reduceExposure.adjustDetectorMap INFO: Softening errors by x=0.028916, y=0.031283 pixels (0.002701 nm) to yield chi^2/dof=1
reduceExposure.adjustDetectorMap INFO: Softened fit: chi2=3809.945100 dof=3844 xRMS=0.035572 yRMS=0.037518 (0.003239 nm) from 1934 lines
reduceExposure.adjustDetectorMap INFO: Softened fit quality from reserved lines: chi2=3788354.083193 xRMS=0.129776 yRMS=0.780470 (0.067386 nm) from 1264 lines
reduceExposure.photometerLines INFO: Photometered 125331 lines
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment