Skip to content

Instantly share code, notes, and snippets.

@Athanaseus
Last active February 1, 2024 09:24
Show Gist options
  • Save Athanaseus/da2c53f0c1ab8573be18ae04ec2de99f to your computer and use it in GitHub Desktop.
Save Athanaseus/da2c53f0c1ab8573be18ae04ec2de99f to your computer and use it in GitHub Desktop.
DDF MosaicKing

Mosaicking for the MeerKAT GPLS using Direction Dependent Faceting

Mosaicking 3 single scan pointings of 3 minutes each.

DDFacet command: DDF.py --Data-MS T16R02C02-corr-scan3-chan512-t16-XXYY.ms,T16R02C04-corr-scan4-chan512-t16-XXYY.ms,T16R02C06-corr-scan5-chan512-t16-XXYY.ms --Data-ColName DATA --Data-ChunkHours 3.0 --Data-Sort 1 --Predict-ColName POINTING_MODEL --Output-Mode Clean --Output-Name mgpls_im_01/T16R02-avg-mosaic-di1/im-di1 --Output-Cubes all --Output-Images all --Image-NPix 18000 --Image-Cell 1.0 --Facets-NFacets 9 --Weight-Robust -1.0 --Weight-SigmoidTaperInnerRolloffStrength 0 --Weight-SigmoidTaperOuterRolloffStrength 0 --RIME-DecorrMode FT --Parallel-NCPU 60 --Cache-Reset 1 --Beam-Model FITS --Beam-NBand 3 --Beam-CenterNorm 1 --Beam-Smooth 1 --Beam-FITSFile mktat_beams/new_mkt_pb_avg_$(xy)_$(reim).fits --Beam-FITSFeed xy --Beam-DtBeamMin 4.0 --Beam-FITSParAngleIncDeg 0.5 --Freq-NBand 3 --Freq-NDegridBand 12 --Deconv-Mode SSD2 --Deconv-MaxMajorIter 4 --Deconv-RMSFactor 3.0 --Deconv-PeakFactor 0.01 --Mask-Auto 1 --SSD2-PolyFreqOrder 3 --Log-Memory 1 --Log-Boring 0

Image Header

Name= im-dil.app.restored.fits
HDU = 0
Shape = [18225, 18225, 1, 1]
Number of channels = 1
Number of stokes = 1
Coordinate type = Right Ascension, Declination
Projection = SIN
Image reference pixels = [9113, 9113]
Image reference coords [14:17:01.3800, -061.10.37.9000]
Image ref coords (deg) = [214.256 deg, -61.1772 deg]
Pixel increment = -1′′, 1′′
Celestial frame = ICRS
Spectral frame = TOPO
Velocity definition = RADIO
Pixel unit = Jy/beam
Restoring beam = 21.8848′′ X 10.2914", 50.8834 deg
@Athanaseus
Copy link
Author

Dirty Images for each pointing

T16R02C02 T16R02C04 T16R02C06
im-di1 dirty corr fits-image-2023-12-13-08-19-13 im-di1 dirty corr fits-image-2023-12-13-08-19-20 im-di1 dirty corr fits-image-2023-12-13-08-19-25

@Athanaseus
Copy link
Author

Athanaseus commented Dec 13, 2023

Mosaics

Apparent image

im-di1 app restored fits-image-2023-12-13-08-42-26

NB: Blue region is the intended phase centre of the mosaic but the value was not parsed correctly.

Intrinsic image

im-di1 int restored fits-image-2023-12-13-08-19-37

@Athanaseus
Copy link
Author

Athanaseus commented Dec 13, 2023

Re-run with:
centre: 14:08:11.2840349216, -61:33:03.2826418686

im-di1 int restored fits-image-2023-12-13-12-47-38

regions

# Region file format: DS9 CARTA 2.0.0
global color=green dashlist=8 3 width=1 font="helvetica 10 normal roman" select=1 highlite=1 dash=0 fixed=0 edit=1 move=1 delete=1 include=1 source=1
icrs
ellipse(212.047017, -61.550912, 189.0064", 243.0908", 90) # color=#1C28F5 width=10 text={['14:07:48.9574', '-61:50:44.893']}
ellipse(214.250288, -61.147219, 4905.4624", 5088.2744", 90) # color=#2EE6D6 width=2
ellipse(212.299429, -61.455582, 4873.5356", 4812.1221", 90) # color=#2EE6D6 width=2
ellipse(210.253342, -61.775583, 4811.0903", 4625.1841", 90) # color=#2EE6D6 width=2

@Athanaseus
Copy link
Author

#  - 23:50:54 - ClassVisServer               [35.9/62.1 45.0/70.7 95.1Gb]   channel Mapping Gridding  : [0, 0, 0, 0, 0, 0, 0, 0,  ...., 3, 3, 3, 3, 3, 3, 3, 3], (len = 4096)
#  - 23:50:54 - ClassVisServer               [35.9/62.1 45.0/70.7 95.1Gb]   channel Mapping DeGridding: [0, 1, 2, 3,  ...., 4092, 4093, 4094, 4095], (len = 4096)
#  - 23:51:12 - ClassVisServer               [43.4/62.1 52.5/70.7 98.9Gb]   using cached BDA mapping
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/1543664600_sdp_l0-T16R02C02-corr.ms.F0.D0.ddfcache/R0:245830/BDA.Grid
#  - 23:51:12 - ClassVisServer               [43.5/62.1 52.6/70.7 99.0Gb]   using cached BDA mapping
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/1543664600_sdp_l0-T16R02C02-corr.ms.F0.D0.ddfcache/R0:245830/BDA.Degrid
#  - 23:51:19 - ClassJones                   [48.1/62.1 57.3/70.7 103.6Gb]   using cached Jones matrices from
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/1543664600_sdp_l0-T16R02C02-corr.ms.F0.D0.ddfcache/R0:245830/JonesNorm_Beam.npz
#  - 23:51:19 - ClassJones                   [48.1/62.1 57.3/70.7 103.6Gb]
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/1543664600_sdp_l0-T16R02C02-corr.ms.F0.D0.ddfcache/R0:245830/JonesNorm_Beam.npz.npz loaded
#  - 23:51:19 - ClassJones                   [48.1/62.1 57.3/70.7 103.6Gb]
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/1543664600_sdp_l0-T16R02C02-corr.ms.F0.D0.ddfcache/R0:245830/JonesNorm_Beam.npz.npy loaded
#  - 23:51:21 - AsyncProcessPool             [25.4/35.4 82.9/82.9 103.6Gb] Reading 1.1: 1 jobs complete, average single-core time 95.16s per job
#  - 23:51:21 - ClassVisServer               [25.4/35.4 82.9/82.9 103.6Gb] scheduling loading of chunk 1.2
# Traceback (most recent call last):
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 467, in <module>
#     main(OP, messages)
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 301, in main
#     Imager.main()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassDeconvMachine.py", line 1347, in main
#     ModelImage = self.FacetMachine.setModelImage(self.DeconvMachine.GiveModelImage(model_freqs))
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/SSD2/ClassImageDeconvMachineSSD.py", line 175, in GiveModelImage
#     return self.ModelMachine.GiveModelImage(*args)
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/SSD2/ClassModelMachineSSD.py", line 282, in GiveModelImage
#     ModelImage = np.zeros((nchan,npol,nx,ny),dtype=np.float32)
# numpy.core._exceptions.MemoryError: Unable to allocate 1.53 TiB for an array with shape (4096, 1, 10125, 10125) and data type float32
2024-01-17 23:51:31 STIMELA.mosaic.mosaic-di1 ERROR: step 'mosaic.mosaic-di1' has failed, aborting the recipe
──────────────────────────────────────────────────────────────────────────── detailed error report follows ────────────────────────────────────────────────────────────────────────────
        ⚠ step 'mosaic.mosaic-di1' has failed, aborting the recipe
        └── error running cab '/home/aramaila/.virtualenvs/DDF/bin/DDF.py'
            └── /home/aramaila/.virtualenvs/DDF/bin/DDF.py returns error code 1 after 7:10:50
────────────────────────────────────────────────────────────────────────────────── profiling results ──────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                  
                                         averages & total I/O                                                                           peaks                                     
    ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
    ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃   Load ┃ R GB/s ┃ W GB/s ┃   R GB ┃   W GB ┃ ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃    Load ┃ R GB/s ┃ W GB/s ┃
    ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
    │ mosaic            │ 7:10:50.4 │ 33.57 │  92.94 │ 255.42 │   0.01 │   0.01 │ 284.46 │ 304.47 │ │ mosaic            │ 7:10:50.4 │ 72.10 │ 303.00 │ 1070.14 │   1.31 │   4.78 │
    │ mosaic.mosaic-di1 │ 7:10:50.1 │ 33.58 │  92.94 │ 255.44 │   0.01 │   0.01 │ 284.46 │ 304.47 │ │ mosaic.mosaic-di1 │ 7:10:50.1 │ 72.10 │ 303.00 │ 1070.14 │   1.31 │   4.78 │
    └───────────────────┴───────────┴───────┴────────┴────────┴────────┴────────┴────────┴────────┘ └───────────────────┴───────────┴───────┴────────┴─────────┴────────┴────────┘

2024-01-17 23:51:31 STIMELA.mosaic INFO: saved full profiling stats to mgpls_im_03/logs/log-20240117-164026/stimela.stats.full
2024-01-17 23:51:31 STIMELA.mosaic INFO: saved summary to mgpls_im_03/logs/log-20240117-164026/stimela.stats.summary.txt
2024-01-17 23:51:31 STIMELA ERROR: run failed, exiting with error code 1

@Athanaseus
Copy link
Author

/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms: Reading next data chunk in [0, 20801] rows
#  - 19:11:36 - ClassMS                      [2.1/2.1 11.9/11.9 63.3Gb] reading cached visibilities from
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/Data.npy
#  - 19:11:39 - ClassMS                      [3.4/4.6 12.1/13.2 62.6Gb] reading cached flags from
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/Flags.npy
#  - 19:11:40 - ClassMS                      [3.5/4.6 12.1/13.2 62.7Gb] Rotate uvw [Mode = ToTarget]
#  - 19:11:40 - ClassMS                      [3.5/4.6 12.1/13.2 62.7Gb]      from [14:17:01.380, -61.10.37.900] [3.739468 -1.067743]
#  - 19:11:40 - ClassMS                      [3.5/4.6 12.1/13.2 62.7Gb]        to [14:08:11.284, -61.33.03.282] [3.700919 -1.074266]
#  - 19:11:40 - ClassVisServer               [3.5/4.6 12.7/13.2 62.7Gb]   channel Mapping Gridding  : [0, 0, 0, 0, 0, 0, 0, 0,  ...., 3, 3, 3, 3, 3, 3, 3, 3], (len = 4096)
#  - 19:11:40 - ClassVisServer               [3.5/4.6 12.7/13.2 62.7Gb]   channel Mapping DeGridding: [0, 1, 2, 3,  ...., 4092, 4093, 4094, 4095], (len = 4096)
#  - 19:11:41 - ClassVisServer               [4.2/4.8 13.3/14.0 63.0Gb]   using cached BDA mapping
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/BDA.Grid
#  - 19:11:41 - ClassVisServer               [4.2/4.8 13.3/14.0 63.0Gb]   using cached BDA mapping
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/BDA.Degrid
#  - 19:11:42 - ClassJones                   [4.6/5.1 13.8/14.3 63.5Gb]   using cached Jones matrices from
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/JonesNorm_Beam.npz
#  - 19:11:42 - ClassJones                   [4.6/5.1 13.8/14.3 63.5Gb]
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/JonesNorm_Beam.npz.npz loaded
#  - 19:11:42 - ClassJones                   [4.6/5.1 13.8/14.3 63.5Gb]
/data6/aramaila/hii-bubbles/mgpls-reductions/msdir/T16R02C02-corr-scan3-chan512-t16-XXYY.ms.F0.D0.ddfcache/R0:20801/JonesNorm_Beam.npz.npy loaded
#  - 19:11:42 - AsyncProcessPool             [25.3/35.4 82.8/82.8 63.5Gb] Reading 1.1: 1 jobs complete, average single-core time 5.88s per job
#  - 19:11:42 - ClassVisServer               [25.3/35.4 82.8/82.8 63.5Gb] scheduling loading of chunk 2.1
# Traceback (most recent call last):
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 467, in <module>
#     main(OP, messages)
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 301, in main
#     Imager.main()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassDeconvMachine.py", line 1347, in main
#     ModelImage = self.FacetMachine.setModelImage(self.DeconvMachine.GiveModelImage(model_freqs))
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/SSD2/ClassImageDeconvMachineSSD.py", line 175, in GiveModelImage
#     return self.ModelMachine.GiveModelImage(*args)
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/SSD2/ClassModelMachineSSD.py", line 282, in GiveModelImage
#     ModelImage = np.zeros((nchan,npol,nx,ny),dtype=np.float32)
# numpy.core._exceptions.MemoryError: Unable to allocate 1.53 TiB for an array with shape (4096, 1, 10125, 10125) and data type float32
2024-01-22 19:11:50 STIMELA.mosaic.mosaic-di1 ERROR: step 'mosaic.mosaic-di1' has failed, aborting the recipe
────────────────────────────────────────────────────────────────────── detailed error report follows ───────────────────────────────────────────────────────────────────────
        ⚠ step 'mosaic.mosaic-di1' has failed, aborting the recipe
        └── error running cab '/home/aramaila/.virtualenvs/DDF/bin/DDF.py'
            └── /home/aramaila/.virtualenvs/DDF/bin/DDF.py returns error code 1 after 2:36:58
──────────────────────────────────────────────────────────────────────────── profiling results ─────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                           
                                     averages & total I/O                                                                         peaks                                    
 ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
 ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃   Load ┃ R GB/s ┃ W GB/s ┃ R GB ┃  W GB ┃ ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃   Load ┃ R GB/s ┃ W GB/s ┃
 ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━┩ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
 │ mosaic            │ 2:36:58.5 │ 35.78 │  87.97 │ 229.56 │   0.00 │   0.01 │ 8.99 │ 47.40 │ │ mosaic            │ 2:36:58.5 │ 77.70 │ 139.00 │ 789.86 │   1.06 │   5.17 │
 │ mosaic.mosaic-di1 │ 2:36:58.2 │ 35.78 │  87.99 │ 229.60 │   0.00 │   0.01 │ 8.99 │ 47.40 │ │ mosaic.mosaic-di1 │ 2:36:58.2 │ 77.70 │ 139.00 │ 789.86 │   1.06 │   5.17 │
 └───────────────────┴───────────┴───────┴────────┴────────┴────────┴────────┴──────┴───────┘ └───────────────────┴───────────┴───────┴────────┴────────┴────────┴────────┘

2024-01-22 19:11:50 STIMELA.mosaic INFO: saved full profiling stats to mgpls_im_001/logs/log-20240122-163437/stimela.stats.full
2024-01-22 19:11:50 STIMELA.mosaic INFO: saved summary to mgpls_im_001/logs/log-20240122-163437/stimela.stats.summary.txt
2024-01-22 19:11:50 STIMELA ERROR: run failed, exiting with error code 1

@Athanaseus
Copy link
Author

# 
# Traceback (most recent call last):
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 457, in <module>
#     main(OP, messages)
#   File "/home/aramaila/.virtualenvs/DDF/bin/DDF.py", line 283, in main
#     Imager.Init()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassDeconvMachine.py", line 313, in Init
#     self.CreateFacetMachines()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassDeconvMachine.py", line 353, in CreateFacetMachines
#     self.FacetMachine.Init()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassFacetMachine.py", line 813, in Init
#     self.setWisdom()
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/Imager/ClassFacetMachine.py", line 874, in setWisdom
#     ModFFTW.learnFFTWWisdom(*TypeKey)
#   File "/home/aramaila/.virtualenvs/DDF/lib/python3.6/site-packages/DDFacet/ToolsDir/ModFFTW.py", line 792, in learnFFTWWisdom
#     print("  Computing fftw wisdom FFTs for shape [%i x %i] and dtype %s" % (npix_x,npix_y,dtype.__name__), file=log)
# TypeError: must be real number, not getset_descriptor
2024-02-01 10:59:01 STIMELA.mosaic.mosaic-di1 ERROR: step 'mosaic.mosaic-di1' has failed, aborting the recipe
──────────────────────────────────────────────────────────────────────────────────────── detailed error report follows ────────────────────────────────────────────────────────────────────────────────────────
        ⚠ step 'mosaic.mosaic-di1' has failed, aborting the recipe
        └── error running cab '/home/aramaila/.virtualenvs/DDF/bin/DDF.py'
            └── /home/aramaila/.virtualenvs/DDF/bin/DDF.py returns error code 1 after 0:00:14
────────────────────────────────────────────────────────────────────────────────────────────── profiling results ──────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                                                                          
                                                       averages & total I/O                                                                       peaks                                   
                     ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┓
                     ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃ Load ┃ R GB/s ┃ W GB/s ┃ R GB ┃ W GB ┃ ┃                   ┃  time hms ┃ CPU % ┃ Mem GB ┃ Load ┃ R GB/s ┃ W GB/s ┃
                     ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━┩ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━┩
                     │ mosaic            │ 0:00:15.2 │  1.92 │  33.00 │ 1.07 │   0.00 │   0.00 │ 0.03 │ 0.01 │ │ mosaic            │ 0:00:15.2 │  9.90 │  33.00 │ 1.15 │   0.01 │   0.01 │
                     │ mosaic.mosaic-di1 │ 0:00:14.9 │  1.86 │  33.00 │ 1.07 │   0.00 │   0.00 │ 0.03 │ 0.01 │ │ mosaic.mosaic-di1 │ 0:00:14.9 │  9.90 │  33.00 │ 1.15 │   0.01 │   0.01 │
                     └───────────────────┴───────────┴───────┴────────┴──────┴────────┴────────┴──────┴──────┘ └───────────────────┴───────────┴───────┴────────┴──────┴────────┴────────┘

2024-02-01 10:59:01 STIMELA.mosaic INFO: saved full profiling stats to mgpls_im_06/logs/log-20240201-105832/stimela.stats.full
2024-02-01 10:59:01 STIMELA.mosaic INFO: saved summary to mgpls_im_06/logs/log-20240201-105832/stimela.stats.summary.txt
2024-02-01 10:59:01 STIMELA ERROR: run failed, exiting with error code 1
2024-02-01 10:59:01 STIMELA.mosaic INFO: last log directory was mgpls_im_06/logs/log-20240201-105832

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment