Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save GenevieveBuckley/12ae179940e78c10d616dc7fbc93ed8c to your computer and use it in GitHub Desktop.
Save GenevieveBuckley/12ae179940e78c10d616dc7fbc93ed8c to your computer and use it in GitHub Desktop.
map-overlap-length-without-materializing-the-task-graph
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@GenevieveBuckley
Copy link
Author

Rules in 1D

  • 1 task per chunk from da.ones (this shouldn't be counted in the layer number of tasks, I don't think?
  • If the depth is larger than the smallest chunksize, that screws up these calculations, but we won't worry about that for now
  • first getitem call has 1 task per input chunk
  • concatenate call - there are three concatenate calls, but if chunks are neigbouring they can share the edge ones
  • overlap - there are three overlap calls, but if chunks are neigbouring they can share the edge ones
  • There is one getitem call per chunk after the overlap part
  • actual function passed to overlap - generates 1 task per input array chunk
  • 2 tasks per chunk if trim=True (these are the block-info_trim and _trim tasks)
  • next getitem calls - the

re are four getitem calls, but if chunks are neigbouring they can share the edge ones

Rules in 2D

  • 1 task per chunk from da.ones (this shouldn't be counted in the layer number of tasks, I don't think?
  • If the depth is larger than the smallest chunksize, that screws up these calculations, but we won't worry about that for now
  • first getitem call has 1 task per input chunk
  • ???concatenate call - there are three concatenate calls, but if chunks are neigbouring they can share the edge ones
  • ???more getitems ??
  • ???more concatenate ??
  • ???overlap - there are three overlap calls, but if chunks are neigbouring they can share the edge ones
  • There is one getitem call per chunk after the overlap part
  • actual function passed to overlap - generates 1 task per input array chunk
  • 2 tasks per chunk if trim=True (these

are the block-info_trim and _trim tasks)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment