Skip to content

Instantly share code, notes, and snippets.

View roggenkemper's full-sized avatar

Richard Roggenkemper roggenkemper

View GitHub Profile
@roggenkemper
roggenkemper / gist:a782981eed3739d9ee1f4b36160365a4
Last active December 12, 2024 21:24
batched-parallel updates
BatchStep(batch_messages_by_fingerprint, next_step=UnbatchStep(return_fingerprint_batch_of_messages, next_step=run_task_with_multiprocessing(
function=process_batch,
next_step=CommitOffsets(commit),
max_batch_size=self.max_batch_size,
max_batch_time=self.max_batch_time,
pool=self.pool,
input_block_size=self.input_block_size,
output_block_size=self.output_block_size,
)