Skip to content

Instantly share code, notes, and snippets.

View RaczeQ's full-sized avatar
🚀
Accelerating data science career

Kamil Raczycki RaczeQ

🚀
Accelerating data science career
View GitHub Profile
@RaczeQ
RaczeQ / pyarrow_multiprocessing_streaming.py
Last active May 19, 2024 19:41
Pyarrow Multiprocessing with streaming the result
import multiprocessing
from pathlib import Path
from queue import Queue
from time import sleep
from typing import Callable
import pyarrow as pa
import pyarrow.parquet as pq
from tqdm import tqdm