maxjustus/pipe.sh

Created February 21, 2024 20:02

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/maxjustus/856ec7de457bb7d7c864b5e2d07b42d3.js"></script>
Save maxjustus/856ec7de457bb7d7c864b5e2d07b42d3 to your computer and use it in GitHub Desktop.

Download ZIP

Pipe a continuous stream into a clickhouse table and insert in chunks

Raw

pipe.sh

./s5cmd ls s3://mybucket/* | parallel --pipe -N 1000 'clickhouse client --query "insert into some_table FORMAT TSVRaw settings async_insert=1"'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment