Skip to content

Instantly share code, notes, and snippets.

@tom-clickhouse
Last active October 17, 2023 19:52
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save tom-clickhouse/4cff7a19f89faac96f53f2e3963403b3 to your computer and use it in GitHub Desktop.
min_insert_block_size_rows = 10_000_000
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- 32 insert threads
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- clickhouse-client output --------------------------------------------------------------------------------------------
0 rows in set. Elapsed: 4759.683 sec. Processed 65.33 billion rows, 9.60 TB (13.73 million rows/s., 2.02 GB/s.)
Peak memory usage: 231.84 GiB.
-- number of active parts at ingest finish -----------------------------------------------------------------------------
┌─parts─┐
│ 4042 │
└───────┘
-- time to less than 3k active parts ----------------------------------------------------------------------------------
┌─seconds─┐
│ 5527 │
└─────────┘
-- number of initial parts written to storage --------------------------------------------------------------------------
┌─parts─────────┬─rows_max──────┬─rows_avg─────┬─rows_min────────┬─size_uncompressed_avg─┬─size_compressed_avg─┬─rows_total────┐
│ 6.54 thousand │ 10.01 million │ 9.98 million │ 196.61 thousand │ 2.23 GiB │ 203.83 MiB │ 65.33 billion │
└───────────────┴───────────────┴──────────────┴─────────────────┴───────────────────────┴─────────────────────┴───────────────┘
-- number of initial parts written to storage per cluster node ---------------------------------------------------------
┌─n─┬─parts─────────┬─rows_total────┐
│ 1 │ 2.22 thousand │ 22.18 billion │
│ 2 │ 2.19 thousand │ 21.93 billion │
│ 3 │ 2.13 thousand │ 21.23 billion │
└───┴───────────────┴───────────────┘
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- 16 insert threads
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- clickhouse-client output --------------------------------------------------------------------------------------------
0 rows in set. Elapsed: 6321.984 sec. Processed 65.33 billion rows, 9.60 TB (10.33 million rows/s., 1.52 GB/s.)
Peak memory usage: 116.92 GiB.
-- number of active parts at ingest finish -----------------------------------------------------------------------------
┌─parts─┐
│ 1734 │
└───────┘
-- time to less than 3k active parts ----------------------------------------------------------------------------------
┌─seconds─┐
│ 6322 │
└─────────┘
-- number of initial parts written to storage --------------------------------------------------------------------------
┌─parts─────────┬─rows_max──────┬─rows_avg──────┬─rows_min─────┬─size_uncompressed_avg─┬─size_compressed_avg─┬─rows_total────┐
│ 6.54 thousand │ 10.01 million │ 10.00 million │ 2.22 million │ 2.24 GiB │ 203.97 MiB │ 65.33 billion │
└───────────────┴───────────────┴───────────────┴──────────────┴───────────────────────┴─────────────────────┴───────────────┘
-- number of initial parts written to storage per cluster node ---------------------------------------------------------
┌─n─┬─parts─────────┬─rows_total────┐
│ 1 │ 1.99 thousand │ 19.94 billion │
│ 2 │ 2.37 thousand │ 23.72 billion │
│ 3 │ 2.17 thousand │ 21.67 billion │
└───┴───────────────┴───────────────┘
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- 8 insert threads
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- clickhouse-client output --------------------------------------------------------------------------------------------
0 rows in set. Elapsed: 9406.904 sec. Processed 65.33 billion rows, 9.60 TB (6.95 million rows/s., 1.02 GB/s.)
Peak memory usage: 31.34 GiB.
-- number of active parts at ingest finish -----------------------------------------------------------------------------
┌─parts─┐
│ 893 │
└───────┘
-- time to less than 3k active parts ----------------------------------------------------------------------------------
┌─seconds─┐
│ 9813 │
└─────────┘
-- number of initial parts written to storage --------------------------------------------------------------------------
┌─parts─────────┬─rows_max──────┬─rows_avg─────┬─rows_min────────┬─size_uncompressed_avg─┬─size_compressed_avg─┬─rows_total────┐
│ 6.54 thousand │ 10.01 million │ 9.99 million │ 530.55 thousand │ 2.25 GiB │ 203.09 MiB │ 65.33 billion │
└───────────────┴───────────────┴──────────────┴─────────────────┴───────────────────────┴─────────────────────┴───────────────┘
-- number of initial parts written to storage per cluster node ---------------------------------------------------------
┌─n─┬─parts─────────┬─rows_total────┐
│ 1 │ 2.08 thousand │ 20.81 billion │
│ 2 │ 2.00 thousand │ 19.93 billion │
│ 3 │ 2.46 thousand │ 24.59 billion │
└───┴───────────────┴───────────────┘
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- 4 insert threads
------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
-- clickhouse-client output --------------------------------------------------------------------------------------------
0 rows in set. Elapsed: 16714.530 sec. Processed 65.33 billion rows, 9.60 TB (3.91 million rows/s., 574.54 MB/s.)
Peak memory usage: 31.30 GiB.
-- number of active parts at ingest finish -----------------------------------------------------------------------------
┌─parts─┐
│ 437 │
└───────┘
-- time to less than 3k active parts ----------------------------------------------------------------------------------
┌─seconds─┐
│ 16715 │
└─────────┘
-- number of initial parts written to storage --------------------------------------------------------------------------
┌─parts─────────┬─rows_max──────┬─rows_avg──────┬─rows_min────────┬─size_uncompressed_avg─┬─size_compressed_avg─┬─rows_total────┐
│ 6.53 thousand │ 10.01 million │ 10.00 million │ 610.14 thousand │ 2.25 GiB │ 201.93 MiB │ 65.33 billion │
└───────────────┴───────────────┴───────────────┴─────────────────┴───────────────────────┴─────────────────────┴───────────────┘
-- number of initial parts written to storage per cluster node ---------------------------------------------------------
┌─n─┬─parts─────────┬─rows_total────┐
│ 1 │ 2.03 thousand │ 20.31 billion │
│ 2 │ 2.17 thousand │ 21.66 billion │
│ 3 │ 2.34 thousand │ 23.37 billion │
└───┴───────────────┴───────────────┘
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment