Skip to content

Instantly share code, notes, and snippets.

@den-crane
Last active June 6, 2023 18:38
Show Gist options
  • Save den-crane/62d3c56138250ed875c748b6154890ba to your computer and use it in GitHub Desktop.
Save den-crane/62d3c56138250ed875c748b6154890ba to your computer and use it in GitHub Desktop.
Using MinMax skip index to improve partition reading
create table test(tenant_id UInt32,  ts1 DateTime, timestamp DateTime)
engine=MergeTree
partition by toDate(ts1)
PRIMARY KEY (tenant_id, toStartOfHour(timestamp))
ORDER BY (tenant_id, toStartOfHour(timestamp),  timestamp)
as select 1, now() - number/10 x, x from numbers(1e8);

insert into test select number%1000, now() - number/10 x, 0 from numbers(1e8);

optimize table test final;

query reads 989.14 thousand rows / all partitions: 117 /117 parts by primary key

select count() from test 
where tenant_id =1 
AND `timestamp` BETWEEN toDateTime (1686020502) AND toDateTime (1686020562);
┌─count()─┐
│     610 │
└─────────┘
Selected 117/117 parts by partition key, 117 parts by primary key, 121/24422 marks by primary key, 121 marks to read from 117 ranges

1 row in set. Elapsed: 0.026 sec. Processed 989.14 thousand rows, 7.91 MB (38.19 million rows/s., 305.55 MB/s.)

if we use ts1 -- partition_by column, it reads 49.15 thousand rows / 1 parts by primary key

select count() from test 
where tenant_id =1 
AND `timestamp` BETWEEN toDateTime (1686020502) AND toDateTime (1686020562) 
and ts1>=toStartOfDay(toDateTime(1686020502));
┌─count()─┐
│     610 │
└─────────┘
Selected 1/117 parts by partition key, 1 parts by primary key, 5/156 marks by primary key, 5 marks to read from 1 ranges

1 row in set. Elapsed: 0.017 sec. Processed 49.15 thousand rows, 589.82 KB (2.94 million rows/s., 35.31 MB/s.)

let's add skip index on timestamp column

alter table test add index x1 timestamp type minmax GRANULARITY 1000;
alter table test materialize index x1 settings mutations_sync=2;

The original query even without ts1 reads only 40.96 thousand rows / 1 parts by primary key

select count() from test 
where tenant_id =1 
AND `timestamp` BETWEEN toDateTime (1686020502) AND toDateTime (1686020562);
┌─count()─┐
│     610 │
└─────────┘
Index `x1` has dropped 116/121 granules.
Selected 1/117 parts by partition key, 1 parts by primary key, 5/156 marks by primary key, 5 marks to read from 1 ranges

1 row in set. Elapsed: 0.032 sec. Processed 40.96 thousand rows, 327.68 KB (1.27 million rows/s., 10.16 MB/s.)

Huge GRANULARITY 1000 -- allows to reduce impact on insertion, the index is very light, and very small, but it works anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment