Skip to content

Instantly share code, notes, and snippets.

@simpl1g

simpl1g/kafka

Last active August 6, 2019 06:35
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save simpl1g/5c357d540e7c9cbcb95dbc0571e88765 to your computer and use it in GitHub Desktop.
Save simpl1g/5c357d540e7c9cbcb95dbc0571e88765 to your computer and use it in GitHub Desktop.
Download ZIP
clichouse-kafka
Raw
kafka
select count(1) from dummy_buf => 0
kafkacat -b kafka_brokers -t dummy -P
{"ad_type":1}
select count(1) from dummy_buf => 1
...
select count(1) from dummy_buf => 200
Raw
log
2019.08.02 12:54:36.043294 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:36.543444 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:36.545347 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:36.545378 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:36.545399 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:36.545419 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:36.545431 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:36.545450 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:36.545465 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:36.545480 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:36.545508 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:36.545520 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:36.545531 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:36.545547 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:36.545903 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1003 ms.
2019.08.02 12:54:37.046005 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:37.046240 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:37.046314 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:37.546545 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:38.046786 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:38.049976 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:38.050039 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:38.050063 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:38.050084 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:38.050110 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:38.050141 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:38.050169 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:38.050190 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:38.050210 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:38.050230 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:38.050253 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:38.050273 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:38.050608 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1004 ms.
2019.08.02 12:54:38.550715 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:38.550883 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:38.550989 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:39.051213 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:39.551394 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:39.552159 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:39.552179 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:39.552183 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:39.552187 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:39.552192 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:39.552197 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:39.552206 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:39.552210 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:39.552213 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:39.552216 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:39.552219 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:39.552226 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:39.552519 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1001 ms.
2019.08.02 12:54:40.052650 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:40.052826 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:40.052907 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:40.553137 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:41.053312 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:41.053947 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:41.053960 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:41.053964 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:41.053967 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:41.053970 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:41.053977 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:41.053980 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:41.053983 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:41.053986 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:41.053989 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:41.053993 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:41.053999 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:41.054262 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1001 ms.
2019.08.02 12:54:41.554372 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:41.554552 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:41.554618 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:42.054831 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:42.555034 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:42.555837 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:42.555897 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:42.555936 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:42.555970 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:42.556002 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:42.556056 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:42.556091 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:42.556123 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:42.556153 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:42.556184 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:42.556227 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:42.556295 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:42.556644 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1002 ms.
2019.08.02 12:54:43.056746 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:43.057086 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:43.057198 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:43.557418 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:44.057677 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:44.061359 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:44.061433 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:44.061487 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:44.061532 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:44.061601 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:44.061645 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:44.061687 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:44.061729 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:44.061806 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:44.061850 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:44.061904 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:44.061947 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:44.062323 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1005 ms.
2019.08.02 12:54:44.562406 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:44.562678 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:44.562747 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:45.062980 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:45.563253 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:45.566723 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:45.566809 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:45.566857 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:45.566934 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:45.566984 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:45.567039 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:45.567083 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:45.567164 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:45.567217 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:45.567261 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:45.567304 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:45.567345 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:45.567773 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1005 ms.
2019.08.02 12:54:46.067860 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:46.068105 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:46.068182 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:46.568471 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
2019.08.02 12:54:47.068749 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Stalled
2019.08.02 12:54:47.071355 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 0)
2019.08.02 12:54:47.071449 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 1)
2019.08.02 12:54:47.071508 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset 1 (topic: dummy, partition: 2)
2019.08.02 12:54:47.071600 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 3)
2019.08.02 12:54:47.071658 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 4)
2019.08.02 12:54:47.071713 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 5)
2019.08.02 12:54:47.071802 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 6)
2019.08.02 12:54:47.071861 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 7)
2019.08.02 12:54:47.071916 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 8)
2019.08.02 12:54:47.071970 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 9)
2019.08.02 12:54:47.072049 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 10)
2019.08.02 12:54:47.072107 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Committed offset -1001 (topic: dummy, partition: 11)
2019.08.02 12:54:47.072592 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Execution took 1004 ms.
2019.08.02 12:54:47.572717 [ 37 ] {} <Debug> StorageKafka (dummy_queue): Started streaming to 1 attached views
2019.08.02 12:54:47.573058 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Subscribed to topics: dummy
2019.08.02 12:54:47.573202 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Assigned to topics: dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy dummy
2019.08.02 12:54:48.073461 [ 37 ] {} <Trace> StorageKafka (dummy_queue): Polled batch of 1 messages
Raw
sql
CREATE TABLE appodeal.dummy_queue ON CLUSTER ch_shards (
ad_type UInt8
) ENGINE = Kafka('{kafka_brokers}', 'dummy', 'stage-dummy_ch_queue', 'JSONEachRow');
CREATE MATERIALIZED VIEW appodeal.dummy_mv ON CLUSTER ch_shards TO appodeal.dummy_buf (
ad_type UInt8
) AS SELECT
toUInt8(ad_type) AS ad_type
FROM appodeal.dummy_queue;
CREATE TABLE appodeal.dummy ON CLUSTER ch_shards (
ad_type UInt8
) ENGINE = MergeTree ORDER BY (ad_type) SETTINGS index_granularity = 8192;
CREATE TABLE appodeal.dummy_buf ON CLUSTER ch_shards AS appodeal.dummy ENGINE = Buffer('appodeal', 'dummy', 16, 60, 300, 100000, 1000000, 104857600, 524288000);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment