Skip to content

Instantly share code, notes, and snippets.

@bhechinger

bhechinger/gist:254ac42394fb9235563b7f66f1c5258b

Created October 10, 2019 20:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bhechinger/254ac42394fb9235563b7f66f1c5258b to your computer and use it in GitHub Desktop.
Save bhechinger/254ac42394fb9235563b7f66f1c5258b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
++ hostname
+ exec /cockroach/cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local --http-host 0.0.0.0 --http-port 8080 --port 26257 --cache 25% --max-sql-memory 25% --join cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257,cockroachdb-shared-cockroachdb-1.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257,cockroachdb-shared-cockroachdb-2.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257
I191010 20:26:27.540069 1 cli/start.go:1033 logging to directory /cockroach/cockroach-data/logs
I191010 20:26:27.555957 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
I191010 20:26:27.556034 1 cli/start.go:1082 CockroachDB CCL v19.1.3 (x86_64-unknown-linux-gnu, built 2019/07/08 18:24:39, go1.11.6)
I191010 20:26:27.725757 1 server/status/recorder.go:610 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
I191010 20:26:27.726054 1 server/config.go:386 system total memory: 1.9 GiB
I191010 20:26:27.726231 1 server/config.go:388 server configuration:
max offset 500000000
cache size 499 MiB
SQL memory pool size 499 MiB
scan interval 10m0s
scan min idle time 10ms
scan max idle time 1s
event log enabled true
I191010 20:26:27.726362 1 cli/start.go:922 using local environment variables: COCKROACH_CHANNEL=kubernetes-helm
I191010 20:26:27.726870 1 cli/start.go:929 process identity: uid 0 euid 0 gid 0 egid 0
I191010 20:26:27.726990 1 cli/start.go:554 starting cockroach node
I191010 20:26:27.763451 21 storage/engine/rocksdb.go:613 opening rocksdb instance at "/cockroach/cockroach-data/cockroach-temp185518525"
I191010 20:26:28.093634 21 server/server.go:879 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I191010 20:26:28.093990 21 base/addr_validation.go:279 [n?] server certificate addresses: IP=127.0.0.1; DNS=localhost,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb,cockroachdb-shared-cockroachdb-public,cockroachdb-shared-cockroachdb-public.data-lake.svc.cluster.local; CN=node
W191010 20:26:28.095130 21 base/addr_validation.go:293 [n?] listen address "::" not in node certificate (IP=127.0.0.1; DNS=localhost,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb,cockroachdb-shared-cockroachdb-public,cockroachdb-shared-cockroachdb-public.data-lake.svc.cluster.local; CN=node)
Secure node-node and SQL connections are likely to fail.
Consider extending the node certificate or tweak --listen-addr/--advertise-addr.
I191010 20:26:28.095166 21 base/addr_validation.go:319 [n?] web UI certificate addresses: IP=127.0.0.1; DNS=localhost,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local,cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb,cockroachdb-shared-cockroachdb-public,cockroachdb-shared-cockroachdb-public.data-lake.svc.cluster.local; CN=node
I191010 20:26:28.097859 21 storage/engine/rocksdb.go:613 opening rocksdb instance at "/cockroach/cockroach-data"
I191010 20:26:28.925085 21 server/config.go:494 [n?] 1 storage engine initialized
I191010 20:26:28.925825 21 server/config.go:497 [n?] RocksDB cache size: 499 MiB
I191010 20:26:28.926081 21 server/config.go:497 [n?] store 0: RocksDB, max size 0 B, max open file limit 1043576
W191010 20:26:28.973868 21 gossip/gossip.go:1496 [n?] no incoming or outgoing connections
I191010 20:26:29.170045 21 gossip/gossip.go:392 [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257" > attrs:<> locality:<> ServerVersion:<major_val:19 minor_val:1 patch:0 unstable:0 > build_tag:"v19.1.3" started_at:1570739189166115023
I191010 20:26:29.571560 75 gossip/client.go:128 [n1] started gossip client to cockroachdb-shared-cockroachdb-1.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257
I191010 20:26:30.605953 72 gossip/gossip.go:1510 [n1] node has connected to cluster via gossip
I191010 20:26:32.139056 21 server/node.go:461 [n1] initialized store [n1,s1]: disk (capacity=976 MiB, available=471 MiB, used=359 MiB, logicalBytes=1.6 GiB), ranges=114, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=146.00 p50=14737120.00 p75=32079077.00 p90=36734210.00 pMax=49931703.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I191010 20:26:32.246132 21 storage/stores.go:244 [n1] read 2 node addresses from persistent storage
I191010 20:26:32.264462 21 server/node.go:699 [n1] connecting to gossip network to verify cluster ID...
I191010 20:26:32.267470 21 server/node.go:719 [n1] node connected via gossip and verified as part of cluster "b626180e-edcf-40a9-b471-57008efbe9b5"
I191010 20:26:32.268948 21 server/node.go:542 [n1] node=1: started with [<no-attributes>=/cockroach/cockroach-data] engine(s) and attributes []
I191010 20:26:32.277580 21 server/status/recorder.go:610 [n1] available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
I191010 20:26:32.560236 21 server/server.go:1592 [n1] starting https server at [::]:8080 (use: 0.0.0.0:8080)
I191010 20:26:32.560732 21 server/server.go:1594 [n1] starting grpc/postgres server at [::]:26257
I191010 20:26:32.560768 21 server/server.go:1595 [n1] advertising CockroachDB node at cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257
W191010 20:26:32.598398 110 storage/store.go:3704 [n1,s1,r2/1:/System/NodeLiveness{-Max}] handle raft ready: 0.8s [processed=20]
W191010 20:26:32.803043 111 storage/store.go:3704 [n1,s1,r1/1:/{Min-System/NodeL…}] handle raft ready: 1.0s [processed=3]
I191010 20:26:32.938016 72 storage/stores.go:263 [n1] wrote 2 node addresses to persistent storage
I191010 20:26:34.828425 21 server/server.go:1660 [n1] done ensuring all necessary migrations have run
I191010 20:26:34.886933 21 server/server.go:1663 [n1] serving sql connections
I191010 20:26:34.900007 21 cli/start.go:689 [config] clusterID: b626180e-edcf-40a9-b471-57008efbe9b5
I191010 20:26:34.900089 21 cli/start.go:697 node startup completed:
CockroachDB node starting at 2019-10-10 20:26:34.897025477 +0000 UTC (took 7.4s)
build: CCL v19.1.3 @ 2019/07/08 18:24:39 (go1.11.6)
webui: https://0.0.0.0:8080
sql: postgresql://root@cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257?sslmode=verify-full&sslrootcert=%2Fcockroach%2Fcockroach-certs%2Fca.crt
client flags: /cockroach/cockroach <client cmd> --host=cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257 --certs-dir=/cockroach/cockroach-certs
logs: /cockroach/cockroach-data/logs
temp dir: /cockroach/cockroach-data/cockroach-temp185518525
external I/O path: /cockroach/cockroach-data/extern
store[0]: path=/cockroach/cockroach-data
status: restarted pre-existing node
clusterID: b626180e-edcf-40a9-b471-57008efbe9b5
nodeID: 1
W191010 20:26:34.866600 178 storage/node_liveness.go:523 [n1,hb] slow heartbeat took 2.2s
I191010 20:26:36.554040 207 server/server_update.go:67 [n1] no need to upgrade, cluster already at the newest version
I191010 20:26:36.835410 210 sql/event_log.go:135 [n1] Event: "node_restart", target: 1, info: {Descriptor:{NodeID:1 Address:cockroachdb-shared-cockroachdb-0.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257 Attrs: Locality: ServerVersion:19.1 BuildTag:v19.1.3 StartedAt:1570739189166115023 LocalityAddress:[] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} ClusterID:b626180e-edcf-40a9-b471-57008efbe9b5 StartedAt:1570739189166115023 LastUp:1570739130465959071}
I191010 20:26:37.551611 212 server/updates.go:266 [n1] A new version is available: 19.1.5, details: https://www.cockroachlabs.com/docs/releases/v19.1.5.html
E191010 20:26:38.988353 319 server.go:2977 http: TLS handshake error from 10.244.7.121:54492: remote error: tls: bad certificate
I191010 20:26:42.920853 161 storage/store.go:4216 [n1,s1] sstables (read amplification = 2):
5 [ 4M 3 ]: 2M[2] 729K
6 [ 249M 64 ]: 4M[60] 2M[3] 1M
I191010 20:26:42.934804 161 storage/store.go:4217 [n1,s1]
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0 1 0.215 0 0
L5 3/0 4.22 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 3.9 3.0 2.5 2 1 1.669 94K 5146
L6 64/0 249.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Sum 67/0 253.65 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.2 2.7 2.6 2 2 0.942 94K 5146
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.2 2.7 2.6 2 2 0.942 94K 5146
Uptime(secs): 14.8 total, 14.8 interval
Flush(GB): cumulative 0.001, interval 0.001
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.33 MB/s write, 0.00 GB read, 0.34 MB/s read, 1.9 seconds
Interval compaction: 0.00 GB write, 0.33 MB/s write, 0.00 GB read, 0.34 MB/s read, 1.9 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
estimated_pending_compaction_bytes: 0 B
I191010 20:26:42.937054 166 server/status/runtime.go:500 [n1] runtime stats: 155 MiB RSS, 156 goroutines, 110 MiB/15 MiB/135 MiB GO alloc/idle/total, 27 MiB/30 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (10x), 395 KiB/145 KiB (r/w)net
W191010 20:26:45.121756 168 server/node.go:880 [n1,summaries] health alerts detected: {Alerts:[{StoreID:0 Category:METRICS Description:round-trip-latency-p90 Value:1.040187391e+09 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}
I191010 20:26:53.644430 166 server/status/runtime.go:500 [n1] runtime stats: 166 MiB RSS, 165 goroutines, 122 MiB/2.1 MiB/135 MiB GO alloc/idle/total, 27 MiB/30 MiB CGO alloc/total, 126.0 CGO/sec, 3.2/2.7 %(u/s)time, 0.0 %gc (1x), 156 KiB/82 KiB (r/w)net
W191010 20:26:53.664809 178 storage/node_liveness.go:523 [n1,hb] slow heartbeat took 2.1s
E191010 20:26:53.950164 420 server.go:2977 http: TLS handshake error from 10.244.7.121:54526: remote error: tls: bad certificate
W191010 20:26:59.778245 178 storage/node_liveness.go:523 [n1,hb] slow heartbeat took 4.0s
W191010 20:27:00.066200 168 server/node.go:880 [n1,summaries] health alerts detected: {Alerts:[{StoreID:0 Category:METRICS Description:round-trip-latency-p90 Value:1.677721599e+09 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}
W191010 20:27:00.225991 117 storage/store.go:3704 [n1,s1,r502/1:/System/tsd/cr.node.sql.m…] handle raft ready: 0.6s [processed=0]
W191010 20:27:00.958761 178 storage/node_liveness.go:523 [n1,hb] slow heartbeat took 1.1s
W191010 20:27:03.645882 113 storage/engine/rocksdb.go:2040 batch [3/5736/0] commit took 617.710985ms (>= warning threshold 500ms)
W191010 20:27:03.668649 114 storage/store.go:3704 [n1,s1,r502/1:/System/tsd/cr.node.sql.m…] handle raft ready: 1.5s [processed=0]
W191010 20:27:03.834309 113 storage/store.go:3704 [n1,s1,r401/1:/System{/tsd/cr.s…-tse}] handle raft ready: 2.4s [processed=0]
I191010 20:27:03.898977 166 server/status/runtime.go:500 [n1] runtime stats: 171 MiB RSS, 172 goroutines, 97 MiB/24 MiB/135 MiB GO alloc/idle/total, 27 MiB/30 MiB CGO alloc/total, 196.3 CGO/sec, 3.2/3.6 %(u/s)time, 0.0 %gc (0x), 179 KiB/104 KiB (r/w)net
W191010 20:27:05.636530 114 storage/store.go:3704 [n1,s1,r478/1:/System/tsd/cr.node.sql.…] handle raft ready: 0.6s [processed=0]
W191010 20:27:09.380634 168 server/node.go:880 [n1,summaries] health alerts detected: {Alerts:[{StoreID:0 Category:METRICS Description:round-trip-latency-p90 Value:1.677721599e+09 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}
I191010 20:27:09.736100 1 cli/start.go:765 received signal 'terminated'
E191010 20:27:09.685520 585 server.go:2977 http: TLS handshake error from 10.244.7.121:54550: remote error: tls: bad certificate
I191010 20:27:09.754269 1 cli/start.go:830 initiating graceful shutdown of server
initiating graceful shutdown of server
I191010 20:27:12.856811 610 storage/store.go:1107 [drain] waiting for 2 replicas to transfer their lease away
I191010 20:27:13.074757 610 util/stop/stopper.go:546 [server drain process] quiescing; tasks left:
2 [async] closedts-subscription
1 [async] stats.Refresher: maybeRefreshStats
1 [async] intent_resolver_ir_batcher
1 [async] intent_resolver_gc_batcher
1 [async] closedts-rangefeed-subscriber
W191010 20:27:14.327897 61 vendor/google.golang.org/grpc/clientconn.go:1440 grpc: addrConn.transportMonitor exits due to: grpc: the connection is closing
W191010 20:27:14.363019 156 storage/raft_transport.go:583 [n1] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
W191010 20:27:14.371036 72 gossip/gossip.go:1496 [n1] no incoming or outgoing connections
W191010 20:27:14.373034 196 storage/raft_transport.go:583 [n1] while processing outgoing Raft queue to node 2: EOF:
I191010 20:27:14.389889 610 util/stop/stopper.go:546 [server drain process] quiescing; tasks left:
2 [async] closedts-subscription
1 [async] intent_resolver_ir_batcher
1 [async] intent_resolver_gc_batcher
1 [async] closedts-rangefeed-subscriber
I191010 20:27:14.524359 985 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:322 [n1] circuitbreaker: rpc [::]:26257->2 tripped: failed to connect to n2 at cockroachdb-shared-cockroachdb-2.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257: context canceled
W191010 20:27:14.742245 122 storage/store.go:1525 [n1,s1,r1/1:/{Min-System/NodeL…}] could not gossip first range descriptor: node unavailable; try another peer
I191010 20:27:14.556986 166 server/status/runtime.go:500 [n1] runtime stats: 166 MiB RSS, 104 goroutines, 114 MiB/9.8 MiB/135 MiB GO alloc/idle/total, 27 MiB/30 MiB CGO alloc/total, 215.8 CGO/sec, 3.9/3.8 %(u/s)time, 0.0 %gc (0x), 217 KiB/165 KiB (r/w)net
I191010 20:27:14.562605 984 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:322 [n1] circuitbreaker: rpc [::]:26257->3 tripped: failed to connect to n3 at cockroachdb-shared-cockroachdb-1.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257: context canceled
I191010 20:27:14.750949 985 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n1] circuitbreaker: rpc [::]:26257->2 event: BreakerTripped
I191010 20:27:14.838940 984 vendor/github.com/cockroachdb/circuitbreaker/circuitbreaker.go:447 [n1] circuitbreaker: rpc [::]:26257->3 event: BreakerTripped
I191010 20:27:14.864378 985 rpc/nodedialer/nodedialer.go:143 [ct-client] unable to connect to n2: failed to connect to n2 at cockroachdb-shared-cockroachdb-2.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257: context canceled
I191010 20:27:14.873237 984 rpc/nodedialer/nodedialer.go:143 [ct-client] unable to connect to n3: failed to connect to n3 at cockroachdb-shared-cockroachdb-1.cockroachdb-shared-cockroachdb.data-lake.svc.cluster.local:26257: context canceled
I191010 20:27:14.898960 610 storage/engine/rocksdb.go:753 closing rocksdb instance at "/cockroach/cockroach-data/cockroach-temp185518525"
I191010 20:27:14.966789 610 storage/engine/rocksdb.go:753 closing rocksdb instance at "/cockroach/cockroach-data"
I191010 20:27:15.000460 1 cli/start.go:876 server drained and shutdown completed
server drained and shutdown completed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment