Skip to content

Instantly share code, notes, and snippets.

@tamird

tamird/TestStatusLocalLogs

Created August 23, 2015 17:15
Show Gist options
  • Save tamird/6df602f3931e4024e23d to your computer and use it in GitHub Desktop.
Save tamird/6df602f3931e4024e23d to your computer and use it in GitHub Desktop.
Download ZIP
TestStatusLocalLogs timeout
Raw
TestStatusLocalLogs
=== RUN TestStatusLocalStacks
I0823 17:08:58.367808 792 base/context.go:141 setting up TLS from certificates directory: test_certs
I0823 17:08:58.372483 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.375198 792 rpc/clock_offset.go:155 monitoring cluster offset
I0823 17:08:58.375371 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.375517 792 raft/raft.go:406 group 1 100000001 became follower at term 5
I0823 17:08:58.375641 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 5, commit: 10, applied: 10, lastindex: 10, lastterm: 5]
I0823 17:08:58.375716 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 5
I0823 17:08:58.375788 792 raft/raft.go:419 group 1 100000001 became candidate at term 6
I0823 17:08:58.375821 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 6
I0823 17:08:58.375872 792 raft/raft.go:447 group 1 100000001 became leader at term 6
I0823 17:08:58.376551 792 storage/replica_command.go:1017 range 1: new leader lease replica 1:1 00:00:00.000 +1440349739.375s
I0823 17:08:58.378822 792 storage/replica.go:1070 gossiping accounting config from store 1, range 1
I0823 17:08:58.378976 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.379116 792 storage/replica.go:1070 gossiping user config from store 1, range 1
I0823 17:08:58.379254 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.379550 792 server/node.go:310 initialized store store=1:1 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.379628 792 server/node.go:217 new node allocated ID 1
I0823 17:08:58.379694 792 gossip/gossip.go:181 gossiping node descriptor node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:56831" > attrs:<>
I0823 17:08:58.379762 792 server/node.go:400 connecting to gossip network to verify cluster ID...
I0823 17:08:58.379820 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.380020 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.380180 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.380300 792 storage/replica.go:997 gossiping cluster id cluster-1 from store 1, range 1
I0823 17:08:58.380361 792 storage/replica.go:1007 gossiping sentinel from store 1, range 1
I0823 17:08:58.380407 792 storage/replica.go:1013 gossiping first range from store 1, range 1
I0823 17:08:58.380615 792 server/node.go:417 node connected via gossip and verified as part of cluster "cluster-1"
I0823 17:08:58.380678 792 server/node.go:275 Started node with [[]=] engine(s) and attributes []
I0823 17:08:58.380723 792 server/server.go:197 starting https server at 127.0.0.1:56831
I0823 17:08:58.380855 792 kv/range_cache.go:124 lookup range descriptor: key="\x00zone"
I0823 17:08:58.380901 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00zone"
I0823 17:08:58.398692 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738398153524.76 127.0.0.1:0 17:08:58.398188 273.732µs node server/node.go:486
c349738398153524.76 127.0.0.1:0 17:08:58.398227 176.26µs ·executing RangeLookup storage/store.go:1298
c349738398153524.76 127.0.0.1:0 17:08:58.398237 165.555µs ··read-only path storage/replica.go:527
I0823 17:08:58.399017 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.399700 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738399244267.82 127.0.0.1:0 17:08:58.399304 208.909µs node server/node.go:486
c349738399244267.82 127.0.0.1:0 17:08:58.399339 97.004µs ·executing RangeLookup storage/store.go:1298
c349738399244267.82 127.0.0.1:0 17:08:58.399347 88.444µs ··read-only path storage/replica.go:527
I0823 17:08:58.399947 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.400086 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.400781 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738380807931.11 127.0.0.1:0 17:08:58.400384 205.848µs node server/node.go:486
c349738380807931.11 127.0.0.1:0 17:08:58.400415 137.933µs ·executing Get storage/store.go:1298
c349738380807931.11 127.0.0.1:0 17:08:58.400423 129.532µs ··read-only path storage/replica.go:527
I0823 17:08:58.401476 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738380807931.11 127.0.0.1:0 17:08:58.380824 20.150759ms sending Get kv/txn_coord_sender.go:301
c349738380807931.11 127.0.0.1:0 17:08:58.380833 19.289819ms ·meta descriptor lookup kv/dist_sender.go:653
c349738380807931.11 127.0.0.1:0 17:08:58.400129 840.026µs ·sending RPC kv/dist_sender.go:553
c349738380807931.11 127.0.0.1:0 17:08:58.400194 0 ··sending to 127.0.0.1:56831 rpc/send.go:171
I0823 17:08:58.401864 792 raft/raft.go:406 group 1 100000001 became follower at term 6
I0823 17:08:58.402032 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 6, commit: 14, applied: 14, lastindex: 14, lastterm: 6]
I0823 17:08:58.402202 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 6
I0823 17:08:58.402300 792 raft/raft.go:419 group 1 100000001 became candidate at term 7
I0823 17:08:58.402352 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 7
I0823 17:08:58.402414 792 raft/raft.go:447 group 1 100000001 became leader at term 7
I0823 17:08:58.403276 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.403721 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.404060 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738401020176.90 127.0.0.1:0 17:08:58.401622 1.825008ms node server/node.go:486
c349738401020176.90 127.0.0.1:0 17:08:58.401654 1.750721ms ·executing Put storage/store.go:1298
c349738401020176.90 127.0.0.1:0 17:08:58.401663 1.741531ms ··read-write path storage/replica.go:530
c349738401020176.90 127.0.0.1:0 17:08:58.401670 7.8µs ···command queue storage/replica.go:687
c349738401020176.90 127.0.0.1:0 17:08:58.401690 1.713829ms ···raft storage/replica.go:728
c349738401020176.90 127.0.0.1:0 17:08:58.402863 511.658µs ····applying Put storage/replica.go:802
I0823 17:08:58.404235 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.405710 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738401020176.90 127.0.0.1:0 17:08:58.401082 3.068982ms sending Put kv/txn_coord_sender.go:301
c349738401020176.90 127.0.0.1:0 17:08:58.401094 8.74µs ·meta descriptor lookup kv/dist_sender.go:653
c349738401020176.90 127.0.0.1:0 17:08:58.401108 3.039362ms ·sending RPC kv/dist_sender.go:553
c349738401020176.90 127.0.0.1:0 17:08:58.401140 0 ··sending to 127.0.0.1:56831 rpc/send.go:171
I0823 17:08:58.422970 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.423503 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738405200382.4 127.0.0.1:0 17:08:58.422198 964.706µs node server/node.go:486
c349738405200382.4 127.0.0.1:0 17:08:58.422239 875.168µs ·executing Put storage/store.go:1298
c349738405200382.4 127.0.0.1:0 17:08:58.422252 862.135µs ··read-write path storage/replica.go:530
c349738405200382.4 127.0.0.1:0 17:08:58.422261 11.48µs ···command queue storage/replica.go:687
c349738405200382.4 127.0.0.1:0 17:08:58.422283 830.108µs ···raft storage/replica.go:728
c349738405200382.4 127.0.0.1:0 17:08:58.422673 417.794µs ····applying Put storage/replica.go:802
I0823 17:08:58.423902 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738405200382.4 127.0.0.1:0 17:08:58.421916 1.683213ms sending Put kv/txn_coord_sender.go:301
c349738405200382.4 127.0.0.1:0 17:08:58.421931 13.401µs ·meta descriptor lookup kv/dist_sender.go:653
c349738405200382.4 127.0.0.1:0 17:08:58.421949 1.646651ms ·sending RPC kv/dist_sender.go:553
c349738405200382.4 127.0.0.1:0 17:08:58.421994 0 ··sending to 127.0.0.1:56831 rpc/send.go:171
I0823 17:08:58.424127 792 server/testserver.go:71 Test server listening on https: 127.0.0.1:56831
I0823 17:08:58.424173 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.453457 792 base/context.go:103 setting up TLS from certificates directory: test_certs
--- PASS: TestStatusLocalStacks (0.13s)
=== RUN TestStatusJson
I0823 17:08:58.495570 792 base/context.go:141 setting up TLS from certificates directory: test_certs
I0823 17:08:58.496818 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.498993 792 rpc/clock_offset.go:155 monitoring cluster offset
I0823 17:08:58.499133 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.499234 792 raft/raft.go:406 group 1 100000001 became follower at term 5
I0823 17:08:58.499297 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 5, commit: 10, applied: 10, lastindex: 10, lastterm: 5]
I0823 17:08:58.499347 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 5
I0823 17:08:58.499367 792 raft/raft.go:419 group 1 100000001 became candidate at term 6
I0823 17:08:58.499382 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 6
I0823 17:08:58.499416 792 raft/raft.go:447 group 1 100000001 became leader at term 6
I0823 17:08:58.500161 792 storage/replica_command.go:1017 range 1: new leader lease replica 1:1 00:00:00.000 +1440349739.499s
I0823 17:08:58.506686 792 storage/replica.go:1070 gossiping accounting config from store 1, range 1
I0823 17:08:58.506865 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.507010 792 storage/replica.go:1070 gossiping user config from store 1, range 1
I0823 17:08:58.507154 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.507520 792 server/node.go:310 initialized store store=1:1 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.507564 792 server/node.go:217 new node allocated ID 1
I0823 17:08:58.507664 792 gossip/gossip.go:181 gossiping node descriptor node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:48476" > attrs:<>
I0823 17:08:58.507701 792 server/node.go:400 connecting to gossip network to verify cluster ID...
I0823 17:08:58.507794 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.508010 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.508180 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.508264 792 storage/replica.go:997 gossiping cluster id cluster-1 from store 1, range 1
I0823 17:08:58.508346 792 storage/replica.go:1007 gossiping sentinel from store 1, range 1
I0823 17:08:58.508397 792 storage/replica.go:1013 gossiping first range from store 1, range 1
I0823 17:08:58.508584 792 server/node.go:417 node connected via gossip and verified as part of cluster "cluster-1"
I0823 17:08:58.508675 792 server/node.go:275 Started node with [[]=] engine(s) and attributes []
I0823 17:08:58.508720 792 server/server.go:197 starting https server at 127.0.0.1:48476
I0823 17:08:58.508870 792 kv/range_cache.go:124 lookup range descriptor: key="\x00zone"
I0823 17:08:58.508912 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00zone"
I0823 17:08:58.528294 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738527833881.33 127.0.0.1:0 17:08:58.527863 203.623µs node server/node.go:486
c349738527833881.33 127.0.0.1:0 17:08:58.527899 122.433µs ·executing RangeLookup storage/store.go:1298
c349738527833881.33 127.0.0.1:0 17:08:58.527908 113.22µs ··read-only path storage/replica.go:527
I0823 17:08:58.528495 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.529018 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738528670192.87 127.0.0.1:0 17:08:58.528694 199.77µs node server/node.go:486
c349738528670192.87 127.0.0.1:0 17:08:58.528725 92.621µs ·executing RangeLookup storage/store.go:1298
c349738528670192.87 127.0.0.1:0 17:08:58.528732 84.741µs ··read-only path storage/replica.go:527
I0823 17:08:58.529228 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.529319 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.529855 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738508775918.62 127.0.0.1:0 17:08:58.529535 161.458µs node server/node.go:486
c349738508775918.62 127.0.0.1:0 17:08:58.529565 97.892µs ·executing Get storage/store.go:1298
c349738508775918.62 127.0.0.1:0 17:08:58.529573 90.002µs ··read-only path storage/replica.go:527
I0823 17:08:58.530294 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738508775918.62 127.0.0.1:0 17:08:58.508838 21.139488ms sending Get kv/txn_coord_sender.go:301
c349738508775918.62 127.0.0.1:0 17:08:58.508848 20.506261ms ·meta descriptor lookup kv/dist_sender.go:653
c349738508775918.62 127.0.0.1:0 17:08:58.529360 613.404µs ·sending RPC kv/dist_sender.go:553
c349738508775918.62 127.0.0.1:0 17:08:58.529389 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
I0823 17:08:58.530567 792 raft/raft.go:406 group 1 100000001 became follower at term 6
I0823 17:08:58.530666 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 6, commit: 14, applied: 14, lastindex: 14, lastterm: 6]
I0823 17:08:58.530761 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 6
I0823 17:08:58.530802 792 raft/raft.go:419 group 1 100000001 became candidate at term 7
I0823 17:08:58.530832 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 7
I0823 17:08:58.530885 792 raft/raft.go:447 group 1 100000001 became leader at term 7
I0823 17:08:58.531436 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.531716 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.531933 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738530011590.85 127.0.0.1:0 17:08:58.530388 1.15606ms node server/node.go:486
c349738530011590.85 127.0.0.1:0 17:08:58.530418 1.088722ms ·executing Put storage/store.go:1298
c349738530011590.85 127.0.0.1:0 17:08:58.530426 1.080574ms ··read-write path storage/replica.go:530
c349738530011590.85 127.0.0.1:0 17:08:58.530432 7.87µs ···command queue storage/replica.go:687
c349738530011590.85 127.0.0.1:0 17:08:58.530450 1.055377ms ···raft storage/replica.go:728
c349738530011590.85 127.0.0.1:0 17:08:58.531198 287.239µs ····applying Put storage/replica.go:802
I0823 17:08:58.532038 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.533118 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738530011590.85 127.0.0.1:0 17:08:58.530026 1.975644ms sending Put kv/txn_coord_sender.go:301
c349738530011590.85 127.0.0.1:0 17:08:58.530035 6.84µs ·meta descriptor lookup kv/dist_sender.go:653
c349738530011590.85 127.0.0.1:0 17:08:58.530045 1.952893ms ·sending RPC kv/dist_sender.go:553
c349738530011590.85 127.0.0.1:0 17:08:58.530069 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
I0823 17:08:58.551648 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.552110 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738532795130.30 127.0.0.1:0 17:08:58.551038 792.456µs node server/node.go:486
c349738532795130.30 127.0.0.1:0 17:08:58.551077 632.983µs ·executing Put storage/store.go:1298
c349738532795130.30 127.0.0.1:0 17:08:58.551087 622.66µs ··read-write path storage/replica.go:530
c349738532795130.30 127.0.0.1:0 17:08:58.551094 9.599µs ···command queue storage/replica.go:687
c349738532795130.30 127.0.0.1:0 17:08:58.551113 595.765µs ···raft storage/replica.go:728
c349738532795130.30 127.0.0.1:0 17:08:58.551409 279.465µs ····applying Put storage/replica.go:802
I0823 17:08:58.552393 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738532795130.30 127.0.0.1:0 17:08:58.550796 1.375048ms sending Put kv/txn_coord_sender.go:301
c349738532795130.30 127.0.0.1:0 17:08:58.550808 11.848µs ·meta descriptor lookup kv/dist_sender.go:653
c349738532795130.30 127.0.0.1:0 17:08:58.550824 1.342681ms ·sending RPC kv/dist_sender.go:553
c349738532795130.30 127.0.0.1:0 17:08:58.550862 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
I0823 17:08:58.552538 792 server/testserver.go:71 Test server listening on https: 127.0.0.1:48476
I0823 17:08:58.569708 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738569214668.34 127.0.0.1:0 17:08:58.569441 127.939µs node server/node.go:486
c349738569214668.34 127.0.0.1:0 17:08:58.569475 63.244µs ·executing Scan storage/store.go:1298
c349738569214668.34 127.0.0.1:0 17:08:58.569484 54.324µs ··read-only path storage/replica.go:527
I0823 17:08:58.574819 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738569214668.34 127.0.0.1:0 17:08:58.569233 625.524µs sending Scan kv/txn_coord_sender.go:301
c349738569214668.34 127.0.0.1:0 17:08:58.569243 8.01µs ·meta descriptor lookup kv/dist_sender.go:653
c349738569214668.34 127.0.0.1:0 17:08:58.569254 600.215µs ·sending RPC kv/dist_sender.go:553
c349738569214668.34 127.0.0.1:0 17:08:58.569281 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
I0823 17:08:58.577017 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738576547838.55 127.0.0.1:0 17:08:58.576760 131.591µs node server/node.go:486
c349738576547838.55 127.0.0.1:0 17:08:58.576792 69.767µs ·executing Scan storage/store.go:1298
c349738576547838.55 127.0.0.1:0 17:08:58.576800 61.735µs ··read-only path storage/replica.go:527
I0823 17:08:58.577418 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738576547838.55 127.0.0.1:0 17:08:58.576565 574.127µs sending Scan kv/txn_coord_sender.go:301
c349738576547838.55 127.0.0.1:0 17:08:58.576574 7.015µs ·meta descriptor lookup kv/dist_sender.go:653
c349738576547838.55 127.0.0.1:0 17:08:58.576584 550.747µs ·sending RPC kv/dist_sender.go:553
c349738576547838.55 127.0.0.1:0 17:08:58.576610 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
I0823 17:08:58.578385 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738577924294.2 127.0.0.1:0 17:08:58.578138 125.335µs node server/node.go:486
c349738577924294.2 127.0.0.1:0 17:08:58.578170 61.836µs ·executing Scan storage/store.go:1298
c349738577924294.2 127.0.0.1:0 17:08:58.578177 54.122µs ··read-only path storage/replica.go:527
I0823 17:08:58.578755 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738577924294.2 127.0.0.1:0 17:08:58.577940 560.403µs sending Scan kv/txn_coord_sender.go:301
c349738577924294.2 127.0.0.1:0 17:08:58.577949 6.789µs ·meta descriptor lookup kv/dist_sender.go:653
c349738577924294.2 127.0.0.1:0 17:08:58.577959 537.904µs ·sending RPC kv/dist_sender.go:553
c349738577924294.2 127.0.0.1:0 17:08:58.577983 0 ··sending to 127.0.0.1:48476 rpc/send.go:171
--- PASS: TestStatusJson (0.10s)
=== RUN TestStatusGossipJson
I0823 17:08:58.599232 792 base/context.go:141 setting up TLS from certificates directory: test_certs
I0823 17:08:58.600272 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.602350 792 rpc/clock_offset.go:155 monitoring cluster offset
I0823 17:08:58.602470 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.602572 792 raft/raft.go:406 group 1 100000001 became follower at term 5
I0823 17:08:58.602636 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 5, commit: 10, applied: 10, lastindex: 10, lastterm: 5]
I0823 17:08:58.602694 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 5
I0823 17:08:58.602714 792 raft/raft.go:419 group 1 100000001 became candidate at term 6
I0823 17:08:58.602728 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 6
I0823 17:08:58.602761 792 raft/raft.go:447 group 1 100000001 became leader at term 6
I0823 17:08:58.603291 792 storage/replica_command.go:1017 range 1: new leader lease replica 1:1 00:00:00.000 +1440349739.602s
I0823 17:08:58.604684 792 storage/replica.go:1070 gossiping accounting config from store 1, range 1
I0823 17:08:58.604795 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.604879 792 storage/replica.go:1070 gossiping user config from store 1, range 1
I0823 17:08:58.604976 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.605241 792 server/node.go:310 initialized store store=1:1 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.605268 792 server/node.go:217 new node allocated ID 1
I0823 17:08:58.605321 792 gossip/gossip.go:181 gossiping node descriptor node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:56385" > attrs:<>
I0823 17:08:58.605343 792 server/node.go:400 connecting to gossip network to verify cluster ID...
I0823 17:08:58.605381 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.605530 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.605642 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.605709 792 storage/replica.go:997 gossiping cluster id cluster-1 from store 1, range 1
I0823 17:08:58.605751 792 storage/replica.go:1007 gossiping sentinel from store 1, range 1
I0823 17:08:58.605779 792 storage/replica.go:1013 gossiping first range from store 1, range 1
I0823 17:08:58.605904 792 server/node.go:417 node connected via gossip and verified as part of cluster "cluster-1"
I0823 17:08:58.605949 792 server/node.go:275 Started node with [[]=] engine(s) and attributes []
I0823 17:08:58.605985 792 server/server.go:197 starting https server at 127.0.0.1:56385
I0823 17:08:58.606068 792 kv/range_cache.go:124 lookup range descriptor: key="\x00zone"
I0823 17:08:58.606092 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00zone"
I0823 17:08:58.628804 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738628273180.41 127.0.0.1:0 17:08:58.628319 272.299µs node server/node.go:486
c349738628273180.41 127.0.0.1:0 17:08:58.628364 166.01µs ·executing RangeLookup storage/store.go:1298
c349738628273180.41 127.0.0.1:0 17:08:58.628376 153.51µs ··read-only path storage/replica.go:527
I0823 17:08:58.629037 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.629688 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738629253239.15 127.0.0.1:0 17:08:58.629281 289.396µs node server/node.go:486
c349738629253239.15 127.0.0.1:0 17:08:58.629316 215.88µs ·executing RangeLookup storage/store.go:1298
c349738629253239.15 127.0.0.1:0 17:08:58.629325 206.298µs ··read-only path storage/replica.go:527
I0823 17:08:58.629853 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.629989 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.630495 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738606023383.63 127.0.0.1:0 17:08:58.630195 168.276µs node server/node.go:486
c349738606023383.63 127.0.0.1:0 17:08:58.630227 99.531µs ·executing Get storage/store.go:1298
c349738606023383.63 127.0.0.1:0 17:08:58.630234 91.96µs ··read-only path storage/replica.go:527
I0823 17:08:58.630987 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738606023383.63 127.0.0.1:0 17:08:58.606038 24.574085ms sending Get kv/txn_coord_sender.go:301
c349738606023383.63 127.0.0.1:0 17:08:58.606047 23.963828ms ·meta descriptor lookup kv/dist_sender.go:653
c349738606023383.63 127.0.0.1:0 17:08:58.630016 591.096µs ·sending RPC kv/dist_sender.go:553
c349738606023383.63 127.0.0.1:0 17:08:58.630046 0 ··sending to 127.0.0.1:56385 rpc/send.go:171
I0823 17:08:58.631268 792 raft/raft.go:406 group 1 100000001 became follower at term 6
I0823 17:08:58.631374 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 6, commit: 14, applied: 14, lastindex: 14, lastterm: 6]
I0823 17:08:58.631473 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 6
I0823 17:08:58.631518 792 raft/raft.go:419 group 1 100000001 became candidate at term 7
I0823 17:08:58.631591 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 7
I0823 17:08:58.631645 792 raft/raft.go:447 group 1 100000001 became leader at term 7
I0823 17:08:58.632238 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.632524 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.632749 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738630648574.78 127.0.0.1:0 17:08:58.631085 1.255257ms node server/node.go:486
c349738630648574.78 127.0.0.1:0 17:08:58.631117 1.19264ms ·executing Put storage/store.go:1298
c349738630648574.78 127.0.0.1:0 17:08:58.631124 1.184463ms ··read-write path storage/replica.go:530
c349738630648574.78 127.0.0.1:0 17:08:58.631131 6.687µs ···command queue storage/replica.go:687
c349738630648574.78 127.0.0.1:0 17:08:58.631148 1.160538ms ···raft storage/replica.go:728
c349738630648574.78 127.0.0.1:0 17:08:58.631975 313.692µs ····applying Put storage/replica.go:802
I0823 17:08:58.632857 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.633981 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738630648574.78 127.0.0.1:0 17:08:58.630663 2.154194ms sending Put kv/txn_coord_sender.go:301
c349738630648574.78 127.0.0.1:0 17:08:58.630673 8.521µs ·meta descriptor lookup kv/dist_sender.go:653
c349738630648574.78 127.0.0.1:0 17:08:58.630686 2.12857ms ·sending RPC kv/dist_sender.go:553
c349738630648574.78 127.0.0.1:0 17:08:58.630712 0 ··sending to 127.0.0.1:56385 rpc/send.go:171
I0823 17:08:58.651646 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.651992 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738633636913.63 127.0.0.1:0 17:08:58.651008 732.34µs node server/node.go:486
c349738633636913.63 127.0.0.1:0 17:08:58.651048 655.282µs ·executing Put storage/store.go:1298
c349738633636913.63 127.0.0.1:0 17:08:58.651058 645.18µs ··read-write path storage/replica.go:530
c349738633636913.63 127.0.0.1:0 17:08:58.651066 8.938µs ···command queue storage/replica.go:687
c349738633636913.63 127.0.0.1:0 17:08:58.651084 618.543µs ···raft storage/replica.go:728
c349738633636913.63 127.0.0.1:0 17:08:58.651411 271.688µs ····applying Put storage/replica.go:802
I0823 17:08:58.652276 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738633636913.63 127.0.0.1:0 17:08:58.650782 1.262606ms sending Put kv/txn_coord_sender.go:301
c349738633636913.63 127.0.0.1:0 17:08:58.650794 10.608µs ·meta descriptor lookup kv/dist_sender.go:653
c349738633636913.63 127.0.0.1:0 17:08:58.650809 1.232437ms ·sending RPC kv/dist_sender.go:553
c349738633636913.63 127.0.0.1:0 17:08:58.650843 0 ··sending to 127.0.0.1:56385 rpc/send.go:171
I0823 17:08:58.652422 792 server/testserver.go:71 Test server listening on https: 127.0.0.1:56385
--- PASS: TestStatusGossipJson (0.10s)
=== RUN TestStatusLocalLogs
I0823 17:08:58.696055 792 base/context.go:141 setting up TLS from certificates directory: test_certs
I0823 17:08:58.698375 792 base/context.go:103 setting up TLS from certificates directory: test_certs
I0823 17:08:58.699942 792 rpc/clock_offset.go:155 monitoring cluster offset
I0823 17:08:58.701985 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.702147 792 raft/raft.go:406 group 1 100000001 became follower at term 5
I0823 17:08:58.702278 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 5, commit: 10, applied: 10, lastindex: 10, lastterm: 5]
I0823 17:08:58.702359 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 5
I0823 17:08:58.702429 792 raft/raft.go:419 group 1 100000001 became candidate at term 6
I0823 17:08:58.702471 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 6
I0823 17:08:58.702532 792 raft/raft.go:447 group 1 100000001 became leader at term 6
I0823 17:08:58.703247 792 storage/replica_command.go:1017 range 1: new leader lease replica 1:1 00:00:00.000 +1440349739.702s
I0823 17:08:58.705144 792 multiraft/multiraft.go:446 node 100000002 starting
I0823 17:08:58.706069 792 multiraft/multiraft.go:446 node 100000003 starting
I0823 17:08:58.707047 792 storage/replica.go:1070 gossiping accounting config from store 1, range 1
I0823 17:08:58.707211 792 storage/replica.go:1070 gossiping permissions config from store 1, range 1
I0823 17:08:58.707321 792 storage/replica.go:1070 gossiping user config from store 1, range 1
I0823 17:08:58.707462 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.707796 792 server/node.go:310 initialized store store=1:1 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.708074 792 multiraft/multiraft.go:446 node 100000001 starting
I0823 17:08:58.708281 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.708468 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.708589 792 storage/replica.go:997 gossiping cluster id cluster-1 from store 1, range 1
I0823 17:08:58.708655 792 storage/replica.go:1007 gossiping sentinel from store 1, range 1
I0823 17:08:58.708708 792 storage/replica.go:1013 gossiping first range from store 1, range 1
I0823 17:08:58.709057 792 server/node.go:310 initialized store store=1:2 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.709502 792 server/node.go:310 initialized store store=1:3 ([]=): {Capacity:257668677632 Available:198379286528 RangeCount:0}
I0823 17:08:58.709550 792 server/node.go:217 new node allocated ID 1
I0823 17:08:58.709653 792 gossip/gossip.go:181 gossiping node descriptor node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:46564" > attrs:<>
I0823 17:08:58.709697 792 server/node.go:400 connecting to gossip network to verify cluster ID...
I0823 17:08:58.709734 792 server/node.go:417 node connected via gossip and verified as part of cluster "cluster-1"
I0823 17:08:58.709836 792 server/node.go:275 Started node with [[]= []= []=] engine(s) and attributes []
I0823 17:08:58.709898 792 server/server.go:197 starting https server at 127.0.0.1:46564
I0823 17:08:58.710001 792 kv/range_cache.go:124 lookup range descriptor: key="\x00zone"
I0823 17:08:58.710049 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00zone"
I0823 17:08:58.710168 792 multiraft/multiraft.go:446 node 100000002 starting
I0823 17:08:58.710313 792 multiraft/multiraft.go:446 node 100000003 starting
I0823 17:08:58.721073 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:08:58.721182 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.721278 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:08:58.721371 792 storage/replicate_queue.go:88 range=1 [""-"\xff\xff") needs 3 nodes; has 1
I0823 17:08:58.721446 792 storage/queue.go:205 adding replica range=1 [""-"\xff\xff") to replicate queue
I0823 17:08:58.721513 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:08:58.721697 792 storage/queue.go:295 processing replica range=1 [""-"\xff\xff") from replicate queue...
I0823 17:08:58.721762 792 storage/replicate_queue.go:88 range=1 [""-"\xff\xff") needs 3 nodes; has 1
E0823 17:08:58.721880 792 storage/queue.go:311 failure processing replica range=1 [""-"\xff\xff") from replicate queue: storage/allocator.go:204: unable to allocate a target store; no candidates available
I0823 17:08:58.727973 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738727588688.86 127.0.0.1:0 17:08:58.727615 194.133µs node server/node.go:486
c349738727588688.86 127.0.0.1:0 17:08:58.727649 115.899µs ·executing RangeLookup storage/store.go:1298
c349738727588688.86 127.0.0.1:0 17:08:58.727657 106.919µs ··read-only path storage/replica.go:527
I0823 17:08:58.728167 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.733123 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738728341160.41 127.0.0.1:0 17:08:58.728364 162.067µs node server/node.go:486
c349738728341160.41 127.0.0.1:0 17:08:58.728394 95.041µs ·executing RangeLookup storage/store.go:1298
c349738728341160.41 127.0.0.1:0 17:08:58.728401 87.438µs ··read-only path storage/replica.go:527
I0823 17:08:58.733349 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:08:58.733419 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.733518 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:08:58.733585 792 storage/replicate_queue.go:88 range=1 [""-"\xff\xff") needs 3 nodes; has 1
I0823 17:08:58.733634 792 storage/queue.go:205 adding replica range=1 [""-"\xff\xff") to replicate queue
I0823 17:08:58.733688 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:08:58.733809 792 storage/queue.go:295 processing replica range=1 [""-"\xff\xff") from replicate queue...
I0823 17:08:58.733868 792 storage/replicate_queue.go:88 range=1 [""-"\xff\xff") needs 3 nodes; has 1
E0823 17:08:58.733933 792 storage/queue.go:311 failure processing replica range=1 [""-"\xff\xff") from replicate queue: storage/allocator.go:204: unable to allocate a target store; no candidates available
I0823 17:08:58.734960 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.735060 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:08:58.735582 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738709954047.5 127.0.0.1:0 17:08:58.735286 166.937µs node server/node.go:486
c349738709954047.5 127.0.0.1:0 17:08:58.735318 101.807µs ·executing Get storage/store.go:1298
c349738709954047.5 127.0.0.1:0 17:08:58.735325 93.776µs ··read-only path storage/replica.go:527
I0823 17:08:58.736039 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738709954047.5 127.0.0.1:0 17:08:58.709970 25.747979ms sending Get kv/txn_coord_sender.go:301
c349738709954047.5 127.0.0.1:0 17:08:58.709980 25.118596ms ·meta descriptor lookup kv/dist_sender.go:653
c349738709954047.5 127.0.0.1:0 17:08:58.735104 610.254µs ·sending RPC kv/dist_sender.go:553
c349738709954047.5 127.0.0.1:0 17:08:58.735135 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.736327 792 raft/raft.go:406 group 1 100000001 became follower at term 6
I0823 17:08:58.736439 792 raft/raft.go:219 group 1 newRaft 100000001 [peers: [100000001], term: 6, commit: 16, applied: 16, lastindex: 16, lastterm: 6]
I0823 17:08:58.736544 792 raft/raft.go:485 group 1 100000001 is starting a new election at term 6
I0823 17:08:58.736593 792 raft/raft.go:419 group 1 100000001 became candidate at term 7
I0823 17:08:58.736636 792 raft/raft.go:468 group 1 100000001 received vote from 100000001 at term 7
I0823 17:08:58.736711 792 raft/raft.go:447 group 1 100000001 became leader at term 7
I0823 17:08:58.737364 792 storage/replica.go:1070 gossiping zones config from store 1, range 1
I0823 17:08:58.737677 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.737915 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738735753343.52 127.0.0.1:0 17:08:58.736147 1.34471ms node server/node.go:486
c349738735753343.52 127.0.0.1:0 17:08:58.736178 1.276806ms ·executing Put storage/store.go:1298
c349738735753343.52 127.0.0.1:0 17:08:58.736186 1.268389ms ··read-write path storage/replica.go:530
c349738735753343.52 127.0.0.1:0 17:08:58.736192 6.638µs ···command queue storage/replica.go:687
c349738735753343.52 127.0.0.1:0 17:08:58.736209 1.244238ms ···raft storage/replica.go:728
c349738735753343.52 127.0.0.1:0 17:08:58.737057 374.882µs ····applying Put storage/replica.go:802
I0823 17:08:58.738334 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738735753343.52 127.0.0.1:0 17:08:58.735768 2.239864ms sending Put kv/txn_coord_sender.go:301
c349738735753343.52 127.0.0.1:0 17:08:58.735777 7.117µs ·meta descriptor lookup kv/dist_sender.go:653
c349738735753343.52 127.0.0.1:0 17:08:58.735787 2.21633ms ·sending RPC kv/dist_sender.go:553
c349738735753343.52 127.0.0.1:0 17:08:58.735810 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.738663 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738738028031.85 127.0.0.1:0 17:08:58.738445 106.027µs node server/node.go:486
c349738738028031.85 127.0.0.1:0 17:08:58.738477 46.106µs ·executing Get storage/store.go:1298
c349738738028031.85 127.0.0.1:0 17:08:58.738485 37.802µs ··read-only path storage/replica.go:527
I0823 17:08:58.739001 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738738028031.85 127.0.0.1:0 17:08:58.738043 756.722µs sending Get kv/txn_coord_sender.go:301
c349738738028031.85 127.0.0.1:0 17:08:58.738052 8.152µs ·meta descriptor lookup kv/dist_sender.go:653
c349738738028031.85 127.0.0.1:0 17:08:58.738064 731.662µs ·sending RPC kv/dist_sender.go:553
c349738738028031.85 127.0.0.1:0 17:08:58.738097 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.740158 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738739198281.99 127.0.0.1:0 17:08:58.739421 544.718µs node server/node.go:486
c349738739198281.99 127.0.0.1:0 17:08:58.739453 483.451µs ·executing Put storage/store.go:1298
c349738739198281.99 127.0.0.1:0 17:08:58.739461 474.77µs ··read-write path storage/replica.go:530
c349738739198281.99 127.0.0.1:0 17:08:58.739469 8.254µs ···command queue storage/replica.go:687
c349738739198281.99 127.0.0.1:0 17:08:58.739485 450.009µs ···raft storage/replica.go:728
c349738739198281.99 127.0.0.1:0 17:08:58.739771 143.275µs ····applying Put storage/replica.go:802
I0823 17:08:58.740328 792 server/server.go:256 recorded status for node 1
I0823 17:08:58.740673 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738739198281.99 127.0.0.1:0 17:08:58.739213 1.087315ms sending Put kv/txn_coord_sender.go:301
c349738739198281.99 127.0.0.1:0 17:08:58.739222 7.732µs ·meta descriptor lookup kv/dist_sender.go:653
c349738739198281.99 127.0.0.1:0 17:08:58.739238 1.058523ms ·sending RPC kv/dist_sender.go:553
c349738739198281.99 127.0.0.1:0 17:08:58.739264 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.740868 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:08:58.740951 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.741030 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:08:58.741091 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:08:58.741142 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:08:58.741969 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738740382162.50 127.0.0.1:0 17:08:58.741327 448.047µs node server/node.go:486
c349738740382162.50 127.0.0.1:0 17:08:58.741363 389.607µs ·executing Put storage/store.go:1298
c349738740382162.50 127.0.0.1:0 17:08:58.741370 381.315µs ··read-write path storage/replica.go:530
c349738740382162.50 127.0.0.1:0 17:08:58.741377 7.356µs ···command queue storage/replica.go:687
c349738740382162.50 127.0.0.1:0 17:08:58.741393 357.643µs ···raft storage/replica.go:728
c349738740382162.50 127.0.0.1:0 17:08:58.741619 110.784µs ····applying Put storage/replica.go:802
I0823 17:08:58.742460 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738740382162.50 127.0.0.1:0 17:08:58.740398 1.715972ms sending Put kv/txn_coord_sender.go:301
c349738740382162.50 127.0.0.1:0 17:08:58.740406 9.456µs ·meta descriptor lookup kv/dist_sender.go:653
c349738740382162.50 127.0.0.1:0 17:08:58.740419 1.691257ms ·sending RPC kv/dist_sender.go:553
c349738740382162.50 127.0.0.1:0 17:08:58.740446 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.743182 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738742139866.48 127.0.0.1:0 17:08:58.742558 440.851µs node server/node.go:486
c349738742139866.48 127.0.0.1:0 17:08:58.742588 383.717µs ·executing Put storage/store.go:1298
c349738742139866.48 127.0.0.1:0 17:08:58.742595 375.769µs ··read-write path storage/replica.go:530
c349738742139866.48 127.0.0.1:0 17:08:58.742601 7.994µs ···command queue storage/replica.go:687
c349738742139866.48 127.0.0.1:0 17:08:58.742619 351.437µs ···raft storage/replica.go:728
c349738742139866.48 127.0.0.1:0 17:08:58.742831 118.714µs ····applying Put storage/replica.go:802
I0823 17:08:58.743599 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738742139866.48 127.0.0.1:0 17:08:58.742153 1.145516ms sending Put kv/txn_coord_sender.go:301
c349738742139866.48 127.0.0.1:0 17:08:58.742161 6.404µs ·meta descriptor lookup kv/dist_sender.go:653
c349738742139866.48 127.0.0.1:0 17:08:58.742171 1.124715ms ·sending RPC kv/dist_sender.go:553
c349738742139866.48 127.0.0.1:0 17:08:58.742194 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.744466 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738743323428.23 127.0.0.1:0 17:08:58.743694 586.415µs node server/node.go:486
c349738743323428.23 127.0.0.1:0 17:08:58.743724 533.052µs ·executing Put storage/store.go:1298
c349738743323428.23 127.0.0.1:0 17:08:58.743731 525.236µs ··read-write path storage/replica.go:530
c349738743323428.23 127.0.0.1:0 17:08:58.743737 6.969µs ···command queue storage/replica.go:687
c349738743323428.23 127.0.0.1:0 17:08:58.743754 501.5µs ···raft storage/replica.go:728
c349738743323428.23 127.0.0.1:0 17:08:58.744122 110.628µs ····applying Put storage/replica.go:802
I0823 17:08:58.744615 792 server/server.go:267 recorded status for 3 stores
E0823 17:08:58.744671 792 server/status_test.go:345 TestStatusLocalLogFile test message-Error
W0823 17:08:58.744709 792 server/status_test.go:347 TestStatusLocalLogFile test message-Warning
I0823 17:08:58.744744 792 server/status_test.go:349 TestStatusLocalLogFile test message-Info
I0823 17:08:58.745084 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349738743323428.23 127.0.0.1:0 17:08:58.743338 1.25353ms sending Put kv/txn_coord_sender.go:301
c349738743323428.23 127.0.0.1:0 17:08:58.743346 6.523µs ·meta descriptor lookup kv/dist_sender.go:653
c349738743323428.23 127.0.0.1:0 17:08:58.743356 1.232179ms ·sending RPC kv/dist_sender.go:553
c349738743323428.23 127.0.0.1:0 17:08:58.743379 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:08:58.748256 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:08:58.748329 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:08:58.748407 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:08:58.748490 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:08:58.748545 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:05.215364 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:05.215685 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:05.215801 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:05.215919 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:05.216025 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:05.216533 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
I0823 17:09:06.211878 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:06.211981 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:06.212081 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:06.212140 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:06.212192 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:14.539681 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:14.540120 792 server/status/runtime.go:118 runtime stats: 67 goroutines, 3.96MiB active, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (59x)
I0823 17:09:14.540677 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
W0823 17:09:14.540710 792 rpc/client.go:233 rpc/client.go:279: heartbeat timed out after 6s
I0823 17:09:14.549708 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:14.549807 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:14.549855 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:14.549891 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:14.557647 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:14.557725 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:14.557801 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:14.557865 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:14.557922 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:14.558800 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349748711549562.20 127.0.0.1:0 17:09:14.556299 2.252806ms node server/node.go:486
c349748711549562.20 127.0.0.1:0 17:09:14.556336 2.185242ms ·executing Put storage/store.go:1298
c349748711549562.20 127.0.0.1:0 17:09:14.556345 2.175951ms ··read-write path storage/replica.go:530
c349748711549562.20 127.0.0.1:0 17:09:14.556357 9.323µs ···command queue storage/replica.go:687
c349748711549562.20 127.0.0.1:0 17:09:14.556371 993.466µs ···request leader lease storage/replica.go:395
c349748711549562.20 127.0.0.1:0 17:09:14.557380 1.140719ms ···raft storage/replica.go:728
c349748711549562.20 127.0.0.1:0 17:09:14.558203 130.449µs ····applying Put storage/replica.go:802
I0823 17:09:14.559323 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754540238592.3 127.0.0.1:0 17:09:14.556755 2.386845ms node server/node.go:486
c349754540238592.3 127.0.0.1:0 17:09:14.556788 2.310178ms ·executing Merge storage/store.go:1298
c349754540238592.3 127.0.0.1:0 17:09:14.556796 2.301967ms ··read-write path storage/replica.go:530
c349754540238592.3 127.0.0.1:0 17:09:14.556802 12.715µs ···command queue storage/replica.go:687
c349754540238592.3 127.0.0.1:0 17:09:14.557981 1.116465ms ···raft storage/replica.go:728
c349754540238592.3 127.0.0.1:0 17:09:14.558353 128.994µs ····applying Merge storage/replica.go:802
I0823 17:09:14.559758 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349748711342588.21 127.0.0.1:0 17:09:14.556841 2.742268ms node server/node.go:486
c349748711342588.21 127.0.0.1:0 17:09:14.556872 2.688445ms ·executing Merge storage/store.go:1298
c349748711342588.21 127.0.0.1:0 17:09:14.556879 2.680793ms ··read-write path storage/replica.go:530
c349748711342588.21 127.0.0.1:0 17:09:14.556885 11.498µs ···command queue storage/replica.go:687
c349748711342588.21 127.0.0.1:0 17:09:14.558897 662.388µs ···raft storage/replica.go:728
c349748711342588.21 127.0.0.1:0 17:09:14.559436 112.232µs ····applying Merge storage/replica.go:802
I0823 17:09:14.560211 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349748711342588.21 127.0.0.1:0 17:09:08.711367 5.848596302s sending Merge kv/txn_coord_sender.go:301
c349748711342588.21 127.0.0.1:0 17:09:08.711379 10.398µs ·meta descriptor lookup kv/dist_sender.go:653
c349748711342588.21 127.0.0.1:0 17:09:08.711392 5.84856161s ·sending RPC kv/dist_sender.go:553
c349748711342588.21 127.0.0.1:0 17:09:14.540484 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:09:14.560377 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349748711549562.20 127.0.0.1:0 17:09:08.711576 5.848487082s sending Put kv/txn_coord_sender.go:301
c349748711549562.20 127.0.0.1:0 17:09:08.711582 4.658µs ·meta descriptor lookup kv/dist_sender.go:653
c349748711549562.20 127.0.0.1:0 17:09:08.711589 5.848470898s ·sending RPC kv/dist_sender.go:553
c349748711549562.20 127.0.0.1:0 17:09:14.540587 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:09:14.560533 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754540238592.3 127.0.0.1:0 17:09:14.540263 19.814421ms sending Merge kv/txn_coord_sender.go:301
c349754540238592.3 127.0.0.1:0 17:09:14.540276 23.252µs ·meta descriptor lookup kv/dist_sender.go:653
c349754540238592.3 127.0.0.1:0 17:09:14.540304 19.771641ms ·sending RPC kv/dist_sender.go:553
c349754540238592.3 127.0.0.1:0 17:09:14.540342 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:09:14.560669 792 server/server.go:256 recorded status for node 1
I0823 17:09:17.402965 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:17.403063 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:17.403142 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:17.403183 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:17.403216 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:19.827746 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:19.827949 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:19.828191 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:19.828298 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:19.828415 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:19.828609 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
W0823 17:09:19.839919 792 rpc/send.go:215 Node.Merge: error reply: rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
I0823 17:09:19.840110 792 kv/range_cache.go:190 evict cached descriptor: key="\x00tsdcr.node.calls.error.1\x00\x01\t\x01\v\x06\x1a\xe1" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:19.840164 792 kv/range_cache.go:190 evict cached descriptor: key="\x00\x00meta2\x00tsdcr.node.calls.error.1\x00\x01\t\x01\v\x06\x1a\xe1" desc=<nil>
I0823 17:09:19.840190 792 kv/range_cache.go:190 evict cached descriptor: key="\x00\x00meta1\x00tsdcr.node.calls.error.1\x00\x01\t\x01\v\x06\x1a\xe1" desc=<nil>
W0823 17:09:19.840227 792 kv/dist_sender.go:698 failed to invoke Merge: failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
W0823 17:09:19.840253 792 kv/dist_sender.go:740 failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
W0823 17:09:19.840278 792 rpc/send.go:215 Node.Put: error reply: rpc/send.go:265: rpc to Node.Put: client not ready after 5s
W0823 17:09:19.840330 792 kv/dist_sender.go:698 failed to invoke Put: failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Put: client not ready after 5s
W0823 17:09:19.840401 792 kv/dist_sender.go:740 failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Put: client not ready after 5s
W0823 17:09:19.840439 792 rpc/send.go:215 Node.Merge: error reply: rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
W0823 17:09:19.840509 792 kv/dist_sender.go:698 failed to invoke Merge: failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
W0823 17:09:19.840548 792 kv/dist_sender.go:740 failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.Merge: client not ready after 5s
I0823 17:09:19.840705 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:19.840758 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:19.840826 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:19.840883 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:19.840928 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:22.216270 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:22.216362 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:22.216448 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:22.216493 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:22.216527 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:22.216696 792 kv/range_cache.go:124 lookup range descriptor: key="\x00tsdcr.node.calls.error.1\x00\x01\t\x01\v\x06\x1a\xe1"
I0823 17:09:22.216727 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00tsdcr.node.calls.error.1\x00\x01\t\x01\v\x06\x1a\xe1"
I0823 17:09:22.216908 792 kv/range_cache.go:124 lookup range descriptor: key="\x00tsdcr.node.sys.goroutines.1\x00\x01\t\x01\v\x06\x1a\xe1"
I0823 17:09:22.216937 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00tsdcr.node.sys.goroutines.1\x00\x01\t\x01\v\x06\x1a\xe1"
I0823 17:09:22.217064 792 kv/range_cache.go:124 lookup range descriptor: key="\x00status-store-\t\x01"
I0823 17:09:22.217090 792 kv/range_cache.go:124 lookup range descriptor: key="\x00\x00meta2\x00status-store-\t\x01"
I0823 17:09:22.217798 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349762217335752.61 127.0.0.1:0 17:09:22.217366 272.928µs node server/node.go:486
c349762217335752.61 127.0.0.1:0 17:09:22.217423 164.252µs ·executing RangeLookup storage/store.go:1298
c349762217335752.61 127.0.0.1:0 17:09:22.217433 154.173µs ··read-only path storage/replica.go:527
I0823 17:09:22.218167 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349762217875255.78 127.0.0.1:0 17:09:22.217899 154.404µs node server/node.go:486
c349762217875255.78 127.0.0.1:0 17:09:22.217929 88.928µs ·executing RangeLookup storage/store.go:1298
c349762217875255.78 127.0.0.1:0 17:09:22.217937 81.133µs ··read-only path storage/replica.go:527
I0823 17:09:22.218528 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349762218237125.36 127.0.0.1:0 17:09:22.218260 147.287µs node server/node.go:486
c349762218237125.36 127.0.0.1:0 17:09:22.218290 84.984µs ·executing RangeLookup storage/store.go:1298
c349762218237125.36 127.0.0.1:0 17:09:22.218297 77.483µs ··read-only path storage/replica.go:527
I0823 17:09:37.393636 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:37.393725 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:37.393812 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:37.393861 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:37.393901 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:48.950411 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
W0823 17:09:48.950481 792 rpc/client.go:233 rpc/client.go:279: heartbeat timed out after 6s
I0823 17:09:48.950537 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:48.950614 792 rpc/clock_offset.go:229 finding offset interval for monitorInterval: 30s, numOffsets 3
I0823 17:09:50.628920 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:50.629013 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:50.629075 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:50.629103 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:50.629133 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:50.629653 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:50.629772 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:50.629945 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:50.630019 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:09:54.182791 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:54.182895 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:54.182983 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:54.183031 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:54.183054 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:09:54.183165 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
W0823 17:09:54.183195 792 rpc/send.go:215 Node.RangeLookup: error reply: rpc/send.go:265: rpc to Node.RangeLookup: client not ready after 5s
W0823 17:09:54.183235 792 kv/dist_sender.go:664 failed to send RPC: too many errors encountered (1 of 1 total): rpc/send.go:265: rpc to Node.RangeLookup: client not ready after 5s
I0823 17:09:56.928771 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349794184524102.78 127.0.0.1:0 17:09:54.184546 2.744074421s node server/node.go:486
c349794184524102.78 127.0.0.1:0 17:09:54.184578 2.743976771s ·executing RangeLookup storage/store.go:1298
c349794184524102.78 127.0.0.1:0 17:09:54.184588 2.743965983s ··read-only path storage/replica.go:527
I0823 17:09:56.929042 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349794184684346.50 127.0.0.1:0 17:09:54.184699 2.744219924s node server/node.go:486
c349794184684346.50 127.0.0.1:0 17:09:54.184719 2.744159351s ·executing RangeLookup storage/store.go:1298
c349794184684346.50 127.0.0.1:0 17:09:54.184724 2.744153506s ··read-only path storage/replica.go:527
I0823 17:09:56.929178 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:09:56.929252 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:09:56.929337 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:09:56.929388 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:09:56.929426 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:00.555306 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:10:00.555402 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:10:00.555911 792 kv/range_cache.go:148 adding descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:10:00.555977 792 kv/range_cache.go:269 clearing overlapping descriptor: key="\x00\x00meta2\xff\xff" desc=range_id:1 start_key:"" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2
I0823 17:10:00.589207 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:00.589366 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:00.589451 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:00.589509 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:00.589555 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:00.589730 792 storage/replica.go:997 gossiping cluster id cluster-1 from store 1, range 1
I0823 17:10:00.589787 792 kv/txn_coord_sender.go:281 txn coordinator: 0.00 txn/sec, 0.00/0.00/0.00 %cmmt/abrt/abnd, 0/0/0 avg/σ/max duration, 0.0/0.0/0.0 avg/σ/max restarts (0 samples)
I0823 17:10:00.590612 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754560574088.27 127.0.0.1:0 17:09:56.930192 3.660038637s node server/node.go:486
c349754560574088.27 127.0.0.1:0 17:09:56.930232 3.659769813s ·executing Merge storage/store.go:1298
c349754560574088.27 127.0.0.1:0 17:09:56.930241 3.659759526s ··read-write path storage/replica.go:530
c349754560574088.27 127.0.0.1:0 17:09:56.930267 11.321µs ···command queue storage/replica.go:687
c349754560574088.27 127.0.0.1:0 17:09:56.930284 3.625987243s ···request leader lease storage/replica.go:395
c349754560574088.27 127.0.0.1:0 17:10:00.556277 33.723415ms ···raft storage/replica.go:728
c349754560574088.27 127.0.0.1:0 17:10:00.589842 145.873µs ····applying Merge storage/replica.go:802
I0823 17:10:00.591269 792 storage/replica.go:1007 gossiping sentinel from store 1, range 1
I0823 17:10:00.591319 792 storage/replica.go:1013 gossiping first range from store 1, range 1
I0823 17:10:00.591628 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754559984503.41 127.0.0.1:0 17:10:00.556973 34.517009ms node server/node.go:486
c349754559984503.41 127.0.0.1:0 17:10:00.557000 34.473123ms ·executing Merge storage/store.go:1298
c349754559984503.41 127.0.0.1:0 17:10:00.557010 34.46371ms ··read-write path storage/replica.go:530
c349754559984503.41 127.0.0.1:0 17:10:00.557016 10.458µs ···command queue storage/replica.go:687
c349754559984503.41 127.0.0.1:0 17:10:00.591103 369.712µs ···raft storage/replica.go:728
c349754559984503.41 127.0.0.1:0 17:10:00.591377 83.452µs ····applying Merge storage/replica.go:802
I0823 17:10:00.592146 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754560714312.86 127.0.0.1:0 17:10:00.556871 35.140043ms node server/node.go:486
c349754560714312.86 127.0.0.1:0 17:10:00.556908 35.088081ms ·executing Put storage/store.go:1298
c349754560714312.86 127.0.0.1:0 17:10:00.556917 35.07858ms ··read-write path storage/replica.go:530
c349754560714312.86 127.0.0.1:0 17:10:00.556923 12.19µs ···command queue storage/replica.go:687
c349754560714312.86 127.0.0.1:0 17:10:00.591705 289.477µs ···raft storage/replica.go:728
c349754560714312.86 127.0.0.1:0 17:10:00.591875 104.505µs ····applying Put storage/replica.go:802
I0823 17:10:00.593018 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754560714312.86 127.0.0.1:0 17:09:14.560728 46.031783137s sending Put kv/txn_coord_sender.go:301
c349754560714312.86 127.0.0.1:0 17:09:14.560736 6.304µs ·meta descriptor lookup kv/dist_sender.go:653
c349754560714312.86 127.0.0.1:0 17:09:14.560745 5.279548515s ·sending RPC kv/dist_sender.go:553
c349754560714312.86 127.0.0.1:0 17:09:14.560767 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
c349754560714312.86 127.0.0.1:0 17:09:19.840303 0 ·send error: rpc.SendError kv/dist_sender.go:679
c349754560714312.86 127.0.0.1:0 17:09:19.840432 0 ·reply error: rpc.SendError kv/dist_sender.go:742
c349754560714312.86 127.0.0.1:0 17:09:20.111989 40.44400797s ·meta descriptor lookup kv/dist_sender.go:653
c349754560714312.86 127.0.0.1:0 17:10:00.556001 36.506656ms ·sending RPC kv/dist_sender.go:553
c349754560714312.86 127.0.0.1:0 17:10:00.556020 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:00.593306 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754560574088.27 127.0.0.1:0 17:09:14.560590 46.032137625s sending Merge kv/txn_coord_sender.go:301
c349754560574088.27 127.0.0.1:0 17:09:14.560598 6.421µs ·meta descriptor lookup kv/dist_sender.go:653
c349754560574088.27 127.0.0.1:0 17:09:14.560608 5.279858015s ·sending RPC kv/dist_sender.go:553
c349754560574088.27 127.0.0.1:0 17:09:14.560631 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
c349754560574088.27 127.0.0.1:0 17:09:19.840476 0 ·send error: rpc.SendError kv/dist_sender.go:679
c349754560574088.27 127.0.0.1:0 17:09:19.840575 0 ·reply error: rpc.SendError kv/dist_sender.go:742
c349754560574088.27 127.0.0.1:0 17:09:20.078473 34.104738824s ·meta descriptor lookup kv/dist_sender.go:653
c349754560574088.27 127.0.0.1:0 17:09:54.732438 19.074µs ·meta descriptor lookup kv/dist_sender.go:653
c349754560574088.27 127.0.0.1:0 17:09:54.732461 5.860264481s ·sending RPC kv/dist_sender.go:553
c349754560574088.27 127.0.0.1:0 17:09:56.929593 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:00.593646 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349754559984503.41 127.0.0.1:0 17:09:14.559999 46.032744237s sending Merge kv/txn_coord_sender.go:301
c349754559984503.41 127.0.0.1:0 17:09:14.560008 7.88µs ·meta descriptor lookup kv/dist_sender.go:653
c349754559984503.41 127.0.0.1:0 17:09:14.560019 5.279929852s ·sending RPC kv/dist_sender.go:553
c349754559984503.41 127.0.0.1:0 17:09:14.560045 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
c349754559984503.41 127.0.0.1:0 17:09:19.839976 0 ·send error: rpc.SendError kv/dist_sender.go:679
c349754559984503.41 127.0.0.1:0 17:09:19.840271 0 ·reply error: rpc.SendError kv/dist_sender.go:742
c349754559984503.41 127.0.0.1:0 17:09:20.055532 40.499902726s ·meta descriptor lookup kv/dist_sender.go:653
c349754559984503.41 127.0.0.1:0 17:10:00.555440 37.301187ms ·sending RPC kv/dist_sender.go:553
c349754559984503.41 127.0.0.1:0 17:10:00.555474 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:00.594776 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:00.594949 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:00.595014 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:00.595056 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:00.595087 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:00.597953 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800593838123.33 127.0.0.1:0 17:10:00.596195 1.435539ms node server/node.go:486
c349800593838123.33 127.0.0.1:0 17:10:00.596230 1.374631ms ·executing Merge storage/store.go:1298
c349800593838123.33 127.0.0.1:0 17:10:00.596238 1.36592ms ··read-write path storage/replica.go:530
c349800593838123.33 127.0.0.1:0 17:10:00.596245 9.263µs ···command queue storage/replica.go:687
c349800593838123.33 127.0.0.1:0 17:10:00.596260 1.343716ms ···raft storage/replica.go:728
c349800593838123.33 127.0.0.1:0 17:10:00.597193 116.659µs ····applying Merge storage/replica.go:802
I0823 17:10:00.598470 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800592545382.80 127.0.0.1:0 17:10:00.596511 1.618294ms node server/node.go:486
c349800592545382.80 127.0.0.1:0 17:10:00.596541 1.561762ms ·executing Put storage/store.go:1298
c349800592545382.80 127.0.0.1:0 17:10:00.596548 1.553607ms ··read-write path storage/replica.go:530
c349800592545382.80 127.0.0.1:0 17:10:00.596554 8.408µs ···command queue storage/replica.go:687
c349800592545382.80 127.0.0.1:0 17:10:00.596571 1.530151ms ···raft storage/replica.go:728
c349800592545382.80 127.0.0.1:0 17:10:00.597331 119.943µs ····applying Put storage/replica.go:802
I0823 17:10:00.598851 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800593674172.72 127.0.0.1:0 17:10:00.596810 1.767407ms node server/node.go:486
c349800593674172.72 127.0.0.1:0 17:10:00.596841 1.710555ms ·executing Merge storage/store.go:1298
c349800593674172.72 127.0.0.1:0 17:10:00.596850 1.700906ms ··read-write path storage/replica.go:530
c349800593674172.72 127.0.0.1:0 17:10:00.596856 10.031µs ···command queue storage/replica.go:687
c349800593674172.72 127.0.0.1:0 17:10:00.596871 1.679474ms ···raft storage/replica.go:728
c349800593674172.72 127.0.0.1:0 17:10:00.597471 119.092µs ····applying Merge storage/replica.go:802
I0823 17:10:01.187678 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:01.187771 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:01.187851 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:01.187897 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:01.187932 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:01.188257 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800593674172.72 127.0.0.1:0 17:10:00.593692 593.739356ms sending Merge kv/txn_coord_sender.go:301
c349800593674172.72 127.0.0.1:0 17:10:00.593700 7.095µs ·meta descriptor lookup kv/dist_sender.go:653
c349800593674172.72 127.0.0.1:0 17:10:00.593711 593.713249ms ·sending RPC kv/dist_sender.go:553
c349800593674172.72 127.0.0.1:0 17:10:00.593735 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:01.188416 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800593838123.33 127.0.0.1:0 17:10:00.593855 594.140114ms sending Merge kv/txn_coord_sender.go:301
c349800593838123.33 127.0.0.1:0 17:10:00.593862 6.755µs ·meta descriptor lookup kv/dist_sender.go:653
c349800593838123.33 127.0.0.1:0 17:10:00.593872 594.117872ms ·sending RPC kv/dist_sender.go:553
c349800593838123.33 127.0.0.1:0 17:10:00.593907 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:01.188560 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349800592545382.80 127.0.0.1:0 17:10:00.592561 595.453805ms sending Put kv/txn_coord_sender.go:301
c349800592545382.80 127.0.0.1:0 17:10:00.592571 9.209µs ·meta descriptor lookup kv/dist_sender.go:653
c349800592545382.80 127.0.0.1:0 17:10:00.592583 595.427877ms ·sending RPC kv/dist_sender.go:553
c349800592545382.80 127.0.0.1:0 17:10:00.592618 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:01.190789 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801188795779.90 127.0.0.1:0 17:10:01.189131 1.457815ms node server/node.go:486
c349801188795779.90 127.0.0.1:0 17:10:01.189196 1.361525ms ·executing Merge storage/store.go:1298
c349801188795779.90 127.0.0.1:0 17:10:01.189206 1.351226ms ··read-write path storage/replica.go:530
c349801188795779.90 127.0.0.1:0 17:10:01.189214 9.922µs ···command queue storage/replica.go:687
c349801188795779.90 127.0.0.1:0 17:10:01.189230 1.327029ms ···raft storage/replica.go:728
c349801188795779.90 127.0.0.1:0 17:10:01.190114 136.218µs ····applying Merge storage/replica.go:802
I0823 17:10:01.191081 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801187461571.95 127.0.0.1:0 17:10:01.189521 1.375152ms node server/node.go:486
c349801187461571.95 127.0.0.1:0 17:10:01.189551 1.318281ms ·executing Merge storage/store.go:1298
c349801187461571.95 127.0.0.1:0 17:10:01.189579 1.289743ms ··read-write path storage/replica.go:530
c349801187461571.95 127.0.0.1:0 17:10:01.189586 10.192µs ···command queue storage/replica.go:687
c349801187461571.95 127.0.0.1:0 17:10:01.189602 1.266172ms ···raft storage/replica.go:728
c349801187461571.95 127.0.0.1:0 17:10:01.190271 122.372µs ····applying Merge storage/replica.go:802
I0823 17:10:01.191377 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801188600603.28 127.0.0.1:0 17:10:01.189832 1.358879ms node server/node.go:486
c349801188600603.28 127.0.0.1:0 17:10:01.189861 1.30321ms ·executing Put storage/store.go:1298
c349801188600603.28 127.0.0.1:0 17:10:01.189869 1.294383ms ··read-write path storage/replica.go:530
c349801188600603.28 127.0.0.1:0 17:10:01.189876 10.331µs ···command queue storage/replica.go:687
c349801188600603.28 127.0.0.1:0 17:10:01.189894 1.268329ms ···raft storage/replica.go:728
c349801188600603.28 127.0.0.1:0 17:10:01.190415 127.429µs ····applying Put storage/replica.go:802
I0823 17:10:02.450843 792 server/server.go:267 recorded status for 3 stores
I0823 17:10:02.451139 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:02.451221 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:02.451302 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:02.451351 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:02.451399 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:02.451625 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801188600603.28 127.0.0.1:0 17:10:01.188616 1.26218895s sending Put kv/txn_coord_sender.go:301
c349801188600603.28 127.0.0.1:0 17:10:01.188625 6.986µs ·meta descriptor lookup kv/dist_sender.go:653
c349801188600603.28 127.0.0.1:0 17:10:01.188635 1.262164178s ·sending RPC kv/dist_sender.go:553
c349801188600603.28 127.0.0.1:0 17:10:01.188667 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:02.451763 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801188795779.90 127.0.0.1:0 17:10:01.188813 1.262282068s sending Merge kv/txn_coord_sender.go:301
c349801188795779.90 127.0.0.1:0 17:10:01.188821 6.868µs ·meta descriptor lookup kv/dist_sender.go:653
c349801188795779.90 127.0.0.1:0 17:10:01.188831 1.262260228s ·sending RPC kv/dist_sender.go:553
c349801188795779.90 127.0.0.1:0 17:10:01.188855 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:02.451867 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349801187461571.95 127.0.0.1:0 17:10:01.187486 1.263621447s sending Merge kv/txn_coord_sender.go:301
c349801187461571.95 127.0.0.1:0 17:10:01.187498 11.802µs ·meta descriptor lookup kv/dist_sender.go:653
c349801187461571.95 127.0.0.1:0 17:10:01.187513 1.263592225s ·sending RPC kv/dist_sender.go:553
c349801187461571.95 127.0.0.1:0 17:10:01.187551 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:03.077857 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:03.077959 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:03.078026 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:03.078068 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:03.078110 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:03.079846 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802452040074.45 127.0.0.1:0 17:10:02.453344 626.281826ms node server/node.go:486
c349802452040074.45 127.0.0.1:0 17:10:02.453395 626.196601ms ·executing Merge storage/store.go:1298
c349802452040074.45 127.0.0.1:0 17:10:02.453405 626.186522ms ··read-write path storage/replica.go:530
c349802452040074.45 127.0.0.1:0 17:10:02.453412 9.128µs ···command queue storage/replica.go:687
c349802452040074.45 127.0.0.1:0 17:10:02.453425 625.352459ms ···request leader lease storage/replica.go:395
c349802452040074.45 127.0.0.1:0 17:10:03.078785 804.751µs ···raft storage/replica.go:728
c349802452040074.45 127.0.0.1:0 17:10:03.079297 135.828µs ····applying Merge storage/replica.go:802
I0823 17:10:03.080565 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802450919460.69 127.0.0.1:0 17:10:02.453461 626.849142ms node server/node.go:486
c349802450919460.69 127.0.0.1:0 17:10:02.453483 626.80025ms ·executing Put storage/store.go:1298
c349802450919460.69 127.0.0.1:0 17:10:02.453489 626.793542ms ··read-write path storage/replica.go:530
c349802450919460.69 127.0.0.1:0 17:10:02.453494 7.428µs ···command queue storage/replica.go:687
c349802450919460.69 127.0.0.1:0 17:10:03.078985 1.296664ms ···raft storage/replica.go:728
c349802450919460.69 127.0.0.1:0 17:10:03.079456 121.719µs ····applying Put storage/replica.go:802
I0823 17:10:03.081115 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802451899120.34 127.0.0.1:0 17:10:02.453519 627.383784ms node server/node.go:486
c349802451899120.34 127.0.0.1:0 17:10:02.453540 627.300622ms ·executing Merge storage/store.go:1298
c349802451899120.34 127.0.0.1:0 17:10:02.453545 627.294316ms ··read-write path storage/replica.go:530
c349802451899120.34 127.0.0.1:0 17:10:02.453550 6.291µs ···command queue storage/replica.go:687
c349802451899120.34 127.0.0.1:0 17:10:03.079971 868.05µs ···raft storage/replica.go:728
c349802451899120.34 127.0.0.1:0 17:10:03.080669 158.206µs ····applying Merge storage/replica.go:802
I0823 17:10:04.242541 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:04.242633 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:04.242707 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:04.242758 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:04.242803 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:04.243073 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802451899120.34 127.0.0.1:0 17:10:02.451915 1.790391506s sending Merge kv/txn_coord_sender.go:301
c349802451899120.34 127.0.0.1:0 17:10:02.451923 7.22µs ·meta descriptor lookup kv/dist_sender.go:653
c349802451899120.34 127.0.0.1:0 17:10:02.451934 1.790366073s ·sending RPC kv/dist_sender.go:553
c349802451899120.34 127.0.0.1:0 17:10:02.451953 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.243208 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802452040074.45 127.0.0.1:0 17:10:02.452050 1.790467113s sending Merge kv/txn_coord_sender.go:301
c349802452040074.45 127.0.0.1:0 17:10:02.452055 6.131µs ·meta descriptor lookup kv/dist_sender.go:653
c349802452040074.45 127.0.0.1:0 17:10:02.452065 1.790449068s ·sending RPC kv/dist_sender.go:553
c349802452040074.45 127.0.0.1:0 17:10:02.452093 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.243323 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349802450919460.69 127.0.0.1:0 17:10:02.450935 1.791596645s sending Put kv/txn_coord_sender.go:301
c349802450919460.69 127.0.0.1:0 17:10:02.450943 8.169µs ·meta descriptor lookup kv/dist_sender.go:653
c349802450919460.69 127.0.0.1:0 17:10:02.450953 1.791575604s ·sending RPC kv/dist_sender.go:553
c349802450919460.69 127.0.0.1:0 17:10:02.450980 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.243370 792 server/server.go:256 recorded status for node 1
I0823 17:10:04.245621 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804243571978.21 127.0.0.1:0 17:10:04.243910 1.553778ms node server/node.go:486
c349804243571978.21 127.0.0.1:0 17:10:04.243944 1.497584ms ·executing Merge storage/store.go:1298
c349804243571978.21 127.0.0.1:0 17:10:04.243951 1.490018ms ··read-write path storage/replica.go:530
c349804243571978.21 127.0.0.1:0 17:10:04.243956 7.829µs ···command queue storage/replica.go:687
c349804243571978.21 127.0.0.1:0 17:10:04.243968 904.416µs ···request leader lease storage/replica.go:395
c349804243571978.21 127.0.0.1:0 17:10:04.244876 564.432µs ···raft storage/replica.go:728
c349804243571978.21 127.0.0.1:0 17:10:04.245223 96.305µs ····applying Merge storage/replica.go:802
I0823 17:10:04.246032 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804242330529.58 127.0.0.1:0 17:10:04.244449 1.445158ms node server/node.go:486
c349804242330529.58 127.0.0.1:0 17:10:04.244480 1.395307ms ·executing Merge storage/store.go:1298
c349804242330529.58 127.0.0.1:0 17:10:04.244488 1.387633ms ··read-write path storage/replica.go:530
c349804242330529.58 127.0.0.1:0 17:10:04.244492 7.717µs ···command queue storage/replica.go:687
c349804242330529.58 127.0.0.1:0 17:10:04.245031 843.415µs ···raft storage/replica.go:728
c349804242330529.58 127.0.0.1:0 17:10:04.245333 100.183µs ····applying Merge storage/replica.go:802
I0823 17:10:04.246401 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804243412151.97 127.0.0.1:0 17:10:04.244515 1.747832ms node server/node.go:486
c349804243412151.97 127.0.0.1:0 17:10:04.244533 1.714675ms ·executing Put storage/store.go:1298
c349804243412151.97 127.0.0.1:0 17:10:04.244537 1.709703ms ··read-write path storage/replica.go:530
c349804243412151.97 127.0.0.1:0 17:10:04.244541 5.306µs ···command queue storage/replica.go:687
c349804243412151.97 127.0.0.1:0 17:10:04.245710 536.422µs ···raft storage/replica.go:728
c349804243412151.97 127.0.0.1:0 17:10:04.246133 95.044µs ····applying Put storage/replica.go:802
I0823 17:10:04.246782 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804243412151.97 127.0.0.1:0 17:10:04.243424 3.112832ms sending Put kv/txn_coord_sender.go:301
c349804243412151.97 127.0.0.1:0 17:10:04.243429 5.247µs ·meta descriptor lookup kv/dist_sender.go:653
c349804243412151.97 127.0.0.1:0 17:10:04.243437 3.097093ms ·sending RPC kv/dist_sender.go:553
c349804243412151.97 127.0.0.1:0 17:10:04.243455 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.246920 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804243571978.21 127.0.0.1:0 17:10:04.243584 3.089466ms sending Merge kv/txn_coord_sender.go:301
c349804243571978.21 127.0.0.1:0 17:10:04.243593 5.158µs ·meta descriptor lookup kv/dist_sender.go:653
c349804243571978.21 127.0.0.1:0 17:10:04.243600 3.070349ms ·sending RPC kv/dist_sender.go:553
c349804243571978.21 127.0.0.1:0 17:10:04.243620 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.247059 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804242330529.58 127.0.0.1:0 17:10:04.242346 4.344354ms sending Merge kv/txn_coord_sender.go:301
c349804242330529.58 127.0.0.1:0 17:10:04.242354 10.188µs ·meta descriptor lookup kv/dist_sender.go:653
c349804242330529.58 127.0.0.1:0 17:10:04.242367 4.321456ms ·sending RPC kv/dist_sender.go:553
c349804242330529.58 127.0.0.1:0 17:10:04.242394 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.786633 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue gc: replica shouldn't be added to queue
I0823 17:10:04.786727 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue split: replica shouldn't be added to queue
I0823 17:10:04.786823 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue verify: replica shouldn't be added to queue
I0823 17:10:04.786883 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue replicate: replica shouldn't be added to queue
I0823 17:10:04.786924 792 storage/queue.go:177 couldn't add range=1 [""-"\xff\xff") to queue rangeGC: replica shouldn't be added to queue
I0823 17:10:04.788159 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804247090884.35 127.0.0.1:0 17:10:04.248298 539.433892ms node server/node.go:486
c349804247090884.35 127.0.0.1:0 17:10:04.248327 539.178945ms ·executing Merge storage/store.go:1298
c349804247090884.35 127.0.0.1:0 17:10:04.248334 539.170853ms ··read-write path storage/replica.go:530
c349804247090884.35 127.0.0.1:0 17:10:04.248341 8.43µs ···command queue storage/replica.go:687
c349804247090884.35 127.0.0.1:0 17:10:04.248355 539.149224ms ···raft storage/replica.go:728
c349804247090884.35 127.0.0.1:0 17:10:04.787352 137.151µs ····applying Merge storage/replica.go:802
I0823 17:10:04.788700 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804247208663.68 127.0.0.1:0 17:10:04.248084 540.377931ms node server/node.go:486
c349804247208663.68 127.0.0.1:0 17:10:04.248138 540.299108ms ·executing Merge storage/store.go:1298
c349804247208663.68 127.0.0.1:0 17:10:04.248147 540.289628ms ··read-write path storage/replica.go:530
c349804247208663.68 127.0.0.1:0 17:10:04.248154 7.821µs ···command queue storage/replica.go:687
c349804247208663.68 127.0.0.1:0 17:10:04.248167 540.268483ms ···raft storage/replica.go:728
c349804247208663.68 127.0.0.1:0 17:10:04.787846 361.681µs ····applying Merge storage/replica.go:802
I0823 17:10:04.788894 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804246555338.56 127.0.0.1:0 17:10:04.248205 540.309216ms node server/node.go:486
c349804246555338.56 127.0.0.1:0 17:10:04.248235 540.171852ms ·executing Put storage/store.go:1298
c349804246555338.56 127.0.0.1:0 17:10:04.248243 540.162907ms ··read-write path storage/replica.go:530
c349804246555338.56 127.0.0.1:0 17:10:04.248250 9.199µs ···command queue storage/replica.go:687
c349804246555338.56 127.0.0.1:0 17:10:04.248270 540.135712ms ···raft storage/replica.go:728
c349804246555338.56 127.0.0.1:0 17:10:04.788230 150.269µs ····applying Put storage/replica.go:802
I0823 17:10:04.789394 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804246555338.56 127.0.0.1:0 17:10:04.246568 542.497222ms sending Put kv/txn_coord_sender.go:301
c349804246555338.56 127.0.0.1:0 17:10:04.246574 4.848µs ·meta descriptor lookup kv/dist_sender.go:653
c349804246555338.56 127.0.0.1:0 17:10:04.246581 542.48086ms ·sending RPC kv/dist_sender.go:553
c349804246555338.56 127.0.0.1:0 17:10:04.246597 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.789498 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804247090884.35 127.0.0.1:0 17:10:04.247105 542.167788ms sending Merge kv/txn_coord_sender.go:301
c349804247090884.35 127.0.0.1:0 17:10:04.247111 5.069µs ·meta descriptor lookup kv/dist_sender.go:653
c349804247090884.35 127.0.0.1:0 17:10:04.247118 542.152304ms ·sending RPC kv/dist_sender.go:553
c349804247090884.35 127.0.0.1:0 17:10:04.247135 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
I0823 17:10:04.789615 792 server/status/monitor.go:218 received trace:
Name Origin Ts Dur Desc File
c349804247208663.68 127.0.0.1:0 17:10:04.247226 542.05958ms sending Merge kv/txn_coord_sender.go:301
c349804247208663.68 127.0.0.1:0 17:10:04.247234 6.201µs ·meta descriptor lookup kv/dist_sender.go:653
c349804247208663.68 127.0.0.1:0 17:10:04.247242 542.041146ms ·sending RPC kv/dist_sender.go:553
c349804247208663.68 127.0.0.1:0 17:10:04.247259 0 ··sending to 127.0.0.1:46564 rpc/send.go:171
panic: test timed out after 1m10s
goroutine 2793 [running]:
testing.startAlarm.func1()
/usr/local/go/src/testing/testing.go:703 +0x132
created by time.goFunc
/usr/local/go/src/time/sleep.go:129 +0x3a
goroutine 1 [chan receive, 1 minutes]:
testing.RunTests(0x137e7a8, 0x1909ca0, 0x1e, 0x1e, 0x7de701)
/usr/local/go/src/testing/testing.go:562 +0x8ad
testing.(*M).Run(0xc820037f08, 0x7decc2)
/usr/local/go/src/testing/testing.go:494 +0x70
github.com/cockroachdb/cockroach/util/leaktest.TestMainWithLeakCheck(0xc820037f08)
/go/src/github.com/cockroachdb/cockroach/util/leaktest/leaktest.go:34 +0x21
github.com/cockroachdb/cockroach/server_test.TestMain(0xc820037f08)
/go/src/github.com/cockroachdb/cockroach/server/main_test.go:35 +0x21
main.main()
github.com/cockroachdb/cockroach/server/_test/_testmain.go:112 +0x113
goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 5 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x19b22a0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1020 +0x64
created by github.com/cockroachdb/cockroach/util/log.init.1
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:607 +0xbc
goroutine 2636 [select]:
github.com/cockroachdb/cockroach/rpc.Send(0x1, 0x1, 0x1dcd6500, 0x12a05f200, 0xc820440910, 0xc8207cb820, 0x8, 0xc8207cb800, 0x1, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/rpc/send.go:189 +0x176c
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc8203192d0, 0xc820440910, 0x1, 0xc8204409b0, 0x1, 0x1, 0x1, 0x7f3ba7dfcbf8, 0xc8207ccf00, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:478 +0x784
github.com/cockroachdb/cockroach/kv.(*DistSender).sendAttempt(0xc8203192d0, 0xc820440910, 0x7f3ba7dfcbf8, 0xc8207ccf00, 0xc820213020, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:589 +0x3a0
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc8203192d0, 0x7f3ba7df80c8, 0xc8207d2750, 0x7f3ba7dfcbf8, 0xc8207ccf00, 0x7f3ba7dff790, 0xc820236be0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:671 +0xf63
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendOne(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2750, 0x7f3ba7dfcbf8, 0xc8207ccf00, 0x7f3ba7dff790, 0xc820236be0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:428 +0x7bd
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2750, 0x7f3ba7dfcbf8, 0xc8207ccf00, 0x7f3ba7dff790, 0xc820236be0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:317 +0x685
github.com/cockroachdb/cockroach/client.(*DB).send(0xc8204eee40, 0xc82028f500, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:447 +0x4ba
github.com/cockroachdb/cockroach/client.(*DB).Run(0xc8204eee40, 0xc8207b8400, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:405 +0x92
github.com/cockroachdb/cockroach/client.runOneResult(0x7f3ba7df7fb8, 0xc8204eee40, 0xc8207b8400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/client/db.go:497 +0x61
github.com/cockroachdb/cockroach/client.(*DB).Put(0xc8204eee40, 0x116e560, 0xc820236bc0, 0x11a3460, 0xc82000d0e0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:279 +0x108
github.com/cockroachdb/cockroach/server.(*Server).writeSummaries(0xc8200a0090, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/server.go:262 +0x419
github.com/cockroachdb/cockroach/server.(*Server).startWriteSummaries.func1.1()
/go/src/github.com/cockroachdb/cockroach/server/server.go:235 +0x24
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc820455eb8, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/server.(*Server).startWriteSummaries.func1()
/go/src/github.com/cockroachdb/cockroach/server/server.go:238 +0x11d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211e60)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2641 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:145 +0x9eb
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2fc0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2616 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1545 +0xeab
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820210e80)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2623 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc8205b2ea0)
/go/src/github.com/coreos/etcd/raft/multinode.go:180 +0x23f2
created by github.com/coreos/etcd/raft.StartMultiNode
/go/src/github.com/coreos/etcd/raft/multinode.go:56 +0x30d
goroutine 2609 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adba0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2604 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc820489800)
/go/src/github.com/coreos/etcd/raft/multinode.go:180 +0x23f2
created by github.com/coreos/etcd/raft.StartMultiNode
/go/src/github.com/coreos/etcd/raft/multinode.go:56 +0x30d
goroutine 2620 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:606 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a20c0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2611 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adbe0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2566 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8200963c0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:215 +0xca5
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:202 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8204159f0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2586 [runnable]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func1()
/go/src/github.com/cockroachdb/cockroach/gossip/server.go:161 +0x12e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205b7a40)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2600 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:587 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205ac740)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2638 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func3()
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:268 +0x4b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211f90)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2738 [IO wait]:
net.runtime_pollWait(0x7f3ba7df7b68, 0x72, 0xc8200121a0)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820197020, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820197020, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820196fc0, 0xc820365000, 0x800, 0x800, 0x0, 0x7f3ba7df1050, 0xc8200121a0)
/usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200289d0, 0xc820365000, 0x800, 0x800, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82037b800, 0x7f3ba7dfe480, 0xc8200289d0, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc820132b00, 0x137ee17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc820132b00, 0xc8204da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:901 +0x167
bufio.(*Reader).fill(0xc8203e5080)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadByte(0xc8203e5080, 0x7dd017, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:229 +0x7a
encoding/binary.ReadUvarint(0x7f3ba7dfebc0, 0xc8203e5080, 0x0, 0x0, 0x0)
/usr/local/go/src/encoding/binary/varint.go:110 +0x52
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc820404f20, 0x7f3ba7dfece0, 0xc820405058, 0x0, 0x137d830, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:91 +0x6d
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).readResponseHeader(0xc820404f20, 0xc820405058, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:155 +0x7d
github.com/cockroachdb/cockroach/rpc/codec.(*clientCodec).ReadResponseHeader(0xc820404f20, 0xc8202cee70, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/client.go:80 +0x4b
net/rpc.(*Client).input(0xc8203e50e0)
/usr/local/go/src/net/rpc/client.go:109 +0xbf
created by net/rpc.NewClientWithCodec
/usr/local/go/src/net/rpc/client.go:201 +0xd2
goroutine 2625 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1545 +0xeab
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211820)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2632 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startPublishStatuses.func1()
/go/src/github.com/cockroachdb/cockroach/server/node.go:456 +0x23e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2520)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2601 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:606 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205ac780)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2596 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.VDepth(0xc800000006, 0x1, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1143 +0x8a
github.com/cockroachdb/cockroach/util/log.V(0xc800000006, 0xc820488720)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:181 +0x28
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:522 +0x101f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820210950)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2637 [IO wait]:
net.runtime_pollWait(0x7f3ba7e099f8, 0x72, 0xc8200121a0)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820196760, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820196760, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820196700, 0x0, 0x7f3ba7df8288, 0xc8205a3d40)
/usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820028480, 0x19b2d60, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820028480, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
crypto/tls.(*listener).Accept(0xc8205b79e0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/tls.go:46 +0x60
net/http.(*Server).Serve(0xc82045a1e0, 0x7f3ba7dfcaf0, 0xc8205b79e0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1887 +0xb3
github.com/cockroachdb/cockroach/rpc.(*Server).Serve.func2()
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:262 +0x3b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2ac0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2639 [select]:
github.com/cockroachdb/cockroach/rpc.(*Client).runHeartbeat(0xc82031ccb0, 0x3b9aca00, 0x6fc23ac00, 0x4000000000000000, 0x0, 0x0, 0xc8204ee120, 0xc8204ee060)
/go/src/github.com/cockroachdb/cockroach/rpc/client.go:239 +0x538
github.com/cockroachdb/cockroach/rpc.NewClient.func1()
/go/src/github.com/cockroachdb/cockroach/rpc/client.go:134 +0x4f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc82036fb80)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2644 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3280)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2790 [chan send]:
github.com/cockroachdb/cockroach/storage.(*Store).ProposeRaftCommand(0xc8205d4a80, 0xc8207e53a0, 0x10, 0x1, 0x100000001, 0x0, 0xc8207ee180, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1513 +0x1db
github.com/cockroachdb/cockroach/storage.(*Replica).proposeRaftCommand(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3260, 0x7f3ba7dfcbf8, 0xc8207ee180, 0x11, 0x20)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:772 +0x839
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3260, 0x7f3ba7dfcbf8, 0xc8207ee180, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:730 +0x5ad
github.com/cockroachdb/cockroach/storage.(*Replica).AddCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3260, 0x7f3ba7dfcbf8, 0xc8207ee180, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:531 +0x328
github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc8205d4a80, 0x7f3ba7df80c8, 0xc8207d3260, 0x7f3ba7dfcbf8, 0xc8207ee180, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1318 +0x929
github.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc8205ce900, 0x7f3ba7df80c8, 0xc8207d3200, 0x7f3ba7dfcbf8, 0xc8207ee180, 0x7f3ba7dff790, 0xc82058e4c0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:162 +0x562
github.com/cockroachdb/cockroach/server.(*Node).executeCmd(0xc820316120, 0x7f3ba7dfd208, 0xc8207ee180, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:489 +0x48e
github.com/cockroachdb/cockroach/server.(*Node).(github.com/cockroachdb/cockroach/server.executeCmd)-fm(0x7f3ba7dfd208, 0xc8207ee180, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:252 +0x50
github.com/cockroachdb/cockroach/rpc.syncAdapter.exec.func1(0xc82028f940, 0xc82043a1e0, 0x7f3ba7dfd208, 0xc8207ee180)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:53 +0x30
created by github.com/cockroachdb/cockroach/rpc.syncAdapter.exec
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:54 +0x53
goroutine 2684 [IO wait]:
net.runtime_pollWait(0x7f3ba7df7268, 0x72, 0xc8200121a0)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820281f00, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820281f00, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820281ea0, 0xc820578000, 0x2000, 0x2000, 0x0, 0x7f3ba7df1050, 0xc8200121a0)
/usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200287b8, 0xc820578000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82037b560, 0x7f3ba7dfe480, 0xc8200287b8, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8201ce580, 0x137ee17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8201ce580, 0xc82039e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7f3ba7dfe368, 0xc8201ce580, 0xc8200ece18, 0xc82039e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820276d40, 0xc82039e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
<autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8201cc600)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8201cc600, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200ecdc0)
/usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:685 +0xc78
goroutine 2643 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3260)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2646 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a32e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2645 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a32c0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2651 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a36e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2628 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:587 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2200)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2647 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.VDepth(0x6, 0x1, 0xc82f38ef00)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1143 +0x8a
github.com/cockroachdb/cockroach/util/log.V(0xc800000006, 0x2f38efe1)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:181 +0x28
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.2()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:230 +0x11c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc82003bf60, 0x2ee85161)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:236 +0x188
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3320)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2653 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3760)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2652 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3700)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2634 [select]:
github.com/cockroachdb/cockroach/rpc.Send(0x1, 0x1, 0x1dcd6500, 0x12a05f200, 0xc820440b90, 0xc8207e4a90, 0xa, 0xc8207e4a70, 0x1, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/rpc/send.go:189 +0x176c
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc8203192d0, 0xc820440b90, 0x1, 0xc820440be0, 0x1, 0x1, 0x1, 0x7f3ba7dfd120, 0xc8207ee000, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:478 +0x784
github.com/cockroachdb/cockroach/kv.(*DistSender).sendAttempt(0xc8203192d0, 0xc820440b90, 0x7f3ba7dfd120, 0xc8207ee000, 0xc820213020, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:589 +0x3a0
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc8203192d0, 0x7f3ba7df80c8, 0xc8207d2bd0, 0x7f3ba7dfd120, 0xc8207ee000, 0x7f3ba7e1d2b8, 0xc82058ece0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:671 +0xf63
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendOne(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2bd0, 0x7f3ba7dfd120, 0xc8207ee000, 0x7f3ba7e1d2b8, 0xc82058ece0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:428 +0x7bd
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2bd0, 0x7f3ba7dfd120, 0xc8207ee000, 0x7f3ba7e1d2b8, 0xc82058ece0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:317 +0x685
github.com/cockroachdb/cockroach/client.(*DB).send(0xc8204eee40, 0xc82028f740, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:447 +0x4ba
github.com/cockroachdb/cockroach/client.(*DB).Run(0xc8204eee40, 0xc8207b8800, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:405 +0x92
github.com/cockroachdb/cockroach/ts.(*DB).StoreData(0xc8200280d8, 0x1, 0xc820285500, 0xa, 0x10, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/ts/db.go:145 +0x6fa
github.com/cockroachdb/cockroach/ts.(*poller).poll.func1()
/go/src/github.com/cockroachdb/cockroach/ts/db.go:96 +0xac
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc820587e70, 0xc8201eaeb8)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/ts.(*poller).poll(0xc8203298f0)
/go/src/github.com/cockroachdb/cockroach/ts/db.go:99 +0x3d
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
/go/src/github.com/cockroachdb/cockroach/ts/db.go:79 +0xf5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211e40)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2792 [chan receive]:
github.com/cockroachdb/cockroach/storage.(*Store).ProposeRaftCommand(0xc8205d4a80, 0xc8207e5140, 0x10, 0x1, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1513 +0x218
github.com/cockroachdb/cockroach/storage.(*Replica).proposeRaftCommand(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3020, 0x7f3ba7dfd120, 0xc8207ee300, 0x0, 0x20)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:772 +0x839
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3020, 0x7f3ba7dfd120, 0xc8207ee300, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:730 +0x5ad
github.com/cockroachdb/cockroach/storage.(*Replica).AddCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d3020, 0x7f3ba7dfd120, 0xc8207ee300, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:531 +0x328
github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc8205d4a80, 0x7f3ba7df80c8, 0xc8207d3020, 0x7f3ba7dfd120, 0xc8207ee300, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1318 +0x929
github.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc8205ce900, 0x7f3ba7df80c8, 0xc8207d2fc0, 0x7f3ba7dfd120, 0xc8207ee300, 0x7f3ba7e1d2b8, 0xc82058e240, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:162 +0x562
github.com/cockroachdb/cockroach/server.(*Node).executeCmd(0xc820316120, 0x7f3ba7dfd588, 0xc8207ee300, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:489 +0x48e
github.com/cockroachdb/cockroach/server.(*Node).(github.com/cockroachdb/cockroach/server.executeCmd)-fm(0x7f3ba7dfd588, 0xc8207ee300, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:252 +0x50
github.com/cockroachdb/cockroach/rpc.syncAdapter.exec.func1(0xc82028f9c0, 0xc82043a880, 0x7f3ba7dfd588, 0xc8207ee300)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:53 +0x30
created by github.com/cockroachdb/cockroach/rpc.syncAdapter.exec
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:54 +0x53
goroutine 2654 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.VDepth(0x6, 0x1, 0xc82f328800)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1143 +0x8a
github.com/cockroachdb/cockroach/util/log.V(0xc800000006, 0x2f32889b)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:181 +0x28
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.2()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:230 +0x11c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc820466f60, 0x2ee29dbc)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:236 +0x188
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a37a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2683 [syscall]:
syscall.Syscall(0x4a, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Fsync(0x3, 0x0, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:492 +0x4a
os.(*File).Sync(0xc820028020, 0x0, 0x0)
/usr/local/go/src/os/file_posix.go:121 +0x61
github.com/cockroachdb/cockroach/util/log.(*syncBuffer).Sync(0xc8201aa150, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:911 +0x31
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushAll(0x19b22a0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1040 +0x96
github.com/cockroachdb/cockroach/util/log.(*loggingT).lockAndFlushAll(0x19b22a0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1028 +0x3b
github.com/cockroachdb/cockroach/util/log.Flush()
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:612 +0x23
github.com/cockroachdb/cockroach/server.(*statusServer).handleLogsLocal(0xc8204cbcb0, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0, 0xc82058e120, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/server/status.go:393 +0x27
github.com/cockroachdb/cockroach/server.(*statusServer).handleLogs(0xc8204cbcb0, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0, 0xc82058e120, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/server/status.go:489 +0x115
github.com/cockroachdb/cockroach/server.(*statusServer).(github.com/cockroachdb/cockroach/server.handleLogs)-fm(0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0, 0xc82058e120, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/server/status.go:135 +0x5c
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc8205b9a40, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0)
/go/src/github.com/julienschmidt/httprouter/router.go:299 +0x193
github.com/cockroachdb/cockroach/server.(*statusServer).ServeHTTP(0xc8204cbcb0, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0)
/go/src/github.com/cockroachdb/cockroach/server/status.go:147 +0x43
net/http.(*ServeMux).ServeHTTP(0xc8204cb6e0, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0)
/usr/local/go/src/net/http/server.go:1699 +0x17d
github.com/cockroachdb/cockroach/server.(*Server).ServeHTTP.func1()
/go/src/github.com/cockroachdb/cockroach/server/server.go:298 +0x206
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc82058b8e8, 0xf)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/server.(*Server).ServeHTTP(0xc8200a0090, 0x7f3ba7e04188, 0xc82058e100, 0xc82079b0a0)
/go/src/github.com/cockroachdb/cockroach/server/server.go:299 +0x56
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc8201dc080, 0x7f3ba7dfea88, 0xc8207c8a50, 0xc82079b0a0)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:160 +0xd91
net/http.serverHandler.ServeHTTP(0xc82045a1e0, 0x7f3ba7dfea88, 0xc8207c8a50, 0xc82079b0a0)
/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8200ed1e0)
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6
goroutine 2791 [chan send]:
github.com/cockroachdb/cockroach/storage.(*Store).ProposeRaftCommand(0xc8205d4a80, 0xc8207e55c0, 0x10, 0x1, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1513 +0x1db
github.com/cockroachdb/cockroach/storage.(*Replica).proposeRaftCommand(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d34d0, 0x7f3ba7dfd120, 0xc8207ee240, 0x0, 0x20)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:772 +0x839
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d34d0, 0x7f3ba7dfd120, 0xc8207ee240, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:730 +0x5ad
github.com/cockroachdb/cockroach/storage.(*Replica).AddCmd(0xc8200a1e60, 0x7f3ba7df80c8, 0xc8207d34d0, 0x7f3ba7dfd120, 0xc8207ee240, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:531 +0x328
github.com/cockroachdb/cockroach/storage.(*Store).ExecuteCmd(0xc8205d4a80, 0x7f3ba7df80c8, 0xc8207d34d0, 0x7f3ba7dfd120, 0xc8207ee240, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1318 +0x929
github.com/cockroachdb/cockroach/kv.(*LocalSender).Send(0xc8205ce900, 0x7f3ba7df80c8, 0xc8207d3470, 0x7f3ba7dfd120, 0xc8207ee240, 0x7f3ba7e1d2b8, 0xc82058e7c0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/local_sender.go:162 +0x562
github.com/cockroachdb/cockroach/server.(*Node).executeCmd(0xc820316120, 0x7f3ba7dfd588, 0xc8207ee240, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:489 +0x48e
github.com/cockroachdb/cockroach/server.(*Node).(github.com/cockroachdb/cockroach/server.executeCmd)-fm(0x7f3ba7dfd588, 0xc8207ee240, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:252 +0x50
github.com/cockroachdb/cockroach/rpc.syncAdapter.exec.func1(0xc82028f980, 0xc82043a880, 0x7f3ba7dfd588, 0xc8207ee240)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:53 +0x30
created by github.com/cockroachdb/cockroach/rpc.syncAdapter.exec
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:54 +0x53
goroutine 2615 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.VDepth(0xc800000008, 0x1, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1143 +0x8a
github.com/cockroachdb/cockroach/util/log.V(0xc800000008, 0xc8204442a0)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:181 +0x28
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:561 +0x1820
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820210e50)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2642 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3200)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2649 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3680)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2564 [select]:
github.com/cockroachdb/cockroach/rpc.(*RemoteClockMonitor).MonitorRemoteOffsets(0xc8205ce7a0, 0xc8204ee120)
/go/src/github.com/cockroachdb/cockroach/rpc/clock_offset.go:158 +0x741
github.com/cockroachdb/cockroach/server.NewServer.func1()
/go/src/github.com/cockroachdb/cockroach/server/server.go:113 +0x2d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205ce7e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2737 [chan receive]:
github.com/cockroachdb/cockroach/rpc.(*Server).sendResponses(0xc8201dc080, 0x7f3ba7dfeb48, 0xc8205d5e00, 0xc8203e5020)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:418 +0x98
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP.func1(0xc8201dc080, 0x7f3ba7dfeb48, 0xc8205d5e00, 0xc8203e5020, 0xc82057b1c0)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:193 +0x3f
created by github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:195 +0x860
goroutine 2650 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a36a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2648 [select, 1 minutes]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:145 +0x9eb
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a3440)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2591 [select]:
github.com/coreos/etcd/raft.(*multiNode).run(0xc820583c80)
/go/src/github.com/coreos/etcd/raft/multinode.go:180 +0x23f2
created by github.com/coreos/etcd/raft.StartMultiNode
/go/src/github.com/coreos/etcd/raft/multinode.go:56 +0x30d
goroutine 2597 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1545 +0xeab
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820210970)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2734 [IO wait]:
net.runtime_pollWait(0x7f3ba7e096f8, 0x72, 0xc8200121a0)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820197100, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820197100, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201970a0, 0xc82037d400, 0x400, 0x400, 0x0, 0x7f3ba7df1050, 0xc8200121a0)
/usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200289d8, 0xc82037d400, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82042f5f0, 0x7f3ba7dfe480, 0xc8200289d8, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc820133080, 0x137ee17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc820133080, 0xc82044c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:901 +0x167
bufio.(*Reader).fill(0xc8203e4fc0)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadByte(0xc8203e4fc0, 0xff6, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:229 +0x7a
encoding/binary.ReadUvarint(0x7f3ba7dfebc0, 0xc8203e4fc0, 0xff6, 0x0, 0x0)
/usr/local/go/src/encoding/binary/varint.go:110 +0x52
github.com/cockroachdb/cockroach/rpc/codec.(*baseConn).recvProto(0xc8205d5e00, 0x7f3ba7dfeb88, 0xc8205d5f50, 0x7f3b00000000, 0x137d830, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/conn.go:91 +0x6d
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).readRequestHeader(0xc8205d5e00, 0xc8203e4fc0, 0xc8205d5f50, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:173 +0x7d
github.com/cockroachdb/cockroach/rpc/codec.(*serverCodec).ReadRequestHeader(0xc8205d5e00, 0xc82058e220, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/codec/server.go:60 +0x65
github.com/cockroachdb/cockroach/rpc.(*Server).readRequest(0xc8201dc080, 0x7f3ba7dfeb48, 0xc8205d5e00, 0xc820428da0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:391 +0xc5
github.com/cockroachdb/cockroach/rpc.(*Server).readRequests(0xc8201dc080, 0x7f3ba7dfeb48, 0xc8205d5e00, 0xc820428da0, 0xc8203e5020)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:359 +0xdc
github.com/cockroachdb/cockroach/rpc.(*Server).ServeHTTP(0xc8201dc080, 0x7f3ba7dfea88, 0xc8200ec4d0, 0xc8203880e0)
/go/src/github.com/cockroachdb/cockroach/rpc/server.go:196 +0x8a2
net/http.serverHandler.ServeHTTP(0xc82045a1e0, 0x7f3ba7dfea88, 0xc8200ec4d0, 0xc8203880e0)
/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc82012afd0)
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6
goroutine 2624 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.VDepth(0xc800000008, 0x1, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1143 +0x8a
github.com/cockroachdb/cockroach/util/log.V(0xc800000008, 0xc8205b3a40)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:181 +0x28
github.com/cockroachdb/cockroach/multiraft.(*state).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/multiraft.go:561 +0x1820
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211810)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2619 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:587 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a20a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2588 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:448 +0x1aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205b7b00)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2565 [select]:
github.com/cockroachdb/cockroach/util.NewFeed.func1()
/go/src/github.com/cockroachdb/cockroach/util/feed.go:62 +0x157
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8204159a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2605 [select]:
github.com/cockroachdb/cockroach/multiraft.(*writeTask).start.func1()
/go/src/github.com/cockroachdb/cockroach/multiraft/storage.go:145 +0x9eb
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205acc00)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2633 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/server/node.go:428 +0x15a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2540)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2635 [select]:
github.com/cockroachdb/cockroach/rpc.Send(0x1, 0x1, 0x1dcd6500, 0x12a05f200, 0xc820440c80, 0xc8207e4c90, 0xa, 0xc8207e4c70, 0x1, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/rpc/send.go:189 +0x176c
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc8203192d0, 0xc820440c80, 0x1, 0xc820440cd0, 0x1, 0x1, 0x1, 0x7f3ba7dfd120, 0xc8207ee0c0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:478 +0x784
github.com/cockroachdb/cockroach/kv.(*DistSender).sendAttempt(0xc8203192d0, 0xc820440c80, 0x7f3ba7dfd120, 0xc8207ee0c0, 0xc820213020, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:589 +0x3a0
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc8203192d0, 0x7f3ba7df80c8, 0xc8207d2d20, 0x7f3ba7dfd120, 0xc8207ee0c0, 0x7f3ba7e1d2b8, 0xc82058eec0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:671 +0xf63
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).sendOne(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2d20, 0x7f3ba7dfd120, 0xc8207ee0c0, 0x7f3ba7e1d2b8, 0xc82058eec0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:428 +0x7bd
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc8200963c0, 0x7f3ba7df80c8, 0xc8207d2d20, 0x7f3ba7dfd120, 0xc8207ee0c0, 0x7f3ba7e1d2b8, 0xc82058eec0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:317 +0x685
github.com/cockroachdb/cockroach/client.(*DB).send(0xc8204eee40, 0xc82028f840, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:447 +0x4ba
github.com/cockroachdb/cockroach/client.(*DB).Run(0xc8204eee40, 0xc8207b8c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/client/db.go:405 +0x92
github.com/cockroachdb/cockroach/ts.(*DB).StoreData(0xc8200280d8, 0x1, 0xc820394000, 0x35, 0x49, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/ts/db.go:145 +0x6fa
github.com/cockroachdb/cockroach/ts.(*poller).poll.func1()
/go/src/github.com/cockroachdb/cockroach/ts/db.go:96 +0xac
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc8205bde70, 0xc8201eb6b8)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/ts.(*poller).poll(0xc8203299b0)
/go/src/github.com/cockroachdb/cockroach/ts/db.go:99 +0x3d
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
/go/src/github.com/cockroachdb/cockroach/ts/db.go:79 +0xf5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc820211e50)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2613 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adc40)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2614 [semacquire]:
sync.runtime_Semacquire(0x19b22e4)
/usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0x19b22e0)
/usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/cockroachdb/cockroach/util/log.(*loggingT).outputLogEntry(0x19b22a0, 0x0, 0x164c2b4, 0x10, 0xb1, 0x0, 0xc8207c8d10)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:729 +0x40
github.com/cockroachdb/cockroach/util/log.AddStructured(0x0, 0x0, 0x0, 0x2, 0x128b0c0, 0x1f, 0xc8205c1a10, 0x3, 0x3)
/go/src/github.com/cockroachdb/cockroach/util/log/structured.go:39 +0x121
github.com/cockroachdb/cockroach/util/log.logDepth(0x0, 0x0, 0x1, 0x0, 0x128b0c0, 0x1f, 0xc8205c1a10, 0x3, 0x3)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:65 +0x7b
github.com/cockroachdb/cockroach/util/log.Infof(0x128b0c0, 0x1f, 0xc8205c1a10, 0x3, 0x3)
/go/src/github.com/cockroachdb/cockroach/util/log/log.go:85 +0x66
github.com/cockroachdb/cockroach/storage.(*baseQueue).MaybeAdd(0xc820583500, 0xc8200a1e60, 0x13fd268422f25738, 0xc800000000)
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:177 +0x2ae
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:186 +0xf1
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc8204ee120, 0xc8205c1cc8, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:106 +0x1e8
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820583440, 0xecd6bf56c, 0x2ee81845, 0x19b1e80, 0xc8204cb710, 0xc8204ee120, 0xc8200a1e60, 0xc800000001)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:188 +0x2de
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc8200a1e60, 0xc82042fb00)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:218 +0x67
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc82042fb60, 0xc82000ef00)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:237 +0x246
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:220 +0x29a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adc80)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2563 [select]:
net/http.(*persistConn).roundTrip(0xc8200ecdc0, 0xc8207a0770, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc820300990, 0xc82079b180, 0xc4, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc82079b180, 0x7f3ba7df57a8, 0xc820300990, 0xc820bcb180, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc820328b40, 0xc82079b180, 0x6f, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc820328b40, 0xc82079b180, 0x137e9f0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc820328b40, 0xc82079b180, 0x6, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:175 +0x9c
github.com/cockroachdb/cockroach/server.getRequest(0xc8200a0000, 0xc820316000, 0x0, 0xc8200a0090, 0x3, 0xc8201cd140, 0x58, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/status_test.go:261 +0x634
github.com/cockroachdb/cockroach/server.TestStatusLocalLogs(0xc8200a0000)
/go/src/github.com/cockroachdb/cockroach/server/status_test.go:448 +0x19de
testing.tRunner(0xc8200a0000, 0x1909f10)
/usr/local/go/src/testing/testing.go:456 +0x98
created by testing.RunTests
/usr/local/go/src/testing/testing.go:561 +0x86d
goroutine 2610 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adbc0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2612 [select, 1 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:252 +0x31d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205adc20)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2685 [select]:
net/http.(*persistConn).writeLoop(0xc8200ecdc0)
/usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:686 +0xc9d
goroutine 2587 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:424 +0x1d6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205b7aa0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
goroutine 2629 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:606 +0x3bf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8204ee120, 0xc8205a2220)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:80 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:81 +0x62
FAIL github.com/cockroachdb/cockroach/server 70.441s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment