Created
June 4, 2018 20:38
-
-
Save a-robinson/14d7740fd58c221cd2bb089d5b18eed8 to your computer and use it in GitHub Desktop.
This file has been truncated, but you can view the full file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| root@:26257/defaultdb> show trace for show testing_ranges from table foo; | |
| +----------------------------------+----------------------+--------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+---------------------------------+-------------------------------------------------------+------+ | |
| | timestamp | age | message | tag | loc | operation | span | | |
| +----------------------------------+----------------------+--------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+---------------------------------+-------------------------------------------------------+------+ | |
| | 2018-06-04 19:47:09.788057+00:00 | 0s | === SPAN START: sql txn === | | | sql txn | 0 | | |
| | 2018-06-04 19:47:09.79127+00:00 | 3ms212µs861ns | === SPAN START: session recording === | | | session recording | 507 | | |
| | 2018-06-04 19:47:09.791273+00:00 | 3ms216µs116ns | === SPAN START: starting plan === | | | starting plan | 1 | | |
| | 2018-06-04 19:47:09.791288+00:00 | 3ms231µs310ns | === SPAN START: dist sender === | | | dist sender | 2 | | |
| | 2018-06-04 19:47:09.791309+00:00 | 3ms252µs718ns | querying next range at /Meta2/Table/52/1/NULL | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 2 | | |
| | 2018-06-04 19:47:09.791336+00:00 | 3ms279µs4ns | === SPAN START: range lookup === | | | range lookup | 3 | | |
| | 2018-06-04 19:47:09.791358+00:00 | 3ms301µs331ns | === SPAN START: dist sender === | | | dist sender | 4 | | |
| | 2018-06-04 19:47:09.791387+00:00 | 3ms330µs548ns | querying next range at /Meta1/Table/52/1/NULL/NULL | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Meta2/Table/52/1/NULL,n1] | | dist sender | 4 | | |
| | 2018-06-04 19:47:09.791403+00:00 | 3ms346µs358ns | === SPAN START: range lookup === | | | range lookup | 5 | | |
| | 2018-06-04 19:47:09.791434+00:00 | 3ms376µs850ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Meta2/Table/52/1/NULL,n1] | | dist sender | 4 | | |
| | 2018-06-04 19:47:09.791457+00:00 | 3ms399µs881ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Meta2/Table/52/1/NULL,n1] | | dist sender | 4 | | |
| | 2018-06-04 19:47:09.791489+00:00 | 3ms432µs557ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Meta2/Table/52/1/NULL,n1] | | dist sender | 4 | | |
| | 2018-06-04 19:47:09.791497+00:00 | 3ms439µs882ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 6 | | |
| | 2018-06-04 19:47:09.791798+00:00 | 3ms740µs828ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.79183+00:00 | 3ms773µs661ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.791838+00:00 | 3ms781µs780ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.791846+00:00 | 3ms789µs659ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.791855+00:00 | 3ms798µs337ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.791857+00:00 | 3ms800µs641ns | operation accepts inconsistent results | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.791859+00:00 | 3ms802µs235ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.792111+00:00 | 4ms54µs467ns | engine stats: {InternalDeleteSkippedCount:2 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.792118+00:00 | 4ms61µs69ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 396 | | |
| | 2018-06-04 19:47:09.792407+00:00 | 4ms350µs768ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 2 | | |
| | 2018-06-04 19:47:09.792433+00:00 | 4ms376µs34ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 2 | | |
| | 2018-06-04 19:47:09.792456+00:00 | 4ms399µs15ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 2 | | |
| | 2018-06-04 19:47:09.792476+00:00 | 4ms419µs782ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 7 | | |
| | 2018-06-04 19:47:09.792693+00:00 | 4ms636µs189ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.792715+00:00 | 4ms658µs515ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.792722+00:00 | 4ms665µs121ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.792727+00:00 | 4ms670µs554ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.792735+00:00 | 4ms678µs226ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.792741+00:00 | 4ms684µs107ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.794156+00:00 | 6ms99µs698ns | engine stats: {InternalDeleteSkippedCount:102 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.794178+00:00 | 6ms121µs645ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 397 | | |
| | 2018-06-04 19:47:09.794569+00:00 | 6ms512µs355ns | === SPAN START: dist sender === | | | dist sender | 8 | | |
| | 2018-06-04 19:47:09.794585+00:00 | 6ms528µs481ns | querying next range at /Meta2/Table/52/2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 8 | | |
| | 2018-06-04 19:47:09.794608+00:00 | 6ms551µs579ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 8 | | |
| | 2018-06-04 19:47:09.794626+00:00 | 6ms568µs997ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 8 | | |
| | 2018-06-04 19:47:09.794641+00:00 | 6ms584µs426ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 9 | | |
| | 2018-06-04 19:47:09.794826+00:00 | 6ms769µs775ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.794848+00:00 | 6ms791µs511ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.794854+00:00 | 6ms797µs342ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.794859+00:00 | 6ms802µs499ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.794866+00:00 | 6ms809µs536ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.794871+00:00 | 6ms814µs443ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.79522+00:00 | 7ms162µs888ns | engine stats: {InternalDeleteSkippedCount:1 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.795224+00:00 | 7ms167µs688ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 398 | | |
| | 2018-06-04 19:47:09.795479+00:00 | 7ms422µs327ns | === SPAN START: consuming rows === | | | consuming rows | 10 | | |
| | 2018-06-04 19:47:09.795506+00:00 | 7ms449µs450ns | === SPAN START: dist sender === | | | dist sender | 11 | | |
| | 2018-06-04 19:47:09.795516+00:00 | 7ms459µs177ns | querying next range at /Table/52 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 11 | | |
| | 2018-06-04 19:47:09.795537+00:00 | 7ms480µs309ns | r23: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 11 | | |
| | 2018-06-04 19:47:09.795554+00:00 | 7ms497µs349ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 11 | | |
| | 2018-06-04 19:47:09.795566+00:00 | 7ms509µs369ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795569+00:00 | 7ms512µs554ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795574+00:00 | 7ms517µs503ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795581+00:00 | 7ms524µs310ns | read-only path | [n1,s1,r23/1:/Table/52{-/1/0}] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795586+00:00 | 7ms528µs856ns | read has no clock uncertainty | [n1,s1,r23/1:/Table/52{-/1/0}] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795589+00:00 | 7ms532µs33ns | command queue | [n1,s1,r23/1:/Table/52{-/1/0}] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795594+00:00 | 7ms537µs133ns | waiting for read lock | [n1,s1,r23/1:/Table/52{-/1/0}] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795605+00:00 | 7ms548µs154ns | read completed | [n1,s1,r23/1:/Table/52{-/1/0}] | | /cockroach.roachpb.Internal/Batch | 12 | | |
| | 2018-06-04 19:47:09.795642+00:00 | 7ms585µs348ns | === SPAN START: dist sender === | | | dist sender | 13 | | |
| | 2018-06-04 19:47:09.79565+00:00 | 7ms593µs519ns | querying next range at /Table/52/1/0 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 13 | | |
| | 2018-06-04 19:47:09.795666+00:00 | 7ms609µs506ns | r24: sending batch 1 LeaseInfo to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 13 | | |
| | 2018-06-04 19:47:09.795685+00:00 | 7ms628µs259ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 13 | | |
| | 2018-06-04 19:47:09.795699+00:00 | 7ms642µs28ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 14 | | |
| | 2018-06-04 19:47:09.79591+00:00 | 7ms853µs244ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.79593+00:00 | 7ms873µs757ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.795936+00:00 | 7ms878µs932ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.795941+00:00 | 7ms884µs458ns | read-only path | [n2,s2,r24/3:/Table/52/1/{0-100}] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.795948+00:00 | 7ms891µs570ns | command queue | [n2,s2,r24/3:/Table/52/1/{0-100}] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.795952+00:00 | 7ms895µs485ns | waiting for read lock | [n2,s2,r24/3:/Table/52/1/{0-100}] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.795961+00:00 | 7ms904µs324ns | read completed | [n2,s2,r24/3:/Table/52/1/{0-100}] | | /cockroach.roachpb.Internal/Batch | 399 | | |
| | 2018-06-04 19:47:09.796194+00:00 | 8ms136µs938ns | === SPAN START: dist sender === | | | dist sender | 15 | | |
| | 2018-06-04 19:47:09.796207+00:00 | 8ms150µs543ns | querying next range at /Table/52/1/100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 15 | | |
| | 2018-06-04 19:47:09.796233+00:00 | 8ms175µs858ns | r25: sending batch 1 LeaseInfo to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 15 | | |
| | 2018-06-04 19:47:09.796255+00:00 | 8ms198µs165ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 15 | | |
| | 2018-06-04 19:47:09.796274+00:00 | 8ms217µs218ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 16 | | |
| | 2018-06-04 19:47:09.796481+00:00 | 8ms424µs126ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796501+00:00 | 8ms443µs871ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796506+00:00 | 8ms449µs407ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796512+00:00 | 8ms455µs502ns | read-only path | [n2,s2,r25/3:/Table/52/1/{100-200}] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796519+00:00 | 8ms462µs88ns | command queue | [n2,s2,r25/3:/Table/52/1/{100-200}] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796523+00:00 | 8ms466µs555ns | waiting for read lock | [n2,s2,r25/3:/Table/52/1/{100-200}] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796532+00:00 | 8ms475µs548ns | read completed | [n2,s2,r25/3:/Table/52/1/{100-200}] | | /cockroach.roachpb.Internal/Batch | 400 | | |
| | 2018-06-04 19:47:09.796749+00:00 | 8ms692µs169ns | === SPAN START: dist sender === | | | dist sender | 17 | | |
| | 2018-06-04 19:47:09.796759+00:00 | 8ms702µs492ns | querying next range at /Table/52/1/200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | 2018-06-04 19:47:09.796776+00:00 | 8ms719µs211ns | r26: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | 2018-06-04 19:47:09.796792+00:00 | 8ms735µs577ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | 2018-06-04 19:47:09.796802+00:00 | 8ms745µs679ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | 2018-06-04 19:47:09.796821+00:00 | 8ms764µs270ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | 2018-06-04 19:47:09.796825+00:00 | 8ms768µs165ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | 2018-06-04 19:47:09.79683+00:00 | 8ms773µs585ns | read-only path | [n1,s1,r26/1:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | 2018-06-04 19:47:09.79685+00:00 | 8ms793µs480ns | replica.Send got error: [NotLeaseHolderError] r26: replica | [n1,s1,r26/1:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=7 | | | | | | |
| | | | start=1528141627.325771938,0 epo=2 pro=1528141627.325774415,0 | | | | | | |
| | 2018-06-04 19:47:09.796858+00:00 | 8ms801µs594ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 18 | | |
| | 2018-06-04 19:47:09.796872+00:00 | 8ms815µs317ns | application error: [NotLeaseHolderError] r26: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=7 | | | | | | |
| | | | start=1528141627.325771938,0 epo=2 pro=1528141627.325774415,0 | | | | | | |
| | 2018-06-04 19:47:09.796881+00:00 | 8ms823µs906ns | error: {(err: [NotLeaseHolderError] r26: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=7 | | | | | | |
| | | | start=1528141627.325771938,0 epo=2 pro=1528141627.325774415,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:47:09.796893+00:00 | 8ms836µs96ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 17 | | |
| | 2018-06-04 19:47:09.796904+00:00 | 8ms847µs660ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 19 | | |
| | 2018-06-04 19:47:09.797058+00:00 | 9ms875ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797075+00:00 | 9ms18µs435ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.79708+00:00 | 9ms22µs825ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797085+00:00 | 9ms28µs640ns | read-only path | [n2,s2,r26/3:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797091+00:00 | 9ms34µs213ns | command queue | [n2,s2,r26/3:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797094+00:00 | 9ms37µs686ns | waiting for read lock | [n2,s2,r26/3:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797102+00:00 | 9ms45µs14ns | read completed | [n2,s2,r26/3:/Table/52/1/{200-300}] | | /cockroach.roachpb.Internal/Batch | 401 | | |
| | 2018-06-04 19:47:09.797286+00:00 | 9ms229µs407ns | === SPAN START: dist sender === | | | dist sender | 20 | | |
| | 2018-06-04 19:47:09.797295+00:00 | 9ms238µs618ns | querying next range at /Table/52/1/300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 20 | | |
| | 2018-06-04 19:47:09.797312+00:00 | 9ms254µs855ns | r27: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 20 | | |
| | 2018-06-04 19:47:09.797326+00:00 | 9ms269µs415ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 20 | | |
| | 2018-06-04 19:47:09.797335+00:00 | 9ms278µs361ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797338+00:00 | 9ms281µs680ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797342+00:00 | 9ms285µs83ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797346+00:00 | 9ms289µs780ns | read-only path | [n1,s1,r27/1:/Table/52/1/{300-400}] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.79735+00:00 | 9ms292µs851ns | read has no clock uncertainty | [n1,s1,r27/1:/Table/52/1/{300-400}] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797352+00:00 | 9ms295µs352ns | command queue | [n1,s1,r27/1:/Table/52/1/{300-400}] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797357+00:00 | 9ms300µs353ns | waiting for read lock | [n1,s1,r27/1:/Table/52/1/{300-400}] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797364+00:00 | 9ms307µs742ns | read completed | [n1,s1,r27/1:/Table/52/1/{300-400}] | | /cockroach.roachpb.Internal/Batch | 21 | | |
| | 2018-06-04 19:47:09.797403+00:00 | 9ms346µs195ns | === SPAN START: dist sender === | | | dist sender | 22 | | |
| | 2018-06-04 19:47:09.79741+00:00 | 9ms353µs80ns | querying next range at /Table/52/1/400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 22 | | |
| | 2018-06-04 19:47:09.797422+00:00 | 9ms365µs234ns | r28: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 22 | | |
| | 2018-06-04 19:47:09.797434+00:00 | 9ms377µs274ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 22 | | |
| | 2018-06-04 19:47:09.797441+00:00 | 9ms384µs160ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797443+00:00 | 9ms386µs417ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797446+00:00 | 9ms389µs80ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.79745+00:00 | 9ms393µs19ns | read-only path | [n1,s1,r28/1:/Table/52/1/{400-500}] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797453+00:00 | 9ms395µs898ns | read has no clock uncertainty | [n1,s1,r28/1:/Table/52/1/{400-500}] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797454+00:00 | 9ms397µs768ns | command queue | [n1,s1,r28/1:/Table/52/1/{400-500}] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797458+00:00 | 9ms401µs372ns | waiting for read lock | [n1,s1,r28/1:/Table/52/1/{400-500}] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.797463+00:00 | 9ms406µs763ns | read completed | [n1,s1,r28/1:/Table/52/1/{400-500}] | | /cockroach.roachpb.Internal/Batch | 23 | | |
| | 2018-06-04 19:47:09.79749+00:00 | 9ms433µs10ns | === SPAN START: dist sender === | | | dist sender | 24 | | |
| | 2018-06-04 19:47:09.797496+00:00 | 9ms439µs278ns | querying next range at /Table/52/1/500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 24 | | |
| | 2018-06-04 19:47:09.797508+00:00 | 9ms451µs353ns | r29: sending batch 1 LeaseInfo to (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 24 | | |
| | 2018-06-04 19:47:09.797519+00:00 | 9ms462µs706ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 24 | | |
| | 2018-06-04 19:47:09.79753+00:00 | 9ms472µs892ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 25 | | |
| | 2018-06-04 19:47:09.797767+00:00 | 9ms710µs533ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797793+00:00 | 9ms736µs171ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797799+00:00 | 9ms741µs977ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797806+00:00 | 9ms749µs282ns | read-only path | [n3,s3,r29/2:/Table/52/1/{500-600}] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797813+00:00 | 9ms755µs899ns | command queue | [n3,s3,r29/2:/Table/52/1/{500-600}] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797816+00:00 | 9ms759µs691ns | waiting for read lock | [n3,s3,r29/2:/Table/52/1/{500-600}] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.797824+00:00 | 9ms767µs168ns | read completed | [n3,s3,r29/2:/Table/52/1/{500-600}] | | /cockroach.roachpb.Internal/Batch | 402 | | |
| | 2018-06-04 19:47:09.798047+00:00 | 9ms990µs327ns | === SPAN START: dist sender === | | | dist sender | 26 | | |
| | 2018-06-04 19:47:09.798056+00:00 | 9ms998µs895ns | querying next range at /Table/52/1/600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 26 | | |
| | 2018-06-04 19:47:09.798071+00:00 | 10ms14µs449ns | r30: sending batch 1 LeaseInfo to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 26 | | |
| | 2018-06-04 19:47:09.798085+00:00 | 10ms28µs171ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 26 | | |
| | 2018-06-04 19:47:09.798095+00:00 | 10ms38µs490ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 27 | | |
| | 2018-06-04 19:47:09.79827+00:00 | 10ms213µs182ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798288+00:00 | 10ms231µs23ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798293+00:00 | 10ms235µs855ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798298+00:00 | 10ms241µs280ns | read-only path | [n2,s2,r30/3:/Table/52/1/{600-700}] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798304+00:00 | 10ms246µs929ns | command queue | [n2,s2,r30/3:/Table/52/1/{600-700}] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798307+00:00 | 10ms250µs581ns | waiting for read lock | [n2,s2,r30/3:/Table/52/1/{600-700}] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798316+00:00 | 10ms259µs777ns | read completed | [n2,s2,r30/3:/Table/52/1/{600-700}] | | /cockroach.roachpb.Internal/Batch | 403 | | |
| | 2018-06-04 19:47:09.798485+00:00 | 10ms428µs602ns | === SPAN START: dist sender === | | | dist sender | 28 | | |
| | 2018-06-04 19:47:09.798494+00:00 | 10ms437µs413ns | querying next range at /Table/52/1/700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 28 | | |
| | 2018-06-04 19:47:09.798511+00:00 | 10ms454µs215ns | r31: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 28 | | |
| | 2018-06-04 19:47:09.798525+00:00 | 10ms468µs301ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 28 | | |
| | 2018-06-04 19:47:09.798533+00:00 | 10ms476µs720ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798536+00:00 | 10ms479µs373ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.79854+00:00 | 10ms482µs884ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798544+00:00 | 10ms487µs205ns | read-only path | [n1,s1,r31/1:/Table/52/1/{700-800}] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798547+00:00 | 10ms490µs85ns | read has no clock uncertainty | [n1,s1,r31/1:/Table/52/1/{700-800}] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798549+00:00 | 10ms492µs285ns | command queue | [n1,s1,r31/1:/Table/52/1/{700-800}] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798553+00:00 | 10ms496µs204ns | waiting for read lock | [n1,s1,r31/1:/Table/52/1/{700-800}] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798559+00:00 | 10ms502µs735ns | read completed | [n1,s1,r31/1:/Table/52/1/{700-800}] | | /cockroach.roachpb.Internal/Batch | 29 | | |
| | 2018-06-04 19:47:09.798587+00:00 | 10ms530µs114ns | === SPAN START: dist sender === | | | dist sender | 30 | | |
| | 2018-06-04 19:47:09.798593+00:00 | 10ms536µs498ns | querying next range at /Table/52/1/800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | 2018-06-04 19:47:09.798606+00:00 | 10ms548µs907ns | r32: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | 2018-06-04 19:47:09.798617+00:00 | 10ms560µs452ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | 2018-06-04 19:47:09.798624+00:00 | 10ms567µs179ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | 2018-06-04 19:47:09.798626+00:00 | 10ms569µs543ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | 2018-06-04 19:47:09.798629+00:00 | 10ms572µs177ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | 2018-06-04 19:47:09.798633+00:00 | 10ms576µs390ns | read-only path | [n1,s1,r32/1:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | 2018-06-04 19:47:09.798649+00:00 | 10ms592µs504ns | replica.Send got error: [NotLeaseHolderError] r32: replica | [n1,s1,r32/1:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=5 | | | | | | |
| | | | start=1528141626.112756917,0 epo=2 pro=1528141626.112760140,0 | | | | | | |
| | 2018-06-04 19:47:09.798656+00:00 | 10ms599µs223ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 31 | | |
| | 2018-06-04 19:47:09.798668+00:00 | 10ms611µs240ns | application error: [NotLeaseHolderError] r32: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=5 | | | | | | |
| | | | start=1528141626.112756917,0 epo=2 pro=1528141626.112760140,0 | | | | | | |
| | 2018-06-04 19:47:09.798674+00:00 | 10ms617µs285ns | error: {(err: [NotLeaseHolderError] r32: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=5 | | | | | | |
| | | | start=1528141626.112756917,0 epo=2 pro=1528141626.112760140,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:47:09.798685+00:00 | 10ms628µs356ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 30 | | |
| | 2018-06-04 19:47:09.798696+00:00 | 10ms638µs861ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 32 | | |
| | 2018-06-04 19:47:09.798848+00:00 | 10ms791µs559ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798876+00:00 | 10ms819µs49ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798881+00:00 | 10ms824µs205ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798887+00:00 | 10ms830µs585ns | read-only path | [n3,s3,r32/2:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798893+00:00 | 10ms836µs689ns | command queue | [n3,s3,r32/2:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798897+00:00 | 10ms840µs441ns | waiting for read lock | [n3,s3,r32/2:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.798905+00:00 | 10ms848µs361ns | read completed | [n3,s3,r32/2:/Table/52/1/{800-900}] | | /cockroach.roachpb.Internal/Batch | 404 | | |
| | 2018-06-04 19:47:09.799081+00:00 | 11ms24µs729ns | === SPAN START: dist sender === | | | dist sender | 33 | | |
| | 2018-06-04 19:47:09.799091+00:00 | 11ms33µs827ns | querying next range at /Table/52/1/900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 33 | | |
| | 2018-06-04 19:47:09.799105+00:00 | 11ms48µs744ns | r33: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 33 | | |
| | 2018-06-04 19:47:09.799119+00:00 | 11ms62µs232ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 33 | | |
| | 2018-06-04 19:47:09.799127+00:00 | 11ms70µs419ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.79913+00:00 | 11ms72µs876ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799133+00:00 | 11ms76µs95ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799137+00:00 | 11ms80µs463ns | read-only path | [n1,s1,r33/1:/Table/52/1/{900-1000}] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799141+00:00 | 11ms84µs174ns | read has no clock uncertainty | [n1,s1,r33/1:/Table/52/1/{900-1000}] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799143+00:00 | 11ms86µs331ns | command queue | [n1,s1,r33/1:/Table/52/1/{900-1000}] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799147+00:00 | 11ms89µs818ns | waiting for read lock | [n1,s1,r33/1:/Table/52/1/{900-1000}] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799153+00:00 | 11ms96µs4ns | read completed | [n1,s1,r33/1:/Table/52/1/{900-1000}] | | /cockroach.roachpb.Internal/Batch | 34 | | |
| | 2018-06-04 19:47:09.799181+00:00 | 11ms124µs803ns | === SPAN START: dist sender === | | | dist sender | 35 | | |
| | 2018-06-04 19:47:09.799188+00:00 | 11ms131µs513ns | querying next range at /Table/52/1/1000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 35 | | |
| | 2018-06-04 19:47:09.799201+00:00 | 11ms144µs41ns | r34: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 35 | | |
| | 2018-06-04 19:47:09.799212+00:00 | 11ms155µs730ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 35 | | |
| | 2018-06-04 19:47:09.799219+00:00 | 11ms162µs489ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799222+00:00 | 11ms164µs925ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799224+00:00 | 11ms167µs586ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799228+00:00 | 11ms171µs190ns | read-only path | [n1,s1,r34/1:/Table/52/1/1{000-100}] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799231+00:00 | 11ms174µs251ns | read has no clock uncertainty | [n1,s1,r34/1:/Table/52/1/1{000-100}] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799233+00:00 | 11ms176µs113ns | command queue | [n1,s1,r34/1:/Table/52/1/1{000-100}] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799236+00:00 | 11ms179µs378ns | waiting for read lock | [n1,s1,r34/1:/Table/52/1/1{000-100}] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799241+00:00 | 11ms184µs431ns | read completed | [n1,s1,r34/1:/Table/52/1/1{000-100}] | | /cockroach.roachpb.Internal/Batch | 36 | | |
| | 2018-06-04 19:47:09.799268+00:00 | 11ms211µs104ns | === SPAN START: dist sender === | | | dist sender | 37 | | |
| | 2018-06-04 19:47:09.799274+00:00 | 11ms217µs289ns | querying next range at /Table/52/1/1100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 37 | | |
| | 2018-06-04 19:47:09.799285+00:00 | 11ms228µs601ns | r35: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 37 | | |
| | 2018-06-04 19:47:09.799297+00:00 | 11ms239µs819ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 37 | | |
| | 2018-06-04 19:47:09.799304+00:00 | 11ms246µs835ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799306+00:00 | 11ms249µs97ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799308+00:00 | 11ms251µs584ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799313+00:00 | 11ms256µs127ns | read-only path | [n1,s1,r35/1:/Table/52/1/1{100-200}] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799316+00:00 | 11ms258µs950ns | read has no clock uncertainty | [n1,s1,r35/1:/Table/52/1/1{100-200}] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799318+00:00 | 11ms260µs864ns | command queue | [n1,s1,r35/1:/Table/52/1/1{100-200}] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799321+00:00 | 11ms263µs931ns | waiting for read lock | [n1,s1,r35/1:/Table/52/1/1{100-200}] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799326+00:00 | 11ms268µs962ns | read completed | [n1,s1,r35/1:/Table/52/1/1{100-200}] | | /cockroach.roachpb.Internal/Batch | 38 | | |
| | 2018-06-04 19:47:09.799349+00:00 | 11ms292µs614ns | === SPAN START: dist sender === | | | dist sender | 39 | | |
| | 2018-06-04 19:47:09.799355+00:00 | 11ms298µs548ns | querying next range at /Table/52/1/1200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 39 | | |
| | 2018-06-04 19:47:09.799367+00:00 | 11ms309µs838ns | r36: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 39 | | |
| | 2018-06-04 19:47:09.799377+00:00 | 11ms320µs560ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 39 | | |
| | 2018-06-04 19:47:09.799384+00:00 | 11ms327µs77ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799386+00:00 | 11ms329µs120ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799388+00:00 | 11ms331µs549ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799392+00:00 | 11ms335µs272ns | read-only path | [n1,s1,r36/1:/Table/52/1/1{200-300}] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799395+00:00 | 11ms337µs823ns | read has no clock uncertainty | [n1,s1,r36/1:/Table/52/1/1{200-300}] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799396+00:00 | 11ms339µs672ns | command queue | [n1,s1,r36/1:/Table/52/1/1{200-300}] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.7994+00:00 | 11ms343µs370ns | waiting for read lock | [n1,s1,r36/1:/Table/52/1/1{200-300}] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.799406+00:00 | 11ms349µs664ns | read completed | [n1,s1,r36/1:/Table/52/1/1{200-300}] | | /cockroach.roachpb.Internal/Batch | 40 | | |
| | 2018-06-04 19:47:09.79943+00:00 | 11ms373µs299ns | === SPAN START: dist sender === | | | dist sender | 41 | | |
| | 2018-06-04 19:47:09.799436+00:00 | 11ms379µs232ns | querying next range at /Table/52/1/1300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 41 | | |
| | 2018-06-04 19:47:09.799447+00:00 | 11ms389µs938ns | r37: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 41 | | |
| | 2018-06-04 19:47:09.799457+00:00 | 11ms400µs382ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 41 | | |
| | 2018-06-04 19:47:09.799463+00:00 | 11ms406µs760ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799466+00:00 | 11ms408µs944ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799468+00:00 | 11ms411µs228ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799471+00:00 | 11ms414µs681ns | read-only path | [n1,s1,r37/1:/Table/52/1/1{300-400}] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799474+00:00 | 11ms417µs409ns | read has no clock uncertainty | [n1,s1,r37/1:/Table/52/1/1{300-400}] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799476+00:00 | 11ms419µs146ns | command queue | [n1,s1,r37/1:/Table/52/1/1{300-400}] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799479+00:00 | 11ms421µs860ns | waiting for read lock | [n1,s1,r37/1:/Table/52/1/1{300-400}] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799484+00:00 | 11ms427µs16ns | read completed | [n1,s1,r37/1:/Table/52/1/1{300-400}] | | /cockroach.roachpb.Internal/Batch | 42 | | |
| | 2018-06-04 19:47:09.799511+00:00 | 11ms454µs779ns | === SPAN START: dist sender === | | | dist sender | 43 | | |
| | 2018-06-04 19:47:09.799518+00:00 | 11ms461µs141ns | querying next range at /Table/52/1/1400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 43 | | |
| | 2018-06-04 19:47:09.799529+00:00 | 11ms472µs389ns | r38: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 43 | | |
| | 2018-06-04 19:47:09.79954+00:00 | 11ms482µs907ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 43 | | |
| | 2018-06-04 19:47:09.799546+00:00 | 11ms489µs232ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799548+00:00 | 11ms491µs269ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.79955+00:00 | 11ms493µs562ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799554+00:00 | 11ms497µs691ns | read-only path | [n1,s1,r38/1:/Table/52/1/1{400-500}] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799557+00:00 | 11ms500µs623ns | read has no clock uncertainty | [n1,s1,r38/1:/Table/52/1/1{400-500}] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799559+00:00 | 11ms502µs327ns | command queue | [n1,s1,r38/1:/Table/52/1/1{400-500}] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799562+00:00 | 11ms505µs180ns | waiting for read lock | [n1,s1,r38/1:/Table/52/1/1{400-500}] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.799567+00:00 | 11ms509µs994ns | read completed | [n1,s1,r38/1:/Table/52/1/1{400-500}] | | /cockroach.roachpb.Internal/Batch | 44 | | |
| | 2018-06-04 19:47:09.79959+00:00 | 11ms532µs820ns | === SPAN START: dist sender === | | | dist sender | 45 | | |
| | 2018-06-04 19:47:09.799595+00:00 | 11ms538µs776ns | querying next range at /Table/52/1/1500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 45 | | |
| | 2018-06-04 19:47:09.799606+00:00 | 11ms548µs902ns | r39: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 45 | | |
| | 2018-06-04 19:47:09.799617+00:00 | 11ms560µs329ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 45 | | |
| | 2018-06-04 19:47:09.799623+00:00 | 11ms566µs670ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.799626+00:00 | 11ms568µs874ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.799628+00:00 | 11ms571µs211ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.799631+00:00 | 11ms574µs749ns | read-only path | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.799635+00:00 | 11ms578µs106ns | request range lease (attempt #1) | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.799639+00:00 | 11ms582µs737ns | === SPAN START: request range lease === | | | request range lease | 47 | | |
| | 2018-06-04 19:47:09.799644+00:00 | 11ms587µs446ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:09.799664+00:00 | 11ms606µs865ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 49 | | |
| | 2018-06-04 19:47:09.79974+00:00 | 11ms683µs198ns | === SPAN START: dist sender === | | | dist sender | 50 | | |
| | 2018-06-04 19:47:09.799756+00:00 | 11ms698µs819ns | querying next range at /System/NodeLiveness/1 | [s1,r39/1:/Table/52/1/1{500-600},txn=c4d396f8,n1] | | dist sender | 50 | | |
| | 2018-06-04 19:47:09.799779+00:00 | 11ms722µs748ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r39/1:/Table/52/1/1{500-600},txn=c4d396f8,n1] | | dist sender | 50 | | |
| | 2018-06-04 19:47:09.799797+00:00 | 11ms739µs976ns | sending request to alex-laptop.local:26259 | [s1,r39/1:/Table/52/1/1{500-600},txn=c4d396f8,n1] | | dist sender | 50 | | |
| | 2018-06-04 19:47:09.799808+00:00 | 11ms751µs751ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 51 | | |
| | 2018-06-04 19:47:09.800025+00:00 | 11ms968µs83ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800045+00:00 | 11ms987µs998ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800051+00:00 | 11ms993µs935ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800057+00:00 | 12ms336ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800071+00:00 | 12ms14µs577ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800089+00:00 | 12ms32µs342ns | === SPAN START: request range lease === | | | request range lease | 406 | | |
| | 2018-06-04 19:47:09.800096+00:00 | 12ms38µs832ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:09.800118+00:00 | 12ms60µs911ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800133+00:00 | 12ms76µs336ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | 2018-06-04 19:47:09.800146+00:00 | 12ms89µs501ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | 2018-06-04 19:47:09.800197+00:00 | 12ms139µs827ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.800201+00:00 | 12ms143µs876ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.80024+00:00 | 12ms183µs91ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | 2018-06-04 19:47:09.800243+00:00 | 12ms186µs633ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | 2018-06-04 19:47:09.800877+00:00 | 12ms820µs320ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.801195+00:00 | 13ms138µs200ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | 2018-06-04 19:47:09.801217+00:00 | 13ms160µs744ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 405 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:09.801272+00:00 | 13ms214µs980ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 407 | | |
| | 2018-06-04 19:47:09.801681+00:00 | 13ms624µs430ns | heartbeat 1528141639.299667112,0 | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | liveness heartbeat | 49 | | |
| | 2018-06-04 19:47:09.801691+00:00 | 13ms634µs18ns | read-write path | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | 2018-06-04 19:47:09.801699+00:00 | 13ms642µs511ns | applied timestamp cache | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | 2018-06-04 19:47:09.801786+00:00 | 13ms729µs92ns | evaluated request | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | 2018-06-04 19:47:09.801794+00:00 | 13ms736µs990ns | acquired {raft,replica}mu | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | 2018-06-04 19:47:09.802375+00:00 | 14ms317µs824ns | applying command | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | 2018-06-04 19:47:09.802458+00:00 | 14ms401µs329ns | new range lease repl=(n1,s1):1 seq=4 start=1528141568.365104551,0 | [n1,s1,r39/1:/Table/52/1/1{500-600}] | storage/replica_proposal.go:202 | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | | | epo=3 pro=1528141629.799637067,0 following repl=(n1,s1):1 seq=3 | | | | | | |
| | | | start=1528141259.602787924,0 epo=1 pro=1528141365.657033828,0 | | | | | | |
| | 2018-06-04 19:47:09.802555+00:00 | 14ms498µs482ns | === SPAN START: [async] storage.Store: gossip on capacity change === | | | [async] storage.Store: gossip on capacity change | 52 | | |
| | 2018-06-04 19:47:09.80257+00:00 | 14ms513µs274ns | not gossiping system config because the replica doesn't contain the | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 48 | | |
| | | | system config's start key | | | | | | |
| | 2018-06-04 19:47:09.802751+00:00 | 14ms694µs592ns | lease acquisition succeeded: repl=(n1,s1):1 seq=3 | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | | | start=1528141259.602787924,0 epo=1 pro=1528141365.657033828,0 | | | | | | |
| | 2018-06-04 19:47:09.802758+00:00 | 14ms700µs902ns | read has no clock uncertainty | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.802761+00:00 | 14ms704µs749ns | command queue | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.802766+00:00 | 14ms709µs244ns | waiting for read lock | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.802775+00:00 | 14ms718µs42ns | read completed | [n1,s1,r39/1:/Table/52/1/1{500-600}] | | /cockroach.roachpb.Internal/Batch | 46 | | |
| | 2018-06-04 19:47:09.802818+00:00 | 14ms761µs237ns | === SPAN START: dist sender === | | | dist sender | 53 | | |
| | 2018-06-04 19:47:09.802827+00:00 | 14ms770µs551ns | querying next range at /Table/52/1/1600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 53 | | |
| | 2018-06-04 19:47:09.802845+00:00 | 14ms788µs722ns | r40: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 53 | | |
| | 2018-06-04 19:47:09.802857+00:00 | 14ms800µs30ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 53 | | |
| | 2018-06-04 19:47:09.802866+00:00 | 14ms809µs157ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:09.802868+00:00 | 14ms811µs383ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:09.802873+00:00 | 14ms816µs327ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:09.802944+00:00 | 14ms887µs715ns | read-only path | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:09.802948+00:00 | 14ms891µs505ns | request range lease (attempt #1) | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:09.802953+00:00 | 14ms896µs507ns | === SPAN START: request range lease === | | | request range lease | 55 | | |
| | 2018-06-04 19:47:09.802959+00:00 | 14ms902µs784ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:09.802979+00:00 | 14ms922µs158ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 57 | | |
| | 2018-06-04 19:47:09.803037+00:00 | 14ms980µs256ns | === SPAN START: dist sender === | | | dist sender | 58 | | |
| | 2018-06-04 19:47:09.803046+00:00 | 14ms989µs734ns | querying next range at /System/NodeLiveness/1 | [s1,r40/1:/Table/52/1/1{600-700},txn=fbaf8b93,n1] | | dist sender | 58 | | |
| | 2018-06-04 19:47:09.803064+00:00 | 15ms6µs909ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r40/1:/Table/52/1/1{600-700},txn=fbaf8b93,n1] | | dist sender | 58 | | |
| | 2018-06-04 19:47:09.803077+00:00 | 15ms20µs643ns | sending request to alex-laptop.local:26259 | [s1,r40/1:/Table/52/1/1{600-700},txn=fbaf8b93,n1] | | dist sender | 58 | | |
| | 2018-06-04 19:47:09.803091+00:00 | 15ms34µs745ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 59 | | |
| | 2018-06-04 19:47:09.803314+00:00 | 15ms257µs415ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803335+00:00 | 15ms278µs808ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803341+00:00 | 15ms284µs734ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803346+00:00 | 15ms289µs624ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803356+00:00 | 15ms298µs994ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803376+00:00 | 15ms318µs859ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803434+00:00 | 15ms377µs359ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.803438+00:00 | 15ms380µs920ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.804008+00:00 | 15ms951µs669ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.804247+00:00 | 16ms190µs5ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | 2018-06-04 19:47:09.804264+00:00 | 16ms207µs194ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 408 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:09.804702+00:00 | 16ms644µs994ns | heartbeat 1528141639.302982820,0 | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | liveness heartbeat | 57 | | |
| | 2018-06-04 19:47:09.804708+00:00 | 16ms651µs26ns | read-write path | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:09.804714+00:00 | 16ms657µs27ns | applied timestamp cache | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:09.804777+00:00 | 16ms720µs664ns | evaluated request | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:09.80478+00:00 | 16ms723µs77ns | acquired {raft,replica}mu | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:12.779563+00:00 | 2s991ms506µs112ns | retry proposal 2c6ad45035f163db: reasonTicks | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:12.779753+00:00 | 2s991ms696µs54ns | applied timestamp cache | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:12.779911+00:00 | 2s991ms854µs220ns | evaluated request | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:12.779916+00:00 | 2s991ms859µs265ns | acquired {raft,replica}mu | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.378666+00:00 | 3s590ms609µs485ns | retry proposal 582601a7fd4ad862: reasonNewLeader | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.378741+00:00 | 3s590ms684µs549ns | applied timestamp cache | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.379324+00:00 | 3s591ms267µs602ns | evaluated request | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.379477+00:00 | 3s591ms420µs344ns | acquired {raft,replica}mu | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.380982+00:00 | 3s592ms925µs348ns | applying command | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | 2018-06-04 19:47:13.381162+00:00 | 3s593ms105µs514ns | new range lease repl=(n1,s1):1 seq=6 start=1528141568.365216808,0 | [n1,s1,r40/1:/Table/52/1/1{600-700}] | storage/replica_proposal.go:202 | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | | | epo=3 pro=1528141629.802950022,0 following repl=(n1,s1):1 seq=5 | | | | | | |
| | | | start=1528141460.337622006,0 epo=1 pro=1528141460.337627515,0 | | | | | | |
| | 2018-06-04 19:47:13.381275+00:00 | 3s593ms218µs570ns | === SPAN START: [async] storage.Store: gossip on capacity change === | | | [async] storage.Store: gossip on capacity change | 60 | | |
| | 2018-06-04 19:47:13.381301+00:00 | 3s593ms244µs752ns | not gossiping system config because the replica doesn't contain the | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 56 | | |
| | | | system config's start key | | | | | | |
| | 2018-06-04 19:47:13.381557+00:00 | 3s593ms500µs372ns | lease acquisition succeeded: repl=(n1,s1):1 seq=5 | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | | | start=1528141460.337622006,0 epo=1 pro=1528141460.337627515,0 | | | | | | |
| | 2018-06-04 19:47:13.381573+00:00 | 3s593ms516µs107ns | read has no clock uncertainty | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:13.381583+00:00 | 3s593ms526µs270ns | command queue | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:13.381604+00:00 | 3s593ms547µs477ns | waiting for read lock | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:13.381642+00:00 | 3s593ms585µs645ns | read completed | [n1,s1,r40/1:/Table/52/1/1{600-700}] | | /cockroach.roachpb.Internal/Batch | 54 | | |
| | 2018-06-04 19:47:13.381844+00:00 | 3s593ms787µs303ns | === SPAN START: dist sender === | | | dist sender | 61 | | |
| | 2018-06-04 19:47:13.381877+00:00 | 3s593ms820µs536ns | querying next range at /Table/52/1/1700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 61 | | |
| | 2018-06-04 19:47:13.381918+00:00 | 3s593ms861µs756ns | === SPAN START: range lookup === | | | range lookup | 62 | | |
| | 2018-06-04 19:47:13.381955+00:00 | 3s593ms898µs450ns | === SPAN START: dist sender === | | | dist sender | 63 | | |
| | 2018-06-04 19:47:13.382059+00:00 | 3s594ms2µs277ns | querying next range at /Meta2/Table/52/1/1700/NULL | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/1700,n1] | | dist sender | 63 | | |
| | 2018-06-04 19:47:13.382189+00:00 | 3s594ms131µs853ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/1700,n1] | | dist sender | 63 | | |
| | 2018-06-04 19:47:13.382245+00:00 | 3s594ms188µs149ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/1700,n1] | | dist sender | 63 | | |
| | 2018-06-04 19:47:13.382256+00:00 | 3s594ms199µs703ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 64 | | |
| | 2018-06-04 19:47:13.383479+00:00 | 3s595ms422µs379ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383569+00:00 | 3s595ms512µs463ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383594+00:00 | 3s595ms537µs413ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383655+00:00 | 3s595ms597µs945ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383693+00:00 | 3s595ms635µs985ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383696+00:00 | 3s595ms639µs678ns | operation accepts inconsistent results | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.383708+00:00 | 3s595ms651µs671ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.384648+00:00 | 3s596ms590µs976ns | engine stats: {InternalDeleteSkippedCount:10 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.384719+00:00 | 3s596ms662µs806ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 409 | | |
| | 2018-06-04 19:47:13.385399+00:00 | 3s597ms342µs233ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 61 | | |
| | 2018-06-04 19:47:13.385444+00:00 | 3s597ms387µs239ns | r41: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 61 | | |
| | 2018-06-04 19:47:13.385473+00:00 | 3s597ms416µs484ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 61 | | |
| | 2018-06-04 19:47:13.38549+00:00 | 3s597ms433µs342ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:13.385498+00:00 | 3s597ms441µs120ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:13.385507+00:00 | 3s597ms449µs940ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:13.385673+00:00 | 3s597ms616µs462ns | read-only path | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:13.385681+00:00 | 3s597ms623µs976ns | request range lease (attempt #1) | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:13.38569+00:00 | 3s597ms633µs245ns | === SPAN START: request range lease === | | | request range lease | 66 | | |
| | 2018-06-04 19:47:13.3857+00:00 | 3s597ms643µs99ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:13.385726+00:00 | 3s597ms669µs538ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 68 | | |
| | 2018-06-04 19:47:13.385828+00:00 | 3s597ms771µs471ns | === SPAN START: dist sender === | | | dist sender | 69 | | |
| | 2018-06-04 19:47:13.385849+00:00 | 3s597ms792µs213ns | querying next range at /System/NodeLiveness/1 | [s1,r41/1:/Table/52/1/1{700-800},txn=aeef0d78,n1] | | dist sender | 69 | | |
| | 2018-06-04 19:47:13.385879+00:00 | 3s597ms822µs67ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r41/1:/Table/52/1/1{700-800},txn=aeef0d78,n1] | | dist sender | 69 | | |
| | 2018-06-04 19:47:13.3859+00:00 | 3s597ms843µs213ns | sending request to alex-laptop.local:26259 | [s1,r41/1:/Table/52/1/1{700-800},txn=aeef0d78,n1] | | dist sender | 69 | | |
| | 2018-06-04 19:47:13.385915+00:00 | 3s597ms858µs285ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 70 | | |
| | 2018-06-04 19:47:13.386305+00:00 | 3s598ms248µs434ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386357+00:00 | 3s598ms300µs293ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386373+00:00 | 3s598ms316µs656ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386388+00:00 | 3s598ms331µs451ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386418+00:00 | 3s598ms361µs366ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386489+00:00 | 3s598ms432µs646ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386621+00:00 | 3s598ms564µs191ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.386629+00:00 | 3s598ms572µs357ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.387739+00:00 | 3s599ms682µs275ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.388183+00:00 | 3s600ms125µs994ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | 2018-06-04 19:47:13.388237+00:00 | 3s600ms180µs2ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 410 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:13.389449+00:00 | 3s601ms392µs381ns | heartbeat 1528141642.885732004,0 | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | liveness heartbeat | 68 | | |
| | 2018-06-04 19:47:13.389479+00:00 | 3s601ms422µs462ns | read-write path | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:13.389507+00:00 | 3s601ms450µs111ns | applied timestamp cache | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:13.389815+00:00 | 3s601ms758µs530ns | evaluated request | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:13.389826+00:00 | 3s601ms769µs139ns | acquired {raft,replica}mu | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.378799+00:00 | 6s590ms742µs654ns | retry proposal 50397e81a2bd7e83: reasonTicks | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.378956+00:00 | 6s590ms899µs155ns | applied timestamp cache | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.379316+00:00 | 6s591ms259µs198ns | evaluated request | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.379326+00:00 | 6s591ms269µs722ns | acquired {raft,replica}mu | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.581442+00:00 | 6s793ms384µs914ns | retry proposal 52d17b0bb61814c0: reasonNewLeader | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.581726+00:00 | 6s793ms669µs21ns | applied timestamp cache | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.583367+00:00 | 6s795ms310µs768ns | evaluated request | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.583413+00:00 | 6s795ms355µs946ns | acquired {raft,replica}mu | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.585453+00:00 | 6s797ms395µs860ns | retry proposal 7ce5ca9e7298e9d4: reasonNewLeaderOrConfigChange | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.585601+00:00 | 6s797ms544µs811ns | applied timestamp cache | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.58639+00:00 | 6s798ms333µs721ns | evaluated request | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.586431+00:00 | 6s798ms373µs989ns | acquired {raft,replica}mu | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | 2018-06-04 19:47:16.589585+00:00 | 6s801ms528µs468ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.365333322,0 epo=3 | | | | | | |
| | | | pro=1528141633.385683914,0 | | | | | | |
| | 2018-06-04 19:47:16.58965+00:00 | 6s801ms593µs206ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | | | seq=4 start=1528141568.365333322,0 epo=3 pro=1528141633.385683914,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.365333322,0 epo=3 | | | | | | |
| | | | pro=1528141633.385683914,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:16.590403+00:00 | 6s802ms346µs692ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 67 | | |
| | | | start=1528141568.365333322,0 epo=3 pro=1528141633.385683914,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.365333322,0 epo=3 | | | | | | |
| | | | pro=1528141633.385683914,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:16.590588+00:00 | 6s802ms530µs888ns | read has no clock uncertainty | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:16.590613+00:00 | 6s802ms556µs777ns | command queue | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:16.590658+00:00 | 6s802ms600µs991ns | waiting for read lock | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:16.590753+00:00 | 6s802ms696µs119ns | read completed | [n1,s1,r41/1:/Table/52/1/1{700-800}] | | /cockroach.roachpb.Internal/Batch | 65 | | |
| | 2018-06-04 19:47:16.591566+00:00 | 6s803ms508µs834ns | === SPAN START: dist sender === | | | dist sender | 71 | | |
| | 2018-06-04 19:47:16.591648+00:00 | 6s803ms591µs268ns | querying next range at /Table/52/1/1800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 71 | | |
| | 2018-06-04 19:47:16.591787+00:00 | 6s803ms730µs341ns | r42: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 71 | | |
| | 2018-06-04 19:47:16.591883+00:00 | 6s803ms826µs588ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 71 | | |
| | 2018-06-04 19:47:16.591939+00:00 | 6s803ms882µs729ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:16.591958+00:00 | 6s803ms901µs4ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:16.592042+00:00 | 6s803ms985µs442ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:16.59259+00:00 | 6s804ms532µs974ns | read-only path | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:16.592616+00:00 | 6s804ms559µs441ns | request range lease (attempt #1) | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:16.592651+00:00 | 6s804ms594µs497ns | === SPAN START: request range lease === | | | request range lease | 73 | | |
| | 2018-06-04 19:47:16.592692+00:00 | 6s804ms635µs251ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:16.592749+00:00 | 6s804ms692µs518ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 75 | | |
| | 2018-06-04 19:47:16.593069+00:00 | 6s805ms12µs297ns | === SPAN START: dist sender === | | | dist sender | 76 | | |
| | 2018-06-04 19:47:16.593121+00:00 | 6s805ms64µs540ns | querying next range at /System/NodeLiveness/1 | [s1,r42/1:/Table/52/1/1{800-900},txn=766a1fed,n1] | | dist sender | 76 | | |
| | 2018-06-04 19:47:16.593302+00:00 | 6s805ms245µs789ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r42/1:/Table/52/1/1{800-900},txn=766a1fed,n1] | | dist sender | 76 | | |
| | 2018-06-04 19:47:16.593394+00:00 | 6s805ms337µs480ns | sending request to alex-laptop.local:26259 | [s1,r42/1:/Table/52/1/1{800-900},txn=766a1fed,n1] | | dist sender | 76 | | |
| | 2018-06-04 19:47:16.593484+00:00 | 6s805ms427µs388ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 77 | | |
| | 2018-06-04 19:47:16.594902+00:00 | 6s806ms844µs864ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.594949+00:00 | 6s806ms892µs505ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.594962+00:00 | 6s806ms905µs144ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.594971+00:00 | 6s806ms913µs881ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.594988+00:00 | 6s806ms931µs200ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.595043+00:00 | 6s806ms985µs930ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.595193+00:00 | 6s807ms136µs166ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.595198+00:00 | 6s807ms141µs695ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.596168+00:00 | 6s808ms111µs146ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.596606+00:00 | 6s808ms549µs593ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | 2018-06-04 19:47:16.596635+00:00 | 6s808ms578µs387ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 411 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:16.597364+00:00 | 6s809ms307µs400ns | heartbeat 1528141646.092766718,0 | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | liveness heartbeat | 75 | | |
| | 2018-06-04 19:47:16.597374+00:00 | 6s809ms317µs238ns | read-write path | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:16.597384+00:00 | 6s809ms327µs605ns | applied timestamp cache | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:16.597491+00:00 | 6s809ms434µs815ns | evaluated request | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:16.597496+00:00 | 6s809ms438µs992ns | acquired {raft,replica}mu | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.576599+00:00 | 9s788ms542µs61ns | retry proposal 6cbb80367c054b66: reasonTicks | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.576893+00:00 | 9s788ms836µs453ns | applied timestamp cache | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.577232+00:00 | 9s789ms175µs36ns | evaluated request | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.577239+00:00 | 9s789ms182µs392ns | acquired {raft,replica}mu | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977112+00:00 | 10s189ms55µs73ns | retry proposal 62b5674dc18cc9ca: reasonNewLeader | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977166+00:00 | 10s189ms109µs101ns | applied timestamp cache | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977469+00:00 | 10s189ms412µs434ns | evaluated request | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977475+00:00 | 10s189ms417µs899ns | acquired {raft,replica}mu | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977823+00:00 | 10s189ms766µs261ns | retry proposal 7c45eda8637f8bad: reasonNewLeaderOrConfigChange | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977852+00:00 | 10s189ms795µs100ns | applied timestamp cache | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977959+00:00 | 10s189ms902µs190ns | evaluated request | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.977963+00:00 | 10s189ms906µs771ns | acquired {raft,replica}mu | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | 2018-06-04 19:47:19.979633+00:00 | 10s191ms576µs171ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.365449377,0 epo=3 | | | | | | |
| | | | pro=1528141636.592623635,0 | | | | | | |
| | 2018-06-04 19:47:19.97968+00:00 | 10s191ms622µs939ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | | | seq=6 start=1528141568.365449377,0 epo=3 pro=1528141636.592623635,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365449377,0 epo=3 | | | | | | |
| | | | pro=1528141636.592623635,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:19.979758+00:00 | 10s191ms701µs740ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 74 | | |
| | | | start=1528141568.365449377,0 epo=3 pro=1528141636.592623635,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365449377,0 epo=3 | | | | | | |
| | | | pro=1528141636.592623635,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:19.979807+00:00 | 10s191ms749µs966ns | read has no clock uncertainty | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:19.979813+00:00 | 10s191ms756µs67ns | command queue | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:19.97982+00:00 | 10s191ms763µs418ns | waiting for read lock | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:19.979836+00:00 | 10s191ms778µs844ns | read completed | [n1,s1,r42/1:/Table/52/1/1{800-900}] | | /cockroach.roachpb.Internal/Batch | 72 | | |
| | 2018-06-04 19:47:19.97992+00:00 | 10s191ms863µs666ns | === SPAN START: dist sender === | | | dist sender | 78 | | |
| | 2018-06-04 19:47:19.979937+00:00 | 10s191ms879µs859ns | querying next range at /Table/52/1/1900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 78 | | |
| | 2018-06-04 19:47:19.979968+00:00 | 10s191ms911µs555ns | r43: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 78 | | |
| | 2018-06-04 19:47:19.979991+00:00 | 10s191ms934µs729ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 78 | | |
| | 2018-06-04 19:47:19.980006+00:00 | 10s191ms949µs496ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:19.98001+00:00 | 10s191ms953µs459ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:19.980016+00:00 | 10s191ms958µs989ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:19.980125+00:00 | 10s192ms67µs840ns | read-only path | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:19.980133+00:00 | 10s192ms76µs329ns | request range lease (attempt #1) | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:19.98014+00:00 | 10s192ms83µs670ns | === SPAN START: request range lease === | | | request range lease | 80 | | |
| | 2018-06-04 19:47:19.980148+00:00 | 10s192ms91µs707ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:19.980184+00:00 | 10s192ms127µs710ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 82 | | |
| | 2018-06-04 19:47:19.980273+00:00 | 10s192ms216µs281ns | === SPAN START: dist sender === | | | dist sender | 83 | | |
| | 2018-06-04 19:47:19.980292+00:00 | 10s192ms235µs678ns | querying next range at /System/NodeLiveness/1 | [s1,r43/1:/Table/52/1/{1900-2000},txn=04a63d23,n1] | | dist sender | 83 | | |
| | 2018-06-04 19:47:19.980323+00:00 | 10s192ms266µs448ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r43/1:/Table/52/1/{1900-2000},txn=04a63d23,n1] | | dist sender | 83 | | |
| | 2018-06-04 19:47:19.980347+00:00 | 10s192ms290µs555ns | sending request to alex-laptop.local:26259 | [s1,r43/1:/Table/52/1/{1900-2000},txn=04a63d23,n1] | | dist sender | 83 | | |
| | 2018-06-04 19:47:19.980379+00:00 | 10s192ms322µs14ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 84 | | |
| | 2018-06-04 19:47:19.980696+00:00 | 10s192ms639µs269ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980761+00:00 | 10s192ms704µs622ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980773+00:00 | 10s192ms715µs881ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980782+00:00 | 10s192ms725µs436ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980802+00:00 | 10s192ms744µs865ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980838+00:00 | 10s192ms781µs496ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980933+00:00 | 10s192ms876µs371ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.980938+00:00 | 10s192ms880µs885ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.98163+00:00 | 10s193ms573µs678ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.981959+00:00 | 10s193ms902µs152ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | 2018-06-04 19:47:19.981983+00:00 | 10s193ms926µs740ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 412 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:19.982563+00:00 | 10s194ms505µs961ns | heartbeat 1528141649.480192922,0 | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | liveness heartbeat | 82 | | |
| | 2018-06-04 19:47:19.982571+00:00 | 10s194ms514µs394ns | read-write path | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:19.982579+00:00 | 10s194ms521µs896ns | applied timestamp cache | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:19.982669+00:00 | 10s194ms612µs258ns | evaluated request | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:19.982673+00:00 | 10s194ms616µs476ns | acquired {raft,replica}mu | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:22.980655+00:00 | 13s192ms598µs98ns | retry proposal 4afe90e9c3732894: reasonTicks | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:22.980711+00:00 | 13s192ms654µs179ns | applied timestamp cache | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:22.98087+00:00 | 13s192ms813µs742ns | evaluated request | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:22.980876+00:00 | 13s192ms818µs968ns | acquired {raft,replica}mu | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.583476+00:00 | 14s795ms419µs621ns | retry proposal 4de6644536330705: reasonNewLeader | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.58352+00:00 | 14s795ms462µs879ns | applied timestamp cache | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.583652+00:00 | 14s795ms595µs48ns | evaluated request | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.583665+00:00 | 14s795ms608µs684ns | acquired {raft,replica}mu | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.584328+00:00 | 14s796ms270µs953ns | retry proposal 47cc4a748939a092: reasonNewLeaderOrConfigChange | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.584363+00:00 | 14s796ms306µs82ns | applied timestamp cache | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.584441+00:00 | 14s796ms384µs246ns | evaluated request | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.584452+00:00 | 14s796ms395µs267ns | acquired {raft,replica}mu | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | 2018-06-04 19:47:24.58564+00:00 | 14s797ms583µs459ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.365564620,0 epo=3 | | | | | | |
| | | | pro=1528141639.980135959,0 | | | | | | |
| | 2018-06-04 19:47:24.585658+00:00 | 14s797ms601µs308ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | | | seq=6 start=1528141568.365564620,0 epo=3 pro=1528141639.980135959,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365564620,0 epo=3 | | | | | | |
| | | | pro=1528141639.980135959,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:24.585733+00:00 | 14s797ms676µs506ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | [async] storage.pendingLeaseRequest: requesting lease | 81 | | |
| | | | start=1528141568.365564620,0 epo=3 pro=1528141639.980135959,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365564620,0 epo=3 | | | | | | |
| | | | pro=1528141639.980135959,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:24.585778+00:00 | 14s797ms721µs742ns | read has no clock uncertainty | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:24.585784+00:00 | 14s797ms727µs217ns | command queue | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:24.585791+00:00 | 14s797ms734µs170ns | waiting for read lock | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:24.585804+00:00 | 14s797ms747µs351ns | read completed | [n1,s1,r43/1:/Table/52/1/{1900-2000}] | | /cockroach.roachpb.Internal/Batch | 79 | | |
| | 2018-06-04 19:47:24.585927+00:00 | 14s797ms869µs975ns | === SPAN START: dist sender === | | | dist sender | 85 | | |
| | 2018-06-04 19:47:24.585951+00:00 | 14s797ms894µs111ns | querying next range at /Table/52/1/2000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 85 | | |
| | 2018-06-04 19:47:24.585982+00:00 | 14s797ms925µs379ns | r44: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 85 | | |
| | 2018-06-04 19:47:24.585998+00:00 | 14s797ms941µs693ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 85 | | |
| | 2018-06-04 19:47:24.586013+00:00 | 14s797ms955µs852ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:24.586017+00:00 | 14s797ms960µs11ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:24.586022+00:00 | 14s797ms965µs693ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:24.586122+00:00 | 14s798ms65µs128ns | read-only path | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:24.586127+00:00 | 14s798ms70µs334ns | request range lease (attempt #1) | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:24.586134+00:00 | 14s798ms77µs310ns | === SPAN START: request range lease === | | | request range lease | 87 | | |
| | 2018-06-04 19:47:24.586142+00:00 | 14s798ms85µs802ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:24.586164+00:00 | 14s798ms107µs742ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 89 | | |
| | 2018-06-04 19:47:24.586248+00:00 | 14s798ms191µs281ns | === SPAN START: dist sender === | | | dist sender | 90 | | |
| | 2018-06-04 19:47:24.586261+00:00 | 14s798ms204µs397ns | querying next range at /System/NodeLiveness/1 | [s1,r44/1:/Table/52/1/2{000-100},txn=22722bef,n1] | | dist sender | 90 | | |
| | 2018-06-04 19:47:24.586287+00:00 | 14s798ms230µs99ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r44/1:/Table/52/1/2{000-100},txn=22722bef,n1] | | dist sender | 90 | | |
| | 2018-06-04 19:47:24.586326+00:00 | 14s798ms268µs931ns | sending request to alex-laptop.local:26259 | [s1,r44/1:/Table/52/1/2{000-100},txn=22722bef,n1] | | dist sender | 90 | | |
| | 2018-06-04 19:47:24.586355+00:00 | 14s798ms298µs445ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 91 | | |
| | 2018-06-04 19:47:24.586712+00:00 | 14s798ms654µs957ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586803+00:00 | 14s798ms746µs613ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586816+00:00 | 14s798ms759µs507ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586826+00:00 | 14s798ms769µs346ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586842+00:00 | 14s798ms784µs910ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586878+00:00 | 14s798ms821µs279ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586994+00:00 | 14s798ms936µs820ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.586998+00:00 | 14s798ms941µs445ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.58772+00:00 | 14s799ms662µs859ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.588091+00:00 | 14s800ms34µs25ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | 2018-06-04 19:47:24.588117+00:00 | 14s800ms60µs729ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 413 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:24.588629+00:00 | 14s800ms571µs870ns | heartbeat 1528141654.086168877,0 | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | liveness heartbeat | 89 | | |
| | 2018-06-04 19:47:24.588639+00:00 | 14s800ms581µs996ns | read-write path | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:24.588646+00:00 | 14s800ms589µs431ns | applied timestamp cache | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:24.588743+00:00 | 14s800ms686µs429ns | evaluated request | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:24.588746+00:00 | 14s800ms689µs677ns | acquired {raft,replica}mu | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:27.580386+00:00 | 17s792ms329µs721ns | retry proposal 59345aa9e3705c65: reasonTicks | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:27.580448+00:00 | 17s792ms391µs316ns | applied timestamp cache | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:27.580613+00:00 | 17s792ms556µs364ns | evaluated request | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:27.580618+00:00 | 17s792ms561µs458ns | acquired {raft,replica}mu | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.584357+00:00 | 18s796ms300µs208ns | retry proposal 39fd5d801bce89f5: reasonNewLeader | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.584446+00:00 | 18s796ms389µs809ns | applied timestamp cache | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.584588+00:00 | 18s796ms531µs126ns | evaluated request | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.584602+00:00 | 18s796ms545µs72ns | acquired {raft,replica}mu | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.585141+00:00 | 18s797ms84µs556ns | retry proposal 0f6b5ac130473dc9: reasonNewLeaderOrConfigChange | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.585181+00:00 | 18s797ms124µs773ns | applied timestamp cache | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.585283+00:00 | 18s797ms226µs300ns | evaluated request | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.585296+00:00 | 18s797ms239µs246ns | acquired {raft,replica}mu | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | 2018-06-04 19:47:28.586535+00:00 | 18s798ms478µs543ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.365680310,0 epo=3 | | | | | | |
| | | | pro=1528141644.586129448,0 | | | | | | |
| | 2018-06-04 19:47:28.586561+00:00 | 18s798ms504µs668ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | | | seq=6 start=1528141568.365680310,0 epo=3 pro=1528141644.586129448,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365680310,0 epo=3 | | | | | | |
| | | | pro=1528141644.586129448,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:28.586651+00:00 | 18s798ms594µs426ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 88 | | |
| | | | start=1528141568.365680310,0 epo=3 pro=1528141644.586129448,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365680310,0 epo=3 | | | | | | |
| | | | pro=1528141644.586129448,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:28.586703+00:00 | 18s798ms646µs198ns | read has no clock uncertainty | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:28.586711+00:00 | 18s798ms654µs413ns | command queue | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:28.586721+00:00 | 18s798ms664µs254ns | waiting for read lock | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:28.586738+00:00 | 18s798ms681µs225ns | read completed | [n1,s1,r44/1:/Table/52/1/2{000-100}] | | /cockroach.roachpb.Internal/Batch | 86 | | |
| | 2018-06-04 19:47:28.586842+00:00 | 18s798ms785µs165ns | === SPAN START: dist sender === | | | dist sender | 92 | | |
| | 2018-06-04 19:47:28.586859+00:00 | 18s798ms802µs219ns | querying next range at /Table/52/1/2100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 92 | | |
| | 2018-06-04 19:47:28.58689+00:00 | 18s798ms833µs338ns | r45: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 92 | | |
| | 2018-06-04 19:47:28.586915+00:00 | 18s798ms857µs816ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 92 | | |
| | 2018-06-04 19:47:28.586929+00:00 | 18s798ms872µs32ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:28.586933+00:00 | 18s798ms876µs68ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:28.586939+00:00 | 18s798ms881µs998ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:28.587033+00:00 | 18s798ms976µs556ns | read-only path | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:28.587039+00:00 | 18s798ms982µs251ns | request range lease (attempt #1) | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:28.587046+00:00 | 18s798ms989µs694ns | === SPAN START: request range lease === | | | request range lease | 94 | | |
| | 2018-06-04 19:47:28.587054+00:00 | 18s798ms997µs646ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:28.587076+00:00 | 18s799ms19µs6ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 96 | | |
| | 2018-06-04 19:47:28.587154+00:00 | 18s799ms97µs107ns | === SPAN START: dist sender === | | | dist sender | 97 | | |
| | 2018-06-04 19:47:28.587167+00:00 | 18s799ms110µs286ns | querying next range at /System/NodeLiveness/1 | [s1,r45/1:/Table/52/1/2{100-200},txn=40864a8f,n1] | | dist sender | 97 | | |
| | 2018-06-04 19:47:28.587191+00:00 | 18s799ms134µs76ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r45/1:/Table/52/1/2{100-200},txn=40864a8f,n1] | | dist sender | 97 | | |
| | 2018-06-04 19:47:28.587208+00:00 | 18s799ms151µs562ns | sending request to alex-laptop.local:26259 | [s1,r45/1:/Table/52/1/2{100-200},txn=40864a8f,n1] | | dist sender | 97 | | |
| | 2018-06-04 19:47:28.587228+00:00 | 18s799ms171µs558ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 98 | | |
| | 2018-06-04 19:47:28.587538+00:00 | 18s799ms481µs485ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587583+00:00 | 18s799ms526µs177ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587594+00:00 | 18s799ms537µs80ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587603+00:00 | 18s799ms545µs959ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587617+00:00 | 18s799ms560µs716ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587651+00:00 | 18s799ms594µs68ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587753+00:00 | 18s799ms696µs654ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.587757+00:00 | 18s799ms700µs710ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.58845+00:00 | 18s800ms393µs292ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.588778+00:00 | 18s800ms721µs560ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | 2018-06-04 19:47:28.588805+00:00 | 18s800ms748µs211ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 414 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:28.589405+00:00 | 18s801ms348µs352ns | heartbeat 1528141658.087079500,0 | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | liveness heartbeat | 96 | | |
| | 2018-06-04 19:47:28.589413+00:00 | 18s801ms356µs457ns | read-write path | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:28.589423+00:00 | 18s801ms366µs152ns | applied timestamp cache | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:28.589508+00:00 | 18s801ms451µs17ns | evaluated request | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:28.589511+00:00 | 18s801ms454µs15ns | acquired {raft,replica}mu | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:31.575748+00:00 | 21s787ms691µs710ns | retry proposal 2c31750b06a3586f: reasonTicks | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:31.57581+00:00 | 21s787ms753µs472ns | applied timestamp cache | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:31.57608+00:00 | 21s788ms22µs979ns | evaluated request | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:31.576085+00:00 | 21s788ms28µs67ns | acquired {raft,replica}mu | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.976391+00:00 | 23s188ms333µs972ns | retry proposal 5e38522aef1635f0: reasonNewLeader | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.976431+00:00 | 23s188ms374µs570ns | applied timestamp cache | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.976576+00:00 | 23s188ms519µs243ns | evaluated request | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.976582+00:00 | 23s188ms525µs626ns | acquired {raft,replica}mu | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.977122+00:00 | 23s189ms64µs936ns | retry proposal 5c3ff670ae7e7ea5: reasonNewLeaderOrConfigChange | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.977171+00:00 | 23s189ms114µs760ns | applied timestamp cache | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.97729+00:00 | 23s189ms233µs21ns | evaluated request | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.977403+00:00 | 23s189ms346µs743ns | acquired {raft,replica}mu | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | 2018-06-04 19:47:32.978672+00:00 | 23s190ms615µs139ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.365795221,0 epo=3 | | | | | | |
| | | | pro=1528141648.587041719,0 | | | | | | |
| | 2018-06-04 19:47:32.97869+00:00 | 23s190ms633µs809ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | | | seq=6 start=1528141568.365795221,0 epo=3 pro=1528141648.587041719,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365795221,0 epo=3 | | | | | | |
| | | | pro=1528141648.587041719,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:32.978769+00:00 | 23s190ms712µs646ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 95 | | |
| | | | start=1528141568.365795221,0 epo=3 pro=1528141648.587041719,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365795221,0 epo=3 | | | | | | |
| | | | pro=1528141648.587041719,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:32.978823+00:00 | 23s190ms766µs139ns | read has no clock uncertainty | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:32.978828+00:00 | 23s190ms771µs715ns | command queue | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:32.978835+00:00 | 23s190ms778µs636ns | waiting for read lock | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:32.97885+00:00 | 23s190ms793µs257ns | read completed | [n1,s1,r45/1:/Table/52/1/2{100-200}] | | /cockroach.roachpb.Internal/Batch | 93 | | |
| | 2018-06-04 19:47:32.978954+00:00 | 23s190ms897µs187ns | === SPAN START: dist sender === | | | dist sender | 99 | | |
| | 2018-06-04 19:47:32.978975+00:00 | 23s190ms918µs36ns | querying next range at /Table/52/1/2200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 99 | | |
| | 2018-06-04 19:47:32.979019+00:00 | 23s190ms962µs17ns | r46: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 99 | | |
| | 2018-06-04 19:47:32.979041+00:00 | 23s190ms983µs839ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 99 | | |
| | 2018-06-04 19:47:32.979055+00:00 | 23s190ms998µs428ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:32.979059+00:00 | 23s191ms2µs468ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:32.979065+00:00 | 23s191ms8µs358ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:32.979169+00:00 | 23s191ms111µs822ns | read-only path | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:32.979174+00:00 | 23s191ms117µs477ns | request range lease (attempt #1) | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:32.979181+00:00 | 23s191ms124µs386ns | === SPAN START: request range lease === | | | request range lease | 101 | | |
| | 2018-06-04 19:47:32.97919+00:00 | 23s191ms133µs593ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:32.979212+00:00 | 23s191ms155µs313ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 103 | | |
| | 2018-06-04 19:47:32.979292+00:00 | 23s191ms235µs262ns | === SPAN START: dist sender === | | | dist sender | 104 | | |
| | 2018-06-04 19:47:32.979305+00:00 | 23s191ms248µs739ns | querying next range at /System/NodeLiveness/1 | [s1,r46/1:/Table/52/1/2{200-300},txn=52cd37c2,n1] | | dist sender | 104 | | |
| | 2018-06-04 19:47:32.979331+00:00 | 23s191ms274µs366ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r46/1:/Table/52/1/2{200-300},txn=52cd37c2,n1] | | dist sender | 104 | | |
| | 2018-06-04 19:47:32.97935+00:00 | 23s191ms293µs123ns | sending request to alex-laptop.local:26259 | [s1,r46/1:/Table/52/1/2{200-300},txn=52cd37c2,n1] | | dist sender | 104 | | |
| | 2018-06-04 19:47:32.97937+00:00 | 23s191ms313µs745ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 105 | | |
| | 2018-06-04 19:47:32.979691+00:00 | 23s191ms634µs38ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979726+00:00 | 23s191ms669µs695ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979735+00:00 | 23s191ms678µs53ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979741+00:00 | 23s191ms684µs690ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979755+00:00 | 23s191ms698µs300ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979785+00:00 | 23s191ms727µs831ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.979856+00:00 | 23s191ms799µs391ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.97986+00:00 | 23s191ms803µs198ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.980559+00:00 | 23s192ms501µs874ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.980949+00:00 | 23s192ms892µs774ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | 2018-06-04 19:47:32.980977+00:00 | 23s192ms920µs164ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 415 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:32.981566+00:00 | 23s193ms509µs186ns | heartbeat 1528141662.479216401,0 | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | liveness heartbeat | 103 | | |
| | 2018-06-04 19:47:32.981576+00:00 | 23s193ms519µs447ns | read-write path | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:32.981584+00:00 | 23s193ms527µs13ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:32.981677+00:00 | 23s193ms620µs309ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:32.981682+00:00 | 23s193ms624µs935ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:35.975824+00:00 | 26s187ms767µs339ns | retry proposal 2a70a2d743d86b3b: reasonTicks | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:35.975872+00:00 | 26s187ms815µs811ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:35.976013+00:00 | 26s187ms956µs649ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:35.976018+00:00 | 26s187ms960µs924ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.381518+00:00 | 26s593ms461µs637ns | retry proposal 6274bf54cc28c241: reasonNewLeader | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.381566+00:00 | 26s593ms509µs205ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.381751+00:00 | 26s593ms693µs990ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.381756+00:00 | 26s593ms699µs629ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.382375+00:00 | 26s594ms318µs201ns | retry proposal 6826022f38185cfb: reasonNewLeaderOrConfigChange | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.382442+00:00 | 26s594ms385µs32ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.382573+00:00 | 26s594ms516µs173ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.382672+00:00 | 26s594ms615µs415ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.383609+00:00 | 26s595ms552µs158ns | retry proposal 5cd3cfd400ba12b5: reasonNewLeader | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.383624+00:00 | 26s595ms566µs979ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.383684+00:00 | 26s595ms626µs926ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.383687+00:00 | 26s595ms630µs465ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.384347+00:00 | 26s596ms290µs655ns | retry proposal 2b1b123aceb70e96: reasonNewLeader | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.384405+00:00 | 26s596ms348µs601ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.384497+00:00 | 26s596ms440µs353ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.384511+00:00 | 26s596ms453µs903ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.384982+00:00 | 26s596ms925µs545ns | retry proposal 385688585d2dcc7b: reasonNewLeaderOrConfigChange | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.385014+00:00 | 26s596ms957µs95ns | applied timestamp cache | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.385081+00:00 | 26s597ms24µs309ns | evaluated request | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.385091+00:00 | 26s597ms34µs739ns | acquired {raft,replica}mu | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | 2018-06-04 19:47:36.385878+00:00 | 26s597ms821µs359ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.365908415,0 epo=3 | | | | | | |
| | | | pro=1528141652.979176881,0 | | | | | | |
| | 2018-06-04 19:47:36.385901+00:00 | 26s597ms844µs368ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | | | seq=6 start=1528141568.365908415,0 epo=3 pro=1528141652.979176881,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365908415,0 epo=3 | | | | | | |
| | | | pro=1528141652.979176881,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:36.386039+00:00 | 26s597ms982µs699ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 102 | | |
| | | | start=1528141568.365908415,0 epo=3 pro=1528141652.979176881,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.365908415,0 epo=3 | | | | | | |
| | | | pro=1528141652.979176881,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:36.386162+00:00 | 26s598ms104µs816ns | read has no clock uncertainty | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:36.386167+00:00 | 26s598ms110µs383ns | command queue | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:36.386175+00:00 | 26s598ms118µs580ns | waiting for read lock | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:36.386193+00:00 | 26s598ms135µs912ns | read completed | [n1,s1,r46/1:/Table/52/1/2{200-300}] | | /cockroach.roachpb.Internal/Batch | 100 | | |
| | 2018-06-04 19:47:36.38629+00:00 | 26s598ms233µs105ns | === SPAN START: dist sender === | | | dist sender | 106 | | |
| | 2018-06-04 19:47:36.386305+00:00 | 26s598ms248µs700ns | querying next range at /Table/52/1/2300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 106 | | |
| | 2018-06-04 19:47:36.386338+00:00 | 26s598ms281µs148ns | r47: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 106 | | |
| | 2018-06-04 19:47:36.386361+00:00 | 26s598ms304µs552ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 106 | | |
| | 2018-06-04 19:47:36.386395+00:00 | 26s598ms337µs950ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:36.386398+00:00 | 26s598ms341µs759ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:36.386404+00:00 | 26s598ms346µs901ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:36.386501+00:00 | 26s598ms444µs85ns | read-only path | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:36.386506+00:00 | 26s598ms449µs74ns | request range lease (attempt #1) | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:36.386512+00:00 | 26s598ms455µs473ns | === SPAN START: request range lease === | | | request range lease | 108 | | |
| | 2018-06-04 19:47:36.386519+00:00 | 26s598ms462µs279ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:36.386539+00:00 | 26s598ms482µs399ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 110 | | |
| | 2018-06-04 19:47:36.386607+00:00 | 26s598ms550µs722ns | === SPAN START: dist sender === | | | dist sender | 111 | | |
| | 2018-06-04 19:47:36.386621+00:00 | 26s598ms564µs355ns | querying next range at /System/NodeLiveness/1 | [s1,r47/1:/Table/52/1/2{300-400},txn=08707ce3,n1] | | dist sender | 111 | | |
| | 2018-06-04 19:47:36.386645+00:00 | 26s598ms588µs736ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r47/1:/Table/52/1/2{300-400},txn=08707ce3,n1] | | dist sender | 111 | | |
| | 2018-06-04 19:47:36.386662+00:00 | 26s598ms605µs665ns | sending request to alex-laptop.local:26259 | [s1,r47/1:/Table/52/1/2{300-400},txn=08707ce3,n1] | | dist sender | 111 | | |
| | 2018-06-04 19:47:36.386681+00:00 | 26s598ms624µs279ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 112 | | |
| | 2018-06-04 19:47:36.386971+00:00 | 26s598ms914µs595ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387016+00:00 | 26s598ms959µs187ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387024+00:00 | 26s598ms966µs848ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.38703+00:00 | 26s598ms973µs764ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387043+00:00 | 26s598ms986µs214ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387085+00:00 | 26s599ms28µs721ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387154+00:00 | 26s599ms97µs583ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387158+00:00 | 26s599ms100µs915ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.387807+00:00 | 26s599ms750µs202ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.388111+00:00 | 26s600ms54µs237ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | 2018-06-04 19:47:36.388133+00:00 | 26s600ms76µs49ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 416 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:36.388564+00:00 | 26s600ms507µs624ns | heartbeat 1528141665.886542838,0 | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | liveness heartbeat | 110 | | |
| | 2018-06-04 19:47:36.388572+00:00 | 26s600ms514µs871ns | read-write path | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:36.388579+00:00 | 26s600ms522µs724ns | applied timestamp cache | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:36.388659+00:00 | 26s600ms601µs979ns | evaluated request | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:36.388661+00:00 | 26s600ms604µs683ns | acquired {raft,replica}mu | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.377164+00:00 | 29s589ms107µs209ns | retry proposal 3bcaa974f751a22a: reasonTicks | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.377221+00:00 | 29s589ms163µs823ns | applied timestamp cache | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.377431+00:00 | 29s589ms374µs402ns | evaluated request | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.377437+00:00 | 29s589ms379µs842ns | acquired {raft,replica}mu | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.975672+00:00 | 30s187ms615µs649ns | retry proposal 350a342b814a416f: reasonNewLeader | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.975711+00:00 | 30s187ms653µs995ns | applied timestamp cache | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.975827+00:00 | 30s187ms770µs358ns | evaluated request | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.975832+00:00 | 30s187ms775µs218ns | acquired {raft,replica}mu | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.976189+00:00 | 30s188ms131µs908ns | retry proposal 63a110c32453933f: reasonNewLeaderOrConfigChange | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.976215+00:00 | 30s188ms157µs990ns | applied timestamp cache | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.976305+00:00 | 30s188ms247µs924ns | evaluated request | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.976308+00:00 | 30s188ms251µs790ns | acquired {raft,replica}mu | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | 2018-06-04 19:47:39.977483+00:00 | 30s189ms426µs586ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.366026368,0 epo=3 | | | | | | |
| | | | pro=1528141656.386508181,0 | | | | | | |
| | 2018-06-04 19:47:39.9775+00:00 | 30s189ms442µs948ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | | | seq=4 start=1528141568.366026368,0 epo=3 pro=1528141656.386508181,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366026368,0 epo=3 | | | | | | |
| | | | pro=1528141656.386508181,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:39.97756+00:00 | 30s189ms502µs953ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 109 | | |
| | | | start=1528141568.366026368,0 epo=3 pro=1528141656.386508181,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366026368,0 epo=3 | | | | | | |
| | | | pro=1528141656.386508181,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:39.977603+00:00 | 30s189ms545µs913ns | read has no clock uncertainty | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:39.977608+00:00 | 30s189ms551µs34ns | command queue | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:39.977614+00:00 | 30s189ms557µs196ns | waiting for read lock | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:39.977625+00:00 | 30s189ms568µs76ns | read completed | [n1,s1,r47/1:/Table/52/1/2{300-400}] | | /cockroach.roachpb.Internal/Batch | 107 | | |
| | 2018-06-04 19:47:39.977706+00:00 | 30s189ms649µs241ns | === SPAN START: dist sender === | | | dist sender | 113 | | |
| | 2018-06-04 19:47:39.97772+00:00 | 30s189ms663µs317ns | querying next range at /Table/52/1/2400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 113 | | |
| | 2018-06-04 19:47:39.97775+00:00 | 30s189ms693µs772ns | r48: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 113 | | |
| | 2018-06-04 19:47:39.977774+00:00 | 30s189ms717µs325ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 113 | | |
| | 2018-06-04 19:47:39.977788+00:00 | 30s189ms730µs912ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:39.977792+00:00 | 30s189ms734µs968ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:39.977797+00:00 | 30s189ms739µs957ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:39.977886+00:00 | 30s189ms828µs913ns | read-only path | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:39.977891+00:00 | 30s189ms834µs206ns | request range lease (attempt #1) | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:39.977897+00:00 | 30s189ms840µs417ns | === SPAN START: request range lease === | | | request range lease | 115 | | |
| | 2018-06-04 19:47:39.977904+00:00 | 30s189ms847µs813ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:39.977925+00:00 | 30s189ms868µs348ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 117 | | |
| | 2018-06-04 19:47:39.978001+00:00 | 30s189ms943µs958ns | === SPAN START: dist sender === | | | dist sender | 118 | | |
| | 2018-06-04 19:47:39.978013+00:00 | 30s189ms955µs866ns | querying next range at /System/NodeLiveness/1 | [s1,r48/1:/Table/52/1/2{400-500},txn=8994b536,n1] | | dist sender | 118 | | |
| | 2018-06-04 19:47:39.978037+00:00 | 30s189ms979µs991ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r48/1:/Table/52/1/2{400-500},txn=8994b536,n1] | | dist sender | 118 | | |
| | 2018-06-04 19:47:39.978054+00:00 | 30s189ms996µs954ns | sending request to alex-laptop.local:26259 | [s1,r48/1:/Table/52/1/2{400-500},txn=8994b536,n1] | | dist sender | 118 | | |
| | 2018-06-04 19:47:39.978073+00:00 | 30s190ms16µs722ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 119 | | |
| | 2018-06-04 19:47:39.978383+00:00 | 30s190ms326µs714ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978433+00:00 | 30s190ms376µs651ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978441+00:00 | 30s190ms384µs622ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978448+00:00 | 30s190ms391µs52ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978461+00:00 | 30s190ms404µs513ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978503+00:00 | 30s190ms446µs8ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978584+00:00 | 30s190ms527µs234ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.978588+00:00 | 30s190ms531µs335ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.979278+00:00 | 30s191ms221µs22ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.979547+00:00 | 30s191ms490µs225ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | 2018-06-04 19:47:39.97957+00:00 | 30s191ms513µs687ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 417 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:39.980001+00:00 | 30s191ms944µs431ns | heartbeat 1528141669.477928955,0 | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | liveness heartbeat | 117 | | |
| | 2018-06-04 19:47:39.98001+00:00 | 30s191ms953µs178ns | read-write path | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:39.980019+00:00 | 30s191ms962µs643ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:39.980097+00:00 | 30s192ms40µs709ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:39.9801+00:00 | 30s192ms43µs612ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:42.978613+00:00 | 33s190ms556µs588ns | retry proposal 2f70fc7d82040ec0: reasonTicks | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:42.978722+00:00 | 33s190ms665µs538ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:42.978905+00:00 | 33s190ms848µs610ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:42.97891+00:00 | 33s190ms853µs655ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.780041+00:00 | 34s991ms984µs557ns | retry proposal 4a6d3173b0e969bb: reasonNewLeader | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.780077+00:00 | 34s992ms20µs814ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.780214+00:00 | 34s992ms157µs671ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.78022+00:00 | 34s992ms163µs286ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.780841+00:00 | 34s992ms784µs276ns | retry proposal 1a6ed69599d54b71: reasonNewLeaderOrConfigChange | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.780905+00:00 | 34s992ms848µs184ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.781028+00:00 | 34s992ms970µs909ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.781144+00:00 | 34s993ms87µs565ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.782201+00:00 | 34s994ms143µs999ns | retry proposal 062f66ad9c729644: reasonNewLeader | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.782217+00:00 | 34s994ms160µs793ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.782278+00:00 | 34s994ms221µs676ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.782281+00:00 | 34s994ms224µs99ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.782923+00:00 | 34s994ms866µs403ns | retry proposal 4a35391fc9318a81: reasonNewLeader | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783008+00:00 | 34s994ms950µs924ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783124+00:00 | 34s995ms67µs226ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783137+00:00 | 34s995ms80µs499ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783608+00:00 | 34s995ms551µs15ns | retry proposal 50cab8e60f41670c: reasonNewLeaderOrConfigChange | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783633+00:00 | 34s995ms576µs498ns | applied timestamp cache | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.78369+00:00 | 34s995ms633µs805ns | evaluated request | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.783699+00:00 | 34s995ms641µs817ns | acquired {raft,replica}mu | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | 2018-06-04 19:47:44.78447+00:00 | 34s996ms412µs943ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.366146142,0 epo=3 | | | | | | |
| | | | pro=1528141659.977893295,0 | | | | | | |
| | 2018-06-04 19:47:44.784492+00:00 | 34s996ms434µs827ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | | | seq=4 start=1528141568.366146142,0 epo=3 pro=1528141659.977893295,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366146142,0 epo=3 | | | | | | |
| | | | pro=1528141659.977893295,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:44.784588+00:00 | 34s996ms531µs335ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 116 | | |
| | | | start=1528141568.366146142,0 epo=3 pro=1528141659.977893295,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366146142,0 epo=3 | | | | | | |
| | | | pro=1528141659.977893295,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:44.784646+00:00 | 34s996ms589µs480ns | read has no clock uncertainty | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:44.784654+00:00 | 34s996ms596µs923ns | command queue | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:44.784663+00:00 | 34s996ms606µs471ns | waiting for read lock | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:44.78468+00:00 | 34s996ms623µs170ns | read completed | [n1,s1,r48/1:/Table/52/1/2{400-500}] | | /cockroach.roachpb.Internal/Batch | 114 | | |
| | 2018-06-04 19:47:44.784778+00:00 | 34s996ms721µs336ns | === SPAN START: dist sender === | | | dist sender | 120 | | |
| | 2018-06-04 19:47:44.784796+00:00 | 34s996ms739µs364ns | querying next range at /Table/52/1/2500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 120 | | |
| | 2018-06-04 19:47:44.784827+00:00 | 34s996ms769µs857ns | r49: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 120 | | |
| | 2018-06-04 19:47:44.784849+00:00 | 34s996ms792µs246ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 120 | | |
| | 2018-06-04 19:47:44.784864+00:00 | 34s996ms806µs993ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:44.784868+00:00 | 34s996ms811µs41ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:44.784876+00:00 | 34s996ms819µs39ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:44.784974+00:00 | 34s996ms916µs858ns | read-only path | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:44.784979+00:00 | 34s996ms922µs144ns | request range lease (attempt #1) | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:44.784989+00:00 | 34s996ms932µs27ns | === SPAN START: request range lease === | | | request range lease | 122 | | |
| | 2018-06-04 19:47:44.784996+00:00 | 34s996ms939µs220ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:44.785017+00:00 | 34s996ms960µs186ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 124 | | |
| | 2018-06-04 19:47:44.785107+00:00 | 34s997ms49µs910ns | === SPAN START: dist sender === | | | dist sender | 125 | | |
| | 2018-06-04 19:47:44.785119+00:00 | 34s997ms62µs447ns | querying next range at /System/NodeLiveness/1 | [s1,r49/1:/Table/52/1/2{500-600},txn=9682a97e,n1] | | dist sender | 125 | | |
| | 2018-06-04 19:47:44.785141+00:00 | 34s997ms84µs335ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r49/1:/Table/52/1/2{500-600},txn=9682a97e,n1] | | dist sender | 125 | | |
| | 2018-06-04 19:47:44.785157+00:00 | 34s997ms100µs112ns | sending request to alex-laptop.local:26259 | [s1,r49/1:/Table/52/1/2{500-600},txn=9682a97e,n1] | | dist sender | 125 | | |
| | 2018-06-04 19:47:44.785175+00:00 | 34s997ms118µs762ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 126 | | |
| | 2018-06-04 19:47:44.785459+00:00 | 34s997ms402µs13ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785501+00:00 | 34s997ms444µs667ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785509+00:00 | 34s997ms452µs218ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785516+00:00 | 34s997ms458µs946ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785528+00:00 | 34s997ms471µs509ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785556+00:00 | 34s997ms499µs335ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785634+00:00 | 34s997ms577µs747ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.785637+00:00 | 34s997ms580µs624ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.786174+00:00 | 34s998ms116µs930ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.786426+00:00 | 34s998ms368µs858ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | 2018-06-04 19:47:44.786448+00:00 | 34s998ms391µs108ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 418 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:44.786902+00:00 | 34s998ms845µs731ns | heartbeat 1528141674.285020605,0 | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | liveness heartbeat | 124 | | |
| | 2018-06-04 19:47:44.786914+00:00 | 34s998ms857µs181ns | read-write path | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:44.786932+00:00 | 34s998ms875µs562ns | applied timestamp cache | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:44.787033+00:00 | 34s998ms975µs888ns | evaluated request | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:44.787036+00:00 | 34s998ms979µs288ns | acquired {raft,replica}mu | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:47.775694+00:00 | 37s987ms637µs129ns | retry proposal 5c26f84bfa7d4b0a: reasonTicks | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:47.77578+00:00 | 37s987ms723µs365ns | applied timestamp cache | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:47.775936+00:00 | 37s987ms878µs945ns | evaluated request | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:47.77594+00:00 | 37s987ms883µs649ns | acquired {raft,replica}mu | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.181303+00:00 | 38s393ms246µs631ns | retry proposal 6313d5e8358a1308: reasonNewLeader | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.181441+00:00 | 38s393ms384µs651ns | applied timestamp cache | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.181607+00:00 | 38s393ms550µs703ns | evaluated request | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.181626+00:00 | 38s393ms569µs740ns | acquired {raft,replica}mu | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.182153+00:00 | 38s394ms96µs173ns | retry proposal 28538949b250a0d0: reasonNewLeaderOrConfigChange | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.18223+00:00 | 38s394ms173µs89ns | applied timestamp cache | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.182365+00:00 | 38s394ms307µs919ns | evaluated request | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.182427+00:00 | 38s394ms370µs668ns | acquired {raft,replica}mu | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | 2018-06-04 19:47:48.183579+00:00 | 38s395ms522µs713ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.366263716,0 epo=3 | | | | | | |
| | | | pro=1528141664.784981501,0 | | | | | | |
| | 2018-06-04 19:47:48.183613+00:00 | 38s395ms555µs856ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | | | seq=6 start=1528141568.366263716,0 epo=3 pro=1528141664.784981501,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366263716,0 epo=3 | | | | | | |
| | | | pro=1528141664.784981501,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:48.183715+00:00 | 38s395ms658µs637ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 123 | | |
| | | | start=1528141568.366263716,0 epo=3 pro=1528141664.784981501,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366263716,0 epo=3 | | | | | | |
| | | | pro=1528141664.784981501,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:48.183781+00:00 | 38s395ms724µs52ns | read has no clock uncertainty | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:48.183789+00:00 | 38s395ms732µs685ns | command queue | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:48.183799+00:00 | 38s395ms742µs697ns | waiting for read lock | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:48.183819+00:00 | 38s395ms762µs575ns | read completed | [n1,s1,r49/1:/Table/52/1/2{500-600}] | | /cockroach.roachpb.Internal/Batch | 121 | | |
| | 2018-06-04 19:47:48.18394+00:00 | 38s395ms883µs566ns | === SPAN START: dist sender === | | | dist sender | 127 | | |
| | 2018-06-04 19:47:48.183966+00:00 | 38s395ms909µs529ns | querying next range at /Table/52/1/2600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 127 | | |
| | 2018-06-04 19:47:48.184005+00:00 | 38s395ms948µs85ns | === SPAN START: range lookup === | | | range lookup | 128 | | |
| | 2018-06-04 19:47:48.184043+00:00 | 38s395ms986µs249ns | === SPAN START: dist sender === | | | dist sender | 129 | | |
| | 2018-06-04 19:47:48.184092+00:00 | 38s396ms35µs699ns | querying next range at /Meta2/Table/52/1/2600/NULL | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/2600,n1] | | dist sender | 129 | | |
| | 2018-06-04 19:47:48.184168+00:00 | 38s396ms111µs431ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/2600,n1] | | dist sender | 129 | | |
| | 2018-06-04 19:47:48.184226+00:00 | 38s396ms168µs991ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/2600,n1] | | dist sender | 129 | | |
| | 2018-06-04 19:47:48.18424+00:00 | 38s396ms183µs347ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 130 | | |
| | 2018-06-04 19:47:48.185344+00:00 | 38s397ms287µs362ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185402+00:00 | 38s397ms344µs973ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185414+00:00 | 38s397ms356µs985ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185426+00:00 | 38s397ms369µs587ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185451+00:00 | 38s397ms394µs564ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185455+00:00 | 38s397ms398µs334ns | operation accepts inconsistent results | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.18546+00:00 | 38s397ms403µs66ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185941+00:00 | 38s397ms884µs722ns | engine stats: {InternalDeleteSkippedCount:10 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.185958+00:00 | 38s397ms900µs905ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 419 | | |
| | 2018-06-04 19:47:48.186809+00:00 | 38s398ms752µs436ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 127 | | |
| | 2018-06-04 19:47:48.186847+00:00 | 38s398ms790µs266ns | r50: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 127 | | |
| | 2018-06-04 19:47:48.18687+00:00 | 38s398ms813µs266ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 127 | | |
| | 2018-06-04 19:47:48.186911+00:00 | 38s398ms854µs107ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:48.186919+00:00 | 38s398ms861µs884ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:48.186927+00:00 | 38s398ms870µs67ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:48.187093+00:00 | 38s399ms36µs184ns | read-only path | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:48.187099+00:00 | 38s399ms42µs340ns | request range lease (attempt #1) | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:48.187109+00:00 | 38s399ms52µs8ns | === SPAN START: request range lease === | | | request range lease | 132 | | |
| | 2018-06-04 19:47:48.187119+00:00 | 38s399ms62µs556ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:48.187146+00:00 | 38s399ms89µs326ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 134 | | |
| | 2018-06-04 19:47:48.187247+00:00 | 38s399ms190µs526ns | === SPAN START: dist sender === | | | dist sender | 135 | | |
| | 2018-06-04 19:47:48.187269+00:00 | 38s399ms212µs536ns | querying next range at /System/NodeLiveness/1 | [s1,r50/1:/Table/52/1/2{600-700},txn=636dd461,n1] | | dist sender | 135 | | |
| | 2018-06-04 19:47:48.1873+00:00 | 38s399ms243µs467ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r50/1:/Table/52/1/2{600-700},txn=636dd461,n1] | | dist sender | 135 | | |
| | 2018-06-04 19:47:48.187324+00:00 | 38s399ms267µs528ns | sending request to alex-laptop.local:26259 | [s1,r50/1:/Table/52/1/2{600-700},txn=636dd461,n1] | | dist sender | 135 | | |
| | 2018-06-04 19:47:48.187349+00:00 | 38s399ms292µs578ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 136 | | |
| | 2018-06-04 19:47:48.187774+00:00 | 38s399ms716µs888ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187812+00:00 | 38s399ms754µs959ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187822+00:00 | 38s399ms765µs483ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187831+00:00 | 38s399ms773µs981ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187845+00:00 | 38s399ms788µs146ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187876+00:00 | 38s399ms819µs600ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.187955+00:00 | 38s399ms898µs780ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.18796+00:00 | 38s399ms903µs588ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.188925+00:00 | 38s400ms868µs521ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.189275+00:00 | 38s401ms217µs896ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | 2018-06-04 19:47:48.189298+00:00 | 38s401ms241µs578ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 420 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:48.189809+00:00 | 38s401ms751µs942ns | heartbeat 1528141677.687150618,0 | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | liveness heartbeat | 134 | | |
| | 2018-06-04 19:47:48.189819+00:00 | 38s401ms762µs264ns | read-write path | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:48.189828+00:00 | 38s401ms771µs584ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:48.189929+00:00 | 38s401ms872µs240ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:48.189932+00:00 | 38s401ms875µs629ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:51.179709+00:00 | 41s391ms652µs241ns | retry proposal 7d54c8db71faac10: reasonTicks | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:51.179767+00:00 | 41s391ms710µs471ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:51.179945+00:00 | 41s391ms888µs14ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:51.17995+00:00 | 41s391ms892µs925ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.575863+00:00 | 42s787ms806µs1ns | retry proposal 6741ab4b56444db1: reasonNewLeader | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.575912+00:00 | 42s787ms855µs339ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.576061+00:00 | 42s788ms4µs256ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.576066+00:00 | 42s788ms9µs272ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.576729+00:00 | 42s788ms672µs172ns | retry proposal 7314d6ab358ee114: reasonNewLeaderOrConfigChange | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.576799+00:00 | 42s788ms742µs358ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.577005+00:00 | 42s788ms948µs176ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.57711+00:00 | 42s789ms53µs710ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.578187+00:00 | 42s790ms130µs205ns | retry proposal 61a9761caf5513fc: reasonNewLeader | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.578202+00:00 | 42s790ms145µs227ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.57829+00:00 | 42s790ms233µs230ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.578293+00:00 | 42s790ms236µs810ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579026+00:00 | 42s790ms969µs274ns | retry proposal 44b8f07bea2f316d: reasonNewLeader | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579099+00:00 | 42s791ms42µs203ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579183+00:00 | 42s791ms126µs273ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579194+00:00 | 42s791ms137µs747ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579654+00:00 | 42s791ms597µs412ns | retry proposal 5fafeaecad95f3e1: reasonNewLeaderOrConfigChange | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579686+00:00 | 42s791ms628µs933ns | applied timestamp cache | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579741+00:00 | 42s791ms684µs206ns | evaluated request | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.579748+00:00 | 42s791ms691µs236ns | acquired {raft,replica}mu | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | 2018-06-04 19:47:52.580487+00:00 | 42s792ms430µs617ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.366379826,0 epo=3 | | | | | | |
| | | | pro=1528141668.187101719,0 | | | | | | |
| | 2018-06-04 19:47:52.580511+00:00 | 42s792ms454µs750ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | | | seq=6 start=1528141568.366379826,0 epo=3 pro=1528141668.187101719,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366379826,0 epo=3 | | | | | | |
| | | | pro=1528141668.187101719,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:52.580615+00:00 | 42s792ms558µs374ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 133 | | |
| | | | start=1528141568.366379826,0 epo=3 pro=1528141668.187101719,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366379826,0 epo=3 | | | | | | |
| | | | pro=1528141668.187101719,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:52.580667+00:00 | 42s792ms609µs873ns | read has no clock uncertainty | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:52.580672+00:00 | 42s792ms615µs623ns | command queue | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:52.58068+00:00 | 42s792ms623µs138ns | waiting for read lock | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:52.580697+00:00 | 42s792ms640µs370ns | read completed | [n1,s1,r50/1:/Table/52/1/2{600-700}] | | /cockroach.roachpb.Internal/Batch | 131 | | |
| | 2018-06-04 19:47:52.580806+00:00 | 42s792ms748µs900ns | === SPAN START: dist sender === | | | dist sender | 137 | | |
| | 2018-06-04 19:47:52.580824+00:00 | 42s792ms767µs650ns | querying next range at /Table/52/1/2700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 137 | | |
| | 2018-06-04 19:47:52.580855+00:00 | 42s792ms797µs984ns | r51: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 137 | | |
| | 2018-06-04 19:47:52.580872+00:00 | 42s792ms815µs33ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 137 | | |
| | 2018-06-04 19:47:52.580885+00:00 | 42s792ms828µs685ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:52.580889+00:00 | 42s792ms832µs335ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:52.580894+00:00 | 42s792ms837µs327ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:52.58102+00:00 | 42s792ms963µs242ns | read-only path | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:52.581025+00:00 | 42s792ms968µs218ns | request range lease (attempt #1) | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:52.581032+00:00 | 42s792ms975µs165ns | === SPAN START: request range lease === | | | request range lease | 139 | | |
| | 2018-06-04 19:47:52.581039+00:00 | 42s792ms982µs314ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:52.581059+00:00 | 42s793ms1µs999ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 141 | | |
| | 2018-06-04 19:47:52.581126+00:00 | 42s793ms69µs163ns | === SPAN START: dist sender === | | | dist sender | 142 | | |
| | 2018-06-04 19:47:52.581137+00:00 | 42s793ms80µs249ns | querying next range at /System/NodeLiveness/1 | [s1,r51/1:/Table/52/1/2{700-800},txn=0d2cc3f8,n1] | | dist sender | 142 | | |
| | 2018-06-04 19:47:52.581159+00:00 | 42s793ms102µs801ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r51/1:/Table/52/1/2{700-800},txn=0d2cc3f8,n1] | | dist sender | 142 | | |
| | 2018-06-04 19:47:52.581175+00:00 | 42s793ms118µs318ns | sending request to alex-laptop.local:26259 | [s1,r51/1:/Table/52/1/2{700-800},txn=0d2cc3f8,n1] | | dist sender | 142 | | |
| | 2018-06-04 19:47:52.581195+00:00 | 42s793ms137µs979ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 143 | | |
| | 2018-06-04 19:47:52.581487+00:00 | 42s793ms430µs299ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581514+00:00 | 42s793ms457µs210ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581522+00:00 | 42s793ms465µs75ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581528+00:00 | 42s793ms471µs413ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581541+00:00 | 42s793ms484µs108ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581566+00:00 | 42s793ms509µs763ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.58164+00:00 | 42s793ms582µs850ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.581643+00:00 | 42s793ms585µs987ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.582229+00:00 | 42s794ms172µs29ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.582517+00:00 | 42s794ms459µs852ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | 2018-06-04 19:47:52.582538+00:00 | 42s794ms481µs518ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 421 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:52.583063+00:00 | 42s795ms6µs476ns | heartbeat 1528141682.081062044,0 | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | liveness heartbeat | 141 | | |
| | 2018-06-04 19:47:52.58307+00:00 | 42s795ms13µs212ns | read-write path | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:52.583076+00:00 | 42s795ms19µs679ns | applied timestamp cache | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:52.583156+00:00 | 42s795ms98µs992ns | evaluated request | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:52.583158+00:00 | 42s795ms101µs479ns | acquired {raft,replica}mu | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:55.579404+00:00 | 45s791ms346µs906ns | retry proposal 354f02bfb169152d: reasonTicks | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:55.579458+00:00 | 45s791ms401µs25ns | applied timestamp cache | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:55.57961+00:00 | 45s791ms553µs241ns | evaluated request | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:55.579614+00:00 | 45s791ms557µs592ns | acquired {raft,replica}mu | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.975657+00:00 | 47s187ms600µs128ns | retry proposal 0e3307c9b66fbc18: reasonNewLeader | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.975694+00:00 | 47s187ms637µs276ns | applied timestamp cache | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.975827+00:00 | 47s187ms770µs232ns | evaluated request | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.975831+00:00 | 47s187ms774µs537ns | acquired {raft,replica}mu | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.976364+00:00 | 47s188ms307µs658ns | retry proposal 1a4b3e4a157db04d: reasonNewLeaderOrConfigChange | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.976445+00:00 | 47s188ms388µs727ns | applied timestamp cache | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.976535+00:00 | 47s188ms478µs411ns | evaluated request | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.976702+00:00 | 47s188ms644µs948ns | acquired {raft,replica}mu | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | 2018-06-04 19:47:56.978081+00:00 | 47s190ms24µs809ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.366493773,0 epo=3 | | | | | | |
| | | | pro=1528141672.581027426,0 | | | | | | |
| | 2018-06-04 19:47:56.978104+00:00 | 47s190ms47µs105ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | | | seq=4 start=1528141568.366493773,0 epo=3 pro=1528141672.581027426,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366493773,0 epo=3 | | | | | | |
| | | | pro=1528141672.581027426,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:56.978168+00:00 | 47s190ms111µs12ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 140 | | |
| | | | start=1528141568.366493773,0 epo=3 pro=1528141672.581027426,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366493773,0 epo=3 | | | | | | |
| | | | pro=1528141672.581027426,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:56.978215+00:00 | 47s190ms158µs327ns | read has no clock uncertainty | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:56.97822+00:00 | 47s190ms163µs444ns | command queue | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:56.978228+00:00 | 47s190ms171µs311ns | waiting for read lock | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:56.97824+00:00 | 47s190ms183µs746ns | read completed | [n1,s1,r51/1:/Table/52/1/2{700-800}] | | /cockroach.roachpb.Internal/Batch | 138 | | |
| | 2018-06-04 19:47:56.978326+00:00 | 47s190ms268µs823ns | === SPAN START: dist sender === | | | dist sender | 144 | | |
| | 2018-06-04 19:47:56.978341+00:00 | 47s190ms284µs50ns | querying next range at /Table/52/1/2800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 144 | | |
| | 2018-06-04 19:47:56.978374+00:00 | 47s190ms317µs423ns | r52: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 144 | | |
| | 2018-06-04 19:47:56.978397+00:00 | 47s190ms340µs764ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 144 | | |
| | 2018-06-04 19:47:56.978411+00:00 | 47s190ms354µs334ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:56.978415+00:00 | 47s190ms358µs52ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:56.978421+00:00 | 47s190ms364µs181ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:56.978515+00:00 | 47s190ms458µs538ns | read-only path | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:56.978521+00:00 | 47s190ms464µs463ns | request range lease (attempt #1) | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:56.978528+00:00 | 47s190ms471µs441ns | === SPAN START: request range lease === | | | request range lease | 146 | | |
| | 2018-06-04 19:47:56.978536+00:00 | 47s190ms479µs225ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:56.978559+00:00 | 47s190ms502µs45ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 148 | | |
| | 2018-06-04 19:47:56.978636+00:00 | 47s190ms579µs557ns | === SPAN START: dist sender === | | | dist sender | 149 | | |
| | 2018-06-04 19:47:56.97865+00:00 | 47s190ms593µs323ns | querying next range at /System/NodeLiveness/1 | [s1,r52/1:/Table/52/1/2{800-900},txn=0efd3681,n1] | | dist sender | 149 | | |
| | 2018-06-04 19:47:56.978674+00:00 | 47s190ms617µs381ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r52/1:/Table/52/1/2{800-900},txn=0efd3681,n1] | | dist sender | 149 | | |
| | 2018-06-04 19:47:56.978693+00:00 | 47s190ms636µs250ns | sending request to alex-laptop.local:26259 | [s1,r52/1:/Table/52/1/2{800-900},txn=0efd3681,n1] | | dist sender | 149 | | |
| | 2018-06-04 19:47:56.978714+00:00 | 47s190ms656µs982ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 150 | | |
| | 2018-06-04 19:47:56.979044+00:00 | 47s190ms987µs473ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979082+00:00 | 47s191ms25µs325ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979095+00:00 | 47s191ms38µs29ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979102+00:00 | 47s191ms45µs696ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979118+00:00 | 47s191ms61µs814ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979155+00:00 | 47s191ms98µs204ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979243+00:00 | 47s191ms185µs866ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979249+00:00 | 47s191ms192µs136ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.979957+00:00 | 47s191ms900µs496ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.980294+00:00 | 47s192ms237µs269ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | 2018-06-04 19:47:56.980319+00:00 | 47s192ms262µs17ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 422 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:56.980836+00:00 | 47s192ms778µs928ns | heartbeat 1528141686.478562725,0 | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | liveness heartbeat | 148 | | |
| | 2018-06-04 19:47:56.980847+00:00 | 47s192ms790µs222ns | read-write path | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:56.980857+00:00 | 47s192ms800µs23ns | applied timestamp cache | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:56.980966+00:00 | 47s192ms908µs850ns | evaluated request | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:56.980969+00:00 | 47s192ms911µs986ns | acquired {raft,replica}mu | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.974274+00:00 | 50s186ms217µs652ns | retry proposal 60047f5d953606bb: reasonTicks | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.974369+00:00 | 50s186ms312µs278ns | applied timestamp cache | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.974509+00:00 | 50s186ms452µs602ns | evaluated request | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.974516+00:00 | 50s186ms458µs825ns | acquired {raft,replica}mu | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.977904+00:00 | 50s189ms847µs619ns | retry proposal 6e90aac970cf90e2: reasonNewLeader | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.97797+00:00 | 50s189ms913µs580ns | applied timestamp cache | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978078+00:00 | 50s190ms21µs640ns | evaluated request | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978094+00:00 | 50s190ms37µs64ns | acquired {raft,replica}mu | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978698+00:00 | 50s190ms640µs823ns | retry proposal 48753e6c4b8bede6: reasonNewLeaderOrConfigChange | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978747+00:00 | 50s190ms690µs412ns | applied timestamp cache | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978818+00:00 | 50s190ms761µs143ns | evaluated request | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.978825+00:00 | 50s190ms768µs757ns | acquired {raft,replica}mu | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | 2018-06-04 19:47:59.979889+00:00 | 50s191ms832µs717ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.366606927,0 epo=3 | | | | | | |
| | | | pro=1528141676.978523513,0 | | | | | | |
| | 2018-06-04 19:47:59.979917+00:00 | 50s191ms860µs247ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | | | seq=6 start=1528141568.366606927,0 epo=3 pro=1528141676.978523513,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366606927,0 epo=3 | | | | | | |
| | | | pro=1528141676.978523513,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:59.980007+00:00 | 50s191ms950µs403ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 147 | | |
| | | | start=1528141568.366606927,0 epo=3 pro=1528141676.978523513,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366606927,0 epo=3 | | | | | | |
| | | | pro=1528141676.978523513,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:47:59.980076+00:00 | 50s192ms18µs989ns | read has no clock uncertainty | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:59.980083+00:00 | 50s192ms26µs694ns | command queue | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:59.980093+00:00 | 50s192ms36µs40ns | waiting for read lock | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:59.980109+00:00 | 50s192ms52µs405ns | read completed | [n1,s1,r52/1:/Table/52/1/2{800-900}] | | /cockroach.roachpb.Internal/Batch | 145 | | |
| | 2018-06-04 19:47:59.980218+00:00 | 50s192ms160µs916ns | === SPAN START: dist sender === | | | dist sender | 151 | | |
| | 2018-06-04 19:47:59.980236+00:00 | 50s192ms178µs821ns | querying next range at /Table/52/1/2900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 151 | | |
| | 2018-06-04 19:47:59.980265+00:00 | 50s192ms208µs212ns | r53: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 151 | | |
| | 2018-06-04 19:47:59.980289+00:00 | 50s192ms232µs90ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 151 | | |
| | 2018-06-04 19:47:59.980303+00:00 | 50s192ms246µs476ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:47:59.980307+00:00 | 50s192ms250µs354ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:47:59.980313+00:00 | 50s192ms256µs124ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:47:59.980412+00:00 | 50s192ms355µs401ns | read-only path | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:47:59.980418+00:00 | 50s192ms360µs948ns | request range lease (attempt #1) | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:47:59.980426+00:00 | 50s192ms369µs289ns | === SPAN START: request range lease === | | | request range lease | 153 | | |
| | 2018-06-04 19:47:59.980434+00:00 | 50s192ms377µs85ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:47:59.980455+00:00 | 50s192ms398µs448ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 155 | | |
| | 2018-06-04 19:47:59.980532+00:00 | 50s192ms474µs896ns | === SPAN START: dist sender === | | | dist sender | 156 | | |
| | 2018-06-04 19:47:59.980545+00:00 | 50s192ms488µs4ns | querying next range at /System/NodeLiveness/1 | [s1,r53/1:/Table/52/1/{2900-3000},txn=98b8aee0,n1] | | dist sender | 156 | | |
| | 2018-06-04 19:47:59.980602+00:00 | 50s192ms545µs117ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r53/1:/Table/52/1/{2900-3000},txn=98b8aee0,n1] | | dist sender | 156 | | |
| | 2018-06-04 19:47:59.980624+00:00 | 50s192ms567µs258ns | sending request to alex-laptop.local:26259 | [s1,r53/1:/Table/52/1/{2900-3000},txn=98b8aee0,n1] | | dist sender | 156 | | |
| | 2018-06-04 19:47:59.980648+00:00 | 50s192ms591µs679ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 157 | | |
| | 2018-06-04 19:47:59.981028+00:00 | 50s192ms971µs186ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981071+00:00 | 50s193ms14µs526ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981081+00:00 | 50s193ms24µs361ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981089+00:00 | 50s193ms31µs859ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981103+00:00 | 50s193ms45µs835ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981133+00:00 | 50s193ms76µs367ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981211+00:00 | 50s193ms153µs978ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981214+00:00 | 50s193ms157µs630ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.981867+00:00 | 50s193ms810µs782ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.982206+00:00 | 50s194ms149µs363ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | 2018-06-04 19:47:59.982232+00:00 | 50s194ms175µs679ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 423 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:47:59.982736+00:00 | 50s194ms679µs524ns | heartbeat 1528141689.480459322,0 | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | liveness heartbeat | 155 | | |
| | 2018-06-04 19:47:59.982744+00:00 | 50s194ms686µs973ns | read-write path | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:47:59.982751+00:00 | 50s194ms694µs219ns | applied timestamp cache | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:47:59.982837+00:00 | 50s194ms780µs638ns | evaluated request | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:47:59.98284+00:00 | 50s194ms783µs534ns | acquired {raft,replica}mu | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:02.979186+00:00 | 53s191ms129µs120ns | retry proposal 521d2ff9e203c9c1: reasonTicks | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:02.979245+00:00 | 53s191ms188µs465ns | applied timestamp cache | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:02.979411+00:00 | 53s191ms354µs600ns | evaluated request | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:02.979416+00:00 | 53s191ms359µs442ns | acquired {raft,replica}mu | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.978627+00:00 | 54s190ms570µs122ns | retry proposal 7c2965dee3e5e9bc: reasonNewLeader | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.978675+00:00 | 54s190ms618µs645ns | applied timestamp cache | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.978878+00:00 | 54s190ms820µs906ns | evaluated request | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.978883+00:00 | 54s190ms825µs906ns | acquired {raft,replica}mu | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.979553+00:00 | 54s191ms495µs940ns | retry proposal 136923c43ffe19f9: reasonNewLeaderOrConfigChange | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.979641+00:00 | 54s191ms583µs875ns | applied timestamp cache | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.979741+00:00 | 54s191ms684µs520ns | evaluated request | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.97985+00:00 | 54s191ms792µs836ns | acquired {raft,replica}mu | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | 2018-06-04 19:48:03.981294+00:00 | 54s193ms237µs744ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.366723061,0 epo=3 | | | | | | |
| | | | pro=1528141679.980420780,0 | | | | | | |
| | 2018-06-04 19:48:03.981317+00:00 | 54s193ms260µs335ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | | | seq=6 start=1528141568.366723061,0 epo=3 pro=1528141679.980420780,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366723061,0 epo=3 | | | | | | |
| | | | pro=1528141679.980420780,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:03.981393+00:00 | 54s193ms336µs414ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | [async] storage.pendingLeaseRequest: requesting lease | 154 | | |
| | | | start=1528141568.366723061,0 epo=3 pro=1528141679.980420780,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366723061,0 epo=3 | | | | | | |
| | | | pro=1528141679.980420780,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:03.981438+00:00 | 54s193ms381µs376ns | read has no clock uncertainty | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:48:03.981444+00:00 | 54s193ms386µs973ns | command queue | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:48:03.981451+00:00 | 54s193ms393µs963ns | waiting for read lock | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:48:03.981463+00:00 | 54s193ms406µs622ns | read completed | [n1,s1,r53/1:/Table/52/1/{2900-3000}] | | /cockroach.roachpb.Internal/Batch | 152 | | |
| | 2018-06-04 19:48:03.981553+00:00 | 54s193ms496µs292ns | === SPAN START: dist sender === | | | dist sender | 158 | | |
| | 2018-06-04 19:48:03.981569+00:00 | 54s193ms512µs275ns | querying next range at /Table/52/1/3000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 158 | | |
| | 2018-06-04 19:48:03.981601+00:00 | 54s193ms544µs337ns | r54: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 158 | | |
| | 2018-06-04 19:48:03.981625+00:00 | 54s193ms568µs270ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 158 | | |
| | 2018-06-04 19:48:03.981639+00:00 | 54s193ms582µs158ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:03.981643+00:00 | 54s193ms586µs502ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:03.981649+00:00 | 54s193ms592µs789ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:03.98178+00:00 | 54s193ms723µs76ns | read-only path | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:03.981788+00:00 | 54s193ms730µs950ns | request range lease (attempt #1) | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:03.981797+00:00 | 54s193ms740µs753ns | === SPAN START: request range lease === | | | request range lease | 160 | | |
| | 2018-06-04 19:48:03.98181+00:00 | 54s193ms752µs991ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:03.981837+00:00 | 54s193ms780µs149ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 162 | | |
| | 2018-06-04 19:48:03.981954+00:00 | 54s193ms897µs810ns | === SPAN START: dist sender === | | | dist sender | 163 | | |
| | 2018-06-04 19:48:03.98199+00:00 | 54s193ms933µs296ns | querying next range at /System/NodeLiveness/1 | [s1,r54/1:/Table/52/1/3{000-100},txn=2ef2a1d3,n1] | | dist sender | 163 | | |
| | 2018-06-04 19:48:03.98204+00:00 | 54s193ms982µs870ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r54/1:/Table/52/1/3{000-100},txn=2ef2a1d3,n1] | | dist sender | 163 | | |
| | 2018-06-04 19:48:03.982063+00:00 | 54s194ms6µs814ns | sending request to alex-laptop.local:26259 | [s1,r54/1:/Table/52/1/3{000-100},txn=2ef2a1d3,n1] | | dist sender | 163 | | |
| | 2018-06-04 19:48:03.982087+00:00 | 54s194ms30µs196ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 164 | | |
| | 2018-06-04 19:48:03.982369+00:00 | 54s194ms311µs970ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982398+00:00 | 54s194ms341µs629ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982408+00:00 | 54s194ms351µs133ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982415+00:00 | 54s194ms358µs760ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982429+00:00 | 54s194ms372µs600ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.98246+00:00 | 54s194ms402µs845ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982533+00:00 | 54s194ms476µs705ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.982537+00:00 | 54s194ms480µs728ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.983209+00:00 | 54s195ms152µs799ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.98355+00:00 | 54s195ms493µs421ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | 2018-06-04 19:48:03.983576+00:00 | 54s195ms518µs819ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 424 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:03.984127+00:00 | 54s196ms70µs510ns | heartbeat 1528141693.481842115,0 | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | liveness heartbeat | 162 | | |
| | 2018-06-04 19:48:03.984136+00:00 | 54s196ms79µs259ns | read-write path | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:03.984144+00:00 | 54s196ms87µs183ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:03.984223+00:00 | 54s196ms166µs33ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:03.984226+00:00 | 54s196ms168µs913ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:06.977506+00:00 | 57s189ms449µs106ns | retry proposal 32deadedcbce0388: reasonTicks | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:06.977566+00:00 | 57s189ms508µs918ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:06.977725+00:00 | 57s189ms668µs580ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:06.97773+00:00 | 57s189ms673µs240ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.375614+00:00 | 57s587ms556µs968ns | retry proposal 09e268d06f1c3bb8: reasonNewLeader | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.375662+00:00 | 57s587ms605µs584ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.375805+00:00 | 57s587ms748µs736ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.37581+00:00 | 57s587ms753µs125ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.376261+00:00 | 57s588ms204µs734ns | retry proposal 7b1ee072c7df699c: reasonNewLeaderOrConfigChange | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.376296+00:00 | 57s588ms239µs219ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.376403+00:00 | 57s588ms346µs283ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.376584+00:00 | 57s588ms527µs138ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.377701+00:00 | 57s589ms644µs545ns | retry proposal 12caad6c99cbca5e: reasonNewLeader | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.377717+00:00 | 57s589ms660µs275ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.377779+00:00 | 57s589ms722µs599ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.377782+00:00 | 57s589ms724µs879ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.378459+00:00 | 57s590ms401µs950ns | retry proposal 5158536746cb0696: reasonNewLeader | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.378527+00:00 | 57s590ms470µs791ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.37865+00:00 | 57s590ms593µs287ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.378668+00:00 | 57s590ms610µs827ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.379128+00:00 | 57s591ms71µs105ns | retry proposal 41efe8b35aad54a8: reasonNewLeaderOrConfigChange | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.379195+00:00 | 57s591ms138µs399ns | applied timestamp cache | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.379278+00:00 | 57s591ms221µs743ns | evaluated request | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.379287+00:00 | 57s591ms230µs710ns | acquired {raft,replica}mu | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | 2018-06-04 19:48:07.379987+00:00 | 57s591ms930µs567ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.366844055,0 epo=3 | | | | | | |
| | | | pro=1528141683.981790853,0 | | | | | | |
| | 2018-06-04 19:48:07.38001+00:00 | 57s591ms953µs345ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | | | seq=6 start=1528141568.366844055,0 epo=3 pro=1528141683.981790853,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366844055,0 epo=3 | | | | | | |
| | | | pro=1528141683.981790853,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:07.380136+00:00 | 57s592ms79µs485ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | [async] storage.pendingLeaseRequest: requesting lease | 161 | | |
| | | | start=1528141568.366844055,0 epo=3 pro=1528141683.981790853,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.366844055,0 epo=3 | | | | | | |
| | | | pro=1528141683.981790853,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:07.380249+00:00 | 57s592ms191µs890ns | read has no clock uncertainty | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:07.380265+00:00 | 57s592ms208µs158ns | command queue | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:07.380276+00:00 | 57s592ms219µs626ns | waiting for read lock | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:07.380317+00:00 | 57s592ms260µs444ns | read completed | [n1,s1,r54/1:/Table/52/1/3{000-100}] | | /cockroach.roachpb.Internal/Batch | 159 | | |
| | 2018-06-04 19:48:07.380404+00:00 | 57s592ms346µs961ns | === SPAN START: dist sender === | | | dist sender | 165 | | |
| | 2018-06-04 19:48:07.380422+00:00 | 57s592ms365µs404ns | querying next range at /Table/52/1/3100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 165 | | |
| | 2018-06-04 19:48:07.380451+00:00 | 57s592ms394µs698ns | r55: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 165 | | |
| | 2018-06-04 19:48:07.380474+00:00 | 57s592ms416µs946ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 165 | | |
| | 2018-06-04 19:48:07.380488+00:00 | 57s592ms431µs122ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:07.380492+00:00 | 57s592ms435µs202ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:07.380497+00:00 | 57s592ms440µs276ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:07.3806+00:00 | 57s592ms543µs570ns | read-only path | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:07.380608+00:00 | 57s592ms551µs718ns | request range lease (attempt #1) | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:07.380615+00:00 | 57s592ms558µs377ns | === SPAN START: request range lease === | | | request range lease | 167 | | |
| | 2018-06-04 19:48:07.380623+00:00 | 57s592ms566µs526ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:07.380646+00:00 | 57s592ms589µs432ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 169 | | |
| | 2018-06-04 19:48:07.380723+00:00 | 57s592ms666µs712ns | === SPAN START: dist sender === | | | dist sender | 170 | | |
| | 2018-06-04 19:48:07.380738+00:00 | 57s592ms681µs127ns | querying next range at /System/NodeLiveness/1 | [s1,r55/1:/Table/52/1/3{100-200},txn=e940293a,n1] | | dist sender | 170 | | |
| | 2018-06-04 19:48:07.380763+00:00 | 57s592ms706µs171ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r55/1:/Table/52/1/3{100-200},txn=e940293a,n1] | | dist sender | 170 | | |
| | 2018-06-04 19:48:07.380781+00:00 | 57s592ms724µs155ns | sending request to alex-laptop.local:26259 | [s1,r55/1:/Table/52/1/3{100-200},txn=e940293a,n1] | | dist sender | 170 | | |
| | 2018-06-04 19:48:07.3808+00:00 | 57s592ms743µs775ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 171 | | |
| | 2018-06-04 19:48:07.381094+00:00 | 57s593ms37µs82ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381122+00:00 | 57s593ms65µs512ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.38113+00:00 | 57s593ms73µs492ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381136+00:00 | 57s593ms79µs573ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381148+00:00 | 57s593ms91µs484ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.38118+00:00 | 57s593ms122µs838ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381245+00:00 | 57s593ms187µs978ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381247+00:00 | 57s593ms190µs766ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.381874+00:00 | 57s593ms817µs282ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.382195+00:00 | 57s594ms138µs111ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | 2018-06-04 19:48:07.382216+00:00 | 57s594ms159µs281ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 425 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:07.382664+00:00 | 57s594ms607µs734ns | heartbeat 1528141696.880650569,0 | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | liveness heartbeat | 169 | | |
| | 2018-06-04 19:48:07.382675+00:00 | 57s594ms618µs131ns | read-write path | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:07.382681+00:00 | 57s594ms624µs447ns | applied timestamp cache | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:07.382762+00:00 | 57s594ms704µs875ns | evaluated request | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:07.382764+00:00 | 57s594ms707µs487ns | acquired {raft,replica}mu | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.374687+00:00 | 1m586ms630µs91ns | retry proposal 1741b3ffc8de467a: reasonTicks | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.374908+00:00 | 1m586ms850µs932ns | applied timestamp cache | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.375054+00:00 | 1m586ms996µs932ns | evaluated request | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.375058+00:00 | 1m587ms1µs740ns | acquired {raft,replica}mu | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981492+00:00 | 1m1s193ms435µs154ns | retry proposal 1731e3ceb868bc28: reasonNewLeader | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981543+00:00 | 1m1s193ms485µs824ns | applied timestamp cache | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981645+00:00 | 1m1s193ms587µs872ns | evaluated request | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981655+00:00 | 1m1s193ms598µs128ns | acquired {raft,replica}mu | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981922+00:00 | 1m1s193ms865µs701ns | retry proposal 354f61f461f985fc: reasonNewLeaderOrConfigChange | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.981959+00:00 | 1m1s193ms902µs5ns | applied timestamp cache | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.98208+00:00 | 1m1s194ms23µs410ns | evaluated request | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.982479+00:00 | 1m1s194ms422µs258ns | acquired {raft,replica}mu | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | 2018-06-04 19:48:10.982932+00:00 | 1m1s194ms874µs935ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.366962026,0 epo=3 | | | | | | |
| | | | pro=1528141687.380611076,0 | | | | | | |
| | 2018-06-04 19:48:10.98295+00:00 | 1m1s194ms893µs443ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | | | seq=4 start=1528141568.366962026,0 epo=3 pro=1528141687.380611076,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366962026,0 epo=3 | | | | | | |
| | | | pro=1528141687.380611076,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:10.983018+00:00 | 1m1s194ms961µs26ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | [async] storage.pendingLeaseRequest: requesting lease | 168 | | |
| | | | start=1528141568.366962026,0 epo=3 pro=1528141687.380611076,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.366962026,0 epo=3 | | | | | | |
| | | | pro=1528141687.380611076,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:10.983065+00:00 | 1m1s195ms8µs284ns | read has no clock uncertainty | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:10.98307+00:00 | 1m1s195ms13µs782ns | command queue | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:10.983078+00:00 | 1m1s195ms21µs403ns | waiting for read lock | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:10.983092+00:00 | 1m1s195ms35µs229ns | read completed | [n1,s1,r55/1:/Table/52/1/3{100-200}] | | /cockroach.roachpb.Internal/Batch | 166 | | |
| | 2018-06-04 19:48:10.983189+00:00 | 1m1s195ms132µs314ns | === SPAN START: dist sender === | | | dist sender | 172 | | |
| | 2018-06-04 19:48:10.983221+00:00 | 1m1s195ms164µs463ns | querying next range at /Table/52/1/3200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 172 | | |
| | 2018-06-04 19:48:10.983251+00:00 | 1m1s195ms194µs752ns | r56: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 172 | | |
| | 2018-06-04 19:48:10.983269+00:00 | 1m1s195ms212µs354ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 172 | | |
| | 2018-06-04 19:48:10.983281+00:00 | 1m1s195ms224µs462ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:10.983284+00:00 | 1m1s195ms227µs536ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:10.983289+00:00 | 1m1s195ms231µs980ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:10.983384+00:00 | 1m1s195ms327µs643ns | read-only path | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:10.983389+00:00 | 1m1s195ms331µs945ns | request range lease (attempt #1) | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:10.983396+00:00 | 1m1s195ms339µs486ns | === SPAN START: request range lease === | | | request range lease | 174 | | |
| | 2018-06-04 19:48:10.983405+00:00 | 1m1s195ms348µs365ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:10.983426+00:00 | 1m1s195ms368µs987ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 176 | | |
| | 2018-06-04 19:48:10.983498+00:00 | 1m1s195ms441µs225ns | === SPAN START: dist sender === | | | dist sender | 177 | | |
| | 2018-06-04 19:48:10.983508+00:00 | 1m1s195ms451µs330ns | querying next range at /System/NodeLiveness/1 | [s1,r56/1:/Table/52/1/3{200-300},txn=99a38b5a,n1] | | dist sender | 177 | | |
| | 2018-06-04 19:48:10.983529+00:00 | 1m1s195ms471µs862ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r56/1:/Table/52/1/3{200-300},txn=99a38b5a,n1] | | dist sender | 177 | | |
| | 2018-06-04 19:48:10.983543+00:00 | 1m1s195ms486µs70ns | sending request to alex-laptop.local:26259 | [s1,r56/1:/Table/52/1/3{200-300},txn=99a38b5a,n1] | | dist sender | 177 | | |
| | 2018-06-04 19:48:10.983559+00:00 | 1m1s195ms502µs582ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 178 | | |
| | 2018-06-04 19:48:10.983807+00:00 | 1m1s195ms750µs386ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983827+00:00 | 1m1s195ms770µs401ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983834+00:00 | 1m1s195ms776µs821ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983839+00:00 | 1m1s195ms781µs954ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983849+00:00 | 1m1s195ms791µs967ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983863+00:00 | 1m1s195ms806µs351ns | === SPAN START: request range lease === | | | request range lease | 427 | | |
| | 2018-06-04 19:48:10.983867+00:00 | 1m1s195ms810µs540ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 428 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:10.983895+00:00 | 1m1s195ms838µs180ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983936+00:00 | 1m1s195ms879µs619ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 428 | | |
| | 2018-06-04 19:48:10.983949+00:00 | 1m1s195ms892µs383ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 428 | | |
| | 2018-06-04 19:48:10.98397+00:00 | 1m1s195ms913µs108ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.983972+00:00 | 1m1s195ms915µs500ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.984028+00:00 | 1m1s195ms971µs527ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 428 | | |
| | 2018-06-04 19:48:10.984178+00:00 | 1m1s196ms121µs431ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | [async] storage.pendingLeaseRequest: requesting lease | 428 | | |
| | 2018-06-04 19:48:10.984497+00:00 | 1m1s196ms439µs867ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.984773+00:00 | 1m1s196ms716µs278ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | 2018-06-04 19:48:10.984792+00:00 | 1m1s196ms734µs819ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 426 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:10.985329+00:00 | 1m1s197ms271µs908ns | heartbeat 1528141700.483430326,0 | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | liveness heartbeat | 176 | | |
| | 2018-06-04 19:48:10.985334+00:00 | 1m1s197ms277µs420ns | read-write path | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:10.985339+00:00 | 1m1s197ms282µs758ns | applied timestamp cache | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:10.985402+00:00 | 1m1s197ms345µs430ns | evaluated request | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:10.985404+00:00 | 1m1s197ms347µs687ns | acquired {raft,replica}mu | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.973608+00:00 | 1m4s185ms551µs280ns | retry proposal 29ebbddfecdf5014: reasonTicks | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.9738+00:00 | 1m4s185ms743µs121ns | applied timestamp cache | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.973956+00:00 | 1m4s185ms899µs502ns | evaluated request | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.973961+00:00 | 1m4s185ms904µs554ns | acquired {raft,replica}mu | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.977609+00:00 | 1m4s189ms552µs729ns | retry proposal 0d6a3e3c365df2dc: reasonNewLeader | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.977637+00:00 | 1m4s189ms580µs274ns | applied timestamp cache | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.977745+00:00 | 1m4s189ms687µs854ns | evaluated request | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.977748+00:00 | 1m4s189ms691µs295ns | acquired {raft,replica}mu | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.978793+00:00 | 1m4s190ms736µs324ns | retry proposal 6a3b9b259baf82a6: reasonNewLeaderOrConfigChange | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.978824+00:00 | 1m4s190ms767µs810ns | applied timestamp cache | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.978919+00:00 | 1m4s190ms862µs409ns | evaluated request | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.978923+00:00 | 1m4s190ms866µs639ns | acquired {raft,replica}mu | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | 2018-06-04 19:48:13.980096+00:00 | 1m4s192ms39µs713ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.367080934,0 epo=3 | | | | | | |
| | | | pro=1528141690.983390542,0 | | | | | | |
| | 2018-06-04 19:48:13.980114+00:00 | 1m4s192ms57µs146ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | | | seq=6 start=1528141568.367080934,0 epo=3 pro=1528141690.983390542,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367080934,0 epo=3 | | | | | | |
| | | | pro=1528141690.983390542,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:13.980198+00:00 | 1m4s192ms141µs547ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 175 | | |
| | | | start=1528141568.367080934,0 epo=3 pro=1528141690.983390542,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367080934,0 epo=3 | | | | | | |
| | | | pro=1528141690.983390542,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:13.980263+00:00 | 1m4s192ms205µs829ns | read has no clock uncertainty | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:13.98027+00:00 | 1m4s192ms213µs424ns | command queue | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:13.980279+00:00 | 1m4s192ms222µs210ns | waiting for read lock | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:13.980291+00:00 | 1m4s192ms234µs390ns | read completed | [n1,s1,r56/1:/Table/52/1/3{200-300}] | | /cockroach.roachpb.Internal/Batch | 173 | | |
| | 2018-06-04 19:48:13.98038+00:00 | 1m4s192ms322µs835ns | === SPAN START: dist sender === | | | dist sender | 179 | | |
| | 2018-06-04 19:48:13.980396+00:00 | 1m4s192ms338µs911ns | querying next range at /Table/52/1/3300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 179 | | |
| | 2018-06-04 19:48:13.980427+00:00 | 1m4s192ms370µs683ns | r57: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 179 | | |
| | 2018-06-04 19:48:13.98045+00:00 | 1m4s192ms393µs320ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 179 | | |
| | 2018-06-04 19:48:13.980465+00:00 | 1m4s192ms408µs96ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:13.980469+00:00 | 1m4s192ms412µs207ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:13.980475+00:00 | 1m4s192ms418µs164ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:13.98058+00:00 | 1m4s192ms522µs817ns | read-only path | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:13.980586+00:00 | 1m4s192ms529µs329ns | request range lease (attempt #1) | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:13.980593+00:00 | 1m4s192ms536µs654ns | === SPAN START: request range lease === | | | request range lease | 181 | | |
| | 2018-06-04 19:48:13.980602+00:00 | 1m4s192ms545µs73ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:13.980625+00:00 | 1m4s192ms568µs774ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 183 | | |
| | 2018-06-04 19:48:13.980717+00:00 | 1m4s192ms659µs982ns | === SPAN START: dist sender === | | | dist sender | 184 | | |
| | 2018-06-04 19:48:13.980735+00:00 | 1m4s192ms678µs403ns | querying next range at /System/NodeLiveness/1 | [s1,r57/1:/Table/52/1/3{300-400},txn=87c7c651,n1] | | dist sender | 184 | | |
| | 2018-06-04 19:48:13.980771+00:00 | 1m4s192ms713µs908ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r57/1:/Table/52/1/3{300-400},txn=87c7c651,n1] | | dist sender | 184 | | |
| | 2018-06-04 19:48:13.980798+00:00 | 1m4s192ms741µs655ns | sending request to alex-laptop.local:26259 | [s1,r57/1:/Table/52/1/3{300-400},txn=87c7c651,n1] | | dist sender | 184 | | |
| | 2018-06-04 19:48:13.980826+00:00 | 1m4s192ms769µs499ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 185 | | |
| | 2018-06-04 19:48:13.981103+00:00 | 1m4s193ms46µs64ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981133+00:00 | 1m4s193ms76µs404ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981141+00:00 | 1m4s193ms84µs654ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981148+00:00 | 1m4s193ms91µs25ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981161+00:00 | 1m4s193ms104µs725ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981192+00:00 | 1m4s193ms134µs833ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981263+00:00 | 1m4s193ms206µs701ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981267+00:00 | 1m4s193ms209µs874ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.981916+00:00 | 1m4s193ms858µs895ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.98223+00:00 | 1m4s194ms172µs883ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | 2018-06-04 19:48:13.982255+00:00 | 1m4s194ms197µs838ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 429 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:13.982789+00:00 | 1m4s194ms731µs998ns | heartbeat 1528141703.480631068,0 | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | liveness heartbeat | 183 | | |
| | 2018-06-04 19:48:13.982797+00:00 | 1m4s194ms740µs340ns | read-write path | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:13.982804+00:00 | 1m4s194ms746µs909ns | applied timestamp cache | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:13.982887+00:00 | 1m4s194ms830µs600ns | evaluated request | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:13.98289+00:00 | 1m4s194ms833µs695ns | acquired {raft,replica}mu | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.974521+00:00 | 1m7s186ms464µs773ns | retry proposal 1c4bff7e5b080dc8: reasonTicks | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.974651+00:00 | 1m7s186ms594µs32ns | applied timestamp cache | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.974928+00:00 | 1m7s186ms871µs284ns | evaluated request | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.974936+00:00 | 1m7s186ms878µs828ns | acquired {raft,replica}mu | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.97675+00:00 | 1m7s188ms692µs970ns | retry proposal 5b4f522c3faa3ed3: reasonNewLeader | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.976775+00:00 | 1m7s188ms718µs394ns | applied timestamp cache | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.976886+00:00 | 1m7s188ms829µs302ns | evaluated request | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.97689+00:00 | 1m7s188ms833µs115ns | acquired {raft,replica}mu | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.977336+00:00 | 1m7s189ms279µs93ns | retry proposal 137e316a319ed617: reasonNewLeaderOrConfigChange | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.97736+00:00 | 1m7s189ms303µs806ns | applied timestamp cache | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.977466+00:00 | 1m7s189ms408µs989ns | evaluated request | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.977716+00:00 | 1m7s189ms659µs274ns | acquired {raft,replica}mu | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | 2018-06-04 19:48:16.979107+00:00 | 1m7s191ms50µs89ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.367198971,0 epo=3 | | | | | | |
| | | | pro=1528141693.980588737,0 | | | | | | |
| | 2018-06-04 19:48:16.979133+00:00 | 1m7s191ms76µs422ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | | | seq=6 start=1528141568.367198971,0 epo=3 pro=1528141693.980588737,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367198971,0 epo=3 | | | | | | |
| | | | pro=1528141693.980588737,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:16.979207+00:00 | 1m7s191ms150µs509ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | [async] storage.pendingLeaseRequest: requesting lease | 182 | | |
| | | | start=1528141568.367198971,0 epo=3 pro=1528141693.980588737,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367198971,0 epo=3 | | | | | | |
| | | | pro=1528141693.980588737,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:16.979264+00:00 | 1m7s191ms207µs335ns | read has no clock uncertainty | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:16.97927+00:00 | 1m7s191ms213µs183ns | command queue | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:16.979278+00:00 | 1m7s191ms221µs783ns | waiting for read lock | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:16.979297+00:00 | 1m7s191ms240µs442ns | read completed | [n1,s1,r57/1:/Table/52/1/3{300-400}] | | /cockroach.roachpb.Internal/Batch | 180 | | |
| | 2018-06-04 19:48:16.979388+00:00 | 1m7s191ms331µs403ns | === SPAN START: dist sender === | | | dist sender | 186 | | |
| | 2018-06-04 19:48:16.979404+00:00 | 1m7s191ms347µs459ns | querying next range at /Table/52/1/3400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 186 | | |
| | 2018-06-04 19:48:16.979436+00:00 | 1m7s191ms379µs137ns | r58: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 186 | | |
| | 2018-06-04 19:48:16.97946+00:00 | 1m7s191ms403µs365ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 186 | | |
| | 2018-06-04 19:48:16.979475+00:00 | 1m7s191ms417µs918ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:16.979478+00:00 | 1m7s191ms421µs569ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:16.979484+00:00 | 1m7s191ms426µs955ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:16.979579+00:00 | 1m7s191ms522µs354ns | read-only path | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:16.979584+00:00 | 1m7s191ms527µs599ns | request range lease (attempt #1) | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:16.979591+00:00 | 1m7s191ms534µs273ns | === SPAN START: request range lease === | | | request range lease | 188 | | |
| | 2018-06-04 19:48:16.979599+00:00 | 1m7s191ms542µs640ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:16.979623+00:00 | 1m7s191ms566µs805ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 190 | | |
| | 2018-06-04 19:48:16.979696+00:00 | 1m7s191ms639µs32ns | === SPAN START: dist sender === | | | dist sender | 191 | | |
| | 2018-06-04 19:48:16.979708+00:00 | 1m7s191ms651µs394ns | querying next range at /System/NodeLiveness/1 | [s1,r58/1:/Table/52/1/3{400-500},txn=ec5df6e3,n1] | | dist sender | 191 | | |
| | 2018-06-04 19:48:16.979732+00:00 | 1m7s191ms675µs598ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r58/1:/Table/52/1/3{400-500},txn=ec5df6e3,n1] | | dist sender | 191 | | |
| | 2018-06-04 19:48:16.97975+00:00 | 1m7s191ms693µs416ns | sending request to alex-laptop.local:26259 | [s1,r58/1:/Table/52/1/3{400-500},txn=ec5df6e3,n1] | | dist sender | 191 | | |
| | 2018-06-04 19:48:16.97977+00:00 | 1m7s191ms713µs373ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 192 | | |
| | 2018-06-04 19:48:16.980135+00:00 | 1m7s192ms78µs670ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980186+00:00 | 1m7s192ms129µs622ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980203+00:00 | 1m7s192ms146µs266ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980223+00:00 | 1m7s192ms166µs791ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980247+00:00 | 1m7s192ms190µs568ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.98029+00:00 | 1m7s192ms233µs651ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980379+00:00 | 1m7s192ms322µs712ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.980383+00:00 | 1m7s192ms326µs477ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.981071+00:00 | 1m7s193ms14µs249ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.98143+00:00 | 1m7s193ms373µs188ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | 2018-06-04 19:48:16.981458+00:00 | 1m7s193ms400µs872ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 430 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:16.982082+00:00 | 1m7s194ms25µs479ns | heartbeat 1528141706.479627712,0 | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | liveness heartbeat | 190 | | |
| | 2018-06-04 19:48:16.982106+00:00 | 1m7s194ms49µs185ns | read-write path | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:16.982118+00:00 | 1m7s194ms60µs847ns | applied timestamp cache | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:16.98223+00:00 | 1m7s194ms173µs531ns | evaluated request | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:16.982236+00:00 | 1m7s194ms179µs315ns | acquired {raft,replica}mu | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:19.975584+00:00 | 1m10s187ms527µs752ns | retry proposal 09ee2cb5c03bfc69: reasonTicks | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:19.975626+00:00 | 1m10s187ms569µs503ns | applied timestamp cache | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:19.975762+00:00 | 1m10s187ms705µs370ns | evaluated request | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:19.975765+00:00 | 1m10s187ms708µs807ns | acquired {raft,replica}mu | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.978642+00:00 | 1m11s190ms585µs767ns | retry proposal 52a298fb4b9f0ded: reasonNewLeader | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.978686+00:00 | 1m11s190ms628µs923ns | applied timestamp cache | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.978831+00:00 | 1m11s190ms774µs570ns | evaluated request | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.978837+00:00 | 1m11s190ms780µs12ns | acquired {raft,replica}mu | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.979666+00:00 | 1m11s191ms609µs300ns | retry proposal 7d768542ae6d53d3: reasonNewLeaderOrConfigChange | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.97973+00:00 | 1m11s191ms673µs364ns | applied timestamp cache | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.979872+00:00 | 1m11s191ms815µs201ns | evaluated request | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.979946+00:00 | 1m11s191ms889µs528ns | acquired {raft,replica}mu | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | 2018-06-04 19:48:20.981183+00:00 | 1m11s193ms126µs184ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.367317917,0 epo=3 | | | | | | |
| | | | pro=1528141696.979586898,0 | | | | | | |
| | 2018-06-04 19:48:20.98121+00:00 | 1m11s193ms152µs877ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | | | seq=6 start=1528141568.367317917,0 epo=3 pro=1528141696.979586898,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367317917,0 epo=3 | | | | | | |
| | | | pro=1528141696.979586898,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.981294+00:00 | 1m11s193ms237µs399ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | [async] storage.pendingLeaseRequest: requesting lease | 189 | | |
| | | | start=1528141568.367317917,0 epo=3 pro=1528141696.979586898,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367317917,0 epo=3 | | | | | | |
| | | | pro=1528141696.979586898,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.981351+00:00 | 1m11s193ms294µs751ns | read has no clock uncertainty | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:20.981357+00:00 | 1m11s193ms300µs263ns | command queue | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:20.981365+00:00 | 1m11s193ms308µs95ns | waiting for read lock | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:20.981377+00:00 | 1m11s193ms320µs542ns | read completed | [n1,s1,r58/1:/Table/52/1/3{400-500}] | | /cockroach.roachpb.Internal/Batch | 187 | | |
| | 2018-06-04 19:48:20.981464+00:00 | 1m11s193ms407µs750ns | === SPAN START: dist sender === | | | dist sender | 193 | | |
| | 2018-06-04 19:48:20.981483+00:00 | 1m11s193ms426µs565ns | querying next range at /Table/52/1/3500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 193 | | |
| | 2018-06-04 19:48:20.981516+00:00 | 1m11s193ms459µs418ns | === SPAN START: range lookup === | | | range lookup | 194 | | |
| | 2018-06-04 19:48:20.981543+00:00 | 1m11s193ms485µs836ns | === SPAN START: dist sender === | | | dist sender | 195 | | |
| | 2018-06-04 19:48:20.981572+00:00 | 1m11s193ms515µs633ns | querying next range at /Meta2/Table/52/1/3500/NULL | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,n1] | | dist sender | 195 | | |
| | 2018-06-04 19:48:20.981591+00:00 | 1m11s193ms534µs754ns | === SPAN START: range lookup === | | | range lookup | 196 | | |
| | 2018-06-04 19:48:20.981608+00:00 | 1m11s193ms551µs313ns | === SPAN START: dist sender === | | | dist sender | 197 | | |
| | 2018-06-04 19:48:20.981641+00:00 | 1m11s193ms584µs657ns | querying next range at /Meta1/Table/52/1/3500/NULL/NULL | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,range-lookup=/Meta2/Table/52/1/3500/NULL,n1] | | dist sender | 197 | | |
| | 2018-06-04 19:48:20.981659+00:00 | 1m11s193ms602µs182ns | === SPAN START: range lookup === | | | range lookup | 198 | | |
| | 2018-06-04 19:48:20.981701+00:00 | 1m11s193ms644µs781ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,range-lookup=/Meta2/Table/52/1/3500/NULL,n1] | | dist sender | 197 | | |
| | 2018-06-04 19:48:20.981736+00:00 | 1m11s193ms679µs796ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,range-lookup=/Meta2/Table/52/1/3500/NULL,n1] | | dist sender | 197 | | |
| | 2018-06-04 19:48:20.981794+00:00 | 1m11s193ms737µs314ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,range-lookup=/Meta2/Table/52/1/3500/NULL,n1] | | dist sender | 197 | | |
| | 2018-06-04 19:48:20.981806+00:00 | 1m11s193ms749µs112ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 199 | | |
| | 2018-06-04 19:48:20.982141+00:00 | 1m11s194ms84µs47ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982172+00:00 | 1m11s194ms114µs926ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982179+00:00 | 1m11s194ms122µs356ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982186+00:00 | 1m11s194ms129µs605ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982195+00:00 | 1m11s194ms138µs799ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982197+00:00 | 1m11s194ms140µs348ns | operation accepts inconsistent results | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982199+00:00 | 1m11s194ms142µs623ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982472+00:00 | 1m11s194ms414µs995ns | engine stats: {InternalDeleteSkippedCount:2 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982478+00:00 | 1m11s194ms420µs920ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 431 | | |
| | 2018-06-04 19:48:20.982761+00:00 | 1m11s194ms704µs60ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,n1] | | dist sender | 195 | | |
| | 2018-06-04 19:48:20.982787+00:00 | 1m11s194ms730µs85ns | r1: sending batch 1 Scan to (n2,s2):3 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,n1] | | dist sender | 195 | | |
| | 2018-06-04 19:48:20.982824+00:00 | 1m11s194ms766µs966ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,range-lookup=/Table/52/1/3500,n1] | | dist sender | 195 | | |
| | 2018-06-04 19:48:20.982833+00:00 | 1m11s194ms776µs94ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 200 | | |
| | 2018-06-04 19:48:20.983049+00:00 | 1m11s194ms991µs871ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983074+00:00 | 1m11s195ms17µs562ns | 1 Scan | [n2] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983081+00:00 | 1m11s195ms23µs882ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983086+00:00 | 1m11s195ms29µs506ns | read-only path | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983093+00:00 | 1m11s195ms35µs846ns | command queue | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983094+00:00 | 1m11s195ms37µs757ns | operation accepts inconsistent results | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983097+00:00 | 1m11s195ms40µs439ns | waiting for read lock | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983437+00:00 | 1m11s195ms380µs790ns | engine stats: {InternalDeleteSkippedCount:10 TimeBoundNumSSTs:0} | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983444+00:00 | 1m11s195ms387µs119ns | read completed | [n2,s2,r1/3:/{Min-System/}] | | /cockroach.roachpb.Internal/Batch | 432 | | |
| | 2018-06-04 19:48:20.983754+00:00 | 1m11s195ms696µs946ns | looked up range descriptor | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 193 | | |
| | 2018-06-04 19:48:20.983778+00:00 | 1m11s195ms721µs311ns | r59: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 193 | | |
| | 2018-06-04 19:48:20.983796+00:00 | 1m11s195ms739µs430ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 193 | | |
| | 2018-06-04 19:48:20.983808+00:00 | 1m11s195ms751µs728ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.983814+00:00 | 1m11s195ms757µs151ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.983819+00:00 | 1m11s195ms762µs231ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.983917+00:00 | 1m11s195ms860µs55ns | read-only path | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.983922+00:00 | 1m11s195ms865µs43ns | request range lease (attempt #1) | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.983928+00:00 | 1m11s195ms870µs954ns | === SPAN START: request range lease === | | | request range lease | 202 | | |
| | 2018-06-04 19:48:20.983934+00:00 | 1m11s195ms877µs393ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:20.983955+00:00 | 1m11s195ms897µs968ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 204 | | |
| | 2018-06-04 19:48:20.984025+00:00 | 1m11s195ms967µs845ns | === SPAN START: dist sender === | | | dist sender | 205 | | |
| | 2018-06-04 19:48:20.984037+00:00 | 1m11s195ms980µs703ns | querying next range at /System/NodeLiveness/1 | [s1,r59/1:/Table/52/1/3{500-600},txn=4a4ee9ce,n1] | | dist sender | 205 | | |
| | 2018-06-04 19:48:20.984059+00:00 | 1m11s196ms2µs335ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r59/1:/Table/52/1/3{500-600},txn=4a4ee9ce,n1] | | dist sender | 205 | | |
| | 2018-06-04 19:48:20.984076+00:00 | 1m11s196ms18µs994ns | sending request to alex-laptop.local:26259 | [s1,r59/1:/Table/52/1/3{500-600},txn=4a4ee9ce,n1] | | dist sender | 205 | | |
| | 2018-06-04 19:48:20.98409+00:00 | 1m11s196ms33µs794ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 206 | | |
| | 2018-06-04 19:48:20.984287+00:00 | 1m11s196ms230µs281ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.98431+00:00 | 1m11s196ms252µs930ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984317+00:00 | 1m11s196ms260µs42ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984323+00:00 | 1m11s196ms266µs17ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984334+00:00 | 1m11s196ms277µs200ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984361+00:00 | 1m11s196ms303µs875ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984426+00:00 | 1m11s196ms368µs989ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.984429+00:00 | 1m11s196ms372µs447ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.985005+00:00 | 1m11s196ms948µs170ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.985262+00:00 | 1m11s197ms205µs86ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | 2018-06-04 19:48:20.985284+00:00 | 1m11s197ms226µs934ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 433 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:20.985768+00:00 | 1m11s197ms711µs57ns | heartbeat 1528141710.483958150,0 | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | liveness heartbeat | 204 | | |
| | 2018-06-04 19:48:20.985775+00:00 | 1m11s197ms718µs89ns | read-write path | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.985781+00:00 | 1m11s197ms724µs401ns | applied timestamp cache | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.985855+00:00 | 1m11s197ms798µs374ns | evaluated request | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.985858+00:00 | 1m11s197ms801µs356ns | acquired {raft,replica}mu | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.987314+00:00 | 1m11s199ms257µs592ns | retry proposal 231e9504c2f2829e: reasonNewLeader | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.987332+00:00 | 1m11s199ms275µs793ns | applied timestamp cache | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.98738+00:00 | 1m11s199ms323µs806ns | evaluated request | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.98739+00:00 | 1m11s199ms333µs599ns | acquired {raft,replica}mu | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.987755+00:00 | 1m11s199ms698µs96ns | retry proposal 29e9808d2a9d5ee2: reasonNewLeaderOrConfigChange | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.987782+00:00 | 1m11s199ms725µs456ns | applied timestamp cache | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.987834+00:00 | 1m11s199ms777µs256ns | evaluated request | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.98784+00:00 | 1m11s199ms783µs725ns | acquired {raft,replica}mu | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | 2018-06-04 19:48:20.98856+00:00 | 1m11s200ms503µs179ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.367447288,0 epo=3 | | | | | | |
| | | | pro=1528141700.983924211,0 | | | | | | |
| | 2018-06-04 19:48:20.988578+00:00 | 1m11s200ms521µs129ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | | | seq=4 start=1528141568.367447288,0 epo=3 pro=1528141700.983924211,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367447288,0 epo=3 | | | | | | |
| | | | pro=1528141700.983924211,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.988656+00:00 | 1m11s200ms599µs673ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | [async] storage.pendingLeaseRequest: requesting lease | 203 | | |
| | | | start=1528141568.367447288,0 epo=3 pro=1528141700.983924211,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367447288,0 epo=3 | | | | | | |
| | | | pro=1528141700.983924211,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.988734+00:00 | 1m11s200ms677µs575ns | read has no clock uncertainty | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.98874+00:00 | 1m11s200ms683µs686ns | command queue | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.988746+00:00 | 1m11s200ms689µs714ns | waiting for read lock | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.988759+00:00 | 1m11s200ms702µs87ns | read completed | [n1,s1,r59/1:/Table/52/1/3{500-600}] | | /cockroach.roachpb.Internal/Batch | 201 | | |
| | 2018-06-04 19:48:20.988826+00:00 | 1m11s200ms769µs806ns | === SPAN START: dist sender === | | | dist sender | 207 | | |
| | 2018-06-04 19:48:20.988838+00:00 | 1m11s200ms781µs30ns | querying next range at /Table/52/1/3600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 207 | | |
| | 2018-06-04 19:48:20.988856+00:00 | 1m11s200ms799µs716ns | r60: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 207 | | |
| | 2018-06-04 19:48:20.988875+00:00 | 1m11s200ms818µs472ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 207 | | |
| | 2018-06-04 19:48:20.988887+00:00 | 1m11s200ms830µs789ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.988891+00:00 | 1m11s200ms834µs37ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.988895+00:00 | 1m11s200ms838µs65ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.988969+00:00 | 1m11s200ms911µs902ns | read-only path | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.988973+00:00 | 1m11s200ms916µs375ns | request range lease (attempt #1) | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.988978+00:00 | 1m11s200ms920µs848ns | === SPAN START: request range lease === | | | request range lease | 209 | | |
| | 2018-06-04 19:48:20.988983+00:00 | 1m11s200ms926µs359ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:20.988999+00:00 | 1m11s200ms941µs862ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 211 | | |
| | 2018-06-04 19:48:20.989049+00:00 | 1m11s200ms992µs166ns | === SPAN START: dist sender === | | | dist sender | 212 | | |
| | 2018-06-04 19:48:20.989061+00:00 | 1m11s201ms4µs85ns | querying next range at /System/NodeLiveness/1 | [s1,r60/1:/Table/52/1/3{600-700},txn=5871726f,n1] | | dist sender | 212 | | |
| | 2018-06-04 19:48:20.989079+00:00 | 1m11s201ms22µs78ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r60/1:/Table/52/1/3{600-700},txn=5871726f,n1] | | dist sender | 212 | | |
| | 2018-06-04 19:48:20.989093+00:00 | 1m11s201ms35µs989ns | sending request to alex-laptop.local:26259 | [s1,r60/1:/Table/52/1/3{600-700},txn=5871726f,n1] | | dist sender | 212 | | |
| | 2018-06-04 19:48:20.989105+00:00 | 1m11s201ms48µs133ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 213 | | |
| | 2018-06-04 19:48:20.989331+00:00 | 1m11s201ms274µs353ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989362+00:00 | 1m11s201ms305µs235ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989382+00:00 | 1m11s201ms325µs352ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989402+00:00 | 1m11s201ms345µs648ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989413+00:00 | 1m11s201ms356µs653ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989436+00:00 | 1m11s201ms379µs23ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989484+00:00 | 1m11s201ms427µs180ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989487+00:00 | 1m11s201ms430µs143ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.989948+00:00 | 1m11s201ms891µs321ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.99017+00:00 | 1m11s202ms112µs950ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | 2018-06-04 19:48:20.990186+00:00 | 1m11s202ms129µs511ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 434 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:20.990521+00:00 | 1m11s202ms464µs467ns | heartbeat 1528141710.489001610,0 | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | liveness heartbeat | 211 | | |
| | 2018-06-04 19:48:20.990527+00:00 | 1m11s202ms470µs271ns | read-write path | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.990532+00:00 | 1m11s202ms475µs143ns | applied timestamp cache | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.990588+00:00 | 1m11s202ms531µs94ns | evaluated request | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.99059+00:00 | 1m11s202ms533µs344ns | acquired {raft,replica}mu | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.991771+00:00 | 1m11s203ms714µs647ns | retry proposal 0c33d4b59cc0a19d: reasonNewLeader | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.991789+00:00 | 1m11s203ms731µs957ns | applied timestamp cache | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.991838+00:00 | 1m11s203ms781µs700ns | evaluated request | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.991845+00:00 | 1m11s203ms787µs838ns | acquired {raft,replica}mu | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.99217+00:00 | 1m11s204ms113µs96ns | retry proposal 1b598d2dac57d46e: reasonNewLeaderOrConfigChange | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.992196+00:00 | 1m11s204ms139µs660ns | applied timestamp cache | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.992246+00:00 | 1m11s204ms188µs846ns | evaluated request | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.992253+00:00 | 1m11s204ms196µs327ns | acquired {raft,replica}mu | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | 2018-06-04 19:48:20.992922+00:00 | 1m11s204ms865µs679ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.367567054,0 epo=3 | | | | | | |
| | | | pro=1528141700.988974905,0 | | | | | | |
| | 2018-06-04 19:48:20.992943+00:00 | 1m11s204ms886µs225ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | | | seq=6 start=1528141568.367567054,0 epo=3 pro=1528141700.988974905,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367567054,0 epo=3 | | | | | | |
| | | | pro=1528141700.988974905,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.992993+00:00 | 1m11s204ms936µs641ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | [async] storage.pendingLeaseRequest: requesting lease | 210 | | |
| | | | start=1528141568.367567054,0 epo=3 pro=1528141700.988974905,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367567054,0 epo=3 | | | | | | |
| | | | pro=1528141700.988974905,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:20.993021+00:00 | 1m11s204ms964µs635ns | read has no clock uncertainty | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.993025+00:00 | 1m11s204ms968µs240ns | command queue | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.99303+00:00 | 1m11s204ms972µs845ns | waiting for read lock | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.993041+00:00 | 1m11s204ms984µs152ns | read completed | [n1,s1,r60/1:/Table/52/1/3{600-700}] | | /cockroach.roachpb.Internal/Batch | 208 | | |
| | 2018-06-04 19:48:20.993091+00:00 | 1m11s205ms34µs253ns | === SPAN START: dist sender === | | | dist sender | 214 | | |
| | 2018-06-04 19:48:20.993104+00:00 | 1m11s205ms47µs322ns | querying next range at /Table/52/1/3700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 214 | | |
| | 2018-06-04 19:48:20.993129+00:00 | 1m11s205ms71µs857ns | r61: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 214 | | |
| | 2018-06-04 19:48:20.993141+00:00 | 1m11s205ms83µs917ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 214 | | |
| | 2018-06-04 19:48:20.99315+00:00 | 1m11s205ms93µs461ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:20.993154+00:00 | 1m11s205ms96µs853ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:20.993158+00:00 | 1m11s205ms101µs272ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:20.993219+00:00 | 1m11s205ms162µs646ns | read-only path | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:20.993223+00:00 | 1m11s205ms166µs360ns | request range lease (attempt #1) | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:20.993227+00:00 | 1m11s205ms170µs740ns | === SPAN START: request range lease === | | | request range lease | 216 | | |
| | 2018-06-04 19:48:20.993232+00:00 | 1m11s205ms175µs477ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:20.993247+00:00 | 1m11s205ms190µs304ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 218 | | |
| | 2018-06-04 19:48:20.993295+00:00 | 1m11s205ms237µs945ns | === SPAN START: dist sender === | | | dist sender | 219 | | |
| | 2018-06-04 19:48:20.993304+00:00 | 1m11s205ms246µs936ns | querying next range at /System/NodeLiveness/1 | [s1,r61/1:/Table/52/1/3{700-800},txn=18b67d2e,n1] | | dist sender | 219 | | |
| | 2018-06-04 19:48:20.993338+00:00 | 1m11s205ms280µs867ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r61/1:/Table/52/1/3{700-800},txn=18b67d2e,n1] | | dist sender | 219 | | |
| | 2018-06-04 19:48:20.99335+00:00 | 1m11s205ms293µs144ns | sending request to alex-laptop.local:26259 | [s1,r61/1:/Table/52/1/3{700-800},txn=18b67d2e,n1] | | dist sender | 219 | | |
| | 2018-06-04 19:48:20.993362+00:00 | 1m11s205ms305µs521ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 220 | | |
| | 2018-06-04 19:48:20.993529+00:00 | 1m11s205ms472µs710ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.99356+00:00 | 1m11s205ms502µs941ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.993566+00:00 | 1m11s205ms509µs406ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.99357+00:00 | 1m11s205ms513µs770ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.993578+00:00 | 1m11s205ms521µs751ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.993598+00:00 | 1m11s205ms541µs514ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.993642+00:00 | 1m11s205ms585µs144ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.993644+00:00 | 1m11s205ms587µs624ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.994068+00:00 | 1m11s206ms11µs270ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.994309+00:00 | 1m11s206ms252µs340ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | 2018-06-04 19:48:20.994327+00:00 | 1m11s206ms270µs41ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 435 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:20.994689+00:00 | 1m11s206ms632µs283ns | heartbeat 1528141710.493249798,0 | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | liveness heartbeat | 218 | | |
| | 2018-06-04 19:48:20.994694+00:00 | 1m11s206ms637µs375ns | read-write path | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:20.994709+00:00 | 1m11s206ms651µs989ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:20.994761+00:00 | 1m11s206ms704µs288ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:20.994763+00:00 | 1m11s206ms706µs393ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:23.973458+00:00 | 1m14s185ms400µs882ns | retry proposal 066c03c114c0716c: reasonTicks | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:23.973505+00:00 | 1m14s185ms448µs628ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:23.973642+00:00 | 1m14s185ms585µs511ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:23.973646+00:00 | 1m14s185ms589µs502ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.17499+00:00 | 1m14s386ms933µs175ns | retry proposal 6c42e51fc5529ace: reasonNewLeader | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.175026+00:00 | 1m14s386ms969µs300ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.175165+00:00 | 1m14s387ms108µs519ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.17517+00:00 | 1m14s387ms113µs235ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.175809+00:00 | 1m14s387ms752µs504ns | retry proposal 01ea8736ff466cf1: reasonNewLeaderOrConfigChange | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.175873+00:00 | 1m14s387ms816µs698ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.175973+00:00 | 1m14s387ms916µs442ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.176031+00:00 | 1m14s387ms974µs385ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177034+00:00 | 1m14s388ms977µs173ns | retry proposal 2f24eb68b91f89d7: reasonNewLeader | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177051+00:00 | 1m14s388ms994µs208ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177127+00:00 | 1m14s389ms70µs45ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.17713+00:00 | 1m14s389ms73µs378ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177733+00:00 | 1m14s389ms676µs623ns | retry proposal 297738324b0ba26e: reasonNewLeader | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177821+00:00 | 1m14s389ms764µs722ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177939+00:00 | 1m14s389ms882µs477ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.177951+00:00 | 1m14s389ms894µs548ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.178419+00:00 | 1m14s390ms362µs319ns | retry proposal 7b051c553073cc5f: reasonNewLeaderOrConfigChange | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.17845+00:00 | 1m14s390ms393µs504ns | applied timestamp cache | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.178512+00:00 | 1m14s390ms455µs315ns | evaluated request | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.17852+00:00 | 1m14s390ms463µs11ns | acquired {raft,replica}mu | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | 2018-06-04 19:48:24.179323+00:00 | 1m14s391ms266µs614ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | | | to repl=(n1,s1):1 seq=4 start=1528141568.367682122,0 epo=3 | | | | | | |
| | | | pro=1528141700.993224760,0 | | | | | | |
| | 2018-06-04 19:48:24.179348+00:00 | 1m14s391ms291µs321ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | | | seq=4 start=1528141568.367682122,0 epo=3 pro=1528141700.993224760,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367682122,0 epo=3 | | | | | | |
| | | | pro=1528141700.993224760,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:24.179455+00:00 | 1m14s391ms398µs673ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=4 | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 217 | | |
| | | | start=1528141568.367682122,0 epo=3 pro=1528141700.993224760,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367682122,0 epo=3 | | | | | | |
| | | | pro=1528141700.993224760,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:24.1795+00:00 | 1m14s391ms443µs767ns | read has no clock uncertainty | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:24.179506+00:00 | 1m14s391ms449µs696ns | command queue | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:24.179517+00:00 | 1m14s391ms459µs880ns | waiting for read lock | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:24.179532+00:00 | 1m14s391ms475µs194ns | read completed | [n1,s1,r61/1:/Table/52/1/3{700-800}] | | /cockroach.roachpb.Internal/Batch | 215 | | |
| | 2018-06-04 19:48:24.179624+00:00 | 1m14s391ms567µs160ns | === SPAN START: dist sender === | | | dist sender | 221 | | |
| | 2018-06-04 19:48:24.179637+00:00 | 1m14s391ms580µs577ns | querying next range at /Table/52/1/3800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 221 | | |
| | 2018-06-04 19:48:24.179668+00:00 | 1m14s391ms611µs185ns | r62: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 221 | | |
| | 2018-06-04 19:48:24.179688+00:00 | 1m14s391ms631µs544ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 221 | | |
| | 2018-06-04 19:48:24.179702+00:00 | 1m14s391ms645µs274ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:24.179706+00:00 | 1m14s391ms649µs44ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:24.179711+00:00 | 1m14s391ms654µs604ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:24.1798+00:00 | 1m14s391ms743µs256ns | read-only path | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:24.179805+00:00 | 1m14s391ms748µs327ns | request range lease (attempt #1) | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:24.179811+00:00 | 1m14s391ms754µs522ns | === SPAN START: request range lease === | | | request range lease | 223 | | |
| | 2018-06-04 19:48:24.179821+00:00 | 1m14s391ms763µs951ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:24.179839+00:00 | 1m14s391ms782µs533ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 225 | | |
| | 2018-06-04 19:48:24.179911+00:00 | 1m14s391ms854µs63ns | === SPAN START: dist sender === | | | dist sender | 226 | | |
| | 2018-06-04 19:48:24.179922+00:00 | 1m14s391ms865µs640ns | querying next range at /System/NodeLiveness/1 | [s1,r62/1:/Table/52/1/3{800-900},txn=dd882069,n1] | | dist sender | 226 | | |
| | 2018-06-04 19:48:24.179944+00:00 | 1m14s391ms887µs325ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r62/1:/Table/52/1/3{800-900},txn=dd882069,n1] | | dist sender | 226 | | |
| | 2018-06-04 19:48:24.179961+00:00 | 1m14s391ms904µs17ns | sending request to alex-laptop.local:26259 | [s1,r62/1:/Table/52/1/3{800-900},txn=dd882069,n1] | | dist sender | 226 | | |
| | 2018-06-04 19:48:24.179982+00:00 | 1m14s391ms925µs186ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 227 | | |
| | 2018-06-04 19:48:24.180229+00:00 | 1m14s392ms172µs662ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180254+00:00 | 1m14s392ms197µs493ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180261+00:00 | 1m14s392ms204µs769ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180268+00:00 | 1m14s392ms210µs927ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.18028+00:00 | 1m14s392ms223µs152ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.18031+00:00 | 1m14s392ms253µs188ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180373+00:00 | 1m14s392ms316µs453ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180376+00:00 | 1m14s392ms319µs658ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.180976+00:00 | 1m14s392ms918µs998ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.181244+00:00 | 1m14s393ms187µs651ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | 2018-06-04 19:48:24.181265+00:00 | 1m14s393ms208µs426ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 436 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:24.181726+00:00 | 1m14s393ms668µs971ns | heartbeat 1528141713.679844278,0 | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | liveness heartbeat | 225 | | |
| | 2018-06-04 19:48:24.181734+00:00 | 1m14s393ms676µs862ns | read-write path | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:24.181742+00:00 | 1m14s393ms685µs501ns | applied timestamp cache | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:24.181843+00:00 | 1m14s393ms786µs573ns | evaluated request | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:24.181846+00:00 | 1m14s393ms789µs402ns | acquired {raft,replica}mu | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:27.176094+00:00 | 1m17s388ms37µs104ns | retry proposal 4d10dddb489fb624: reasonTicks | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:27.176139+00:00 | 1m17s388ms82µs122ns | applied timestamp cache | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:27.176286+00:00 | 1m17s388ms229µs179ns | evaluated request | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:27.17629+00:00 | 1m17s388ms233µs585ns | acquired {raft,replica}mu | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.577983+00:00 | 1m18s789ms925µs944ns | retry proposal 29570825176b0efa: reasonNewLeader | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.578104+00:00 | 1m18s790ms46µs966ns | applied timestamp cache | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.57827+00:00 | 1m18s790ms213µs149ns | evaluated request | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.578289+00:00 | 1m18s790ms231µs822ns | acquired {raft,replica}mu | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.578901+00:00 | 1m18s790ms844µs385ns | retry proposal 76226963df5b874c: reasonNewLeaderOrConfigChange | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.578946+00:00 | 1m18s790ms888µs845ns | applied timestamp cache | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.579037+00:00 | 1m18s790ms980µs628ns | evaluated request | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.579051+00:00 | 1m18s790ms994µs784ns | acquired {raft,replica}mu | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | 2018-06-04 19:48:28.580183+00:00 | 1m18s792ms126µs516ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.367800593,0 epo=3 | | | | | | |
| | | | pro=1528141704.179807377,0 | | | | | | |
| | 2018-06-04 19:48:28.580209+00:00 | 1m18s792ms152µs278ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | | | seq=6 start=1528141568.367800593,0 epo=3 pro=1528141704.179807377,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367800593,0 epo=3 | | | | | | |
| | | | pro=1528141704.179807377,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:28.580296+00:00 | 1m18s792ms239µs581ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | [async] storage.pendingLeaseRequest: requesting lease | 224 | | |
| | | | start=1528141568.367800593,0 epo=3 pro=1528141704.179807377,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.367800593,0 epo=3 | | | | | | |
| | | | pro=1528141704.179807377,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:28.58035+00:00 | 1m18s792ms293µs389ns | read has no clock uncertainty | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:28.580357+00:00 | 1m18s792ms299µs942ns | command queue | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:28.580366+00:00 | 1m18s792ms309µs349ns | waiting for read lock | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:28.580381+00:00 | 1m18s792ms324µs518ns | read completed | [n1,s1,r62/1:/Table/52/1/3{800-900}] | | /cockroach.roachpb.Internal/Batch | 222 | | |
| | 2018-06-04 19:48:28.580523+00:00 | 1m18s792ms466µs315ns | === SPAN START: dist sender === | | | dist sender | 228 | | |
| | 2018-06-04 19:48:28.580545+00:00 | 1m18s792ms487µs855ns | querying next range at /Table/52/1/3900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | 2018-06-04 19:48:28.580578+00:00 | 1m18s792ms521µs592ns | r63: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | 2018-06-04 19:48:28.580602+00:00 | 1m18s792ms545µs571ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | 2018-06-04 19:48:28.580618+00:00 | 1m18s792ms561µs253ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:28.580622+00:00 | 1m18s792ms565µs673ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:28.580628+00:00 | 1m18s792ms571µs649ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:28.580727+00:00 | 1m18s792ms670µs631ns | read-only path | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:28.580733+00:00 | 1m18s792ms675µs931ns | request range lease (attempt #1) | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:28.58074+00:00 | 1m18s792ms683µs452ns | === SPAN START: request range lease === | | | request range lease | 230 | | |
| | 2018-06-04 19:48:28.580747+00:00 | 1m18s792ms690µs472ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:28.580768+00:00 | 1m18s792ms711µs469ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 232 | | |
| | 2018-06-04 19:48:28.580844+00:00 | 1m18s792ms787µs22ns | === SPAN START: dist sender === | | | dist sender | 233 | | |
| | 2018-06-04 19:48:28.580856+00:00 | 1m18s792ms799µs358ns | querying next range at /System/NodeLiveness/1 | [s1,r63/1:/Table/52/1/{3900-4000},txn=a744fba4,n1] | | dist sender | 233 | | |
| | 2018-06-04 19:48:28.58088+00:00 | 1m18s792ms823µs257ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n2,s2):2 | [s1,r63/1:/Table/52/1/{3900-4000},txn=a744fba4,n1] | | dist sender | 233 | | |
| | 2018-06-04 19:48:28.580899+00:00 | 1m18s792ms842µs23ns | sending request to alex-laptop.local:26259 | [s1,r63/1:/Table/52/1/{3900-4000},txn=a744fba4,n1] | | dist sender | 233 | | |
| | 2018-06-04 19:48:28.580919+00:00 | 1m18s792ms862µs172ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 234 | | |
| | 2018-06-04 19:48:28.581297+00:00 | 1m18s793ms239µs947ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581339+00:00 | 1m18s793ms282µs349ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n2] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581348+00:00 | 1m18s793ms291µs514ns | executing 3 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581356+00:00 | 1m18s793ms298µs837ns | read-write path | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581372+00:00 | 1m18s793ms315µs424ns | command queue | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581412+00:00 | 1m18s793ms355µs622ns | applied timestamp cache | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581494+00:00 | 1m18s793ms437µs656ns | evaluated request | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.581499+00:00 | 1m18s793ms442µs128ns | acquired {raft,replica}mu | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.582163+00:00 | 1m18s794ms106µs510ns | applying command | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.582496+00:00 | 1m18s794ms439µs530ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | 2018-06-04 19:48:28.582524+00:00 | 1m18s794ms467µs499ns | gossiping 1 node liveness record(s) from span | [n2,s2,r3/2:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 437 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:28.583001+00:00 | 1m18s794ms944µs292ns | heartbeat 1528141718.080771949,0 | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | liveness heartbeat | 232 | | |
| | 2018-06-04 19:48:28.583011+00:00 | 1m18s794ms954µs686ns | read-write path | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:28.583021+00:00 | 1m18s794ms964µs141ns | applied timestamp cache | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:28.583114+00:00 | 1m18s795ms56µs863ns | evaluated request | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:28.583117+00:00 | 1m18s795ms59µs995ns | acquired {raft,replica}mu | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:31.572854+00:00 | 1m21s784ms797µs175ns | retry proposal 0aced7a81d41d02a: reasonTicks | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:31.572965+00:00 | 1m21s784ms908µs383ns | applied timestamp cache | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:31.573082+00:00 | 1m21s785ms25µs15ns | evaluated request | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:31.573085+00:00 | 1m21s785ms27µs968ns | acquired {raft,replica}mu | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.774857+00:00 | 1m22s986ms799µs896ns | retry proposal 75125e91f1bffc71: reasonNewLeader | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.774887+00:00 | 1m22s986ms829µs837ns | applied timestamp cache | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.774968+00:00 | 1m22s986ms911µs53ns | evaluated request | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.774971+00:00 | 1m22s986ms914µs750ns | acquired {raft,replica}mu | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.776201+00:00 | 1m22s988ms144µs297ns | retry proposal 6a471c176141d518: reasonNewLeaderOrConfigChange | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.776235+00:00 | 1m22s988ms178µs390ns | applied timestamp cache | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.776325+00:00 | 1m22s988ms268µs323ns | evaluated request | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.776418+00:00 | 1m22s988ms361µs692ns | acquired {raft,replica}mu | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | 2018-06-04 19:48:32.778426+00:00 | 1m22s990ms369µs353ns | command proposed from replica (n1,s1):1 with lease #3 incompatible | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | | | to repl=(n3,s3):2 seq=4 start=1528141709.302721661,0 epo=2 | | | | | | |
| | | | pro=1528141709.302724098,0 | | | | | | |
| | 2018-06-04 19:48:32.778451+00:00 | 1m22s990ms394µs638ns | applying command with forced error: cannot replace lease repl=(n3,s3):2 | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | | | seq=4 start=1528141709.302721661,0 epo=2 pro=1528141709.302724098,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367917517,0 epo=3 | | | | | | |
| | | | pro=1528141708.580735302,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:32.778539+00:00 | 1m22s990ms482µs716ns | replica.Send got error: cannot replace lease repl=(n3,s3):2 seq=4 | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | [async] storage.pendingLeaseRequest: requesting lease | 231 | | |
| | | | start=1528141709.302721661,0 epo=2 pro=1528141709.302724098,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=4 start=1528141568.367917517,0 epo=3 | | | | | | |
| | | | pro=1528141708.580735302,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:32.778621+00:00 | 1m22s990ms563µs993ns | replica.Send got error: [NotLeaseHolderError] r63: replica | [n1,s1,r63/1:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141709.302721661,0 epo=2 pro=1528141709.302724098,0 | | | | | | |
| | 2018-06-04 19:48:32.778636+00:00 | 1m22s990ms579µs457ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 229 | | |
| | 2018-06-04 19:48:32.778677+00:00 | 1m22s990ms620µs417ns | application error: [NotLeaseHolderError] r63: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141709.302721661,0 epo=2 pro=1528141709.302724098,0 | | | | | | |
| | 2018-06-04 19:48:32.77869+00:00 | 1m22s990ms633µs163ns | error: {(err: [NotLeaseHolderError] r63: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141709.302721661,0 epo=2 pro=1528141709.302724098,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:32.778714+00:00 | 1m22s990ms657µs286ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 228 | | |
| | 2018-06-04 19:48:32.778739+00:00 | 1m22s990ms682µs427ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 235 | | |
| | 2018-06-04 19:48:32.779063+00:00 | 1m22s991ms6µs442ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779097+00:00 | 1m22s991ms39µs937ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779104+00:00 | 1m22s991ms47µs68ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779111+00:00 | 1m22s991ms54µs55ns | read-only path | [n3,s3,r63/2:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779135+00:00 | 1m22s991ms78µs577ns | command queue | [n3,s3,r63/2:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779141+00:00 | 1m22s991ms84µs609ns | waiting for read lock | [n3,s3,r63/2:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779156+00:00 | 1m22s991ms99µs530ns | read completed | [n3,s3,r63/2:/Table/52/1/{3900-4000}] | | /cockroach.roachpb.Internal/Batch | 438 | | |
| | 2018-06-04 19:48:32.779598+00:00 | 1m22s991ms541µs675ns | === SPAN START: dist sender === | | | dist sender | 236 | | |
| | 2018-06-04 19:48:32.77963+00:00 | 1m22s991ms573µs151ns | querying next range at /Table/52/1/4000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | 2018-06-04 19:48:32.779692+00:00 | 1m22s991ms634µs953ns | r64: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | 2018-06-04 19:48:32.779711+00:00 | 1m22s991ms653µs967ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | 2018-06-04 19:48:32.779727+00:00 | 1m22s991ms670µs581ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | 2018-06-04 19:48:32.779733+00:00 | 1m22s991ms676µs216ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | 2018-06-04 19:48:32.779739+00:00 | 1m22s991ms682µs512ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | 2018-06-04 19:48:32.779746+00:00 | 1m22s991ms689µs759ns | read-only path | [n1,s1,r64/1:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | 2018-06-04 19:48:32.77977+00:00 | 1m22s991ms712µs956ns | replica.Send got error: [NotLeaseHolderError] r64: replica | [n1,s1,r64/1:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141709.300520764,0 epo=2 pro=1528141709.300522636,0 | | | | | | |
| | 2018-06-04 19:48:32.779782+00:00 | 1m22s991ms725µs489ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 237 | | |
| | 2018-06-04 19:48:32.779813+00:00 | 1m22s991ms756µs465ns | application error: [NotLeaseHolderError] r64: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141709.300520764,0 epo=2 pro=1528141709.300522636,0 | | | | | | |
| | 2018-06-04 19:48:32.779826+00:00 | 1m22s991ms769µs696ns | error: {(err: [NotLeaseHolderError] r64: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141709.300520764,0 epo=2 pro=1528141709.300522636,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:32.779856+00:00 | 1m22s991ms799µs774ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 236 | | |
| | 2018-06-04 19:48:32.7799+00:00 | 1m22s991ms843µs537ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 238 | | |
| | 2018-06-04 19:48:32.780364+00:00 | 1m22s992ms307µs391ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.7804+00:00 | 1m22s992ms342µs973ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.780407+00:00 | 1m22s992ms350µs565ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.780415+00:00 | 1m22s992ms358µs560ns | read-only path | [n2,s2,r64/3:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.780425+00:00 | 1m22s992ms368µs160ns | command queue | [n2,s2,r64/3:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.780432+00:00 | 1m22s992ms374µs967ns | waiting for read lock | [n2,s2,r64/3:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.78046+00:00 | 1m22s992ms402µs992ns | read completed | [n2,s2,r64/3:/Table/52/1/4{000-100}] | | /cockroach.roachpb.Internal/Batch | 439 | | |
| | 2018-06-04 19:48:32.780884+00:00 | 1m22s992ms827µs401ns | === SPAN START: dist sender === | | | dist sender | 239 | | |
| | 2018-06-04 19:48:32.780902+00:00 | 1m22s992ms845µs749ns | querying next range at /Table/52/1/4100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 239 | | |
| | 2018-06-04 19:48:32.780934+00:00 | 1m22s992ms877µs622ns | r65: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 239 | | |
| | 2018-06-04 19:48:32.780952+00:00 | 1m22s992ms895µs632ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 239 | | |
| | 2018-06-04 19:48:32.780969+00:00 | 1m22s992ms912µs676ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.780975+00:00 | 1m22s992ms918µs14ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.780982+00:00 | 1m22s992ms924µs821ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.780989+00:00 | 1m22s992ms932µs468ns | read-only path | [n1,s1,r65/1:/Table/52/1/4{100-200}] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.780995+00:00 | 1m22s992ms938µs86ns | read has no clock uncertainty | [n1,s1,r65/1:/Table/52/1/4{100-200}] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.780999+00:00 | 1m22s992ms942µs246ns | command queue | [n1,s1,r65/1:/Table/52/1/4{100-200}] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.781005+00:00 | 1m22s992ms948µs377ns | waiting for read lock | [n1,s1,r65/1:/Table/52/1/4{100-200}] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.781017+00:00 | 1m22s992ms959µs834ns | read completed | [n1,s1,r65/1:/Table/52/1/4{100-200}] | | /cockroach.roachpb.Internal/Batch | 240 | | |
| | 2018-06-04 19:48:32.781078+00:00 | 1m22s993ms21µs104ns | === SPAN START: dist sender === | | | dist sender | 241 | | |
| | 2018-06-04 19:48:32.781163+00:00 | 1m22s993ms105µs877ns | querying next range at /Table/52/1/4200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | 2018-06-04 19:48:32.78119+00:00 | 1m22s993ms133µs269ns | r66: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | 2018-06-04 19:48:32.781206+00:00 | 1m22s993ms149µs525ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | 2018-06-04 19:48:32.781238+00:00 | 1m22s993ms181µs246ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.781244+00:00 | 1m22s993ms187µs341ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.78125+00:00 | 1m22s993ms193µs308ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.781257+00:00 | 1m22s993ms199µs969ns | read-only path | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.781263+00:00 | 1m22s993ms205µs915ns | request range lease (attempt #1) | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.781272+00:00 | 1m22s993ms214µs839ns | === SPAN START: request range lease === | | | request range lease | 243 | | |
| | 2018-06-04 19:48:32.781285+00:00 | 1m22s993ms228µs309ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:32.781324+00:00 | 1m22s993ms267µs577ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 245 | | |
| | 2018-06-04 19:48:32.781471+00:00 | 1m22s993ms414µs400ns | === SPAN START: dist sender === | | | dist sender | 246 | | |
| | 2018-06-04 19:48:32.781492+00:00 | 1m22s993ms435µs505ns | querying next range at /System/NodeLiveness/1 | [s1,r66/1:/Table/52/1/4{200-300},txn=77460716,n1] | | dist sender | 246 | | |
| | 2018-06-04 19:48:32.781523+00:00 | 1m22s993ms465µs832ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n3,s3):3 | [s1,r66/1:/Table/52/1/4{200-300},txn=77460716,n1] | | dist sender | 246 | | |
| | 2018-06-04 19:48:32.781559+00:00 | 1m22s993ms501µs895ns | sending request to alex-laptop.local:26261 | [s1,r66/1:/Table/52/1/4{200-300},txn=77460716,n1] | | dist sender | 246 | | |
| | 2018-06-04 19:48:32.781578+00:00 | 1m22s993ms521µs532ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 247 | | |
| | 2018-06-04 19:48:32.781984+00:00 | 1m22s993ms927µs642ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782016+00:00 | 1m22s993ms959µs674ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n3] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782024+00:00 | 1m22s993ms967µs736ns | executing 3 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782032+00:00 | 1m22s993ms975µs344ns | read-write path | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782044+00:00 | 1m22s993ms987µs689ns | command queue | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782079+00:00 | 1m22s994ms22µs125ns | applied timestamp cache | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782195+00:00 | 1m22s994ms138µs86ns | evaluated request | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.782201+00:00 | 1m22s994ms144µs404ns | acquired {raft,replica}mu | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.783017+00:00 | 1m22s994ms960µs198ns | applying command | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.783743+00:00 | 1m22s995ms685µs888ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | 2018-06-04 19:48:32.783783+00:00 | 1m22s995ms726µs374ns | gossiping 1 node liveness record(s) from span | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 440 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:32.784251+00:00 | 1m22s996ms194µs613ns | heartbeat 1528141722.281330382,0 | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | liveness heartbeat | 245 | | |
| | 2018-06-04 19:48:32.784263+00:00 | 1m22s996ms205µs844ns | read-write path | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | 2018-06-04 19:48:32.78427+00:00 | 1m22s996ms212µs956ns | applied timestamp cache | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | 2018-06-04 19:48:32.784377+00:00 | 1m22s996ms320µs302ns | evaluated request | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | 2018-06-04 19:48:32.78438+00:00 | 1m22s996ms323µs608ns | acquired {raft,replica}mu | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | 2018-06-04 19:48:32.78579+00:00 | 1m22s997ms733µs25ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | | | to repl=(n3,s3):2 seq=6 start=1528141712.778205832,0 epo=2 | | | | | | |
| | | | pro=1528141712.778209059,0 | | | | | | |
| | 2018-06-04 19:48:32.785812+00:00 | 1m22s997ms755µs212ns | applying command with forced error: cannot replace lease repl=(n3,s3):2 | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | | | seq=6 start=1528141712.778205832,0 epo=2 pro=1528141712.778209059,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=7 start=1528141712.781259755,0 epo=3 | | | | | | |
| | | | pro=1528141712.781265746,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:32.786022+00:00 | 1m22s997ms965µs672ns | replica.Send got error: cannot replace lease repl=(n3,s3):2 seq=6 | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | [async] storage.pendingLeaseRequest: requesting lease | 244 | | |
| | | | start=1528141712.778205832,0 epo=2 pro=1528141712.778209059,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=7 start=1528141712.781259755,0 epo=3 | | | | | | |
| | | | pro=1528141712.781265746,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:32.786073+00:00 | 1m22s998ms16µs506ns | replica.Send got error: [NotLeaseHolderError] r66: replica | [n1,s1,r66/1:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.778205832,0 epo=2 pro=1528141712.778209059,0 | | | | | | |
| | 2018-06-04 19:48:32.786087+00:00 | 1m22s998ms30µs435ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 242 | | |
| | 2018-06-04 19:48:32.786105+00:00 | 1m22s998ms47µs965ns | application error: [NotLeaseHolderError] r66: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.778205832,0 epo=2 pro=1528141712.778209059,0 | | | | | | |
| | 2018-06-04 19:48:32.786116+00:00 | 1m22s998ms58µs880ns | error: {(err: [NotLeaseHolderError] r66: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.778205832,0 epo=2 pro=1528141712.778209059,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:32.786135+00:00 | 1m22s998ms78µs135ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 241 | | |
| | 2018-06-04 19:48:32.786159+00:00 | 1m22s998ms102µs205ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 248 | | |
| | 2018-06-04 19:48:32.786622+00:00 | 1m22s998ms565µs644ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786671+00:00 | 1m22s998ms614µs413ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786679+00:00 | 1m22s998ms622µs407ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786686+00:00 | 1m22s998ms628µs865ns | read-only path | [n3,s3,r66/2:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786692+00:00 | 1m22s998ms635µs470ns | command queue | [n3,s3,r66/2:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786696+00:00 | 1m22s998ms639µs806ns | waiting for read lock | [n3,s3,r66/2:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.786759+00:00 | 1m22s998ms702µs377ns | read completed | [n3,s3,r66/2:/Table/52/1/4{200-300}] | | /cockroach.roachpb.Internal/Batch | 441 | | |
| | 2018-06-04 19:48:32.787073+00:00 | 1m22s999ms16µs21ns | === SPAN START: dist sender === | | | dist sender | 249 | | |
| | 2018-06-04 19:48:32.787092+00:00 | 1m22s999ms35µs733ns | querying next range at /Table/52/1/4300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | 2018-06-04 19:48:32.787122+00:00 | 1m22s999ms64µs857ns | r67: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | 2018-06-04 19:48:32.787137+00:00 | 1m22s999ms80µs565ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | 2018-06-04 19:48:32.787182+00:00 | 1m22s999ms125µs258ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | 2018-06-04 19:48:32.787188+00:00 | 1m22s999ms131µs11ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | 2018-06-04 19:48:32.787194+00:00 | 1m22s999ms137µs528ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | 2018-06-04 19:48:32.7872+00:00 | 1m22s999ms143µs98ns | read-only path | [n1,s1,r67/1:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | 2018-06-04 19:48:32.78722+00:00 | 1m22s999ms163µs350ns | replica.Send got error: [NotLeaseHolderError] r67: replica | [n1,s1,r67/1:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141709.303521970,0 epo=2 pro=1528141709.303525049,0 | | | | | | |
| | 2018-06-04 19:48:32.787234+00:00 | 1m22s999ms176µs816ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 250 | | |
| | 2018-06-04 19:48:32.787247+00:00 | 1m22s999ms189µs963ns | application error: [NotLeaseHolderError] r67: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141709.303521970,0 epo=2 pro=1528141709.303525049,0 | | | | | | |
| | 2018-06-04 19:48:32.787257+00:00 | 1m22s999ms200µs267ns | error: {(err: [NotLeaseHolderError] r67: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141709.303521970,0 epo=2 pro=1528141709.303525049,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:32.78727+00:00 | 1m22s999ms213µs727ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 249 | | |
| | 2018-06-04 19:48:32.78729+00:00 | 1m22s999ms233µs272ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 251 | | |
| | 2018-06-04 19:48:32.787604+00:00 | 1m22s999ms547µs568ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787631+00:00 | 1m22s999ms573µs859ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787636+00:00 | 1m22s999ms579µs395ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787644+00:00 | 1m22s999ms586µs978ns | read-only path | [n2,s2,r67/3:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787653+00:00 | 1m22s999ms595µs834ns | command queue | [n2,s2,r67/3:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787657+00:00 | 1m22s999ms600µs245ns | waiting for read lock | [n2,s2,r67/3:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.787667+00:00 | 1m22s999ms610µs270ns | read completed | [n2,s2,r67/3:/Table/52/1/4{300-400}] | | /cockroach.roachpb.Internal/Batch | 442 | | |
| | 2018-06-04 19:48:32.78799+00:00 | 1m22s999ms933µs100ns | === SPAN START: dist sender === | | | dist sender | 252 | | |
| | 2018-06-04 19:48:32.788006+00:00 | 1m22s999ms949µs693ns | querying next range at /Table/52/1/4400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 252 | | |
| | 2018-06-04 19:48:32.788046+00:00 | 1m22s999ms989µs72ns | r68: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 252 | | |
| | 2018-06-04 19:48:32.788075+00:00 | 1m23s18µs777ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 252 | | |
| | 2018-06-04 19:48:32.788101+00:00 | 1m23s44µs221ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:32.788107+00:00 | 1m23s49µs959ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:32.788113+00:00 | 1m23s55µs952ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:32.788118+00:00 | 1m23s61µs687ns | read-only path | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:32.788127+00:00 | 1m23s69µs976ns | request range lease (attempt #1) | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:35.776205+00:00 | 1m25s988ms148µs586ns | read has no clock uncertainty | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:35.776213+00:00 | 1m25s988ms156µs96ns | command queue | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:35.776224+00:00 | 1m25s988ms167µs770ns | waiting for read lock | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:35.77624+00:00 | 1m25s988ms183µs432ns | read completed | [n1,s1,r68/1:/Table/52/1/4{400-500}] | | /cockroach.roachpb.Internal/Batch | 253 | | |
| | 2018-06-04 19:48:35.776419+00:00 | 1m25s988ms362µs299ns | === SPAN START: dist sender === | | | dist sender | 254 | | |
| | 2018-06-04 19:48:35.776442+00:00 | 1m25s988ms385µs221ns | querying next range at /Table/52/1/4500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | 2018-06-04 19:48:35.776507+00:00 | 1m25s988ms450µs632ns | r69: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | 2018-06-04 19:48:35.776541+00:00 | 1m25s988ms483µs915ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | 2018-06-04 19:48:35.776571+00:00 | 1m25s988ms514µs426ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | 2018-06-04 19:48:35.776576+00:00 | 1m25s988ms518µs978ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | 2018-06-04 19:48:35.776581+00:00 | 1m25s988ms524µs565ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | 2018-06-04 19:48:35.776587+00:00 | 1m25s988ms530µs635ns | read-only path | [n1,s1,r69/1:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | 2018-06-04 19:48:35.776605+00:00 | 1m25s988ms548µs598ns | replica.Send got error: [NotLeaseHolderError] r69: replica | [n1,s1,r69/1:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.783689295,0 epo=2 pro=1528141712.783691617,0 | | | | | | |
| | 2018-06-04 19:48:35.776629+00:00 | 1m25s988ms572µs727ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 255 | | |
| | 2018-06-04 19:48:35.776646+00:00 | 1m25s988ms589µs702ns | application error: [NotLeaseHolderError] r69: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.783689295,0 epo=2 pro=1528141712.783691617,0 | | | | | | |
| | 2018-06-04 19:48:35.776655+00:00 | 1m25s988ms598µs359ns | error: {(err: [NotLeaseHolderError] r69: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141712.783689295,0 epo=2 pro=1528141712.783691617,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:35.776664+00:00 | 1m25s988ms607µs451ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 254 | | |
| | 2018-06-04 19:48:35.776685+00:00 | 1m25s988ms628µs601ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 256 | | |
| | 2018-06-04 19:48:35.777003+00:00 | 1m25s988ms946µs327ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777053+00:00 | 1m25s988ms996µs551ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777062+00:00 | 1m25s989ms5µs316ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.77707+00:00 | 1m25s989ms13µs196ns | read-only path | [n3,s3,r69/2:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777089+00:00 | 1m25s989ms32µs721ns | command queue | [n3,s3,r69/2:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777098+00:00 | 1m25s989ms41µs602ns | waiting for read lock | [n3,s3,r69/2:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777112+00:00 | 1m25s989ms55µs365ns | read completed | [n3,s3,r69/2:/Table/52/1/4{500-600}] | | /cockroach.roachpb.Internal/Batch | 443 | | |
| | 2018-06-04 19:48:35.777386+00:00 | 1m25s989ms329µs418ns | === SPAN START: dist sender === | | | dist sender | 257 | | |
| | 2018-06-04 19:48:35.7774+00:00 | 1m25s989ms343µs87ns | querying next range at /Table/52/1/4600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | 2018-06-04 19:48:35.777423+00:00 | 1m25s989ms366µs353ns | r70: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | 2018-06-04 19:48:35.777453+00:00 | 1m25s989ms396µs336ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | 2018-06-04 19:48:35.777473+00:00 | 1m25s989ms416µs464ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | 2018-06-04 19:48:35.777477+00:00 | 1m25s989ms420µs39ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | 2018-06-04 19:48:35.777489+00:00 | 1m25s989ms432µs397ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | 2018-06-04 19:48:35.777494+00:00 | 1m25s989ms437µs621ns | read-only path | [n1,s1,r70/1:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | 2018-06-04 19:48:35.777518+00:00 | 1m25s989ms460µs938ns | replica.Send got error: [NotLeaseHolderError] r70: replica | [n1,s1,r70/1:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141712.777878890,0 epo=2 pro=1528141712.777881902,0 | | | | | | |
| | 2018-06-04 19:48:35.777525+00:00 | 1m25s989ms468µs626ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 258 | | |
| | 2018-06-04 19:48:35.777538+00:00 | 1m25s989ms481µs472ns | application error: [NotLeaseHolderError] r70: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141712.777878890,0 epo=2 pro=1528141712.777881902,0 | | | | | | |
| | 2018-06-04 19:48:35.777545+00:00 | 1m25s989ms488µs69ns | error: {(err: [NotLeaseHolderError] r70: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141712.777878890,0 epo=2 pro=1528141712.777881902,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:35.777557+00:00 | 1m25s989ms499µs978ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 257 | | |
| | 2018-06-04 19:48:35.777575+00:00 | 1m25s989ms518µs240ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 259 | | |
| | 2018-06-04 19:48:35.777768+00:00 | 1m25s989ms710µs972ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777792+00:00 | 1m25s989ms735µs43ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777798+00:00 | 1m25s989ms741µs240ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777804+00:00 | 1m25s989ms747µs413ns | read-only path | [n2,s2,r70/3:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777818+00:00 | 1m25s989ms761µs119ns | command queue | [n2,s2,r70/3:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777822+00:00 | 1m25s989ms765µs362ns | waiting for read lock | [n2,s2,r70/3:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.777831+00:00 | 1m25s989ms774µs718ns | read completed | [n2,s2,r70/3:/Table/52/1/4{600-700}] | | /cockroach.roachpb.Internal/Batch | 444 | | |
| | 2018-06-04 19:48:35.778087+00:00 | 1m25s990ms29µs970ns | === SPAN START: dist sender === | | | dist sender | 260 | | |
| | 2018-06-04 19:48:35.778107+00:00 | 1m25s990ms49µs937ns | querying next range at /Table/52/1/4700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 260 | | |
| | 2018-06-04 19:48:35.778137+00:00 | 1m25s990ms80µs190ns | r71: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 260 | | |
| | 2018-06-04 19:48:35.778156+00:00 | 1m25s990ms99µs387ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 260 | | |
| | 2018-06-04 19:48:35.778218+00:00 | 1m25s990ms161µs68ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:35.778222+00:00 | 1m25s990ms165µs173ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:35.778227+00:00 | 1m25s990ms169µs931ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:35.778348+00:00 | 1m25s990ms291µs763ns | read-only path | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:35.778353+00:00 | 1m25s990ms296µs438ns | request range lease (attempt #1) | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:35.778359+00:00 | 1m25s990ms302µs720ns | === SPAN START: request range lease === | | | request range lease | 262 | | |
| | 2018-06-04 19:48:35.778366+00:00 | 1m25s990ms309µs759ns | === SPAN START: [async] storage.pendingLeaseRequest: requesting lease | | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | | | === | | | | | | |
| | 2018-06-04 19:48:35.778385+00:00 | 1m25s990ms328µs63ns | === SPAN START: liveness heartbeat === | | | liveness heartbeat | 264 | | |
| | 2018-06-04 19:48:35.778447+00:00 | 1m25s990ms389µs937ns | === SPAN START: dist sender === | | | dist sender | 265 | | |
| | 2018-06-04 19:48:35.778457+00:00 | 1m25s990ms400µs712ns | querying next range at /System/NodeLiveness/1 | [s1,r71/1:/Table/52/1/4{700-800},txn=4cd7e55d,n1] | | dist sender | 265 | | |
| | 2018-06-04 19:48:35.778486+00:00 | 1m25s990ms429µs383ns | r3: sending batch 1 CPut, 1 BeginTxn, 1 EndTxn to (n3,s3):3 | [s1,r71/1:/Table/52/1/4{700-800},txn=4cd7e55d,n1] | | dist sender | 265 | | |
| | 2018-06-04 19:48:35.778506+00:00 | 1m25s990ms449µs496ns | sending request to alex-laptop.local:26261 | [s1,r71/1:/Table/52/1/4{700-800},txn=4cd7e55d,n1] | | dist sender | 265 | | |
| | 2018-06-04 19:48:35.778518+00:00 | 1m25s990ms461µs383ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 266 | | |
| | 2018-06-04 19:48:35.778726+00:00 | 1m25s990ms669µs350ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778746+00:00 | 1m25s990ms689µs642ns | 1 CPut, 1 BeginTxn, 1 EndTxn | [n3] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778752+00:00 | 1m25s990ms695µs516ns | executing 3 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778758+00:00 | 1m25s990ms701µs235ns | read-write path | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778768+00:00 | 1m25s990ms710µs863ns | command queue | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778794+00:00 | 1m25s990ms737µs243ns | applied timestamp cache | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778906+00:00 | 1m25s990ms849µs470ns | evaluated request | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.778909+00:00 | 1m25s990ms852µs739ns | acquired {raft,replica}mu | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.77947+00:00 | 1m25s991ms413µs655ns | applying command | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.779742+00:00 | 1m25s991ms685µs352ns | engine stats: {InternalDeleteSkippedCount:0 TimeBoundNumSSTs:0} | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | 2018-06-04 19:48:35.779762+00:00 | 1m25s991ms704µs828ns | gossiping 1 node liveness record(s) from span | [n3,s3,r3/3:/System/NodeLiveness{-Max}] | | /cockroach.roachpb.Internal/Batch | 445 | | |
| | | | /System/NodeLiveness/1{-/NULL} | | | | | | |
| | 2018-06-04 19:48:35.780107+00:00 | 1m25s992ms50µs486ns | heartbeat 1528141725.278387864,0 | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | liveness heartbeat | 264 | | |
| | 2018-06-04 19:48:35.780114+00:00 | 1m25s992ms57µs11ns | read-write path | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:35.780119+00:00 | 1m25s992ms62µs292ns | applied timestamp cache | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:35.780215+00:00 | 1m25s992ms158µs212ns | evaluated request | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:35.780218+00:00 | 1m25s992ms161µs229ns | acquired {raft,replica}mu | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:38.773623+00:00 | 1m28s985ms566µs40ns | retry proposal 39f3e964a996535b: reasonTicks | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:38.773676+00:00 | 1m28s985ms618µs956ns | applied timestamp cache | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:38.773855+00:00 | 1m28s985ms798µs497ns | evaluated request | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:38.77386+00:00 | 1m28s985ms802µs919ns | acquired {raft,replica}mu | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.174293+00:00 | 1m29s386ms236µs36ns | retry proposal 1ec080f28b1da913: reasonNewLeader | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.17435+00:00 | 1m29s386ms293µs528ns | applied timestamp cache | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.174526+00:00 | 1m29s386ms468µs886ns | evaluated request | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.174538+00:00 | 1m29s386ms481µs554ns | acquired {raft,replica}mu | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.174891+00:00 | 1m29s386ms834µs600ns | retry proposal 158d33b63000930a: reasonNewLeaderOrConfigChange | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.174933+00:00 | 1m29s386ms876µs312ns | applied timestamp cache | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.175024+00:00 | 1m29s386ms967µs162ns | evaluated request | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.175047+00:00 | 1m29s386ms990µs463ns | acquired {raft,replica}mu | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | 2018-06-04 19:48:39.175929+00:00 | 1m29s387ms872µs428ns | command proposed from replica (n1,s1):1 with lease #5 incompatible | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | | | to repl=(n1,s1):1 seq=6 start=1528141568.368896594,0 epo=3 | | | | | | |
| | | | pro=1528141715.778355836,0 | | | | | | |
| | 2018-06-04 19:48:39.175949+00:00 | 1m29s387ms892µs17ns | applying command with forced error: cannot replace lease repl=(n1,s1):1 | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | | | seq=6 start=1528141568.368896594,0 epo=3 pro=1528141715.778355836,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.368896594,0 epo=3 | | | | | | |
| | | | pro=1528141715.778355836,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:39.176039+00:00 | 1m29s387ms982µs715ns | replica.Send got error: cannot replace lease repl=(n1,s1):1 seq=6 | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | [async] storage.pendingLeaseRequest: requesting lease | 263 | | |
| | | | start=1528141568.368896594,0 epo=3 pro=1528141715.778355836,0 | | | | | | |
| | | | with repl=(n1,s1):1 seq=6 start=1528141568.368896594,0 epo=3 | | | | | | |
| | | | pro=1528141715.778355836,0: proposed under invalid lease | | | | | | |
| | 2018-06-04 19:48:39.176095+00:00 | 1m29s388ms38µs163ns | read has no clock uncertainty | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:39.176101+00:00 | 1m29s388ms43µs978ns | command queue | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:39.176108+00:00 | 1m29s388ms51µs645ns | waiting for read lock | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:39.176128+00:00 | 1m29s388ms70µs947ns | read completed | [n1,s1,r71/1:/Table/52/1/4{700-800}] | | /cockroach.roachpb.Internal/Batch | 261 | | |
| | 2018-06-04 19:48:39.176281+00:00 | 1m29s388ms224µs233ns | === SPAN START: dist sender === | | | dist sender | 267 | | |
| | 2018-06-04 19:48:39.176373+00:00 | 1m29s388ms315µs862ns | querying next range at /Table/52/1/4800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | 2018-06-04 19:48:39.17641+00:00 | 1m29s388ms353µs500ns | r72: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | 2018-06-04 19:48:39.176429+00:00 | 1m29s388ms372µs271ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | 2018-06-04 19:48:39.176446+00:00 | 1m29s388ms388µs831ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | 2018-06-04 19:48:39.17645+00:00 | 1m29s388ms393µs660ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | 2018-06-04 19:48:39.176456+00:00 | 1m29s388ms399µs794ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | 2018-06-04 19:48:39.176463+00:00 | 1m29s388ms406µs763ns | read-only path | [n1,s1,r72/1:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | 2018-06-04 19:48:39.176485+00:00 | 1m29s388ms428µs151ns | replica.Send got error: [NotLeaseHolderError] r72: replica | [n1,s1,r72/1:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141715.924066503,0 epo=2 pro=1528141715.924070784,0 | | | | | | |
| | 2018-06-04 19:48:39.176496+00:00 | 1m29s388ms439µs678ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 268 | | |
| | 2018-06-04 19:48:39.176515+00:00 | 1m29s388ms458µs370ns | application error: [NotLeaseHolderError] r72: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141715.924066503,0 epo=2 pro=1528141715.924070784,0 | | | | | | |
| | 2018-06-04 19:48:39.176526+00:00 | 1m29s388ms469µs220ns | error: {(err: [NotLeaseHolderError] r72: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141715.924066503,0 epo=2 pro=1528141715.924070784,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:39.176542+00:00 | 1m29s388ms485µs569ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 267 | | |
| | 2018-06-04 19:48:39.176566+00:00 | 1m29s388ms509µs252ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 269 | | |
| | 2018-06-04 19:48:39.177012+00:00 | 1m29s388ms955µs497ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177086+00:00 | 1m29s389ms29µs655ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177093+00:00 | 1m29s389ms36µs109ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.1771+00:00 | 1m29s389ms42µs867ns | read-only path | [n3,s3,r72/2:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177107+00:00 | 1m29s389ms50µs450ns | command queue | [n3,s3,r72/2:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177114+00:00 | 1m29s389ms56µs878ns | waiting for read lock | [n3,s3,r72/2:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177123+00:00 | 1m29s389ms66µs571ns | read completed | [n3,s3,r72/2:/Table/52/1/4{800-900}] | | /cockroach.roachpb.Internal/Batch | 446 | | |
| | 2018-06-04 19:48:39.177385+00:00 | 1m29s389ms328µs487ns | === SPAN START: dist sender === | | | dist sender | 270 | | |
| | 2018-06-04 19:48:39.177397+00:00 | 1m29s389ms340µs110ns | querying next range at /Table/52/1/4900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | 2018-06-04 19:48:39.177414+00:00 | 1m29s389ms357µs373ns | r73: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | 2018-06-04 19:48:39.177434+00:00 | 1m29s389ms377µs658ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | 2018-06-04 19:48:39.177448+00:00 | 1m29s389ms390µs915ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | 2018-06-04 19:48:39.177452+00:00 | 1m29s389ms395µs485ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | 2018-06-04 19:48:39.177457+00:00 | 1m29s389ms400µs605ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | 2018-06-04 19:48:39.177463+00:00 | 1m29s389ms406µs24ns | read-only path | [n1,s1,r73/1:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | 2018-06-04 19:48:39.177483+00:00 | 1m29s389ms425µs958ns | replica.Send got error: [NotLeaseHolderError] r73: replica | [n1,s1,r73/1:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141712.782373828,0 epo=2 pro=1528141712.782376306,0 | | | | | | |
| | 2018-06-04 19:48:39.177493+00:00 | 1m29s389ms436µs692ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 271 | | |
| | 2018-06-04 19:48:39.177505+00:00 | 1m29s389ms448µs619ns | application error: [NotLeaseHolderError] r73: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141712.782373828,0 epo=2 pro=1528141712.782376306,0 | | | | | | |
| | 2018-06-04 19:48:39.17753+00:00 | 1m29s389ms473µs15ns | error: {(err: [NotLeaseHolderError] r73: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141712.782373828,0 epo=2 pro=1528141712.782376306,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:39.177579+00:00 | 1m29s389ms522µs278ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 270 | | |
| | 2018-06-04 19:48:39.177601+00:00 | 1m29s389ms544µs677ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 272 | | |
| | 2018-06-04 19:48:39.177987+00:00 | 1m29s389ms930µs474ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178035+00:00 | 1m29s389ms978µs152ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178047+00:00 | 1m29s389ms990µs440ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178055+00:00 | 1m29s389ms998µs19ns | read-only path | [n2,s2,r73/3:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178064+00:00 | 1m29s390ms7µs163ns | command queue | [n2,s2,r73/3:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178069+00:00 | 1m29s390ms12µs435ns | waiting for read lock | [n2,s2,r73/3:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178082+00:00 | 1m29s390ms25µs360ns | read completed | [n2,s2,r73/3:/Table/52/1/{4900-5000}] | | /cockroach.roachpb.Internal/Batch | 447 | | |
| | 2018-06-04 19:48:39.178498+00:00 | 1m29s390ms441µs286ns | === SPAN START: dist sender === | | | dist sender | 273 | | |
| | 2018-06-04 19:48:39.178531+00:00 | 1m29s390ms474µs342ns | querying next range at /Table/52/1/5000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 273 | | |
| | 2018-06-04 19:48:39.178591+00:00 | 1m29s390ms534µs746ns | r74: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 273 | | |
| | 2018-06-04 19:48:39.178644+00:00 | 1m29s390ms587µs296ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 273 | | |
| | 2018-06-04 19:48:39.178698+00:00 | 1m29s390ms641µs430ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:39.178705+00:00 | 1m29s390ms647µs880ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:39.178711+00:00 | 1m29s390ms654µs274ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:39.178716+00:00 | 1m29s390ms659µs425ns | read-only path | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:39.178721+00:00 | 1m29s390ms663µs877ns | request range lease (attempt #1) | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:44.179428+00:00 | 1m34s391ms371µs694ns | read has no clock uncertainty | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:44.179436+00:00 | 1m34s391ms379µs480ns | command queue | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:44.179445+00:00 | 1m34s391ms388µs595ns | waiting for read lock | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:44.179462+00:00 | 1m34s391ms405µs508ns | read completed | [n1,s1,r74/1:/Table/52/1/5{000-100}] | | /cockroach.roachpb.Internal/Batch | 274 | | |
| | 2018-06-04 19:48:44.179565+00:00 | 1m34s391ms508µs525ns | === SPAN START: dist sender === | | | dist sender | 275 | | |
| | 2018-06-04 19:48:44.179583+00:00 | 1m34s391ms525µs836ns | querying next range at /Table/52/1/5100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | 2018-06-04 19:48:44.179615+00:00 | 1m34s391ms558µs173ns | r75: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | 2018-06-04 19:48:44.179637+00:00 | 1m34s391ms580µs730ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | 2018-06-04 19:48:44.179652+00:00 | 1m34s391ms594µs902ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | 2018-06-04 19:48:44.179656+00:00 | 1m34s391ms599µs42ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | 2018-06-04 19:48:44.179661+00:00 | 1m34s391ms604µs497ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | 2018-06-04 19:48:44.179667+00:00 | 1m34s391ms610µs685ns | read-only path | [n1,s1,r75/1:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | 2018-06-04 19:48:44.179684+00:00 | 1m34s391ms627µs518ns | replica.Send got error: [NotLeaseHolderError] r75: replica | [n1,s1,r75/1:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141716.382532425,0 epo=2 pro=1528141716.382537502,0 | | | | | | |
| | 2018-06-04 19:48:44.179694+00:00 | 1m34s391ms637µs289ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 276 | | |
| | 2018-06-04 19:48:44.17971+00:00 | 1m34s391ms653µs196ns | application error: [NotLeaseHolderError] r75: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141716.382532425,0 epo=2 pro=1528141716.382537502,0 | | | | | | |
| | 2018-06-04 19:48:44.17972+00:00 | 1m34s391ms662µs921ns | error: {(err: [NotLeaseHolderError] r75: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141716.382532425,0 epo=2 pro=1528141716.382537502,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:44.179735+00:00 | 1m34s391ms677µs840ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 275 | | |
| | 2018-06-04 19:48:44.179757+00:00 | 1m34s391ms700µs334ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 277 | | |
| | 2018-06-04 19:48:44.180061+00:00 | 1m34s392ms4µs148ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180096+00:00 | 1m34s392ms39µs534ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180104+00:00 | 1m34s392ms47µs469ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180112+00:00 | 1m34s392ms55µs222ns | read-only path | [n3,s3,r75/2:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.18012+00:00 | 1m34s392ms63µs404ns | command queue | [n3,s3,r75/2:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180127+00:00 | 1m34s392ms70µs180ns | waiting for read lock | [n3,s3,r75/2:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180141+00:00 | 1m34s392ms84µs333ns | read completed | [n3,s3,r75/2:/Table/52/1/5{100-200}] | | /cockroach.roachpb.Internal/Batch | 448 | | |
| | 2018-06-04 19:48:44.180483+00:00 | 1m34s392ms426µs580ns | === SPAN START: dist sender === | | | dist sender | 278 | | |
| | 2018-06-04 19:48:44.18053+00:00 | 1m34s392ms472µs961ns | querying next range at /Table/52/1/5200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | 2018-06-04 19:48:44.180561+00:00 | 1m34s392ms503µs880ns | r76: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | 2018-06-04 19:48:44.180581+00:00 | 1m34s392ms524µs381ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | 2018-06-04 19:48:44.180593+00:00 | 1m34s392ms536µs441ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | 2018-06-04 19:48:44.180597+00:00 | 1m34s392ms540µs264ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | 2018-06-04 19:48:44.180602+00:00 | 1m34s392ms545µs591ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | 2018-06-04 19:48:44.180608+00:00 | 1m34s392ms550µs923ns | read-only path | [n1,s1,r76/1:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | 2018-06-04 19:48:44.180626+00:00 | 1m34s392ms569µs295ns | replica.Send got error: [NotLeaseHolderError] r76: replica | [n1,s1,r76/1:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141715.923999313,0 epo=2 pro=1528141715.924002790,0 | | | | | | |
| | 2018-06-04 19:48:44.180636+00:00 | 1m34s392ms579µs14ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 279 | | |
| | 2018-06-04 19:48:44.180653+00:00 | 1m34s392ms596µs13ns | application error: [NotLeaseHolderError] r76: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141715.923999313,0 epo=2 pro=1528141715.924002790,0 | | | | | | |
| | 2018-06-04 19:48:44.180663+00:00 | 1m34s392ms606µs593ns | error: {(err: [NotLeaseHolderError] r76: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141715.923999313,0 epo=2 pro=1528141715.924002790,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:44.180684+00:00 | 1m34s392ms627µs180ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 278 | | |
| | 2018-06-04 19:48:44.180702+00:00 | 1m34s392ms645µs638ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 280 | | |
| | 2018-06-04 19:48:44.180988+00:00 | 1m34s392ms931µs53ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.181016+00:00 | 1m34s392ms959µs467ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.181023+00:00 | 1m34s392ms966µs360ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.18103+00:00 | 1m34s392ms973µs615ns | read-only path | [n2,s2,r76/3:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.181038+00:00 | 1m34s392ms981µs260ns | command queue | [n2,s2,r76/3:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.181043+00:00 | 1m34s392ms986µs294ns | waiting for read lock | [n2,s2,r76/3:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.181068+00:00 | 1m34s393ms11µs171ns | read completed | [n2,s2,r76/3:/Table/52/1/5{200-300}] | | /cockroach.roachpb.Internal/Batch | 449 | | |
| | 2018-06-04 19:48:44.18147+00:00 | 1m34s393ms412µs909ns | === SPAN START: dist sender === | | | dist sender | 281 | | |
| | 2018-06-04 19:48:44.181483+00:00 | 1m34s393ms426µs140ns | querying next range at /Table/52/1/5300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 281 | | |
| | 2018-06-04 19:48:44.181507+00:00 | 1m34s393ms450µs74ns | r77: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 281 | | |
| | 2018-06-04 19:48:44.18153+00:00 | 1m34s393ms473µs278ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 281 | | |
| | 2018-06-04 19:48:44.181548+00:00 | 1m34s393ms490µs979ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:44.181553+00:00 | 1m34s393ms496µs784ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:44.18156+00:00 | 1m34s393ms503µs432ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:44.181567+00:00 | 1m34s393ms510µs2ns | read-only path | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:44.181573+00:00 | 1m34s393ms516µs362ns | request range lease (attempt #1) | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:47.179094+00:00 | 1m37s391ms37µs86ns | read has no clock uncertainty | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:47.179101+00:00 | 1m37s391ms44µs286ns | command queue | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:47.17911+00:00 | 1m37s391ms53µs414ns | waiting for read lock | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:47.179137+00:00 | 1m37s391ms80µs629ns | read completed | [n1,s1,r77/1:/Table/52/1/5{300-400}] | | /cockroach.roachpb.Internal/Batch | 282 | | |
| | 2018-06-04 19:48:47.179231+00:00 | 1m37s391ms174µs717ns | === SPAN START: dist sender === | | | dist sender | 283 | | |
| | 2018-06-04 19:48:47.179247+00:00 | 1m37s391ms190µs337ns | querying next range at /Table/52/1/5400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | 2018-06-04 19:48:47.179279+00:00 | 1m37s391ms222µs616ns | r78: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | 2018-06-04 19:48:47.179297+00:00 | 1m37s391ms240µs32ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | 2018-06-04 19:48:47.179309+00:00 | 1m37s391ms252µs741ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | 2018-06-04 19:48:47.179313+00:00 | 1m37s391ms256µs544ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | 2018-06-04 19:48:47.179318+00:00 | 1m37s391ms261µs722ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | 2018-06-04 19:48:47.179324+00:00 | 1m37s391ms267µs787ns | read-only path | [n1,s1,r78/1:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | 2018-06-04 19:48:47.179342+00:00 | 1m37s391ms285µs656ns | replica.Send got error: [NotLeaseHolderError] r78: replica | [n1,s1,r78/1:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141719.775159447,0 epo=2 pro=1528141719.775164018,0 | | | | | | |
| | 2018-06-04 19:48:47.179352+00:00 | 1m37s391ms294µs940ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 284 | | |
| | 2018-06-04 19:48:47.179368+00:00 | 1m37s391ms311µs675ns | application error: [NotLeaseHolderError] r78: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141719.775159447,0 epo=2 pro=1528141719.775164018,0 | | | | | | |
| | 2018-06-04 19:48:47.179377+00:00 | 1m37s391ms320µs121ns | error: {(err: [NotLeaseHolderError] r78: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141719.775159447,0 epo=2 pro=1528141719.775164018,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:47.17939+00:00 | 1m37s391ms332µs995ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 283 | | |
| | 2018-06-04 19:48:47.179413+00:00 | 1m37s391ms355µs913ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 285 | | |
| | 2018-06-04 19:48:47.179655+00:00 | 1m37s391ms598µs35ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179683+00:00 | 1m37s391ms625µs838ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179689+00:00 | 1m37s391ms632µs216ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179696+00:00 | 1m37s391ms639µs133ns | read-only path | [n3,s3,r78/2:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179703+00:00 | 1m37s391ms646µs125ns | command queue | [n3,s3,r78/2:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179707+00:00 | 1m37s391ms650µs133ns | waiting for read lock | [n3,s3,r78/2:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179719+00:00 | 1m37s391ms662µs567ns | read completed | [n3,s3,r78/2:/Table/52/1/5{400-500}] | | /cockroach.roachpb.Internal/Batch | 450 | | |
| | 2018-06-04 19:48:47.179969+00:00 | 1m37s391ms911µs903ns | === SPAN START: dist sender === | | | dist sender | 286 | | |
| | 2018-06-04 19:48:47.179979+00:00 | 1m37s391ms922µs463ns | querying next range at /Table/52/1/5500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | 2018-06-04 19:48:47.18+00:00 | 1m37s391ms943µs560ns | r79: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | 2018-06-04 19:48:47.180017+00:00 | 1m37s391ms960µs742ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | 2018-06-04 19:48:47.180028+00:00 | 1m37s391ms971µs68ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | 2018-06-04 19:48:47.180031+00:00 | 1m37s391ms974µs212ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | 2018-06-04 19:48:47.180035+00:00 | 1m37s391ms978µs400ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | 2018-06-04 19:48:47.18004+00:00 | 1m37s391ms983µs639ns | read-only path | [n1,s1,r79/1:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | 2018-06-04 19:48:47.180057+00:00 | 1m37s392ms628ns | replica.Send got error: [NotLeaseHolderError] r79: replica | [n1,s1,r79/1:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141716.175438217,0 epo=2 pro=1528141716.175442204,0 | | | | | | |
| | 2018-06-04 19:48:47.180066+00:00 | 1m37s392ms9µs332ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 287 | | |
| | 2018-06-04 19:48:47.180083+00:00 | 1m37s392ms26µs494ns | application error: [NotLeaseHolderError] r79: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141716.175438217,0 epo=2 pro=1528141716.175442204,0 | | | | | | |
| | 2018-06-04 19:48:47.180093+00:00 | 1m37s392ms36µs721ns | error: {(err: [NotLeaseHolderError] r79: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=4 | | | | | | |
| | | | start=1528141716.175438217,0 epo=2 pro=1528141716.175442204,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:47.180108+00:00 | 1m37s392ms51µs116ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 286 | | |
| | 2018-06-04 19:48:47.180128+00:00 | 1m37s392ms71µs423ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 288 | | |
| | 2018-06-04 19:48:47.180379+00:00 | 1m37s392ms322µs590ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180443+00:00 | 1m37s392ms386µs5ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180451+00:00 | 1m37s392ms394µs758ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180461+00:00 | 1m37s392ms403µs838ns | read-only path | [n2,s2,r79/3:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180468+00:00 | 1m37s392ms411µs253ns | command queue | [n2,s2,r79/3:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180473+00:00 | 1m37s392ms415µs833ns | waiting for read lock | [n2,s2,r79/3:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180485+00:00 | 1m37s392ms428µs14ns | read completed | [n2,s2,r79/3:/Table/52/1/5{500-600}] | | /cockroach.roachpb.Internal/Batch | 451 | | |
| | 2018-06-04 19:48:47.180848+00:00 | 1m37s392ms791µs100ns | === SPAN START: dist sender === | | | dist sender | 289 | | |
| | 2018-06-04 19:48:47.180865+00:00 | 1m37s392ms807µs874ns | querying next range at /Table/52/1/5600 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 289 | | |
| | 2018-06-04 19:48:47.180893+00:00 | 1m37s392ms836µs436ns | r80: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 289 | | |
| | 2018-06-04 19:48:47.180912+00:00 | 1m37s392ms855µs413ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 289 | | |
| | 2018-06-04 19:48:47.180923+00:00 | 1m37s392ms866µs291ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:47.180926+00:00 | 1m37s392ms869µs799ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:47.180931+00:00 | 1m37s392ms874µs75ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:47.180936+00:00 | 1m37s392ms879µs176ns | read-only path | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:47.18094+00:00 | 1m37s392ms883µs479ns | request range lease (attempt #1) | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:50.775404+00:00 | 1m40s987ms347µs1ns | read has no clock uncertainty | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:50.775408+00:00 | 1m40s987ms351µs679ns | command queue | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:50.775415+00:00 | 1m40s987ms358µs44ns | waiting for read lock | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:50.775426+00:00 | 1m40s987ms369µs458ns | read completed | [n1,s1,r80/1:/Table/52/1/5{600-700}] | | /cockroach.roachpb.Internal/Batch | 290 | | |
| | 2018-06-04 19:48:50.775508+00:00 | 1m40s987ms451µs45ns | === SPAN START: dist sender === | | | dist sender | 291 | | |
| | 2018-06-04 19:48:50.775522+00:00 | 1m40s987ms465µs499ns | querying next range at /Table/52/1/5700 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | 2018-06-04 19:48:50.775555+00:00 | 1m40s987ms498µs393ns | r81: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | 2018-06-04 19:48:50.775572+00:00 | 1m40s987ms515µs381ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | 2018-06-04 19:48:50.775585+00:00 | 1m40s987ms528µs11ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | 2018-06-04 19:48:50.775588+00:00 | 1m40s987ms530µs952ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | 2018-06-04 19:48:50.775595+00:00 | 1m40s987ms538µs723ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | 2018-06-04 19:48:50.775611+00:00 | 1m40s987ms553µs897ns | read-only path | [n1,s1,r81/1:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | 2018-06-04 19:48:50.775625+00:00 | 1m40s987ms568µs658ns | replica.Send got error: [NotLeaseHolderError] r81: replica | [n1,s1,r81/1:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141722.983021047,0 epo=2 pro=1528141722.983025952,0 | | | | | | |
| | 2018-06-04 19:48:50.775638+00:00 | 1m40s987ms581µs68ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 292 | | |
| | 2018-06-04 19:48:50.775651+00:00 | 1m40s987ms594µs201ns | application error: [NotLeaseHolderError] r81: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141722.983021047,0 epo=2 pro=1528141722.983025952,0 | | | | | | |
| | 2018-06-04 19:48:50.775662+00:00 | 1m40s987ms605µs372ns | error: {(err: [NotLeaseHolderError] r81: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=4 | | | | | | |
| | | | start=1528141722.983021047,0 epo=2 pro=1528141722.983025952,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:50.775673+00:00 | 1m40s987ms616µs589ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 291 | | |
| | 2018-06-04 19:48:50.775699+00:00 | 1m40s987ms642µs630ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 293 | | |
| | 2018-06-04 19:48:50.776025+00:00 | 1m40s987ms968µs602ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.776063+00:00 | 1m40s988ms6µs377ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.77607+00:00 | 1m40s988ms13µs627ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.77608+00:00 | 1m40s988ms22µs926ns | read-only path | [n3,s3,r81/2:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.776089+00:00 | 1m40s988ms32µs31ns | command queue | [n3,s3,r81/2:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.776093+00:00 | 1m40s988ms36µs664ns | waiting for read lock | [n3,s3,r81/2:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.776106+00:00 | 1m40s988ms49µs33ns | read completed | [n3,s3,r81/2:/Table/52/1/5{700-800}] | | /cockroach.roachpb.Internal/Batch | 452 | | |
| | 2018-06-04 19:48:50.776368+00:00 | 1m40s988ms311µs507ns | === SPAN START: dist sender === | | | dist sender | 294 | | |
| | 2018-06-04 19:48:50.776378+00:00 | 1m40s988ms321µs526ns | querying next range at /Table/52/1/5800 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | 2018-06-04 19:48:50.776396+00:00 | 1m40s988ms339µs117ns | r82: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | 2018-06-04 19:48:50.776412+00:00 | 1m40s988ms355µs439ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | 2018-06-04 19:48:50.776422+00:00 | 1m40s988ms365µs807ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | 2018-06-04 19:48:50.776427+00:00 | 1m40s988ms370µs159ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | 2018-06-04 19:48:50.776431+00:00 | 1m40s988ms374µs411ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | 2018-06-04 19:48:50.776436+00:00 | 1m40s988ms379µs330ns | read-only path | [n1,s1,r82/1:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | 2018-06-04 19:48:50.776453+00:00 | 1m40s988ms396µs163ns | replica.Send got error: [NotLeaseHolderError] r82: replica | [n1,s1,r82/1:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141719.176882111,0 epo=2 pro=1528141719.176887570,0 | | | | | | |
| | 2018-06-04 19:48:50.776466+00:00 | 1m40s988ms409µs131ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 295 | | |
| | 2018-06-04 19:48:50.776484+00:00 | 1m40s988ms427µs444ns | application error: [NotLeaseHolderError] r82: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141719.176882111,0 epo=2 pro=1528141719.176887570,0 | | | | | | |
| | 2018-06-04 19:48:50.776493+00:00 | 1m40s988ms436µs2ns | error: {(err: [NotLeaseHolderError] r82: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141719.176882111,0 epo=2 pro=1528141719.176887570,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:50.776506+00:00 | 1m40s988ms449µs647ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 294 | | |
| | 2018-06-04 19:48:50.77652+00:00 | 1m40s988ms463µs550ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 296 | | |
| | 2018-06-04 19:48:50.776976+00:00 | 1m40s988ms919µs164ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777041+00:00 | 1m40s988ms984µs681ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.77705+00:00 | 1m40s988ms993µs391ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777059+00:00 | 1m40s989ms2µs300ns | read-only path | [n2,s2,r82/3:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777069+00:00 | 1m40s989ms12µs41ns | command queue | [n2,s2,r82/3:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777075+00:00 | 1m40s989ms18µs306ns | waiting for read lock | [n2,s2,r82/3:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777104+00:00 | 1m40s989ms47µs738ns | read completed | [n2,s2,r82/3:/Table/52/1/5{800-900}] | | /cockroach.roachpb.Internal/Batch | 453 | | |
| | 2018-06-04 19:48:50.777423+00:00 | 1m40s989ms365µs988ns | === SPAN START: dist sender === | | | dist sender | 297 | | |
| | 2018-06-04 19:48:50.777443+00:00 | 1m40s989ms386µs126ns | querying next range at /Table/52/1/5900 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 297 | | |
| | 2018-06-04 19:48:50.777474+00:00 | 1m40s989ms417µs298ns | r83: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 297 | | |
| | 2018-06-04 19:48:50.777501+00:00 | 1m40s989ms444µs536ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 297 | | |
| | 2018-06-04 19:48:50.777521+00:00 | 1m40s989ms463µs977ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:50.777526+00:00 | 1m40s989ms469µs315ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:50.777533+00:00 | 1m40s989ms476µs553ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:50.77754+00:00 | 1m40s989ms483µs584ns | read-only path | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:50.777547+00:00 | 1m40s989ms490µs480ns | request range lease (attempt #1) | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:54.375749+00:00 | 1m44s587ms691µs843ns | read has no clock uncertainty | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:54.375757+00:00 | 1m44s587ms700µs582ns | command queue | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:54.375767+00:00 | 1m44s587ms709µs887ns | waiting for read lock | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:54.375784+00:00 | 1m44s587ms727µs250ns | read completed | [n1,s1,r83/1:/Table/52/1/{5900-6000}] | | /cockroach.roachpb.Internal/Batch | 298 | | |
| | 2018-06-04 19:48:54.375908+00:00 | 1m44s587ms851µs191ns | === SPAN START: dist sender === | | | dist sender | 299 | | |
| | 2018-06-04 19:48:54.375925+00:00 | 1m44s587ms868µs458ns | querying next range at /Table/52/1/6000 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | 2018-06-04 19:48:54.37597+00:00 | 1m44s587ms913µs456ns | r84: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | 2018-06-04 19:48:54.375995+00:00 | 1m44s587ms938µs451ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | 2018-06-04 19:48:54.376011+00:00 | 1m44s587ms954µs104ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | 2018-06-04 19:48:54.376016+00:00 | 1m44s587ms959µs27ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | 2018-06-04 19:48:54.376021+00:00 | 1m44s587ms964µs585ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | 2018-06-04 19:48:54.376028+00:00 | 1m44s587ms971µs621ns | read-only path | [n1,s1,r84/1:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | 2018-06-04 19:48:54.376047+00:00 | 1m44s587ms990µs125ns | replica.Send got error: [NotLeaseHolderError] r84: replica | [n1,s1,r84/1:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141727.975738894,0 epo=2 pro=1528141727.975745473,0 | | | | | | |
| | 2018-06-04 19:48:54.376057+00:00 | 1m44s588ms436ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 300 | | |
| | 2018-06-04 19:48:54.376075+00:00 | 1m44s588ms17µs863ns | application error: [NotLeaseHolderError] r84: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141727.975738894,0 epo=2 pro=1528141727.975745473,0 | | | | | | |
| | 2018-06-04 19:48:54.376085+00:00 | 1m44s588ms28µs263ns | error: {(err: [NotLeaseHolderError] r84: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | | | not lease holder; current lease is repl=(n3,s3):2 seq=6 | | | | | | |
| | | | start=1528141727.975738894,0 epo=2 pro=1528141727.975745473,0) <nil>}; | | | | | | |
| | | | trying next peer (n3,s3):2 | | | | | | |
| | 2018-06-04 19:48:54.376101+00:00 | 1m44s588ms43µs847ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 299 | | |
| | 2018-06-04 19:48:54.376124+00:00 | 1m44s588ms67µs690ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 301 | | |
| | 2018-06-04 19:48:54.376445+00:00 | 1m44s588ms388µs684ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376476+00:00 | 1m44s588ms419µs417ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376483+00:00 | 1m44s588ms426µs692ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376494+00:00 | 1m44s588ms437µs50ns | read-only path | [n3,s3,r84/2:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376502+00:00 | 1m44s588ms444µs989ns | command queue | [n3,s3,r84/2:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376506+00:00 | 1m44s588ms449µs729ns | waiting for read lock | [n3,s3,r84/2:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.37652+00:00 | 1m44s588ms463µs170ns | read completed | [n3,s3,r84/2:/Table/52/1/6{000-100}] | | /cockroach.roachpb.Internal/Batch | 454 | | |
| | 2018-06-04 19:48:54.376915+00:00 | 1m44s588ms857µs970ns | === SPAN START: dist sender === | | | dist sender | 302 | | |
| | 2018-06-04 19:48:54.37694+00:00 | 1m44s588ms883µs25ns | querying next range at /Table/52/1/6100 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | 2018-06-04 19:48:54.376973+00:00 | 1m44s588ms915µs966ns | r85: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | 2018-06-04 19:48:54.377+00:00 | 1m44s588ms943µs508ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | 2018-06-04 19:48:54.377019+00:00 | 1m44s588ms962µs238ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | 2018-06-04 19:48:54.377025+00:00 | 1m44s588ms968µs232ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | 2018-06-04 19:48:54.377032+00:00 | 1m44s588ms975µs206ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | 2018-06-04 19:48:54.37704+00:00 | 1m44s588ms983µs177ns | read-only path | [n1,s1,r85/1:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | 2018-06-04 19:48:54.377066+00:00 | 1m44s589ms8µs957ns | replica.Send got error: [NotLeaseHolderError] r85: replica | [n1,s1,r85/1:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | | | (n1,s1):1 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141723.376944386,0 epo=2 pro=1528141723.376949091,0 | | | | | | |
| | 2018-06-04 19:48:54.377081+00:00 | 1m44s589ms23µs971ns | *roachpb.NotLeaseHolderError | [n1] | | /cockroach.roachpb.Internal/Batch | 303 | | |
| | 2018-06-04 19:48:54.377113+00:00 | 1m44s589ms56µs104ns | application error: [NotLeaseHolderError] r85: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141723.376944386,0 epo=2 pro=1528141723.376949091,0 | | | | | | |
| | 2018-06-04 19:48:54.377124+00:00 | 1m44s589ms67µs456ns | error: {(err: [NotLeaseHolderError] r85: replica (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141723.376944386,0 epo=2 pro=1528141723.376949091,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:54.377141+00:00 | 1m44s589ms83µs949ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 302 | | |
| | 2018-06-04 19:48:54.377162+00:00 | 1m44s589ms105µs301ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 304 | | |
| | 2018-06-04 19:48:54.377462+00:00 | 1m44s589ms405µs250ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377499+00:00 | 1m44s589ms442µs317ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377506+00:00 | 1m44s589ms449µs536ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377513+00:00 | 1m44s589ms455µs977ns | read-only path | [n2,s2,r85/3:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377529+00:00 | 1m44s589ms472µs276ns | command queue | [n2,s2,r85/3:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377535+00:00 | 1m44s589ms478µs281ns | waiting for read lock | [n2,s2,r85/3:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.377545+00:00 | 1m44s589ms488µs155ns | read completed | [n2,s2,r85/3:/Table/52/1/6{100-200}] | | /cockroach.roachpb.Internal/Batch | 455 | | |
| | 2018-06-04 19:48:54.37787+00:00 | 1m44s589ms812µs978ns | === SPAN START: dist sender === | | | dist sender | 305 | | |
| | 2018-06-04 19:48:54.37789+00:00 | 1m44s589ms832µs972ns | querying next range at /Table/52/1/6200 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 305 | | |
| | 2018-06-04 19:48:54.377921+00:00 | 1m44s589ms863µs860ns | r86: sending batch 1 LeaseInfo to (n1,s1):1 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 305 | | |
| | 2018-06-04 19:48:54.377946+00:00 | 1m44s589ms889µs595ns | sending request to local server | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 305 | | |
| | 2018-06-04 19:48:54.377959+00:00 | 1m44s589ms902µs27ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:54.377963+00:00 | 1m44s589ms906µs765ns | 1 LeaseInfo | [n1] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:54.377968+00:00 | 1m44s589ms911µs371ns | executing 1 requests | [n1,s1] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:54.377973+00:00 | 1m44s589ms916µs70ns | read-only path | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:54.377977+00:00 | 1m44s589ms919µs931ns | request range lease (attempt #1) | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:57.581048+00:00 | 1m47s792ms991µs169ns | read has no clock uncertainty | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:57.581054+00:00 | 1m47s792ms996µs881ns | command queue | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:57.581061+00:00 | 1m47s793ms4µs155ns | waiting for read lock | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:57.581074+00:00 | 1m47s793ms17µs65ns | read completed | [n1,s1,r86/1:/Table/52/1/6{200-300}] | | /cockroach.roachpb.Internal/Batch | 306 | | |
| | 2018-06-04 19:48:57.581156+00:00 | 1m47s793ms99µs559ns | === SPAN START: dist sender === | | | dist sender | 307 | | |
| | 2018-06-04 19:48:57.581172+00:00 | 1m47s793ms115µs415ns | querying next range at /Table/52/1/6300 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 307 | | |
| | 2018-06-04 19:48:57.581206+00:00 | 1m47s793ms149µs177ns | r87: sending batch 1 LeaseInfo to (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 307 | | |
| | 2018-06-04 19:48:57.581222+00:00 | 1m47s793ms165µs591ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 307 | | |
| | 2018-06-04 19:48:57.581245+00:00 | 1m47s793ms188µs812ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 308 | | |
| | 2018-06-04 19:48:57.581587+00:00 | 1m47s793ms530µs515ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.581621+00:00 | 1m47s793ms564µs278ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.58163+00:00 | 1m47s793ms573µs605ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.581641+00:00 | 1m47s793ms584µs367ns | read-only path | [n3,s3,r87/2:/Table/52/1/6{300-400}] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.581651+00:00 | 1m47s793ms594µs779ns | command queue | [n3,s3,r87/2:/Table/52/1/6{300-400}] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.581658+00:00 | 1m47s793ms601µs465ns | waiting for read lock | [n3,s3,r87/2:/Table/52/1/6{300-400}] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.58167+00:00 | 1m47s793ms613µs148ns | read completed | [n3,s3,r87/2:/Table/52/1/6{300-400}] | | /cockroach.roachpb.Internal/Batch | 456 | | |
| | 2018-06-04 19:48:57.581954+00:00 | 1m47s793ms897µs591ns | === SPAN START: dist sender === | | | dist sender | 309 | | |
| | 2018-06-04 19:48:57.581967+00:00 | 1m47s793ms910µs471ns | querying next range at /Table/52/1/6400 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | 2018-06-04 19:48:57.581991+00:00 | 1m47s793ms934µs19ns | r88: sending batch 1 LeaseInfo to (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | 2018-06-04 19:48:57.58201+00:00 | 1m47s793ms953µs60ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | 2018-06-04 19:48:57.582025+00:00 | 1m47s793ms968µs385ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 310 | | |
| | 2018-06-04 19:48:57.582251+00:00 | 1m47s794ms194µs336ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | 2018-06-04 19:48:57.582287+00:00 | 1m47s794ms230µs211ns | 1 LeaseInfo | [n3] | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | 2018-06-04 19:48:57.582295+00:00 | 1m47s794ms237µs954ns | executing 1 requests | [n3,s3] | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | 2018-06-04 19:48:57.582302+00:00 | 1m47s794ms245µs764ns | read-only path | [n3,s3,r88/2:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | 2018-06-04 19:48:57.582372+00:00 | 1m47s794ms315µs251ns | replica.Send got error: [NotLeaseHolderError] r88: replica | [n3,s3,r88/2:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | | | (n3,s3):2 not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141726.779738033,0 epo=2 pro=1528141726.779746542,0 | | | | | | |
| | 2018-06-04 19:48:57.582385+00:00 | 1m47s794ms328µs488ns | *roachpb.NotLeaseHolderError | [n3] | | /cockroach.roachpb.Internal/Batch | 457 | | |
| | 2018-06-04 19:48:57.582678+00:00 | 1m47s794ms621µs643ns | application error: [NotLeaseHolderError] r88: replica (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141726.779738033,0 epo=2 pro=1528141726.779746542,0 | | | | | | |
| | 2018-06-04 19:48:57.582694+00:00 | 1m47s794ms636µs876ns | error: {(err: [NotLeaseHolderError] r88: replica (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | | | not lease holder; current lease is repl=(n2,s2):3 seq=6 | | | | | | |
| | | | start=1528141726.779738033,0 epo=2 pro=1528141726.779746542,0) <nil>}; | | | | | | |
| | | | trying next peer (n2,s2):3 | | | | | | |
| | 2018-06-04 19:48:57.58271+00:00 | 1m47s794ms653µs508ns | sending request to alex-laptop.local:26259 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 309 | | |
| | 2018-06-04 19:48:57.582732+00:00 | 1m47s794ms674µs974ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 311 | | |
| | 2018-06-04 19:48:57.583076+00:00 | 1m47s795ms18µs976ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583119+00:00 | 1m47s795ms61µs950ns | 1 LeaseInfo | [n2] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583127+00:00 | 1m47s795ms70µs498ns | executing 1 requests | [n2,s2] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583136+00:00 | 1m47s795ms79µs711ns | read-only path | [n2,s2,r88/3:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583147+00:00 | 1m47s795ms89µs933ns | command queue | [n2,s2,r88/3:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583153+00:00 | 1m47s795ms96µs700ns | waiting for read lock | [n2,s2,r88/3:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583168+00:00 | 1m47s795ms111µs481ns | read completed | [n2,s2,r88/3:/Table/52/1/6{400-500}] | | /cockroach.roachpb.Internal/Batch | 458 | | |
| | 2018-06-04 19:48:57.583645+00:00 | 1m47s795ms588µs597ns | === SPAN START: dist sender === | | | dist sender | 312 | | |
| | 2018-06-04 19:48:57.583658+00:00 | 1m47s795ms601µs459ns | querying next range at /Table/52/1/6500 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 312 | | |
| | 2018-06-04 19:48:57.58368+00:00 | 1m47s795ms623µs662ns | r89: sending batch 1 LeaseInfo to (n3,s3):2 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 312 | | |
| | 2018-06-04 19:48:57.583698+00:00 | 1m47s795ms641µs108ns | sending request to alex-laptop.local:26261 | [client=[::1]:57501,user=root,txn=9352587e,n1] | | dist sender | 312 | | |
| | 2018-06-04 19:48:57.583713+00:00 | 1m47s795ms655µs918ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 313 | | |
| | 2018-06-04 19:48:57.583886+00:00 | 1m47s795ms829µs159ns | === SPAN START: /cockroach.roachpb.Internal/Batch === | | | /cockroach.roachpb.Internal/Batch | 459 | | |
| | 2018-06-04 19:48:57.583908+00:00 | 1m47s795ms851µs120ns | 1 LeaseInfo | [n3] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment