Skip to content

Instantly share code, notes, and snippets.

@menzow

menzow/bistro.tests.output.log

Created Sep 17, 2017
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Raw
bistro.tests.output.log
UpdateCTestConfiguration from :/home/bistro/bistro/cmake/Debug/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/bistro/bistro/cmake/Debug/DartConfiguration.tcl
Test project /home/bistro/bistro/cmake/Debug
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: test_sqlite
1: Test command: /home/bistro/bistro/cmake/Debug/sqlite/test_sqlite
1: Test timeout computed to be: 9.99988e+06
1: Running main() from gtest_main.cc
1: [==========] Running 5 tests from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 5 tests from TestSQLite
1: [ RUN ] TestSQLite.TestCreateInsertSelect
1: [ OK ] TestSQLite.TestCreateInsertSelect (0 ms)
1: [ RUN ] TestSQLite.TestBind
1: [ OK ] TestSQLite.TestBind (1 ms)
1: [ RUN ] TestSQLite.TestInvalidParameterCount
1: [ OK ] TestSQLite.TestInvalidParameterCount (0 ms)
1: [ RUN ] TestSQLite.TestTooManyColumns
1: [ OK ] TestSQLite.TestTooManyColumns (0 ms)
1: [ RUN ] TestSQLite.TestInvalidType
1: [ OK ] TestSQLite.TestInvalidType (0 ms)
1: [----------] 5 tests from TestSQLite (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 5 tests from 1 test case ran. (1 ms total)
1: [ PASSED ] 5 tests.
1/55 Test #1: test_sqlite ........................... Passed 0.01 sec
test 2
Start 2: test_remote_worker_state
2: Test command: /home/bistro/bistro/cmake/Debug/remote/test/test_remote_worker_state
2: Test timeout computed to be: 9.99988e+06
2: Running main() from gtest_main.cc
2: [==========] Running 1 test from 1 test case.
2: [----------] Global test environment set-up.
2: [----------] 1 test from TestRemoteWorkerState
2: [ RUN ] TestRemoteWorkerState.HandleAll
2: [ OK ] TestRemoteWorkerState.HandleAll (0 ms)
2: [----------] 1 test from TestRemoteWorkerState (0 ms total)
2:
2: [----------] Global test environment tear-down
2: [==========] 1 test from 1 test case ran. (0 ms total)
2: [ PASSED ] 1 test.
2/55 Test #2: test_remote_worker_state .............. Passed 0.00 sec
test 3
Start 3: test_remote_worker
3: Test command: /home/bistro/bistro/cmake/Debug/remote/test/test_remote_worker
3: Test timeout computed to be: 9.99988e+06
3: Running main() from gtest_main.cc
3: [==========] Running 6 tests from 1 test case.
3: [----------] Global test environment set-up.
3: [----------] 6 tests from TestRemoteWorker
3: [ RUN ] TestRemoteWorker.HandleNormal
3: stderr: WARNING: Logging before InitGoogleLogging() is written to STDERR
3: I0917 06:40:15.813313 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.813408 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.813421 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.813426 114 RemoteWorker.cpp:203] Worker became healthy
3: stderr: W0917 06:40:15.813606 114 RemoteWorker.cpp:189] Worker became unhealthy
3: stderr: I0917 06:40:15.813694 114 RemoteWorker.cpp:203] Worker became healthy
3: [ OK ] TestRemoteWorker.HandleNormal (1 ms)
3: [ RUN ] TestRemoteWorker.DoNotDieDueToLackOfConsensus
3: stderr: I0917 06:40:15.814081 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.814172 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: I0917 06:40:15.814239 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: stderr: I0917 06:40:15.814499 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.814539 114 RemoteWorker.cpp:281] Good healthcheck from sent -1
3: I0917 06:40:15.814546 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.814554 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: [ OK ] TestRemoteWorker.DoNotDieDueToLackOfConsensus (1 ms)
3: [ RUN ] TestRemoteWorker.HandleMustDieAndLostTasks
3: stderr: I0917 06:40:15.814808 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 0,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: I0917 06:40:15.814883 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 0,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: [ OK ] TestRemoteWorker.HandleMustDieAndLostTasks (1 ms)
3: [ RUN ] TestRemoteWorker.WorkerReplacement
3: stderr: I0917 06:40:15.815259 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.815289 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.815296 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.815302 114 RemoteWorker.cpp:203] Worker became healthy
3: I0917 06:40:15.815378 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.815399 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.815405 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.815410 114 RemoteWorker.cpp:203] Worker became healthy
3: stderr: W0917 06:40:15.815526 114 RemoteWorker.cpp:137] Ignored dead worker heartbeat: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: stderr: E0917 06:40:15.815589 114 RemoteWorker.cpp:143] Heartbeat error: same start time 2718281829 but different rands: 0, 1
3: I0917 06:40:15.815661 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: stderr: I0917 06:40:15.815713 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.815724 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.815732 114 RemoteWorker.cpp:203] Worker became healthy
3: I0917 06:40:15.815799 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: The old worker is still okay
3: I0917 06:40:15.815871 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: The old worker is still okay
3: stderr: W0917 06:40:15.815922 114 RemoteWorker.cpp:189] Worker became unhealthy
3: stderr: I0917 06:40:15.816017 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: The old worker is still okay
3: I0917 06:40:15.816090 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: The old worker is still okay
3: I0917 06:40:15.816157 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current unhealthy worker was replaced
3: I0917 06:40:15.816224 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.816287 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current unhealthy worker was replaced
3: I0917 06:40:15.816362 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current unhealthy worker was replaced
3: I0917 06:40:15.816427 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.816489 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 1,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current unhealthy worker was replaced
3: I0917 06:40:15.816557 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: I0917 06:40:15.816619 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker just became lost
3: I0917 06:40:15.816694 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker was already lost
3: I0917 06:40:15.816757 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current worker was already lost
3: I0917 06:40:15.816824 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current lost worker was replaced
3: I0917 06:40:15.816889 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 3,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.816951 114 RemoteWorkerUpdate.h:73] Telling BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 2,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281829,
3: 2: rand (i64) = 1,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: } to commit suicide: Current lost worker was replaced
3: [ OK ] TestRemoteWorker.WorkerReplacement (2 ms)
3: [ RUN ] TestRemoteWorker.UnsureIfRunning
3: stderr: I0917 06:40:15.817342 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.817370 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.817379 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.817386 114 RemoteWorker.cpp:203] Worker became healthy
3: I0917 06:40:15.817399 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.817420 114 RemoteWorker.cpp:281] Good healthcheck from sent 1
3: I0917 06:40:15.817430 114 RemoteWorker.cpp:281] Good healthcheck from sent 2
3: I0917 06:40:15.817446 114 RemoteWorker.cpp:281] Good healthcheck from sent 5
3: I0917 06:40:15.817456 114 RemoteWorker.cpp:281] Good healthcheck from sent 6
3: I0917 06:40:15.817478 114 RemoteWorker.cpp:281] Good healthcheck from sent 13
3: I0917 06:40:15.817488 114 RemoteWorker.cpp:281] Good healthcheck from sent 14
3: I0917 06:40:15.817504 114 RemoteWorker.cpp:281] Good healthcheck from sent 29
3: I0917 06:40:15.817517 114 RemoteWorker.cpp:281] Good healthcheck from sent 30
3: I0917 06:40:15.817533 114 RemoteWorker.cpp:281] Good healthcheck from sent 61
3: I0917 06:40:15.817546 114 RemoteWorker.cpp:281] Good healthcheck from sent 62
3: I0917 06:40:15.817562 114 RemoteWorker.cpp:281] Good healthcheck from sent 125
3: I0917 06:40:15.817574 114 RemoteWorker.cpp:281] Good healthcheck from sent 126
3: I0917 06:40:15.817590 114 RemoteWorker.cpp:281] Good healthcheck from sent 253
3: I0917 06:40:15.817605 114 RemoteWorker.cpp:281] Good healthcheck from sent 254
3: I0917 06:40:15.817621 114 RemoteWorker.cpp:281] Good healthcheck from sent 509
3: I0917 06:40:15.817633 114 RemoteWorker.cpp:281] Good healthcheck from sent 510
3: I0917 06:40:15.817652 114 RemoteWorker.cpp:281] Good healthcheck from sent 765
3: I0917 06:40:15.817665 114 RemoteWorker.cpp:281] Good healthcheck from sent 766
3: I0917 06:40:15.817688 114 RemoteWorker.cpp:281] Good healthcheck from sent 767
3: I0917 06:40:15.817708 114 RemoteWorker.cpp:281] Good healthcheck from sent 768
3: I0917 06:40:15.817718 114 RemoteWorker.cpp:281] Good healthcheck from sent 769
3: I0917 06:40:15.817734 114 RemoteWorker.cpp:281] Good healthcheck from sent 772
3: I0917 06:40:15.817744 114 RemoteWorker.cpp:281] Good healthcheck from sent 773
3: I0917 06:40:15.817759 114 RemoteWorker.cpp:281] Good healthcheck from sent 780
3: I0917 06:40:15.817770 114 RemoteWorker.cpp:281] Good healthcheck from sent 781
3: I0917 06:40:15.817785 114 RemoteWorker.cpp:281] Good healthcheck from sent 796
3: I0917 06:40:15.817795 114 RemoteWorker.cpp:281] Good healthcheck from sent 797
3: I0917 06:40:15.817811 114 RemoteWorker.cpp:281] Good healthcheck from sent 828
3: I0917 06:40:15.817823 114 RemoteWorker.cpp:281] Good healthcheck from sent 829
3: I0917 06:40:15.817842 114 RemoteWorker.cpp:281] Good healthcheck from sent 892
3: I0917 06:40:15.817857 114 RemoteWorker.cpp:281] Good healthcheck from sent 893
3: I0917 06:40:15.817873 114 RemoteWorker.cpp:281] Good healthcheck from sent 1020
3: I0917 06:40:15.817884 114 RemoteWorker.cpp:281] Good healthcheck from sent 1021
3: I0917 06:40:15.817901 114 RemoteWorker.cpp:281] Good healthcheck from sent 1276
3: I0917 06:40:15.817914 114 RemoteWorker.cpp:281] Good healthcheck from sent 1277
3: I0917 06:40:15.817929 114 RemoteWorker.cpp:281] Good healthcheck from sent 1532
3: I0917 06:40:15.817940 114 RemoteWorker.cpp:281] Good healthcheck from sent 1533
3: I0917 06:40:15.817989 114 RemoteWorker.cpp:281] Good healthcheck from sent 1534
3: I0917 06:40:15.818014 114 RemoteWorker.cpp:281] Good healthcheck from sent 1535
3: I0917 06:40:15.818024 114 RemoteWorker.cpp:281] Good healthcheck from sent 1536
3: I0917 06:40:15.818040 114 RemoteWorker.cpp:281] Good healthcheck from sent 1539
3: I0917 06:40:15.818050 114 RemoteWorker.cpp:281] Good healthcheck from sent 1540
3: I0917 06:40:15.818068 114 RemoteWorker.cpp:281] Good healthcheck from sent 1547
3: I0917 06:40:15.818089 114 RemoteWorker.cpp:281] Good healthcheck from sent 1548
3: I0917 06:40:15.818104 114 RemoteWorker.cpp:281] Good healthcheck from sent 1563
3: I0917 06:40:15.818114 114 RemoteWorker.cpp:281] Good healthcheck from sent 1564
3: I0917 06:40:15.818130 114 RemoteWorker.cpp:281] Good healthcheck from sent 1595
3: I0917 06:40:15.818143 114 RemoteWorker.cpp:281] Good healthcheck from sent 1596
3: I0917 06:40:15.818158 114 RemoteWorker.cpp:281] Good healthcheck from sent 1659
3: I0917 06:40:15.818171 114 RemoteWorker.cpp:281] Good healthcheck from sent 1660
3: I0917 06:40:15.818187 114 RemoteWorker.cpp:281] Good healthcheck from sent 1787
3: I0917 06:40:15.818202 114 RemoteWorker.cpp:281] Good healthcheck from sent 1788
3: I0917 06:40:15.818220 114 RemoteWorker.cpp:281] Good healthcheck from sent 2043
3: I0917 06:40:15.818234 114 RemoteWorker.cpp:281] Good healthcheck from sent 2044
3: I0917 06:40:15.818250 114 RemoteWorker.cpp:281] Good healthcheck from sent 2299
3: I0917 06:40:15.818261 114 RemoteWorker.cpp:281] Good healthcheck from sent 2300
3: [ OK ] TestRemoteWorker.UnsureIfRunning (1 ms)
3: [ RUN ] TestRemoteWorker.RecordStatuses
3: stderr: I0917 06:40:15.818809 114 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
3: 1: shard (string) = "",
3: 2: machineLock (struct) = MachinePortLock {
3: 1: hostname (string) = "",
3: 2: port (i32) = 0,
3: },
3: 3: addr (struct) = ServiceAddress {
3: 1: ip_or_host (string) = "",
3: 2: port (i32) = 0,
3: },
3: 4: id (struct) = BistroInstanceID {
3: 1: startTime (i64) = 2718281828,
3: 2: rand (i64) = 0,
3: },
3: 5: heartbeatPeriodSec (i32) = 0,
3: 6: protocolVersion (i16) = 2,
3: 7: usableResources (struct) = UsablePhysicalResources {
3: 1: msSinceEpoch (i64) = 0,
3: 2: cpuCores (double) = 0,
3: 3: memoryMB (double) = 0,
3: 4: gpus (list) = list<struct>[0] {
3: },
3: },
3: }
3: I0917 06:40:15.818836 114 RemoteWorker.cpp:281] Good healthcheck from sent 0
3: I0917 06:40:15.818843 114 RemoteWorker.cpp:173] Worker can be healthy but lacks WorkerSetID consensus
3: I0917 06:40:15.818850 114 RemoteWorker.cpp:203] Worker became healthy
3: E0917 06:40:15.818895 114 RemoteWorker.cpp:285] Unsuccessful healthcheck status {"time":1505630415,"result_bits":16} from sent 0
3: stderr: W0917 06:40:15.819386 114 RemoteWorker.cpp:329] Ignoring severely delayed updateStatus message with incorrect invocation ID, new task RunningTask {
3: 1: job (string) = "foobar",
3: 2: node (string) = "",
3: 3: nodeResources (list) = list<struct>[0] {
3: },
3: 4: workerShard (string) = "",
3: 5: invocationID (struct) = BistroInstanceID {
3: 1: startTime (i64) = 1,
3: 2: rand (i64) = 0,
3: },
3: 6: nextBackoffDuration (struct) = BackoffDuration {
3: 1: noMoreBackoffs (bool) = false,
3: 2: seconds (i32) = 0,
3: },
3: 8: workerSuicideTaskKillWaitMs (i32) = 0,
3: } vs new task RunningTask {
3: 1: job (string) = "foobar",
3: 2: node (string) = "",
3: 3: nodeResources (list) = list<struct>[0] {
3: },
3: 4: workerShard (string) = "",
3: 5: invocationID (struct) = BistroInstanceID {
3: 1: startTime (i64) = 0,
3: 2: rand (i64) = 0,
3: },
3: 6: nextBackoffDuration (struct) = BackoffDuration {
3: 1: noMoreBackoffs (bool) = false,
3: 2: seconds (i32) = 0,
3: },
3: 8: workerSuicideTaskKillWaitMs (i32) = 0,
3: } with status {"time":1505630415,"result_bits":4}
3: stderr: E0917 06:40:15.819502 114 RemoteWorker.cpp:352] RemoteWorker already knew that RunningTask {
3: 1: job (string) = "foobar",
3: 2: node (string) = "",
3: 3: nodeResources (list) = list<struct>[0] {
3: },
3: 4: workerShard (string) = "",
3: 5: invocationID (struct) = BistroInstanceID {
3: 1: startTime (i64) = 0,
3: 2: rand (i64) = 0,
3: },
3: 6: nextBackoffDuration (struct) = BackoffDuration {
3: 1: noMoreBackoffs (bool) = false,
3: 2: seconds (i32) = 0,
3: },
3: 8: workerSuicideTaskKillWaitMs (i32) = 0,
3: } was NOT running
3: stderr: I0917 06:40:15.819595 114 RemoteWorker.cpp:345] Ignoring overwriteable status for RunningTask {
3: 1: job (string) = "foobar",
3: 2: node (string) = "",
3: 3: nodeResources (list) = list<struct>[0] {
3: },
3: 4: workerShard (string) = "",
3: 5: invocationID (struct) = BistroInstanceID {
3: 1: startTime (i64) = 0,
3: 2: rand (i64) = 0,
3: },
3: 6: nextBackoffDuration (struct) = BackoffDuration {
3: 1: noMoreBackoffs (bool) = false,
3: 2: seconds (i32) = 0,
3: },
3: 8: workerSuicideTaskKillWaitMs (i32) = 0,
3: }: {"time":1505630415,"data":{"exception":"d'oh"},"result_bits":1120}
3: [ OK ] TestRemoteWorker.RecordStatuses (1 ms)
3: [----------] 6 tests from TestRemoteWorker (7 ms total)
3:
3: [----------] Global test environment tear-down
3: [==========] 6 tests from 1 test case ran. (7 ms total)
3: [ PASSED ] 6 tests.
3/55 Test #3: test_remote_worker .................... Passed 0.03 sec
test 4
Start 4: test_remote_workers
4: Test command: /home/bistro/bistro/cmake/Debug/remote/test/test_remote_workers
4: Test timeout computed to be: 9.99988e+06
4: Running main() from gtest_main.cc
4: [==========] Running 7 tests from 2 test cases.
4: [----------] Global test environment set-up.
4: [----------] 2 tests from TestRemoteWorkers
4: [ RUN ] TestRemoteWorkers.ProtocolMismatch
4: WARNING: Logging before InitGoogleLogging() is written to STDERR
4: W0917 06:40:15.842478 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843118 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -4715471109881888844,
4: 2: rand (i64) = 7878781317863822364,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: [ OK ] TestRemoteWorkers.ProtocolMismatch (1 ms)
4: [ RUN ] TestRemoteWorkers.WorkerPools
4: W0917 06:40:15.843191 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843264 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host1",
4: 2: port (i32) = 123,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -1256632111275438969,
4: 2: rand (i64) = -7372892837036078385,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.843298 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843363 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w2",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host1",
4: 2: port (i32) = 456,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 56176210434975651,
4: 2: rand (i64) = 7435001310217622509,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.843395 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843467 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host2",
4: 2: port (i32) = 123,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -2755307969578708347,
4: 2: rand (i64) = 239661822597965937,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.843518 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843585 115 RemoteWorkerUpdate.h:73] Telling BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host2",
4: 2: port (i32) = 789,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -7536639974719130799,
4: 2: rand (i64) = -7762534492942141496,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: } to commit suicide: The old worker is still okay
4: W0917 06:40:15.843631 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843693 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host1",
4: 2: port (i32) = 123,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -1256632111275438969,
4: 2: rand (i64) = -7372892837036078385,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.843742 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.843803 115 RemoteWorkerUpdate.h:73] Telling BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host1",
4: 2: port (i32) = 123,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -1256632111275438969,
4: 2: rand (i64) = -7372892837036078385,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: } to commit suicide: Current lost worker was replaced
4: I0917 06:40:15.843879 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "host2",
4: 2: port (i32) = 789,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 8631888041097148469,
4: 2: rand (i64) = -3734994617478273143,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: [ OK ] TestRemoteWorkers.WorkerPools (0 ms)
4: [----------] 2 tests from TestRemoteWorkers (1 ms total)
4:
4: [----------] 5 tests from TestRemoteWorkersInitialWait
4: [ RUN ] TestRemoteWorkersInitialWait.NoWorkers
4: [ OK ] TestRemoteWorkersInitialWait.NoWorkers (1 ms)
4: [ RUN ] TestRemoteWorkersInitialWait.OneWorkerWithRunningTasks
4: W0917 06:40:15.844139 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.844214 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8900224945384854272,
4: 2: rand (i64) = 4428558160287101735,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.844233 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.844240 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.844310 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8900224945384854272,
4: 2: rand (i64) = 4428558160287101735,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.844344 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.844348 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.844413 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8900224945384854272,
4: 2: rand (i64) = 4428558160287101735,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.844441 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.844445 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.844506 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8900224945384854272,
4: 2: rand (i64) = 4428558160287101735,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.844547 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.844552 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.844615 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8900224945384854272,
4: 2: rand (i64) = 4428558160287101735,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.844643 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.844647 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: [ OK ] TestRemoteWorkersInitialWait.OneWorkerWithRunningTasks (0 ms)
4: [ RUN ] TestRemoteWorkersInitialWait.AchieveAndMaintainWorkerSetConsensus
4: stderr: W0917 06:40:15.845007 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845115 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w2",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -3675597157560704077,
4: 2: rand (i64) = -6802114097076032037,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.845129 115 RemoteWorker.cpp:281] Good healthcheck from w2 sent 1
4: stderr: W0917 06:40:15.845212 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.845221 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845293 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w2",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -3675597157560704077,
4: 2: rand (i64) = -6802114097076032037,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: stderr: W0917 06:40:15.845405 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.845414 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845418 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.845469 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845554 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 256683427399804019,
4: 2: rand (i64) = -1162634313819206480,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.845566 115 RemoteWorker.cpp:281] Good healthcheck from w3 sent 1
4: stderr: W0917 06:40:15.845616 115 RemoteWorkers.cpp:368] Not pruning history until 2 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.845625 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845696 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 256683427399804019,
4: 2: rand (i64) = -1162634313819206480,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.845731 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845734 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.845816 115 RemoteWorkers.cpp:368] Not pruning history until 2 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.845823 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845827 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.845831 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845835 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.845882 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.845957 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -2601880369146780436,
4: 2: rand (i64) = -195729221048062709,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.845968 115 RemoteWorker.cpp:281] Good healthcheck from w1 sent 1
4: stderr: W0917 06:40:15.846011 115 RemoteWorkers.cpp:368] Not pruning history until 3 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.846019 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846087 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -2601880369146780436,
4: 2: rand (i64) = -195729221048062709,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.846094 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846098 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.846101 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846105 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.846189 115 RemoteWorkers.cpp:368] Not pruning history until 3 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.846195 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846201 115 RemoteWorker.cpp:173] Worker w1 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.846205 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846210 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.846213 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846217 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.846257 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846276 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.846283 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846356 115 RemoteWorkerUpdate.h:73] Telling BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 256683427399804019,
4: 2: rand (i64) = -1162634313819206480,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: } to commit suicide: Current worker just became lost
4: W0917 06:40:15.846385 115 RemoteWorkers.cpp:368] Not pruning history until 2 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.846390 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846395 115 RemoteWorker.cpp:173] Worker w1 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.846472 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = -133652101150545321,
4: 2: rand (i64) = -7017312132007331775,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -6020794099307680494,
4: 2: xorAll (i64) = 1483382449814381356,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8160477631943301226,
4: 2: xorAll (i64) = -5543976996368855968,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = -133652101150545321,
4: 2: rand (i64) = -7017312132007331775,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -6277477526707484513,
4: 2: xorAll (i64) = 1664612556902827871,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -6997843318124094746,
4: 2: xorAll (i64) = 6688593165712031952,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: }
4: W0917 06:40:15.846482 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846485 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: [ OK ] TestRemoteWorkersInitialWait.AchieveAndMaintainWorkerSetConsensus (2 ms)
4: [ RUN ] TestRemoteWorkersInitialWait.NoConsensusWithEmptyWorkerSetID
4: stderr: W0917 06:40:15.846851 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.846947 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 5889178550894790785,
4: 2: rand (i64) = 4386784837397645022,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.846961 115 RemoteWorker.cpp:281] Good healthcheck from w sent 1
4: stderr: W0917 06:40:15.847031 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.847039 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847112 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 5889178550894790785,
4: 2: rand (i64) = 4386784837397645022,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: stderr: W0917 06:40:15.847193 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.847199 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847204 115 RemoteWorker.cpp:173] Worker w can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.847239 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.847245 115 RemoteWorkers.cpp:73] Worker w is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847249 115 RemoteWorker.cpp:173] Worker w can be healthy but lacks WorkerSetID consensus
4: [ OK ] TestRemoteWorkersInitialWait.NoConsensusWithEmptyWorkerSetID (1 ms)
4: [ RUN ] TestRemoteWorkersInitialWait.HistoryAndWorkerSetIDPropagation
4: stderr: W0917 06:40:15.847529 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847640 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -3552866646201315003,
4: 2: rand (i64) = -2962300819907356689,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.847656 115 RemoteWorker.cpp:281] Good healthcheck from w1 sent 1
4: stderr: W0917 06:40:15.847723 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.847733 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847815 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -3552866646201315003,
4: 2: rand (i64) = -2962300819907356689,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: stderr: W0917 06:40:15.847914 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.847923 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.847928 115 RemoteWorker.cpp:173] Worker w1 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.848036 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: W0917 06:40:15.848050 115 RemoteWorkers.cpp:73] Worker w1 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.848059 115 RemoteWorker.cpp:173] Worker w1 can be healthy but lacks WorkerSetID consensus
4: I0917 06:40:15.848085 115 RemoteWorkers.cpp:137] Worker w1 has not been healthy, but WorkerSetID consensus allows it.
4: I0917 06:40:15.848093 115 RemoteWorker.cpp:203] Worker w1 became healthy
4: stderr: W0917 06:40:15.848263 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.848361 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w2",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -4620495272771086811,
4: 2: rand (i64) = 5191828878308021719,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.848373 115 RemoteWorker.cpp:281] Good healthcheck from w2 sent 1
4: stderr: W0917 06:40:15.848438 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.848455 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.848533 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w2",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -4620495272771086811,
4: 2: rand (i64) = 5191828878308021719,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: stderr: W0917 06:40:15.848706 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.848721 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.848726 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.848994 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: W0917 06:40:15.849007 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.849014 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.849037 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.849045 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.849184 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.849206 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.849212 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.849351 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: stderr: W0917 06:40:15.849506 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.849591 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8034718570719457587,
4: 2: rand (i64) = 3694327980500278814,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.849603 115 RemoteWorker.cpp:281] Good healthcheck from w3 sent 1
4: stderr: W0917 06:40:15.849653 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.849668 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.849743 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w3",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -8034718570719457587,
4: 2: rand (i64) = 3694327980500278814,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.849824 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.849831 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.850044 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.850061 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.850066 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850152 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.850167 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850495 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: W0917 06:40:15.850507 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: W0917 06:40:15.850512 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.850519 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850567 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.850572 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850652 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.850661 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850874 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: W0917 06:40:15.850886 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.850908 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.850911 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.850986 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.850989 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851212 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: W0917 06:40:15.851223 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.851302 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 2,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = -8173361918972401814,
4: 2: xorAll (i64) = 8165336858172966752,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 2229528058400665030,
4: 2: xorAll (i64) = -6994407146444504520,
4: },
4: 3: numWorkers (i32) = 2,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.851310 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851341 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.851344 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851347 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.851349 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851492 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.851497 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.851524 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.851532 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851536 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.851537 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.851680 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.851691 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 3 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: stderr: W0917 06:40:15.851856 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.851948 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w4",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 2729845074988230816,
4: 2: rand (i64) = 4259238104661866343,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: I0917 06:40:15.851960 115 RemoteWorker.cpp:281] Good healthcheck from w4 sent 1
4: stderr: W0917 06:40:15.852015 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.852032 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.852102 115 RemoteWorker.cpp:184] Initializing new worker: BistroWorker {
4: 1: shard (string) = "w4",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = 2729845074988230816,
4: 2: rand (i64) = 4259238104661866343,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: }
4: W0917 06:40:15.852180 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.852185 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.852258 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.852269 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: stderr: W0917 06:40:15.852478 115 RemoteWorkers.cpp:368] Not pruning history until 1 workers echo their first WorkerSetID from this scheduler
4: W0917 06:40:15.852496 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.852501 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.852579 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.852586 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.852659 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.852665 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853029 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.853111 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.853116 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has no WorkerSetID
4: W0917 06:40:15.853121 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has no WorkerSetID
4: I0917 06:40:15.853127 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853157 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.853159 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853232 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.853235 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853307 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.853312 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853530 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.853615 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.853619 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.853644 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.853648 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853724 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.853729 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.853798 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.853802 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854005 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.854086 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.854092 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.854167 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.854176 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854218 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.854223 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854300 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.854303 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854307 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.854310 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854450 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.854519 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.854524 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.854560 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.854563 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854636 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.854640 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854643 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.854645 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854789 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.854866 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.854877 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.854912 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.854915 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.854990 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: I0917 06:40:15.854996 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.855017 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: I0917 06:40:15.855020 115 RemoteWorker.cpp:173] Worker w2 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.855162 115 RemoteWorkers.cpp:73] Worker w2 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 2 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.855238 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: }
4: W0917 06:40:15.855248 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It first appeared in the following WorkerSetID: 4 but an earlier one is indirectly required by all non-MUST_DIE workers: 1
4: W0917 06:40:15.855329 115 RemoteWorker.cpp:189] Worker w1 became unhealthy
4: I0917 06:40:15.855469 115 RemoteWorkerUpdate.h:73] Telling BistroWorker {
4: 1: shard (string) = "w1",
4: 2: machineLock (struct) = MachinePortLock {
4: 1: hostname (string) = "",
4: 2: port (i32) = 0,
4: },
4: 3: addr (struct) = ServiceAddress {
4: 1: ip_or_host (string) = "",
4: 2: port (i32) = 0,
4: },
4: 4: id (struct) = BistroInstanceID {
4: 1: startTime (i64) = -3552866646201315003,
4: 2: rand (i64) = -2962300819907356689,
4: },
4: 5: heartbeatPeriodSec (i32) = 0,
4: 6: protocolVersion (i16) = 2,
4: 7: usableResources (struct) = UsablePhysicalResources {
4: 1: msSinceEpoch (i64) = 0,
4: 2: cpuCores (double) = 0,
4: 3: memoryMB (double) = 0,
4: 4: gpus (list) = list<struct>[0] {
4: },
4: },
4: } to commit suicide: Current worker just became lost
4: W0917 06:40:15.855607 115 RemoteWorkers.cpp:73] Worker w4 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 4,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 4968508659005923031,
4: 2: xorAll (i64) = -4265556209584545523,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -8263649930146741429,
4: 2: xorAll (i64) = -7588030979704083647,
4: },
4: 3: numWorkers (i32) = 4,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 5,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 8521375305207238034,
4: 2: xorAll (i64) = 755511418519171144,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -5301349110239384740,
4: 2: xorAll (i64) = 4634816579992222894,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.855617 115 RemoteWorker.cpp:173] Worker w4 can be healthy but lacks WorkerSetID consensus
4: W0917 06:40:15.855691 115 RemoteWorkers.cpp:73] Worker w3 is not yet able to achieve consensus: It has WorkerSetID WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 3,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 2238663584017692215,
4: 2: xorAll (i64) = -2220300335417292371,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = 5923856038900943844,
4: 2: xorAll (i64) = -5932863057775977434,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: } while the scheduler has WorkerSetID {
4: 1: schedulerID (struct) = BistroInstanceID {
4: 1: startTime (i64) = 168302408787067780,
4: 2: rand (i64) = 929900080386923079,
4: },
4: 2: version (i64) = 5,
4: 3: hash (struct) = WorkerSetHash {
4: 1: startTime (struct) = SetHash {
4: 1: addAll (i64) = 8521375305207238034,
4: 2: xorAll (i64) = 755511418519171144,
4: },
4: 2: rand (struct) = SetHash {
4: 1: addAll (i64) = -5301349110239384740,
4: 2: xorAll (i64) = 4634816579992222894,
4: },
4: 3: numWorkers (i32) = 3,
4: },
4: }
4: I0917 06:40:15.855695 115 RemoteWorker.cpp:173] Worker w3 can be healthy but lacks WorkerSetID consensus