Skip to content

Instantly share code, notes, and snippets.

@trbngr

trbngr/10.75.138.85-cluster-node.log Secret

Created February 17, 2015 22:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save trbngr/d3e3f6ba996410012b31 to your computer and use it in GitHub Desktop.
Save trbngr/d3e3f6ba996410012b31 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
10.75.138.85-cluster-node.log
ES VERSION: 3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)
OS: Unknown (Unix 3.16.0.29)
RUNTIME: 3.8.0 ((detached/45d0ba1) (64-bit)
GC: 2 GENERATIONS
LOGS: /var/log/eventstore
HELP: False (<DEFAULT>)
VERSION: False (<DEFAULT>)
LOG: /var/log/eventstore (Config File)
CONFIG: /var/lib/eventstore/eventstore.config (Command Line)
DEFINES: <empty> (<DEFAULT>)
WHAT IF: False (<DEFAULT>)
INT IP: 127.0.0.1 (<DEFAULT>)
EXT IP: 10.75.138.85 (Config File)
INT HTTP PORT: 2112 (<DEFAULT>)
EXT HTTP PORT: 2113 (<DEFAULT>)
INT TCP PORT: 1112 (<DEFAULT>)
INT SECURE TCP PORT: 0 (<DEFAULT>)
EXT TCP PORT: 1113 (<DEFAULT>)
EXT SECURE TCP PORT: 0 (<DEFAULT>)
INT TCP HEARTBEAT TIMEOUT: 700 (<DEFAULT>)
EXT TCP HEARTBEAT TIMEOUT: 1000 (<DEFAULT>)
INT TCP HEARTBEAT INTERVAL: 700 (<DEFAULT>)
EXT TCP HEARTBEAT INTERVAL: 2000 (<DEFAULT>)
FORCE: False (<DEFAULT>)
CLUSTER SIZE: 3 (Config File)
NODE PRIORITY: 0 (<DEFAULT>)
MIN FLUSH DELAY MS: 2 (<DEFAULT>)
COMMIT COUNT: -1 (<DEFAULT>)
PREPARE COUNT: -1 (<DEFAULT>)
ADMIN ON EXT: True (<DEFAULT>)
STATS ON EXT: True (<DEFAULT>)
GOSSIP ON EXT: True (<DEFAULT>)
DISABLE SCAVENGE MERGING: False (<DEFAULT>)
DISCOVER VIA DNS: false (Config File)
CLUSTER DNS: fake.dns (<DEFAULT>)
CLUSTER GOSSIP PORT: 2113 (Config File)
GOSSIP SEED: [ 65.52.20.2:2113, 65.52.20.15:2113 ] (Config File)
STATS PERIOD SEC: 30 (<DEFAULT>)
CACHED CHUNKS: -1 (<DEFAULT>)
CHUNKS CACHE SIZE: 536871424 (<DEFAULT>)
MAX MEM TABLE SIZE: 1000000 (<DEFAULT>)
DB: /var/lib/eventstore/db (Config File)
MEM DB: False (<DEFAULT>)
SKIP DB VERIFY: False (<DEFAULT>)
RUN PROJECTIONS: All (Config File)
PROJECTION THREADS: 3 (<DEFAULT>)
WORKER THREADS: 5 (<DEFAULT>)
HTTP PREFIXES: [ http://*:2113/ ] (Config File)
ENABLE TRUSTED AUTH: False (<DEFAULT>)
CERTIFICATE STORE LOCATION: <empty> (<DEFAULT>)
CERTIFICATE STORE NAME: <empty> (<DEFAULT>)
CERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>)
CERTIFICATE THUMBPRINT: <empty> (<DEFAULT>)
CERTIFICATE FILE: <empty> (<DEFAULT>)
CERTIFICATE PASSWORD: <empty> (<DEFAULT>)
USE INTERNAL SSL: False (<DEFAULT>)
SSL TARGET HOST: n/a (<DEFAULT>)
SSL VALIDATE SERVER: True (<DEFAULT>)
AUTHENTICATION TYPE: internal (<DEFAULT>)
PREPARE TIMEOUT MS: 2000 (<DEFAULT>)
COMMIT TIMEOUT MS: 2000 (<DEFAULT>)
UNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>)
GOSSIP INTERVAL MS: 1000 (<DEFAULT>)
GOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>)
GOSSIP TIMEOUT MS: 500 (<DEFAULT>)
[PID:08168:001 2015.02.17 22:39:50.163 INFO ProgramBase`1 ] Quorum size set to 2
[PID:08168:001 2015.02.17 22:39:50.177 INFO ProgramBase`1 ] Can't find plugins path: plugins
[PID:08168:001 2015.02.17 22:39:50.214 INFO ProgramBase`1 ]
INSTANCE ID: 34cdf6e1-3141-4eeb-a26b-233bf1ef328b
DATABASE: /var/lib/eventstore/db
WRITER CHECKPOINT: 0 (0x0)
CHASER CHECKPOINT: 0 (0x0)
EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[PID:08168:001 2015.02.17 22:39:50.416 TRACE MessageHierarchy ] MessageHierarchy initialization took 00:00:00.1539668.
[PID:08168:001 2015.02.17 22:39:50.514 TRACE TFChunk ] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0033083.
[PID:08168:001 2015.02.17 22:39:50.674 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/projections
[PID:08168:001 2015.02.17 22:39:50.675 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Prelude
[PID:08168:001 2015.02.17 22:39:50.675 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/resources ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/web-resources/js
[PID:08168:001 2015.02.17 22:39:50.676 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.676 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.677 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.678 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.678 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.678 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.692 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08168:001 2015.02.17 22:39:50.692 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.694 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08168:001 2015.02.17 22:39:50.694 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.694 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08168:001 2015.02.17 22:39:50.694 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08168:001 2015.02.17 22:39:50.694 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08168:001 2015.02.17 22:39:50.694 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08168:010 2015.02.17 22:39:50.727 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM INIT...
[PID:08168:010 2015.02.17 22:39:50.756 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 10.75.138.85:1113.
[PID:08168:010 2015.02.17 22:39:50.760 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 127.0.0.1:1112.
[PID:08168:015 2015.02.17 22:39:50.779 INFO IndexCommitter ] TableIndex initialization...
[PID:08168:015 2015.02.17 22:39:50.786 INFO IndexCommitter ] ReadIndex building...
[PID:08168:015 2015.02.17 22:39:50.787 DEBUG IndexCommitter ] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0009460.
[PID:08168:010 2015.02.17 22:39:50.807 INFO HttpAsyncServer ] Starting HTTP server on [http://*:2113/]...
[PID:08168:010 2015.02.17 22:39:50.811 INFO HttpAsyncServer ] HTTP server is up and listening on [http://*:2113/]
[PID:08168:010 2015.02.17 22:39:50.834 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: SystemInit - 51ms. Handler: HttpService.
[PID:08168:010 2015.02.17 22:39:50.834 INFO HttpAsyncServer ] Starting HTTP server on [http://127.0.0.1:2112/]...
[PID:08168:010 2015.02.17 22:39:50.834 INFO HttpAsyncServer ] HTTP server is up and listening on [http://127.0.0.1:2112/]
[PID:08168:010 2015.02.17 22:39:50.860 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: SystemInit - 142ms. Q: 0/8.
[PID:08168:010 2015.02.17 22:39:50.860 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageReader' initialized.
[PID:08168:010 2015.02.17 22:39:50.863 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageWriter' initialized.
[PID:08168:010 2015.02.17 22:39:50.864 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageChaser' initialized.
[PID:08168:010 2015.02.17 22:39:50.897 TRACE GossipServiceBase ] CLUSTER HAS CHANGED
Old:
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:50.835
New:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:39:50.866
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:39:50.866
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:50.835
--------------------------------------------------------------------------------
[PID:08168:010 2015.02.17 22:39:50.900 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM START...
[PID:08168:010 2015.02.17 22:39:50.905 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] IS UNKNOWN!!! WHOA!!!
[PID:08168:010 2015.02.17 22:39:50.955 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS.
[PID:08168:010 2015.02.17 22:39:50.956 DEBUG ElectionsService ] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:50.958 DEBUG ElectionsService ] ELECTIONS: (V=0) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:51.590 TRACE GossipServiceBase ] Looks like node [65.52.20.2:2113] is DEAD (Gossip send failed).
[PID:08168:010 2015.02.17 22:39:51.592 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.20.2:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:39:50.866
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:39:50.866
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:51.547
New:
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:39:51.591
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:39:50.866
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:51.547
--------------------------------------------------------------------------------
[PID:08168:010 2015.02.17 22:39:51.679 TRACE GossipServiceBase ] Looks like node [65.52.20.15:2113] is DEAD (Gossip send failed).
[PID:08168:010 2015.02.17 22:39:51.679 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.20.15:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:39:51.591
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:39:50.866
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:51.655
New:
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:39:51.591
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:39:51.680
VND {34cdf6e1-3141-4eeb-a26b-233bf1ef328b} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.138.85:1113, n/a, 127.0.0.1:2112, 10.75.138.85:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:39:51.655
--------------------------------------------------------------------------------
[PID:08168:010 2015.02.17 22:39:51.968 DEBUG ElectionsService ] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:51.968 DEBUG ElectionsService ] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:51.968 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:52.975 DEBUG ElectionsService ] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:52.975 DEBUG ElectionsService ] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:52.975 DEBUG ElectionsService ] ELECTIONS: (V=2) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:53.983 DEBUG ElectionsService ] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:53.983 DEBUG ElectionsService ] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:53.983 DEBUG ElectionsService ] ELECTIONS: (V=3) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:54.987 DEBUG ElectionsService ] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:54.987 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:54.991 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:021 2015.02.17 22:39:55.396 INFO TcpService ] External TCP connection accepted: [Normal, 10.75.138.130:55896, L10.75.138.85:1113, {04e5fb4e-aa4e-4514-a161-1a04ee7a3026}].
[PID:08168:010 2015.02.17 22:39:55.995 DEBUG ElectionsService ] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:55.995 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:55.995 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:57.003 DEBUG ElectionsService ] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:57.003 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:57.003 DEBUG ElectionsService ] ELECTIONS: (V=6) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:39:58.011 DEBUG ElectionsService ] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:58.011 DEBUG ElectionsService ] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:58.011 DEBUG ElectionsService ] ELECTIONS: (V=7) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:008 2015.02.17 22:39:58.433 TRACE TcpConnectionManager] Closing connection 'external-normal' [10.75.138.130:55896, L10.75.138.85:1113, {04e5fb4e-aa4e-4514-a161-1a04ee7a3026}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0
[PID:08168:008 2015.02.17 22:39:58.458 INFO TcpConnection ] ES TcpConnection closed [22:39:58.435: N10.75.138.130:55896, L10.75.138.85:1113, {04e5fb4e-aa4e-4514-a161-1a04ee7a3026}]:
Received bytes: 0, Sent bytes: 22
Send calls: 1, callbacks: 1
Receive calls: 1, callbacks: 0
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0
[PID:08168:008 2015.02.17 22:39:58.461 INFO TcpConnectionManager] Connection 'external-normal' [10.75.138.130:55896, {04e5fb4e-aa4e-4514-a161-1a04ee7a3026}] closed: Success.
[PID:08168:010 2015.02.17 22:39:59.019 DEBUG ElectionsService ] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:39:59.019 DEBUG ElectionsService ] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:39:59.019 DEBUG ElectionsService ] ELECTIONS: (V=8) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:00.031 DEBUG ElectionsService ] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:00.031 DEBUG ElectionsService ] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:00.031 DEBUG ElectionsService ] ELECTIONS: (V=9) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:01.039 DEBUG ElectionsService ] ELECTIONS: (V=9) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:01.039 DEBUG ElectionsService ] ELECTIONS: (V=10) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:01.039 DEBUG ElectionsService ] ELECTIONS: (V=10) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:02.047 DEBUG ElectionsService ] ELECTIONS: (V=10) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:02.047 DEBUG ElectionsService ] ELECTIONS: (V=11) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:02.048 DEBUG ElectionsService ] ELECTIONS: (V=11) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:03.051 DEBUG ElectionsService ] ELECTIONS: (V=11) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:03.051 DEBUG ElectionsService ] ELECTIONS: (V=12) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:03.051 DEBUG ElectionsService ] ELECTIONS: (V=12) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:04.055 DEBUG ElectionsService ] ELECTIONS: (V=12) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:04.055 DEBUG ElectionsService ] ELECTIONS: (V=13) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:04.055 DEBUG ElectionsService ] ELECTIONS: (V=13) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:05.059 DEBUG ElectionsService ] ELECTIONS: (V=13) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:05.059 DEBUG ElectionsService ] ELECTIONS: (V=14) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:05.059 DEBUG ElectionsService ] ELECTIONS: (V=14) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:06.063 DEBUG ElectionsService ] ELECTIONS: (V=14) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:06.063 DEBUG ElectionsService ] ELECTIONS: (V=15) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:06.063 DEBUG ElectionsService ] ELECTIONS: (V=15) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:07.072 DEBUG ElectionsService ] ELECTIONS: (V=15) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:07.072 DEBUG ElectionsService ] ELECTIONS: (V=16) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:07.072 DEBUG ElectionsService ] ELECTIONS: (V=16) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:08.079 DEBUG ElectionsService ] ELECTIONS: (V=16) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:08.079 DEBUG ElectionsService ] ELECTIONS: (V=17) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:08.079 DEBUG ElectionsService ] ELECTIONS: (V=17) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:09.087 DEBUG ElectionsService ] ELECTIONS: (V=17) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:09.087 DEBUG ElectionsService ] ELECTIONS: (V=18) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:09.088 DEBUG ElectionsService ] ELECTIONS: (V=18) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:10.096 DEBUG ElectionsService ] ELECTIONS: (V=18) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:10.096 DEBUG ElectionsService ] ELECTIONS: (V=19) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:10.096 DEBUG ElectionsService ] ELECTIONS: (V=19) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:019 2015.02.17 22:40:10.360 INFO TcpService ] External TCP connection accepted: [Normal, 10.75.138.130:55906, L10.75.138.85:1113, {ed6514e1-c199-4aec-8e65-bd28de1b72cf}].
[PID:08168:010 2015.02.17 22:40:11.099 DEBUG ElectionsService ] ELECTIONS: (V=19) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:11.099 DEBUG ElectionsService ] ELECTIONS: (V=20) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:11.099 DEBUG ElectionsService ] ELECTIONS: (V=20) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:12.107 DEBUG ElectionsService ] ELECTIONS: (V=20) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:12.107 DEBUG ElectionsService ] ELECTIONS: (V=21) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:12.108 DEBUG ElectionsService ] ELECTIONS: (V=21) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:010 2015.02.17 22:40:13.115 DEBUG ElectionsService ] ELECTIONS: (V=21) TIMED OUT! (S=ElectingLeader, M=).
[PID:08168:010 2015.02.17 22:40:13.115 DEBUG ElectionsService ] ELECTIONS: (V=22) SHIFT TO LEADER ELECTION.
[PID:08168:010 2015.02.17 22:40:13.115 DEBUG ElectionsService ] ELECTIONS: (V=22) VIEWCHANGE FROM [127.0.0.1:2112, {34cdf6e1-3141-4eeb-a26b-233bf1ef328b}].
[PID:08168:008 2015.02.17 22:40:13.371 TRACE TcpConnectionManager] Closing connection 'external-normal' [10.75.138.130:55906, L10.75.138.85:1113, {ed6514e1-c199-4aec-8e65-bd28de1b72cf}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0
[PID:08168:008 2015.02.17 22:40:13.371 INFO TcpConnection ] ES TcpConnection closed [22:40:13.372: N10.75.138.130:55906, L10.75.138.85:1113, {ed6514e1-c199-4aec-8e65-bd28de1b72cf}]:
Received bytes: 0, Sent bytes: 22
Send calls: 1, callbacks: 1
Receive calls: 1, callbacks: 0
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0
[PID:08168:008 2015.02.17 22:40:13.371 INFO TcpConnectionManager] Connection 'external-normal' [10.75.138.130:55906, {ed6514e1-c199-4aec-8e65-bd28de1b72cf}] closed: Success.
Raw
10.75.166.146-cluster-node.log
ES VERSION: 3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)
OS: Unknown (Unix 3.16.0.29)
RUNTIME: 3.8.0 ((detached/45d0ba1) (64-bit)
GC: 2 GENERATIONS
LOGS: /var/log/eventstore
HELP: False (<DEFAULT>)
VERSION: False (<DEFAULT>)
LOG: /var/log/eventstore (Config File)
CONFIG: /var/lib/eventstore/eventstore.config (Command Line)
DEFINES: <empty> (<DEFAULT>)
WHAT IF: False (<DEFAULT>)
INT IP: 127.0.0.1 (<DEFAULT>)
EXT IP: 10.75.166.146 (Config File)
INT HTTP PORT: 2112 (<DEFAULT>)
EXT HTTP PORT: 2113 (<DEFAULT>)
INT TCP PORT: 1112 (<DEFAULT>)
INT SECURE TCP PORT: 0 (<DEFAULT>)
EXT TCP PORT: 1113 (<DEFAULT>)
EXT SECURE TCP PORT: 0 (<DEFAULT>)
INT TCP HEARTBEAT TIMEOUT: 700 (<DEFAULT>)
EXT TCP HEARTBEAT TIMEOUT: 1000 (<DEFAULT>)
INT TCP HEARTBEAT INTERVAL: 700 (<DEFAULT>)
EXT TCP HEARTBEAT INTERVAL: 2000 (<DEFAULT>)
FORCE: False (<DEFAULT>)
CLUSTER SIZE: 3 (Config File)
NODE PRIORITY: 0 (<DEFAULT>)
MIN FLUSH DELAY MS: 2 (<DEFAULT>)
COMMIT COUNT: -1 (<DEFAULT>)
PREPARE COUNT: -1 (<DEFAULT>)
ADMIN ON EXT: True (<DEFAULT>)
STATS ON EXT: True (<DEFAULT>)
GOSSIP ON EXT: True (<DEFAULT>)
DISABLE SCAVENGE MERGING: False (<DEFAULT>)
DISCOVER VIA DNS: false (Config File)
CLUSTER DNS: fake.dns (<DEFAULT>)
CLUSTER GOSSIP PORT: 2113 (Config File)
GOSSIP SEED: [ 65.52.20.15:2113, 65.52.19.21:2113 ] (Config File)
STATS PERIOD SEC: 30 (<DEFAULT>)
CACHED CHUNKS: -1 (<DEFAULT>)
CHUNKS CACHE SIZE: 536871424 (<DEFAULT>)
MAX MEM TABLE SIZE: 1000000 (<DEFAULT>)
DB: /var/lib/eventstore/db (Config File)
MEM DB: False (<DEFAULT>)
SKIP DB VERIFY: False (<DEFAULT>)
RUN PROJECTIONS: All (Config File)
PROJECTION THREADS: 3 (<DEFAULT>)
WORKER THREADS: 5 (<DEFAULT>)
HTTP PREFIXES: [ http://*:2113/ ] (Config File)
ENABLE TRUSTED AUTH: False (<DEFAULT>)
CERTIFICATE STORE LOCATION: <empty> (<DEFAULT>)
CERTIFICATE STORE NAME: <empty> (<DEFAULT>)
CERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>)
CERTIFICATE THUMBPRINT: <empty> (<DEFAULT>)
CERTIFICATE FILE: <empty> (<DEFAULT>)
CERTIFICATE PASSWORD: <empty> (<DEFAULT>)
USE INTERNAL SSL: False (<DEFAULT>)
SSL TARGET HOST: n/a (<DEFAULT>)
SSL VALIDATE SERVER: True (<DEFAULT>)
AUTHENTICATION TYPE: internal (<DEFAULT>)
PREPARE TIMEOUT MS: 2000 (<DEFAULT>)
COMMIT TIMEOUT MS: 2000 (<DEFAULT>)
UNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>)
GOSSIP INTERVAL MS: 1000 (<DEFAULT>)
GOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>)
GOSSIP TIMEOUT MS: 500 (<DEFAULT>)
[PID:08421:001 2015.02.17 22:40:13.873 INFO ProgramBase`1 ] Quorum size set to 2
[PID:08421:001 2015.02.17 22:40:13.878 INFO ProgramBase`1 ] Can't find plugins path: plugins
[PID:08421:001 2015.02.17 22:40:13.922 INFO ProgramBase`1 ]
INSTANCE ID: 80181742-da63-47b1-9106-38d5d6823f8e
DATABASE: /var/lib/eventstore/db
WRITER CHECKPOINT: 0 (0x0)
CHASER CHECKPOINT: 0 (0x0)
EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[PID:08421:001 2015.02.17 22:40:14.125 TRACE MessageHierarchy ] MessageHierarchy initialization took 00:00:00.1566824.
[PID:08421:001 2015.02.17 22:40:14.225 TRACE TFChunk ] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0023842.
[PID:08421:001 2015.02.17 22:40:14.386 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/projections
[PID:08421:001 2015.02.17 22:40:14.387 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Prelude
[PID:08421:001 2015.02.17 22:40:14.387 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/resources ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/web-resources/js
[PID:08421:001 2015.02.17 22:40:14.388 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.388 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.389 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.390 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.390 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.390 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.404 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08421:001 2015.02.17 22:40:14.404 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.405 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08421:001 2015.02.17 22:40:14.405 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.406 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08421:001 2015.02.17 22:40:14.406 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08421:001 2015.02.17 22:40:14.406 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08421:001 2015.02.17 22:40:14.406 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08421:010 2015.02.17 22:40:14.440 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM INIT...
[PID:08421:010 2015.02.17 22:40:14.486 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 10.75.166.146:1113.
[PID:08421:015 2015.02.17 22:40:14.509 INFO IndexCommitter ] TableIndex initialization...
[PID:08421:015 2015.02.17 22:40:14.516 INFO IndexCommitter ] ReadIndex building...
[PID:08421:015 2015.02.17 22:40:14.518 DEBUG IndexCommitter ] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0009610.
[PID:08421:010 2015.02.17 22:40:14.536 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: SystemInit - 49ms. Handler: TcpService.
[PID:08421:010 2015.02.17 22:40:14.537 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 127.0.0.1:1112.
[PID:08421:010 2015.02.17 22:40:14.561 INFO HttpAsyncServer ] Starting HTTP server on [http://*:2113/]...
[PID:08421:010 2015.02.17 22:40:14.565 INFO HttpAsyncServer ] HTTP server is up and listening on [http://*:2113/]
[PID:08421:010 2015.02.17 22:40:14.565 INFO HttpAsyncServer ] Starting HTTP server on [http://127.0.0.1:2112/]...
[PID:08421:010 2015.02.17 22:40:14.565 INFO HttpAsyncServer ] HTTP server is up and listening on [http://127.0.0.1:2112/]
[PID:08421:010 2015.02.17 22:40:14.590 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: SystemInit - 160ms. Q: 0/8.
[PID:08421:010 2015.02.17 22:40:14.591 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageReader' initialized.
[PID:08421:010 2015.02.17 22:40:14.593 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageWriter' initialized.
[PID:08421:010 2015.02.17 22:40:14.594 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageChaser' initialized.
[PID:08421:010 2015.02.17 22:40:14.627 TRACE GossipServiceBase ] CLUSTER HAS CHANGED
Old:
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:14.566
New:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:40:14.595
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:14.595
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:14.566
--------------------------------------------------------------------------------
[PID:08421:010 2015.02.17 22:40:14.630 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM START...
[PID:08421:010 2015.02.17 22:40:14.635 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] IS UNKNOWN!!! WHOA!!!
[PID:08421:010 2015.02.17 22:40:14.678 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS.
[PID:08421:010 2015.02.17 22:40:14.678 DEBUG ElectionsService ] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:14.680 DEBUG ElectionsService ] ELECTIONS: (V=0) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:15.322 TRACE GossipServiceBase ] Looks like node [65.52.19.21:2113] is DEAD (Gossip send failed).
[PID:08421:010 2015.02.17 22:40:15.323 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.19.21:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:40:14.595
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:14.595
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:15.260
New:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:40:14.595
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:15.323
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:15.260
--------------------------------------------------------------------------------
[PID:08421:010 2015.02.17 22:40:15.376 TRACE GossipServiceBase ] Looks like node [65.52.20.15:2113] is DEAD (Gossip send failed).
[PID:08421:010 2015.02.17 22:40:15.376 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.20.15:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:40:14.595
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:15.323
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:15.368
New:
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.15:2113, 65.52.20.15:2113] | 2015-02-17 22:40:15.376
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:15.323
VND {80181742-da63-47b1-9106-38d5d6823f8e} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.166.146:1113, n/a, 127.0.0.1:2112, 10.75.166.146:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:15.368
--------------------------------------------------------------------------------
[PID:08421:010 2015.02.17 22:40:15.685 DEBUG ElectionsService ] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:15.685 DEBUG ElectionsService ] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:15.685 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:16.688 DEBUG ElectionsService ] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:16.688 DEBUG ElectionsService ] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:16.688 DEBUG ElectionsService ] ELECTIONS: (V=2) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:17.692 DEBUG ElectionsService ] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:17.692 DEBUG ElectionsService ] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:17.692 DEBUG ElectionsService ] ELECTIONS: (V=3) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:18.700 DEBUG ElectionsService ] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:18.704 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:18.704 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:19.708 DEBUG ElectionsService ] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:19.708 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:19.708 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:20.712 DEBUG ElectionsService ] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:20.712 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:20.712 DEBUG ElectionsService ] ELECTIONS: (V=6) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:21.716 DEBUG ElectionsService ] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:21.716 DEBUG ElectionsService ] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:21.716 DEBUG ElectionsService ] ELECTIONS: (V=7) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:22.720 DEBUG ElectionsService ] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:22.720 DEBUG ElectionsService ] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:22.721 DEBUG ElectionsService ] ELECTIONS: (V=8) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:23.728 DEBUG ElectionsService ] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:23.728 DEBUG ElectionsService ] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:23.728 DEBUG ElectionsService ] ELECTIONS: (V=9) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:24.732 DEBUG ElectionsService ] ELECTIONS: (V=9) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:24.733 DEBUG ElectionsService ] ELECTIONS: (V=10) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:24.733 DEBUG ElectionsService ] ELECTIONS: (V=10) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:023 2015.02.17 22:40:25.245 INFO TcpService ] External TCP connection accepted: [Normal, 10.75.166.110:52605, L10.75.166.146:1113, {50f27a1c-1af5-4728-850a-a5d4c0e5cd72}].
[PID:08421:010 2015.02.17 22:40:25.736 DEBUG ElectionsService ] ELECTIONS: (V=10) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:25.736 DEBUG ElectionsService ] ELECTIONS: (V=11) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:25.736 DEBUG ElectionsService ] ELECTIONS: (V=11) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:26.740 DEBUG ElectionsService ] ELECTIONS: (V=11) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:26.740 DEBUG ElectionsService ] ELECTIONS: (V=12) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:26.741 DEBUG ElectionsService ] ELECTIONS: (V=12) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:010 2015.02.17 22:40:27.744 DEBUG ElectionsService ] ELECTIONS: (V=12) TIMED OUT! (S=ElectingLeader, M=).
[PID:08421:010 2015.02.17 22:40:27.744 DEBUG ElectionsService ] ELECTIONS: (V=13) SHIFT TO LEADER ELECTION.
[PID:08421:010 2015.02.17 22:40:27.744 DEBUG ElectionsService ] ELECTIONS: (V=13) VIEWCHANGE FROM [127.0.0.1:2112, {80181742-da63-47b1-9106-38d5d6823f8e}].
[PID:08421:008 2015.02.17 22:40:28.320 TRACE TcpConnectionManager] Closing connection 'external-normal' [10.75.166.110:52605, L10.75.166.146:1113, {50f27a1c-1af5-4728-850a-a5d4c0e5cd72}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0
[PID:08421:008 2015.02.17 22:40:28.349 INFO TcpConnection ] ES TcpConnection closed [22:40:28.323: N10.75.166.110:52605, L10.75.166.146:1113, {50f27a1c-1af5-4728-850a-a5d4c0e5cd72}]:
Received bytes: 0, Sent bytes: 22
Send calls: 1, callbacks: 1
Receive calls: 1, callbacks: 0
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0
[PID:08421:008 2015.02.17 22:40:28.353 INFO TcpConnectionManager] Connection 'external-normal' [10.75.166.110:52605, {50f27a1c-1af5-4728-850a-a5d4c0e5cd72}] closed: Success.
Raw
10.75.176.127-cluster-node.log
ES VERSION: 3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)
OS: Unknown (Unix 3.16.0.29)
RUNTIME: 3.8.0 ((detached/45d0ba1) (64-bit)
GC: 2 GENERATIONS
LOGS: /var/log/eventstore
HELP: False (<DEFAULT>)
VERSION: False (<DEFAULT>)
LOG: /var/log/eventstore (Config File)
CONFIG: /var/lib/eventstore/eventstore.config (Command Line)
DEFINES: <empty> (<DEFAULT>)
WHAT IF: False (<DEFAULT>)
INT IP: 127.0.0.1 (<DEFAULT>)
EXT IP: 10.75.176.127 (Config File)
INT HTTP PORT: 2112 (<DEFAULT>)
EXT HTTP PORT: 2113 (<DEFAULT>)
INT TCP PORT: 1112 (<DEFAULT>)
INT SECURE TCP PORT: 0 (<DEFAULT>)
EXT TCP PORT: 1113 (<DEFAULT>)
EXT SECURE TCP PORT: 0 (<DEFAULT>)
INT TCP HEARTBEAT TIMEOUT: 700 (<DEFAULT>)
EXT TCP HEARTBEAT TIMEOUT: 1000 (<DEFAULT>)
INT TCP HEARTBEAT INTERVAL: 700 (<DEFAULT>)
EXT TCP HEARTBEAT INTERVAL: 2000 (<DEFAULT>)
FORCE: False (<DEFAULT>)
CLUSTER SIZE: 3 (Config File)
NODE PRIORITY: 0 (<DEFAULT>)
MIN FLUSH DELAY MS: 2 (<DEFAULT>)
COMMIT COUNT: -1 (<DEFAULT>)
PREPARE COUNT: -1 (<DEFAULT>)
ADMIN ON EXT: True (<DEFAULT>)
STATS ON EXT: True (<DEFAULT>)
GOSSIP ON EXT: True (<DEFAULT>)
DISABLE SCAVENGE MERGING: False (<DEFAULT>)
DISCOVER VIA DNS: false (Config File)
CLUSTER DNS: fake.dns (<DEFAULT>)
CLUSTER GOSSIP PORT: 2113 (Config File)
GOSSIP SEED: [ 65.52.19.21:2113, 65.52.20.2:2113 ] (Config File)
STATS PERIOD SEC: 30 (<DEFAULT>)
CACHED CHUNKS: -1 (<DEFAULT>)
CHUNKS CACHE SIZE: 536871424 (<DEFAULT>)
MAX MEM TABLE SIZE: 1000000 (<DEFAULT>)
DB: /var/lib/eventstore/db (Config File)
MEM DB: False (<DEFAULT>)
SKIP DB VERIFY: False (<DEFAULT>)
RUN PROJECTIONS: All (Config File)
PROJECTION THREADS: 3 (<DEFAULT>)
WORKER THREADS: 5 (<DEFAULT>)
HTTP PREFIXES: [ http://*:2113/ ] (Config File)
ENABLE TRUSTED AUTH: False (<DEFAULT>)
CERTIFICATE STORE LOCATION: <empty> (<DEFAULT>)
CERTIFICATE STORE NAME: <empty> (<DEFAULT>)
CERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>)
CERTIFICATE THUMBPRINT: <empty> (<DEFAULT>)
CERTIFICATE FILE: <empty> (<DEFAULT>)
CERTIFICATE PASSWORD: <empty> (<DEFAULT>)
USE INTERNAL SSL: False (<DEFAULT>)
SSL TARGET HOST: n/a (<DEFAULT>)
SSL VALIDATE SERVER: True (<DEFAULT>)
AUTHENTICATION TYPE: internal (<DEFAULT>)
PREPARE TIMEOUT MS: 2000 (<DEFAULT>)
COMMIT TIMEOUT MS: 2000 (<DEFAULT>)
UNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>)
GOSSIP INTERVAL MS: 1000 (<DEFAULT>)
GOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>)
GOSSIP TIMEOUT MS: 500 (<DEFAULT>)
[PID:08730:001 2015.02.17 22:40:19.002 INFO ProgramBase`1 ] Quorum size set to 2
[PID:08730:001 2015.02.17 22:40:19.011 INFO ProgramBase`1 ] Can't find plugins path: plugins
[PID:08730:001 2015.02.17 22:40:19.049 INFO ProgramBase`1 ]
INSTANCE ID: 115667ea-d2e2-4412-80ad-f66b6b3df508
DATABASE: /var/lib/eventstore/db
WRITER CHECKPOINT: 0 (0x0)
CHASER CHECKPOINT: 0 (0x0)
EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF)
[PID:08730:001 2015.02.17 22:40:19.249 TRACE MessageHierarchy ] MessageHierarchy initialization took 00:00:00.1547682.
[PID:08730:001 2015.02.17 22:40:19.349 TRACE TFChunk ] CACHED TFChunk #0-0 (chunk-000000.000000) in 00:00:00.0024189.
[PID:08730:001 2015.02.17 22:40:19.508 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/projections
[PID:08730:001 2015.02.17 22:40:19.508 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Prelude
[PID:08730:001 2015.02.17 22:40:19.508 INFO MiniWeb ] Starting MiniWeb for /web/es/js/projections/resources ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/web-resources/js
[PID:08730:001 2015.02.17 22:40:19.510 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.510 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.510 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.511 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.511 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.511 TRACE MiniWeb ] Binding MiniWeb to /web/es/js/projections/resources/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.525 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08730:001 2015.02.17 22:40:19.526 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.527 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08730:001 2015.02.17 22:40:19.527 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.527 INFO MiniWeb ] Starting MiniWeb for /web ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/clusternode-web
[PID:08730:001 2015.02.17 22:40:19.527 TRACE MiniWeb ] Binding MiniWeb to /web/{*remaining_path}
[PID:08730:001 2015.02.17 22:40:19.527 INFO MiniWeb ] Starting MiniWeb for /web/users ==> /var/lib/eventstore/EventStore-OSS-Linux-v3.0.1/Users/web
[PID:08730:001 2015.02.17 22:40:19.527 TRACE MiniWeb ] Binding MiniWeb to /web/users/{*remaining_path}
[PID:08730:010 2015.02.17 22:40:19.556 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM INIT...
[PID:08730:010 2015.02.17 22:40:19.588 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 10.75.176.127:1113.
[PID:08730:010 2015.02.17 22:40:19.593 INFO TcpServerListener ] Starting Normal TCP listening on TCP endpoint: 127.0.0.1:1112.
[PID:08730:015 2015.02.17 22:40:19.610 INFO IndexCommitter ] TableIndex initialization...
[PID:08730:015 2015.02.17 22:40:19.618 INFO IndexCommitter ] ReadIndex building...
[PID:08730:015 2015.02.17 22:40:19.619 DEBUG IndexCommitter ] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0009660.
[PID:08730:010 2015.02.17 22:40:19.638 INFO HttpAsyncServer ] Starting HTTP server on [http://*:2113/]...
[PID:08730:010 2015.02.17 22:40:19.642 INFO HttpAsyncServer ] HTTP server is up and listening on [http://*:2113/]
[PID:08730:010 2015.02.17 22:40:19.666 TRACE InMemoryBus ] SLOW BUS MSG [MainBus]: SystemInit - 49ms. Handler: HttpService.
[PID:08730:010 2015.02.17 22:40:19.666 INFO HttpAsyncServer ] Starting HTTP server on [http://127.0.0.1:2112/]...
[PID:08730:010 2015.02.17 22:40:19.666 INFO HttpAsyncServer ] HTTP server is up and listening on [http://127.0.0.1:2112/]
[PID:08730:010 2015.02.17 22:40:19.691 TRACE QueuedHandlerAutoRes] SLOW QUEUE MSG [MainQueue]: SystemInit - 140ms. Q: 0/8.
[PID:08730:010 2015.02.17 22:40:19.693 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageWriter' initialized.
[PID:08730:010 2015.02.17 22:40:19.693 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageReader' initialized.
[PID:08730:010 2015.02.17 22:40:19.695 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] Service 'StorageChaser' initialized.
[PID:08730:010 2015.02.17 22:40:19.728 TRACE GossipServiceBase ] CLUSTER HAS CHANGED
Old:
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:19.667
New:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:40:19.697
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:19.697
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:19.667
--------------------------------------------------------------------------------
[PID:08730:010 2015.02.17 22:40:19.731 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] SYSTEM START...
[PID:08730:010 2015.02.17 22:40:19.736 INFO ClusterVNodeControll] ========== [127.0.0.1:2112] IS UNKNOWN!!! WHOA!!!
[PID:08730:010 2015.02.17 22:40:19.784 DEBUG ElectionsService ] ELECTIONS: STARTING ELECTIONS.
[PID:08730:010 2015.02.17 22:40:19.784 DEBUG ElectionsService ] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:19.785 DEBUG ElectionsService ] ELECTIONS: (V=0) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:20.430 TRACE GossipServiceBase ] Looks like node [65.52.19.21:2113] is DEAD (Gossip send failed).
[PID:08730:010 2015.02.17 22:40:20.432 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.19.21:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:40:19.697
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:19.697
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:20.360
New:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:40:19.697
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:20.431
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:20.360
--------------------------------------------------------------------------------
[PID:08730:010 2015.02.17 22:40:20.789 DEBUG ElectionsService ] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:20.789 DEBUG ElectionsService ] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:20.789 DEBUG ElectionsService ] ELECTIONS: (V=1) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:20.936 TRACE GossipServiceBase ] Looks like node [65.52.20.2:2113] is DEAD (Gossip send failed).
[PID:08730:010 2015.02.17 22:40:20.936 TRACE GossipServiceBase ] CLUSTER HAS CHANGED (gossip send failed to [65.52.20.2:2113])
Old:
MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:40:19.697
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:20.431
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:20.880
New:
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.20.2:2113, 65.52.20.2:2113] | 2015-02-17 22:40:20.937
MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 65.52.19.21:2113, 65.52.19.21:2113] | 2015-02-17 22:40:20.431
VND {115667ea-d2e2-4412-80ad-f66b6b3df508} <LIVE> [Unknown, 127.0.0.1:1112, n/a, 10.75.176.127:1113, n/a, 127.0.0.1:2112, 10.75.176.127:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-02-17 22:40:20.880
--------------------------------------------------------------------------------
[PID:08730:010 2015.02.17 22:40:21.792 DEBUG ElectionsService ] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:21.792 DEBUG ElectionsService ] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:21.792 DEBUG ElectionsService ] ELECTIONS: (V=2) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:22.796 DEBUG ElectionsService ] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:22.796 DEBUG ElectionsService ] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:22.800 DEBUG ElectionsService ] ELECTIONS: (V=3) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:23.805 DEBUG ElectionsService ] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:23.805 DEBUG ElectionsService ] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:23.805 DEBUG ElectionsService ] ELECTIONS: (V=4) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:24.808 DEBUG ElectionsService ] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:24.808 DEBUG ElectionsService ] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:24.808 DEBUG ElectionsService ] ELECTIONS: (V=5) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:25.812 DEBUG ElectionsService ] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:25.812 DEBUG ElectionsService ] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:25.812 DEBUG ElectionsService ] ELECTIONS: (V=6) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:26.816 DEBUG ElectionsService ] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:26.816 DEBUG ElectionsService ] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:26.816 DEBUG ElectionsService ] ELECTIONS: (V=7) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:022 2015.02.17 22:40:27.727 INFO TcpService ] External TCP connection accepted: [Normal, 10.75.176.140:51420, L10.75.176.127:1113, {197469f3-9eda-459f-8339-ec4b95d97951}].
[PID:08730:010 2015.02.17 22:40:27.820 DEBUG ElectionsService ] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:27.820 DEBUG ElectionsService ] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:27.820 DEBUG ElectionsService ] ELECTIONS: (V=8) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:28.824 DEBUG ElectionsService ] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:28.824 DEBUG ElectionsService ] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:28.824 DEBUG ElectionsService ] ELECTIONS: (V=9) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:010 2015.02.17 22:40:29.828 DEBUG ElectionsService ] ELECTIONS: (V=9) TIMED OUT! (S=ElectingLeader, M=).
[PID:08730:010 2015.02.17 22:40:29.828 DEBUG ElectionsService ] ELECTIONS: (V=10) SHIFT TO LEADER ELECTION.
[PID:08730:010 2015.02.17 22:40:29.828 DEBUG ElectionsService ] ELECTIONS: (V=10) VIEWCHANGE FROM [127.0.0.1:2112, {115667ea-d2e2-4412-80ad-f66b6b3df508}].
[PID:08730:008 2015.02.17 22:40:30.774 TRACE TcpConnectionManager] Closing connection 'external-normal' [10.75.176.140:51420, L10.75.176.127:1113, {197469f3-9eda-459f-8339-ec4b95d97951}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 0
[PID:08730:008 2015.02.17 22:40:30.806 INFO TcpConnection ] ES TcpConnection closed [22:40:30.777: N10.75.176.140:51420, L10.75.176.127:1113, {197469f3-9eda-459f-8339-ec4b95d97951}]:
Received bytes: 0, Sent bytes: 22
Send calls: 1, callbacks: 1
Receive calls: 1, callbacks: 0
Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 0
[PID:08730:008 2015.02.17 22:40:30.811 INFO TcpConnectionManager] Connection 'external-normal' [10.75.176.140:51420, {197469f3-9eda-459f-8339-ec4b95d97951}] closed: Success.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment