Created
September 16, 2015 06:31
-
-
Save stuartrexking/e23d9bd96b1f457c4be9 to your computer and use it in GitHub Desktop.
Event Stored logs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| vagrant@n1:~$ eventstored --mem-db -log /var/log/eventstore.log --int-ip 172.20.20.10 --ext-ip 172.20.20.10 --cluster-size=3 --cluster-dns eventstore.service.consul | |
| [04273,01,06:05:28.629] | |
| ES VERSION: 3.2.1.0 (release-v3.2.1/7994e0fec6b2dc6740dfb54cbf3139e98c3fc592, Wed, 9 Sep 2015 19:11:22 +0200) | |
| OS: Linux (Unix 3.13.0.63) | |
| RUNTIME: 3.12.1 (tarball) (64-bit) | |
| GC: 2 GENERATIONS | |
| LOGS: /var/log/eventstore.log | |
| MODIFIED OPTIONS: | |
| MEM DB: true (Command Line) | |
| LOG: /var/log/eventstore.log (Command Line) | |
| INT IP: 172.20.20.10 (Command Line) | |
| EXT IP: 172.20.20.10 (Command Line) | |
| CLUSTER SIZE: 3 (Command Line) | |
| CLUSTER DNS: eventstore.service.consul (Command Line) | |
| CONFIG: /etc/eventstore/eventstore.conf (Config File) | |
| RUN PROJECTIONS: None (Config File) | |
| DEFAULT OPTIONS: | |
| HELP: False (<DEFAULT>) | |
| VERSION: False (<DEFAULT>) | |
| DEFINES: <empty> (<DEFAULT>) | |
| WHAT IF: False (<DEFAULT>) | |
| DEVELOPMENT MODE: False (<DEFAULT>) | |
| MONO MIN THREADPOOL SIZE: 10 (<DEFAULT>) | |
| INT HTTP PORT: 2112 (<DEFAULT>) | |
| EXT HTTP PORT: 2113 (<DEFAULT>) | |
| INT TCP PORT: 1112 (<DEFAULT>) | |
| INT SECURE TCP PORT: 0 (<DEFAULT>) | |
| EXT TCP PORT: 1113 (<DEFAULT>) | |
| EXT SECURE TCP PORT: 0 (<DEFAULT>) | |
| EXT IP ADVERTISE AS: <empty> (<DEFAULT>) | |
| EXT TCP PORT ADVERTISE AS: 0 (<DEFAULT>) | |
| EXT HTTP PORT ADVERTISE AS: 0 (<DEFAULT>) | |
| INT IP ADVERTISE AS: <empty> (<DEFAULT>) | |
| INT TCP PORT ADVERTISE AS: 0 (<DEFAULT>) | |
| INT HTTP PORT ADVERTISE AS: 0 (<DEFAULT>) | |
| INT TCP HEARTBEAT TIMEOUT: 700 (<DEFAULT>) | |
| EXT TCP HEARTBEAT TIMEOUT: 1000 (<DEFAULT>) | |
| INT TCP HEARTBEAT INTERVAL: 700 (<DEFAULT>) | |
| EXT TCP HEARTBEAT INTERVAL: 2000 (<DEFAULT>) | |
| FORCE: False (<DEFAULT>) | |
| NODE PRIORITY: 0 (<DEFAULT>) | |
| MIN FLUSH DELAY MS: 2 (<DEFAULT>) | |
| COMMIT COUNT: -1 (<DEFAULT>) | |
| PREPARE COUNT: -1 (<DEFAULT>) | |
| ADMIN ON EXT: True (<DEFAULT>) | |
| STATS ON EXT: True (<DEFAULT>) | |
| GOSSIP ON EXT: True (<DEFAULT>) | |
| DISABLE SCAVENGE MERGING: False (<DEFAULT>) | |
| DISCOVER VIA DNS: True (<DEFAULT>) | |
| CLUSTER GOSSIP PORT: 30777 (<DEFAULT>) | |
| GOSSIP SEED: <empty> (<DEFAULT>) | |
| STATS PERIOD SEC: 30 (<DEFAULT>) | |
| CACHED CHUNKS: -1 (<DEFAULT>) | |
| CHUNKS CACHE SIZE: 536871424 (<DEFAULT>) | |
| MAX MEM TABLE SIZE: 1000000 (<DEFAULT>) | |
| DB: /var/lib/eventstore (<DEFAULT>) | |
| SKIP DB VERIFY: False (<DEFAULT>) | |
| PROJECTION THREADS: 3 (<DEFAULT>) | |
| WORKER THREADS: 5 (<DEFAULT>) | |
| INT HTTP PREFIXES: <empty> (<DEFAULT>) | |
| EXT HTTP PREFIXES: <empty> (<DEFAULT>) | |
| ENABLE TRUSTED AUTH: False (<DEFAULT>) | |
| ADD INTERFACE PREFIXES: True (<DEFAULT>) | |
| CERTIFICATE STORE LOCATION: <empty> (<DEFAULT>) | |
| CERTIFICATE STORE NAME: <empty> (<DEFAULT>) | |
| CERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>) | |
| CERTIFICATE THUMBPRINT: <empty> (<DEFAULT>) | |
| CERTIFICATE FILE: <empty> (<DEFAULT>) | |
| CERTIFICATE PASSWORD: <empty> (<DEFAULT>) | |
| USE INTERNAL SSL: False (<DEFAULT>) | |
| SSL TARGET HOST: n/a (<DEFAULT>) | |
| SSL VALIDATE SERVER: True (<DEFAULT>) | |
| AUTHENTICATION TYPE: internal (<DEFAULT>) | |
| PREPARE TIMEOUT MS: 2000 (<DEFAULT>) | |
| COMMIT TIMEOUT MS: 2000 (<DEFAULT>) | |
| UNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>) | |
| INDEX CACHE DEPTH: 16 (<DEFAULT>) | |
| GOSSIP INTERVAL MS: 1000 (<DEFAULT>) | |
| GOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>) | |
| GOSSIP TIMEOUT MS: 500 (<DEFAULT>) | |
| ENABLE HISTOGRAMS: False (<DEFAULT>) | |
| [04273,01,06:05:28.647] Quorum size set to 2 | |
| [04273,01,06:05:28.652] Cannot find plugins path: plugins | |
| [04273,01,06:05:28.689] | |
| INSTANCE ID: 1570c98e-4a09-49f4-b806-49de67bae864 | |
| DATABASE: /var/lib/eventstore | |
| WRITER CHECKPOINT: 0 (0x0) | |
| CHASER CHECKPOINT: 0 (0x0) | |
| EPOCH CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) | |
| TRUNCATE CHECKPOINT: -1 (0xFFFFFFFFFFFFFFFF) | |
| [04273,01,06:05:28.787] MessageHierarchy initialization took 00:00:00.0745040. | |
| [04273,01,06:05:28.907] Starting MiniWeb for /web/es/js/projections ==> /usr/share/eventstore/projections | |
| [04273,01,06:05:28.907] Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /usr/share/eventstore/Prelude | |
| [04273,01,06:05:28.907] Binding MiniWeb to /web/es/js/projections/{*remaining_path} | |
| [04273,01,06:05:28.907] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} | |
| [04273,01,06:05:28.907] Binding MiniWeb to /web/es/js/projections/{*remaining_path} | |
| [04273,01,06:05:28.907] Binding MiniWeb to /web/es/js/projections/v8/Prelude/{*remaining_path} | |
| [04273,01,06:05:28.917] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web | |
| [04273,01,06:05:28.919] Binding MiniWeb to /web/{*remaining_path} | |
| [04273,01,06:05:28.919] Starting MiniWeb for /web ==> /usr/share/eventstore/clusternode-web | |
| [04273,01,06:05:28.919] Binding MiniWeb to /web/{*remaining_path} | |
| [04273,19,06:05:28.934] ========== [172.20.20.10:2112] SYSTEM INIT... | |
| [04273,19,06:05:28.960] Starting Normal TCP listening on TCP endpoint: 172.20.20.10:1113. | |
| [04273,19,06:05:28.970] Starting Normal TCP listening on TCP endpoint: 172.20.20.10:1112. | |
| [04273,23,06:05:28.977] TableIndex initialization... | |
| [04273,23,06:05:28.982] ReadIndex building... | |
| [04273,23,06:05:28.982] ReadIndex rebuilding done: total processed 0 records, time elapsed: 00:00:00.0004460. | |
| [04273,19,06:05:28.992] Starting HTTP server on [http://172.20.20.10:2113/]... | |
| [04273,19,06:05:28.996] HTTP server is up and listening on [http://172.20.20.10:2113/] | |
| [04273,19,06:05:28.996] Starting HTTP server on [http://172.20.20.10:2112/]... | |
| [04273,19,06:05:28.996] HTTP server is up and listening on [http://172.20.20.10:2112/] | |
| [04273,19,06:05:29.013] SLOW QUEUE MSG [MainQueue]: SystemInit - 80ms. Q: 0/7. | |
| [04273,19,06:05:29.013] ========== [172.20.20.10:2112] Service 'StorageWriter' initialized. | |
| [04273,19,06:05:29.013] ========== [172.20.20.10:2112] Service 'StorageReader' initialized. | |
| [04273,19,06:05:29.016] ========== [172.20.20.10:2112] Service 'StorageChaser' initialized. | |
| [04273,19,06:05:29.034] CLUSTER HAS CHANGED | |
| Old: | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:28.998 | |
| New: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.016 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:28.998 | |
| -------------------------------------------------------------------------------- | |
| [04273,19,06:05:29.036] ========== [172.20.20.10:2112] SYSTEM START... | |
| [04273,19,06:05:29.039] ========== [172.20.20.10:2112] IS UNKNOWN... | |
| [04273,12,06:05:29.077] Subscriptions received state change to Unknown stopping listening. | |
| [04273,19,06:05:29.100] ELECTIONS: STARTING ELECTIONS. | |
| [04273,19,06:05:29.100] ELECTIONS: (V=0) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:29.100] ELECTIONS: (V=0) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:29.185] Looks like node [172.20.20.10:30777] is DEAD (Gossip send failed). | |
| [04273,19,06:05:29.186] CLUSTER HAS CHANGED (gossip send failed to [172.20.20.10:30777]) | |
| Old: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.016 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.149 | |
| New: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.185 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.149 | |
| -------------------------------------------------------------------------------- | |
| [04273,19,06:05:29.252] Looks like node [172.20.20.11:30777] is DEAD (Gossip send failed). | |
| [04273,19,06:05:29.252] CLUSTER HAS CHANGED (gossip send failed to [172.20.20.11:30777]) | |
| Old: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.185 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.251 | |
| New: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.252 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.185 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.251 | |
| -------------------------------------------------------------------------------- | |
| [04273,19,06:05:29.959] Looks like node [172.20.20.12:30777] is DEAD (Gossip send failed). | |
| [04273,19,06:05:29.959] CLUSTER HAS CHANGED (gossip send failed to [172.20.20.12:30777]) | |
| Old: | |
| MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.016 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.252 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.185 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.957 | |
| New: | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.12:30777, 172.20.20.12:30777] | 2015-09-16 06:05:29.959 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.11:30777, 172.20.20.11:30777] | 2015-09-16 06:05:29.252 | |
| MAN {00000000-0000-0000-0000-000000000000} <DEAD> [Manager, 172.20.20.10:30777, 172.20.20.10:30777] | 2015-09-16 06:05:29.185 | |
| VND {1570c98e-4a09-49f4-b806-49de67bae864} <LIVE> [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.957 | |
| -------------------------------------------------------------------------------- | |
| [04273,19,06:05:30.107] ELECTIONS: (V=0) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:30.107] ELECTIONS: (V=1) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:30.107] ELECTIONS: (V=1) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:31.109] ELECTIONS: (V=1) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:31.109] ELECTIONS: (V=2) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:31.109] ELECTIONS: (V=2) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:32.114] ELECTIONS: (V=2) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:32.114] ELECTIONS: (V=3) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:32.114] ELECTIONS: (V=3) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:33.116] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:33.116] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:33.116] ELECTIONS: (V=4) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:34.117] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:34.117] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:34.117] ELECTIONS: (V=5) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:35.118] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:35.118] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:35.118] ELECTIONS: (V=6) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:36.119] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:36.119] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:36.119] ELECTIONS: (V=7) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:37.120] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:37.120] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:37.120] ELECTIONS: (V=8) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| [04273,19,06:05:38.121] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=). | |
| [04273,19,06:05:38.121] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION. | |
| [04273,19,06:05:38.121] ELECTIONS: (V=9) VIEWCHANGE FROM [172.20.20.10:2112, {1570c98e-4a09-49f4-b806-49de67bae864}]. | |
| vagrant@n1:~$ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It looks like the nodes are unable to talk to each other. And they have never been able to talk to each other. As an example this node only knows about itself:
VND {1570c98e-4a09-49f4-b806-49de67bae864} [Unknown, 172.20.20.10:1112, n/a, 172.20.20.10:1113, n/a, 172.20.20.10:2112, 172.20.20.10:2113] -1/0/0/E-1@-1:{00000000-0000-0000-0000-000000000000} | 2015-09-16 06:05:29.149 None of the other nodes show up there as nodes.
Can you curl from one node to another on 2113/2112?