Skip to content

Instantly share code, notes, and snippets.

@sega-yarkin

sega-yarkin/nats1.conf

Created March 2, 2018 23:37
Show Gist options
  • Save sega-yarkin/2abfd960af7a85004724f7277437cfa9 to your computer and use it in GitHub Desktop.
Save sega-yarkin/2abfd960af7a85004724f7277437cfa9 to your computer and use it in GitHub Desktop.
Download ZIP
Nats Streaming crash
Raw
nats1.conf
port: 4231
http_port: 8231
store: "file"
dir: "/data"
cluster {
host: '10.10.0.100'
port: 5231
bootstrap: true
raft_logging: true
log_path: "/data"
routes = [
nats-route://10.10.0.100:5232
nats-route://10.10.0.100:5233
]
}
Raw
nats1.console
$ docker run -it --name nats1 -v ${PWD}/srv1:/data --network=host nats-streaming -c /data/nats.conf
[1] 2018/03/02 22:30:55.316194 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.8.0-beta
[1] 2018/03/02 22:30:55.316244 [INF] STREAM: ServerID: sAt1uRelj8TWHvlfT0fTFT
[1] 2018/03/02 22:30:55.316250 [INF] STREAM: Go version: go1.9.3
[1] 2018/03/02 22:30:55.316475 [INF] Starting nats-server version 1.0.4
[1] 2018/03/02 22:30:55.316773 [INF] Starting http monitor on 0.0.0.0:8231
[1] 2018/03/02 22:30:55.316953 [INF] Listening for client connections on 0.0.0.0:4231
[1] 2018/03/02 22:30:55.316970 [INF] Server is ready
[1] 2018/03/02 22:30:55.318015 [INF] Listening for route connections on 10.10.0.100:5231
[1] 2018/03/02 22:30:55.344011 [INF] STREAM: Recovering the state...
[1] 2018/03/02 22:30:55.344209 [INF] STREAM: No recovered state
[1] 2018/03/02 22:30:55.345793 [INF] STREAM: raft: Initial configuration (index=0): []
[1] 2018/03/02 22:30:55.345925 [INF] STREAM: raft: Node at test-cluster.sAt1uRelj8TWHvlfT0fTHO.stan [Follower] entering Follower state (Leader: "")
[1] 2018/03/02 22:30:55.346644 [INF] STREAM: Message store is RAFT_FILE
[1] 2018/03/02 22:30:55.346697 [INF] STREAM: ---------- Store Limits ----------
[1] 2018/03/02 22:30:55.346706 [INF] STREAM: Channels: 100 *
[1] 2018/03/02 22:30:55.346713 [INF] STREAM: --------- Channels Limits --------
[1] 2018/03/02 22:30:55.346724 [INF] STREAM: Subscriptions: 1000 *
[1] 2018/03/02 22:30:55.346733 [INF] STREAM: Messages : 1000000 *
[1] 2018/03/02 22:30:55.346743 [INF] STREAM: Bytes : 976.56 MB *
[1] 2018/03/02 22:30:55.346753 [INF] STREAM: Age : unlimited *
[1] 2018/03/02 22:30:55.346760 [INF] STREAM: ----------------------------------
[1] 2018/03/02 22:30:56.718712 [INF] STREAM: raft: Heartbeat timeout from "" reached, starting election
[1] 2018/03/02 22:30:56.718757 [INF] STREAM: raft: Node at test-cluster.sAt1uRelj8TWHvlfT0fTHO.stan [Candidate] entering Candidate state in term 2
[1] 2018/03/02 22:30:56.719067 [INF] STREAM: raft: Election won. Tally: 1
[1] 2018/03/02 22:30:56.719085 [INF] STREAM: raft: Node at test-cluster.sAt1uRelj8TWHvlfT0fTHO.stan [Leader] entering Leader state
[1] 2018/03/02 22:30:56.719204 [INF] STREAM: server became leader, performing leader promotion actions
[1] 2018/03/02 22:30:56.719772 [INF] STREAM: finished leader promotion actions
[1] 2018/03/02 22:31:20.164979 [INF] STREAM: raft: Updating configuration with AddStaging (Jchw4RbZTYkbcD5dMoP1Yn, test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan) to [{Suffrage:Voter ID:sAt1uRelj8TWHvlfT0fTHO Address:test-cluster.sAt1uRelj8TWHvlfT0fTHO.stan} {Suffrage:Voter ID:Jchw4RbZTYkbcD5dMoP1Yn Address:test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}]
[1] 2018/03/02 22:31:20.165155 [INF] STREAM: raft: Added peer Jchw4RbZTYkbcD5dMoP1Yn, starting replication
[1] 2018/03/02 22:31:20.166985 [INF] STREAM: raft: AppendEntries to {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan} rejected, sending older logs (next: 1)
[1] 2018/03/02 22:31:20.169720 [INF] STREAM: raft: pipelining replication to peer {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}
[1] 2018/03/02 22:31:40.080920 [INF] STREAM: raft: Updating configuration with AddStaging (T0qMxJDUPbProHXcH5X69q, test-cluster.T0qMxJDUPbProHXcH5X69q.stan) to [{Suffrage:Voter ID:sAt1uRelj8TWHvlfT0fTHO Address:test-cluster.sAt1uRelj8TWHvlfT0fTHO.stan} {Suffrage:Voter ID:Jchw4RbZTYkbcD5dMoP1Yn Address:test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan} {Suffrage:Voter ID:T0qMxJDUPbProHXcH5X69q Address:test-cluster.T0qMxJDUPbProHXcH5X69q.stan}]
[1] 2018/03/02 22:31:40.081123 [INF] STREAM: raft: Added peer T0qMxJDUPbProHXcH5X69q, starting replication
[1] 2018/03/02 22:31:40.085092 [INF] STREAM: raft: AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan} rejected, sending older logs (next: 1)
[1] 2018/03/02 22:31:40.086227 [INF] STREAM: raft: pipelining replication to peer {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}
[1] 2018/03/02 22:35:06.991415 [INF] STREAM: WARNING: Reached limits for store "test" (msgs=963326/1000000 bytes=976.56 MB/976.56 MB), dropping old messages to make room for new ones
[1] 2018/03/02 22:36:25.526722 [INF] STREAM: raft: aborting pipeline replication to peer {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}
[1] 2018/03/02 22:36:26.416340 [INF] STREAM: raft: pipelining replication to peer {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}
[1] 2018/03/02 22:36:31.639673 [INF] STREAM: raft: aborting pipeline replication to peer {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}
[1] 2018/03/02 22:36:32.564515 [INF] STREAM: raft: pipelining replication to peer {Voter Jchw4RbZTYkbcD5dMoP1Yn test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan}
[1] 2018/03/02 22:36:42.942305 [INF] STREAM: raft: Starting snapshot up to 101165
[1] 2018/03/02 22:36:42.942344 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-101165-1520030202942.tmp
[1] 2018/03/02 22:36:56.575205 [INF] STREAM: raft: Compacting logs from 1 to 96561
[1] 2018/03/02 22:36:57.742712 [INF] STREAM: raft: Snapshot to 101165 complete
[1] 2018/03/02 22:39:08.043522 [INF] STREAM: raft: Starting snapshot up to 115738
[1] 2018/03/02 22:39:08.043567 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-115738-1520030348043.tmp
[1] 2018/03/02 22:39:08.059343 [INF] STREAM: raft: Compacting logs from 96562 to 105503
[1] 2018/03/02 22:39:08.167858 [INF] STREAM: raft: Snapshot to 115738 complete
[1] 2018/03/02 22:40:23.602459 [INF] STREAM: raft: Failed to contact T0qMxJDUPbProHXcH5X69q in 528.397413ms
[1] 2018/03/02 22:40:24.104654 [INF] STREAM: raft: Failed to contact T0qMxJDUPbProHXcH5X69q in 1.030599589s
[1] 2018/03/02 22:40:25.081053 [INF] STREAM: raft: aborting pipeline replication to peer {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}
[1] 2018/03/02 22:40:25.229890 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: natslog: read timeout
[1] 2018/03/02 22:40:25.585381 [INF] STREAM: raft: Failed to contact T0qMxJDUPbProHXcH5X69q in 504.395095ms
[1] 2018/03/02 22:40:26.095817 [INF] STREAM: raft: Failed to contact T0qMxJDUPbProHXcH5X69q in 1.014837551s
[1] 2018/03/02 22:40:27.081576 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:27.411856 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:29.092181 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:29.544993 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:31.102747 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:31.710252 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:33.126740 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:33.865919 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:35.168832 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:36.126167 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:37.250728 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:38.453152 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:39.412466 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:40.909640 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:41.735495 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:43.671934 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:44.377076 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:47.111779 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:47.661146 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:51.807911 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:52.223082 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:40:59.094576 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:40:59.346766 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:41:11.499089 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:41:11.591724 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:41:23.833392 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:41:23.851984 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:41:36.074874 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:41:36.213946 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:41:48.316340 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:41:48.608826 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:00.557631 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:42:01.032235 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:12.799393 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:42:13.413796 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:25.043129 [ERR] STREAM: raft: Failed to AppendEntries to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:42:25.827497 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:34.669382 [INF] STREAM: raft: Starting snapshot up to 131956
[1] 2018/03/02 22:42:34.669423 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-131956-1520030554669.tmp
[1] 2018/03/02 22:42:34.673415 [INF] STREAM: snapshot: reaping snapshot /data/stan/snapshots/2-101165-1520030202942
[1] 2018/03/02 22:42:34.673865 [INF] STREAM: raft: Compacting logs from 105504 to 121721
[1] 2018/03/02 22:42:34.866697 [INF] STREAM: raft: Snapshot to 131956 complete
[1] 2018/03/02 22:42:35.283343 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:42:37.284231 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:42:37.284280 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:42:38.192263 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:47.524505 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:42:49.525969 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:42:49.526019 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:42:50.561077 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:42:59.766300 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:43:01.770035 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:43:01.770090 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:43:02.948721 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:43:12.010249 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:43:14.011194 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:43:14.011244 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:43:15.308043 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:43:24.251499 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:43:26.252764 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:43:26.252805 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:43:27.665558 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:43:36.492973 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:43:38.493858 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:43:38.493927 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:43:40.079836 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:43:48.734131 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:43:50.735462 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:43:50.735515 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:43:52.458559 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:44:00.977227 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:44:02.978249 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:44:02.978330 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:44:04.899683 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:44:13.218553 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:44:15.219777 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:44:15.219830 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:44:17.258075 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:44:25.460002 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:44:27.461018 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:44:27.461074 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:44:29.657693 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:44:37.701228 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:44:39.702730 [ERR] STREAM: raft: Failed to install snapshot 2-131956-1520030554669: nats: timeout
[1] 2018/03/02 22:44:39.702839 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:44:42.037190 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:44:47.576773 [INF] STREAM: raft: Starting snapshot up to 141915
[1] 2018/03/02 22:44:47.576808 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-141915-1520030687576.tmp
[1] 2018/03/02 22:44:47.580203 [INF] STREAM: snapshot: reaping snapshot /data/stan/snapshots/2-115738-1520030348043
[1] 2018/03/02 22:44:47.580485 [INF] STREAM: raft: Compacting logs from 121722 to 131682
[1] 2018/03/02 22:44:47.705253 [INF] STREAM: raft: Snapshot to 141915 complete
[1] 2018/03/02 22:44:49.943035 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:44:51.944123 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:44:51.944184 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:44:54.454190 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:45:02.184387 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:45:04.185501 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:45:04.185551 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:45:06.823720 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:45:14.425812 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:45:16.427105 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:45:16.427176 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:45:19.231986 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:45:26.669912 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:45:28.670737 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:45:28.670786 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:45:31.657129 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:45:38.910986 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:45:40.912087 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:45:40.912216 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
[1] 2018/03/02 22:45:44.000875 [ERR] STREAM: raft: Failed to heartbeat to test-cluster.T0qMxJDUPbProHXcH5X69q.stan: nats: timeout
[1] 2018/03/02 22:45:51.152412 [ERR] STREAM: raft: Failed to get log at index 121253: log not found
[1] 2018/03/02 22:45:53.153526 [ERR] STREAM: raft: Failed to install snapshot 2-141915-1520030687576: nats: timeout
[1] 2018/03/02 22:45:53.153576 [ERR] STREAM: raft: Failed to send snapshot to {Voter T0qMxJDUPbProHXcH5X69q test-cluster.T0qMxJDUPbProHXcH5X69q.stan}: nats: timeout
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x87e359]
goroutine 60 [running]:
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).removeFirstMsg(0xc420092340, 0x0, 0xc42435bf00)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2647 +0x69
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).enforceLimits(0xc420092340, 0x740001, 0xc425d83e00, 0xc426b2af01)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2589 +0xa2
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).Store(0xc420092340, 0xc425d83e00, 0x0, 0x0, 0x0)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2482 +0xb99
github.com/nats-io/nats-streaming-server/server.(*StanServer).Apply(0xc420170500, 0xc429c0a620, 0xdfe320, 0xbe9e92e8c4cd0c97)
/go/src/github.com/nats-io/nats-streaming-server/server/server.go:4498 +0x176
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc4201d0190)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:57 +0x17b
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc420176b00)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:120 +0x31e
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).(github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.runFSM)-fm()
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/api.go:506 +0x2a
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc420176b00, 0xc4201ecae0)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:146 +0x53
created by github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:144 +0x66
Raw
nats2.conf
port: 4232
http_port: 8232
store: "file"
dir: "/data"
cluster {
host: '10.10.0.100'
port: 5232
raft_logging: true
log_path: "/data"
routes = [
nats-route://10.10.0.100:5231
nats-route://10.10.0.100:5233
]
}
Raw
nats2.console
$ docker run -it --name nats2 -v ${PWD}/srv2:/data --network=host nats-streaming -c /data/nats.conf
[1] 2018/03/02 22:31:20.134462 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.8.0-beta
[1] 2018/03/02 22:31:20.134513 [INF] STREAM: ServerID: Jchw4RbZTYkbcD5dMoP1TT
[1] 2018/03/02 22:31:20.134520 [INF] STREAM: Go version: go1.9.3
[1] 2018/03/02 22:31:20.134778 [INF] Starting nats-server version 1.0.4
[1] 2018/03/02 22:31:20.135111 [INF] Starting http monitor on 0.0.0.0:8232
[1] 2018/03/02 22:31:20.135268 [INF] Listening for client connections on 0.0.0.0:4232
[1] 2018/03/02 22:31:20.135281 [INF] Server is ready
[1] 2018/03/02 22:31:20.136505 [INF] Listening for route connections on 10.10.0.100:5232
[1] 2018/03/02 22:31:20.162400 [INF] STREAM: Recovering the state...
[1] 2018/03/02 22:31:20.162601 [INF] STREAM: No recovered state
[1] 2018/03/02 22:31:20.164455 [INF] STREAM: raft: Initial configuration (index=0): []
[1] 2018/03/02 22:31:20.164615 [INF] STREAM: raft: Node at test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan [Follower] entering Follower state (Leader: "")
[1] 2018/03/02 22:31:20.165817 [INF] STREAM: Message store is RAFT_FILE
[1] 2018/03/02 22:31:20.165864 [INF] STREAM: ---------- Store Limits ----------
[1] 2018/03/02 22:31:20.165875 [INF] STREAM: Channels: 100 *
[1] 2018/03/02 22:31:20.165884 [INF] STREAM: --------- Channels Limits --------
[1] 2018/03/02 22:31:20.165890 [INF] STREAM: Subscriptions: 1000 *
[1] 2018/03/02 22:31:20.165897 [INF] STREAM: Messages : 1000000 *
[1] 2018/03/02 22:31:20.165903 [INF] STREAM: Bytes : 976.56 MB *
[1] 2018/03/02 22:31:20.165909 [INF] STREAM: Age : unlimited *
[1] 2018/03/02 22:31:20.165916 [INF] STREAM: ----------------------------------
[1] 2018/03/02 22:31:20.166613 [INF] STREAM: raft: Failed to get previous log: 4 log not found (last: 0)
[1] 2018/03/02 22:35:06.997235 [INF] STREAM: WARNING: Reached limits for store "test" (msgs=963326/1000000 bytes=976.56 MB/976.56 MB), dropping old messages to make room for new ones
[1] 2018/03/02 22:36:16.990290 [INF] STREAM: raft: Starting snapshot up to 83102
[1] 2018/03/02 22:36:16.990344 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-83102-1520030176990.tmp
[1] 2018/03/02 22:36:21.570762 [INF] STREAM: raft: Compacting logs from 1 to 75846
[1] 2018/03/02 22:36:22.370118 [INF] STREAM: raft: Snapshot to 83102 complete
[1] 2018/03/02 22:36:23.402374 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.412767 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.446887 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.457226 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.468883 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.482707 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.514714 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.546243 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.567433 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.584467 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.585293 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.613029 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.621555 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.632864 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.666074 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.680308 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.694641 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.714167 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.729543 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.744241 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.761162 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.777693 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.784864 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.796882 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.807693 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.841465 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.873431 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.888850 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.905797 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.944530 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.977006 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:23.993088 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.005155 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.016679 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.049108 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.064647 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.078361 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.095863 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.114301 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.134358 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.149749 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.166832 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.187021 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.211482 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.232543 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.244284 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.257274 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.273130 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.286372 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:24.301488 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1ol: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:25.525993 [ERR] STREAM: raft-net: Failed to decode incoming command: codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 0, dec: 0
[1] 2018/03/02 22:36:29.833878 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1u5: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:29.960360 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1u5: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:30.038223 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1u5: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:30.038808 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1u5: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:30.487511 [ERR] STREAM: Asynchronous error on connection _NSS-test-cluster-raft, subject raft.test-cluster.Jchw4RbZTYkbcD5dMoP1Yn.stan.request._INBOX.Jchw4RbZTYkbcD5dMoP1u5: nats: slow consumer, messages dropped
[1] 2018/03/02 22:36:31.639202 [ERR] STREAM: raft-net: Failed to decode incoming command: codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 0, dec: 0
[1] 2018/03/02 22:39:19.167997 [INF] STREAM: raft: Starting snapshot up to 116531
[1] 2018/03/02 22:39:19.168036 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-116531-1520030359168.tmp
[1] 2018/03/02 22:39:19.170969 [INF] STREAM: raft: Compacting logs from 75847 to 106294
[1] 2018/03/02 22:39:19.376234 [INF] STREAM: raft: Snapshot to 116531 complete
[1] 2018/03/02 22:41:42.652278 [INF] STREAM: raft: Starting snapshot up to 127497
[1] 2018/03/02 22:41:42.652315 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-127497-1520030502652.tmp
[1] 2018/03/02 22:41:42.655717 [INF] STREAM: snapshot: reaping snapshot /data/stan/snapshots/2-83102-1520030176990
[1] 2018/03/02 22:41:42.657613 [INF] STREAM: raft: Compacting logs from 106295 to 117263
[1] 2018/03/02 22:41:42.765757 [INF] STREAM: raft: Snapshot to 127497 complete
[1] 2018/03/02 22:45:17.994109 [INF] STREAM: raft: Starting snapshot up to 143778
[1] 2018/03/02 22:45:17.994153 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-143778-1520030717994.tmp
[1] 2018/03/02 22:45:18.218319 [INF] STREAM: snapshot: reaping snapshot /data/stan/snapshots/2-116531-1520030359168
[1] 2018/03/02 22:45:18.218645 [INF] STREAM: raft: Compacting logs from 117264 to 133538
[1] 2018/03/02 22:45:18.381283 [INF] STREAM: raft: Snapshot to 143778 complete
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x87e359]
goroutine 31 [running]:
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).removeFirstMsg(0xc4202564e0, 0x0, 0xc436d79c00)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2647 +0x69
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).enforceLimits(0xc4202564e0, 0x740001, 0xc4201eaa50, 0xc4251eff01)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2589 +0xa2
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).Store(0xc4202564e0, 0xc4201eaa50, 0x0, 0x0, 0x0)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2482 +0xb99
github.com/nats-io/nats-streaming-server/server.(*StanServer).Apply(0xc4201f4000, 0xc4243e75f0, 0xdfe320, 0xbe9e92e8c556db13)
/go/src/github.com/nats-io/nats-streaming-server/server/server.go:4498 +0x176
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc421a520b0)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:57 +0x17b
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc4200ed340)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:120 +0x31e
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).(github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.runFSM)-fm()
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/api.go:506 +0x2a
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc4200ed340, 0xc420186350)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:146 +0x53
created by github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:144 +0x66
Raw
nats3.conf
port: 4233
http_port: 8233
store: "file"
dir: "/data"
cluster {
host: '10.10.0.100'
port: 5233
raft_logging: true
log_path: "/data"
routes = [
nats-route://10.10.0.100:5231
nats-route://10.10.0.100:5232
]
}
Raw
nats3.console
$ docker run -it --name nats3 -v ${PWD}/srv3:/data --network=host nats-streaming -c /data/nats.conf
[1] 2018/03/02 22:31:40.049962 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.8.0-beta
[1] 2018/03/02 22:31:40.050008 [INF] STREAM: ServerID: T0qMxJDUPbProHXcH5X68N
[1] 2018/03/02 22:31:40.050020 [INF] STREAM: Go version: go1.9.3
[1] 2018/03/02 22:31:40.050262 [INF] Starting nats-server version 1.0.4
[1] 2018/03/02 22:31:40.050617 [INF] Starting http monitor on 0.0.0.0:8233
[1] 2018/03/02 22:31:40.050774 [INF] Listening for client connections on 0.0.0.0:4233
[1] 2018/03/02 22:31:40.050785 [INF] Server is ready
[1] 2018/03/02 22:31:40.052156 [INF] Listening for route connections on 10.10.0.100:5233
[1] 2018/03/02 22:31:40.078077 [INF] STREAM: Recovering the state...
[1] 2018/03/02 22:31:40.078307 [INF] STREAM: No recovered state
[1] 2018/03/02 22:31:40.080278 [INF] STREAM: raft: Initial configuration (index=0): []
[1] 2018/03/02 22:31:40.080449 [INF] STREAM: raft: Node at test-cluster.T0qMxJDUPbProHXcH5X69q.stan [Follower] entering Follower state (Leader: "")
[1] 2018/03/02 22:31:40.082543 [INF] STREAM: raft: Failed to get previous log: 5 log not found (last: 0)
[1] 2018/03/02 22:31:40.085282 [INF] STREAM: Message store is RAFT_FILE
[1] 2018/03/02 22:31:40.085363 [INF] STREAM: ---------- Store Limits ----------
[1] 2018/03/02 22:31:40.085373 [INF] STREAM: Channels: 100 *
[1] 2018/03/02 22:31:40.085383 [INF] STREAM: --------- Channels Limits --------
[1] 2018/03/02 22:31:40.085391 [INF] STREAM: Subscriptions: 1000 *
[1] 2018/03/02 22:31:40.085399 [INF] STREAM: Messages : 1000000 *
[1] 2018/03/02 22:31:40.085406 [INF] STREAM: Bytes : 976.56 MB *
[1] 2018/03/02 22:31:40.085413 [INF] STREAM: Age : unlimited *
[1] 2018/03/02 22:31:40.085421 [INF] STREAM: ----------------------------------
[1] 2018/03/02 22:35:06.997374 [INF] STREAM: WARNING: Reached limits for store "test" (msgs=963326/1000000 bytes=976.56 MB/976.56 MB), dropping old messages to make room for new ones
[1] 2018/03/02 22:35:32.622281 [INF] STREAM: raft: Starting snapshot up to 45011
[1] 2018/03/02 22:35:32.622310 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-45011-1520030132622.tmp
[1] 2018/03/02 22:35:32.624219 [INF] STREAM: raft: Compacting logs from 1 to 34778
[1] 2018/03/02 22:35:33.081613 [INF] STREAM: raft: Snapshot to 45011 complete
[1] 2018/03/02 22:37:44.370274 [INF] STREAM: raft: Starting snapshot up to 109631
[1] 2018/03/02 22:37:44.370334 [INF] STREAM: snapshot: Creating new snapshot at /data/stan/snapshots/2-109631-1520030264370.tmp
[1] 2018/03/02 22:37:44.373553 [INF] STREAM: raft: Compacting logs from 34779 to 99397
[1] 2018/03/02 22:37:45.104976 [INF] STREAM: raft: Snapshot to 109631 complete
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x87e359]
goroutine 44 [running]:
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).removeFirstMsg(0xc42028a4e0, 0x0, 0xc426614b00)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2647 +0x69
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).enforceLimits(0xc42028a4e0, 0x590001, 0xc4251c5950, 0xc424387001)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2589 +0xa2
github.com/nats-io/nats-streaming-server/stores.(*FileMsgStore).Store(0xc42028a4e0, 0xc4251c5950, 0x0, 0x0, 0x0)
/go/src/github.com/nats-io/nats-streaming-server/stores/filestore.go:2482 +0xb99
github.com/nats-io/nats-streaming-server/server.(*StanServer).Apply(0xc420206000, 0xc420c42480, 0xdfe320, 0x3)
/go/src/github.com/nats-io/nats-streaming-server/server/server.go:4498 +0x176
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc4201643f0)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:57 +0x17b
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc420376000)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/fsm.go:120 +0x31e
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*Raft).(github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.runFSM)-fm()
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/api.go:506 +0x2a
github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc420376000, 0xc420165140)
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:146 +0x53
created by github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft.(*raftState).goFunc
/go/src/github.com/nats-io/nats-streaming-server/vendor/github.com/hashicorp/raft/state.go:144 +0x66
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment