Skip to content

Instantly share code, notes, and snippets.

@rozzaaq

rozzaaq/rdy-elastic.log

Created May 13, 2019 07:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rozzaaq/b21e67d7a8d40d7dfdd48ae4e54d6c1a to your computer and use it in GitHub Desktop.
Save rozzaaq/b21e67d7a8d40d7dfdd48ae4e54d6c1a to your computer and use it in GitHub Desktop.
Download ZIP
Master node log
Raw
rdy-elastic.log
[2019-05-13T07:45:34,961][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-join[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 2978, reason: added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:35,051][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2978, reason: Publication{term=2, version=2978}
[2019-05-13T07:45:35,082][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [rdy-elastic3] failed to execute on node [5oO-CIf3TCyjtr1y1ssCVA]
org.elasticsearch.transport.RemoteTransportException: [rdy-elastic6][10.148.0.35:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [cluster:monitor/nodes/stats[n]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:650) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:466) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:471) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:355) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:326) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-05-13T07:45:36,996][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-left[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} followers check retry count exceeded], term: 2, version: 2981, reason: removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:37,008][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2981, reason: Publication{term=2, version=2981}
[2019-05-13T07:45:37,337][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-join[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 2982, reason: added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:37,420][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2982, reason: Publication{term=2, version=2982}
[2019-05-13T07:45:37,425][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [rdy-elastic3] failed to execute on node [5oO-CIf3TCyjtr1y1ssCVA]
org.elasticsearch.transport.RemoteTransportException: [rdy-elastic6][10.148.0.35:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [cluster:monitor/nodes/stats[n]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:650) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:466) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:471) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:355) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:326) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-05-13T07:45:39,347][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-left[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} followers check retry count exceeded], term: 2, version: 2984, reason: removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:39,360][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2984, reason: Publication{term=2, version=2984}
[2019-05-13T07:45:39,362][WARN ][o.e.c.c.Coordinator ] [rdy-elastic3] failed to validate incoming join request from node [{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.NodeDisconnectedException: [rdy-elastic6][10.148.0.35:9300][internal:cluster/coordination/join/validate] disconnected
[2019-05-13T07:45:40,346][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-join[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 2985, reason: added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:40,404][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2985, reason: Publication{term=2, version=2985}
[2019-05-13T07:45:40,408][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [rdy-elastic3] failed to execute on node [5oO-CIf3TCyjtr1y1ssCVA]
org.elasticsearch.transport.RemoteTransportException: [rdy-elastic6][10.148.0.35:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [cluster:monitor/nodes/stats[n]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:650) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:466) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:471) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:355) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:326) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-05-13T07:45:42,398][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-left[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} followers check retry count exceeded], term: 2, version: 2988, reason: removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:42,410][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2988, reason: Publication{term=2, version=2988}
[2019-05-13T07:45:43,339][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-join[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 2989, reason: added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:43,377][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2989, reason: Publication{term=2, version=2989}
[2019-05-13T07:45:43,380][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [rdy-elastic3] failed to execute on node [5oO-CIf3TCyjtr1y1ssCVA]
org.elasticsearch.transport.RemoteTransportException: [rdy-elastic6][10.148.0.35:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [cluster:monitor/nodes/stats[n]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:650) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:466) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:471) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:355) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:326) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-05-13T07:45:45,351][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-left[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} followers check retry count exceeded], term: 2, version: 2991, reason: removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:45,364][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] removed {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2991, reason: Publication{term=2, version=2991}
[2019-05-13T07:45:45,366][INFO ][o.e.c.s.MasterService ] [rdy-elastic3] node-join[{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} join existing leader], term: 2, version: 2992, reason: added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}
[2019-05-13T07:45:45,406][INFO ][o.e.c.s.ClusterApplierService] [rdy-elastic3] added {{rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true},}, term: 2, version: 2992, reason: Publication{term=2, version=2992}
[2019-05-13T07:45:45,411][WARN ][o.e.c.NodeConnectionsService] [rdy-elastic3] failed to connect to node {rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [rdy-elastic6][10.148.0.35:9300] general node connection failure
at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:127) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:343) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:330) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalStateException: handshake failed with {rdy-elastic6}{5oO-CIf3TCyjtr1y1ssCVA}{ZGjHlVu2SCagPhPKLSvAJw}{10.148.0.35}{10.148.0.35:9300}{ml.machine_memory=1771659264, ml.max_open_jobs=20, xpack.installed=true}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:418) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TransportService.lambda$connectionValidator$4(TransportService.java:349) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:105) ~[elasticsearch-7.0.0.jar:7.0.0]
... 9 more
Caused by: org.elasticsearch.transport.RemoteTransportException: [rdy-elastic6][10.148.0.35:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for action [internal:transport/handshake]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:154) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:650) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$19(AuthenticationService.java:466) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:471) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:355) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$9(AuthenticationService.java:326) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:345) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1036) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:922) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
@weekyuan
Copy link

Hello, friend, is this problem solved? I also encountered the same situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment