Skip to content

Instantly share code, notes, and snippets.

@oravecz

oravecz/gist:1023472

Created June 13, 2011 19:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save oravecz/1023472 to your computer and use it in GitHub Desktop.
Save oravecz/1023472 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
19:05:32,376 INFO main ontext.ContextLoader: 187 - Root WebApplicationContext: initialization started
19:05:32,463 INFO main ebApplicationContext: 456 - Refreshing Root WebApplicationContext: startup date [Mon Jun 13 19:05:32 UTC 2011]; root of context hierarchy
19:05:32,585 INFO main BeanDefinitionReader: 315 - Loading XML bean definitions from class path resource [spring-context.xml]
19:05:33,770 INFO main BeanDefinitionReader: 315 - Loading XML bean definitions from class path resource [spring-security.xml]
19:05:34,099 INFO main icationConfigBuilder: 338 - No login page configured. The default internal one will be used. Use the 'login-page' attribute to set the URL of the login page.
19:05:34,106 INFO main BeanDefinitionParser: 187 - Checking sorted filter chain: [Root bean: class [org.springframework.security.web.context.SecurityContextPersistenceFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 300, Root bean: class [org.springframework.security.web.authentication.logout.LogoutFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 400, Root bean: class [org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 800, Root bean: class [org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1000, Root bean: class [org.springframework.security.web.authentication.www.BasicAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1200, Root bean: class [org.springframework.security.web.savedrequest.RequestCacheAwareFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1300, Root bean: class [org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1400, Root bean: class [org.springframework.security.web.authentication.AnonymousAuthenticationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1600, Root bean: class [org.springframework.security.web.session.SessionManagementFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1700, Root bean: class [org.springframework.security.web.access.ExceptionTranslationFilter]; scope=; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null, order = 1800, <org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0>, order = 1900]
19:05:34,399 INFO main laceholderConfigurer: 177 - Loading properties file from class path resource [server.properties]
19:05:34,552 INFO main tListableBeanFactory: 549 - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1315415: defining beans [systemPlaceholderConfigurer,configParams,mailSender,elasticSearchConfiguration,esServer,esDataStore,delegatingStore,mapStoreManager,hzServer,backup,baseReflector,baseReflector$child#0,baseReflector$child#1,baseReflector$child#2,baseReflector$child#3,baseReflector$child#4,baseReflector$child#5,baseReflector$child#6,baseReflector$child#7,baseReflector$child#8,baseReflector$child#9,baseReflector$child#10,baseReflector$child#11,baseReflector$child#12,baseReflector$child#13,baseReflector$child#14,cronJobs,myScheduler,org.springframework.scheduling.support.MethodInvokingRunnable#0,org.springframework.scheduling.support.MethodInvokingRunnable#1,org.springframework.scheduling.config.ScheduledTaskRegistrar#0,org.springframework.security.web.PortMapperImpl#0,org.springframework.security.web.context.HttpSessionSecurityContextRepository#0,org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy#0,org.springframework.security.authentication.ProviderManager#0,org.springframework.security.access.vote.AffirmativeBased#0,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#0,org.springframework.security.authentication.AnonymousAuthenticationProvider#0,org.springframework.security.web.savedrequest.HttpSessionRequestCache#0,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#0,org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter#0,org.springframework.security.config.http.UserDetailsServiceInjectionBeanPostProcessor#0,org.springframework.security.filterChainProxy,restUserDetailsService,org.springframework.security.authentication.dao.DaoAuthenticationProvider#0,org.springframework.security.authentication.DefaultAuthenticationEventPublisher#0,org.springframework.security.authenticationManager]; root of factory hierarchy
19:05:34,691 INFO main ropertiesFactoryBean: 177 - Loading properties file from class path resource [elasticsearch-server.properties]
19:05:34,719 INFO main .ElasticSearchServer: 47 - Starting the Elastic Search server node
19:05:34,899 INFO main g.elasticsearch.node: 78 - [Crusader] {elasticsearch/0.16.2}[1106]: initializing ...
19:05:34,922 INFO main lasticsearch.plugins: 78 - [Crusader] loaded [cloud-aws]
19:05:38,646 DEBUG main csearch.cache.memory: 70 - [Crusader] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
19:05:38,686 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [cached], type [cached], keep_alive [5m]
19:05:38,690 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [index], type [cached], keep_alive [5m]
19:05:38,690 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [search], type [cached], keep_alive [5m]
19:05:38,691 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [percolate], type [cached], keep_alive [5m]
19:05:38,692 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [management], type [scaling], min [1], size [20], keep_alive [5m]
19:05:38,695 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [merge], type [scaling], min [1], size [20], keep_alive [5m]
19:05:38,695 DEBUG main ticsearch.threadpool: 70 - [Crusader] creating thread_pool [snapshot], type [scaling], min [1], size [5], keep_alive [5m]
19:05:38,755 DEBUG main y.zen.ping.multicast: 70 - [Crusader] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
19:05:38,761 DEBUG main ery.zen.ping.unicast: 70 - [Crusader] using initial hosts []
19:05:38,762 DEBUG main search.discovery.ec2: 70 - [Crusader] using initial_ping_timeout [30s]
19:05:38,767 DEBUG main rch.discovery.zen.fd: 70 - [Crusader] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
19:05:38,770 DEBUG main rch.discovery.zen.fd: 70 - [Crusader] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
19:05:39,033 TRACE main rg.elasticsearch.env: 62 - [Crusader] obtaining node lock on /var/local/es/data/elasticsearch-dev/nodes/0 ...
19:05:39,037 DEBUG main rg.elasticsearch.env: 70 - [Crusader] using node location [/var/local/es/data/elasticsearch-dev/nodes/0], local_node_id [0]
19:05:39,406 DEBUG main r.routing.allocation: 70 - [Crusader] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
19:05:39,406 DEBUG main r.routing.allocation: 70 - [Crusader] using [allow_rebalance] with [indices_all_active]
19:05:39,407 DEBUG main r.routing.allocation: 70 - [Crusader] using [cluster_concurrent_rebalance] with [2]
19:05:39,468 DEBUG main ticsearch.monitor.os: 70 - [Crusader] Using probe [org.elasticsearch.monitor.os.JmxOsProbe@ef9525] with refresh_interval [5s]
19:05:39,469 DEBUG main arch.monitor.process: 70 - [Crusader] Using probe [org.elasticsearch.monitor.process.JmxProcessProbe@90288e] with refresh_interval [5s]
19:05:39,470 DEBUG main arch.monitor.network: 70 - [Crusader] Using probe [org.elasticsearch.monitor.network.JmxNetworkProbe@1d1695e] with refresh_interval [5s]
19:05:39,477 DEBUG main arch.monitor.network: 70 - [Crusader] net_info
host [domU-12-31-39-09-48-D4]
eth0 display_name [eth0]
address [/fe80:0:0:0:1031:39ff:fe09:48d4%2] [/10.210.79.34]
mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false]
lo display_name [lo]
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]
mtu [16436] multicast [false] ptp [false] loopback [true] up [true] virtual [false]
19:05:39,477 TRACE main arch.monitor.network: 62 - [Crusader] ifconfig
NA
19:05:40,139 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute TotalNumberOfRequests[r=true,w=false,is=false,type=long]
19:05:40,140 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute BoundAddress[r=true,w=false,is=false,type=java.lang.String]
19:05:40,140 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute PublishAddress[r=true,w=false,is=false,type=java.lang.String]
19:05:40,141 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute TcpNoDelay[r=true,w=false,is=false,type=java.lang.Boolean]
19:05:40,142 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute NumberOfOutboundConnections[r=true,w=false,is=false,type=long]
19:05:40,142 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute Port[r=true,w=false,is=false,type=java.lang.String]
19:05:40,142 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute WorkerCount[r=true,w=false,is=false,type=int]
19:05:40,142 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute TcpReceiveBufferSize[r=true,w=false,is=false,type=java.lang.String]
19:05:40,142 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute ReuseAddress[r=true,w=false,is=false,type=java.lang.Boolean]
19:05:40,143 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute ConnectTimeout[r=true,w=false,is=false,type=java.lang.String]
19:05:40,143 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute TcpKeepAlive[r=true,w=false,is=false,type=java.lang.Boolean]
19:05:40,143 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute PublishHost[r=true,w=false,is=false,type=java.lang.String]
19:05:40,143 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute TcpSendBufferSize[r=true,w=false,is=false,type=java.lang.String]
19:05:40,143 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Attribute BindHost[r=true,w=false,is=false,type=java.lang.String]
19:05:40,148 DEBUG main index.shard.recovery: 70 - [Crusader] using concurrent_streams [5], file_chunk_size [100kb], translog_size [100kb], translog_ops [1000], and compress [true]
19:05:40,149 DEBUG main earch.indices.memory: 70 - [Crusader] using index_buffer_size [116mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
19:05:40,195 DEBUG main indices.cache.filter: 70 - [Crusader] using [node] filter cache with size [232mb]
19:05:40,195 DEBUG main ticsearch.gateway.s3: 70 - [Crusader] using bucket [ppkc-es-gateway-dev], region [null], chunk_size [100mb], concurrent_streams [5]
19:05:40,990 TRACE main ticsearch.gateway.s3: 62 - [Crusader] [findLatestMetadata]: Processing [metadata-75]
19:05:41,233 DEBUG main ticsearch.gateway.s3: 70 - [Crusader] Latest metadata found at index [75]
19:05:41,247 INFO main g.elasticsearch.node: 78 - [Crusader] {elasticsearch/0.16.2}[1106]: initialized
19:05:41,247 INFO main .ElasticSearchServer: 64 - Starting the Elastic Search server node with these settings:
19:05:41,247 INFO main .ElasticSearchServer: 69 - cloud.aws.access_key : AKIAJWQRTNTMFXIMX3WA
19:05:41,247 INFO main .ElasticSearchServer: 69 - cloud.aws.secret_key : <HIDDEN>
19:05:41,248 INFO main .ElasticSearchServer: 69 - cluster.name : elasticsearch-dev
19:05:41,248 INFO main .ElasticSearchServer: 69 - discovery.ec2.ping_timeout : 30s
19:05:41,248 INFO main .ElasticSearchServer: 69 - discovery.type : ec2
19:05:41,248 INFO main .ElasticSearchServer: 69 - gateway.s3.bucket : ppkc-es-gateway-dev
19:05:41,248 INFO main .ElasticSearchServer: 69 - gateway.type : s3
19:05:41,248 INFO main .ElasticSearchServer: 69 - http.enabled : true
19:05:41,249 INFO main .ElasticSearchServer: 69 - http.port : 9311
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email.filter.0 : lowercase
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email.tokenizer : uax_url_email
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email.type : custom
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_regex.lowercase : true
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_regex.pattern : [@.]
19:05:41,249 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_regex.stopwords : _none_
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_regex.type : pattern
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_split.filter.0 : lowercase
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_split.tokenizer : letter
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.email_split.type : custom
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.lowercase_keyword.filter.0 : lowercase
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.lowercase_keyword.tokenizer : keyword
19:05:41,250 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.lowercase_keyword.type : custom
19:05:41,251 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.char_filter.0 : html_strip
19:05:41,251 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.filter.0 : standard
19:05:41,252 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.filter.1 : lowercase
19:05:41,252 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.filter.2 : stop
19:05:41,252 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.filter.3 : porterStem
19:05:41,252 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.tokenizer : standard
19:05:41,272 INFO main .ElasticSearchServer: 69 - index.analysis.analyzer.porter_standard.type : custom
19:05:41,273 INFO main .ElasticSearchServer: 69 - index.mapping._id.indexed : true
19:05:41,273 INFO main .ElasticSearchServer: 69 - index.store.fs.memory.direct : true
19:05:41,273 INFO main .ElasticSearchServer: 69 - index.store.memory.direct : true
19:05:41,273 INFO main .ElasticSearchServer: 69 - index.store.memory.warm_cache : true
19:05:41,273 INFO main .ElasticSearchServer: 69 - index.store.type : niofs
19:05:41,273 INFO main .ElasticSearchServer: 69 - name : Crusader
19:05:41,274 INFO main .ElasticSearchServer: 69 - network.host : 0.0.0.0
19:05:41,274 INFO main .ElasticSearchServer: 69 - node.data : true
19:05:41,274 INFO main .ElasticSearchServer: 69 - path.conf : /usr/share/tomcat6/webapps/ROOT//WEB-INF/config
19:05:41,274 INFO main .ElasticSearchServer: 69 - path.data : /var/local/es/data
19:05:41,274 INFO main .ElasticSearchServer: 69 - path.data_with_cluster : /var/local/es/data/elasticsearch-dev
19:05:41,274 INFO main .ElasticSearchServer: 69 - path.home : /var/local/es
19:05:41,275 INFO main .ElasticSearchServer: 69 - path.logs : /var/local/es/logs
19:05:41,275 INFO main .ElasticSearchServer: 69 - path.work : /var/local/es/work
19:05:41,275 INFO main .ElasticSearchServer: 69 - path.work_with_cluster : /var/local/es/work/elasticsearch-dev
19:05:41,275 INFO main .ElasticSearchServer: 69 - transport.tcp.port : 9310
19:05:41,276 INFO main g.elasticsearch.node: 78 - [Crusader] {elasticsearch/0.16.2}[1106]: starting ...
19:05:41,356 DEBUG main .NioProviderMetadata: 70 - Using the autodetected NIO constraint level: 0
19:05:41,491 DEBUG main arch.transport.netty: 70 - [Crusader] Bound to address [/0:0:0:0:0:0:0:0:9310]
19:05:41,493 INFO main sticsearch.transport: 78 - [Crusader] bound_address {inet[/0:0:0:0:0:0:0:0:9310]}, publish_address {inet[/10.210.79.34:9310]}
19:05:41,537 TRACE main sticsearch.discovery: 62 - [Crusader] waiting for 30s for the initial state to be set by the discovery
19:05:41,840 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] building dynamic unicast discovery nodes...
19:05:41,844 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] adding i-c767d9a9, address 10.91.25.118, transport_address inet[/10.91.25.118:9310]
19:05:41,844 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] adding i-7164da1f, address 10.122.62.201, transport_address inet[/10.122.62.201:9310]
19:05:41,844 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] adding i-5b553335, address 10.254.91.0, transport_address inet[/10.254.91.0:9310]
19:05:41,844 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] adding i-4bcbbc25, address 10.210.79.34, transport_address inet[/10.210.79.34:9310]
19:05:41,844 TRACE thread-1 search.discovery.ec2: 62 - [Crusader] adding i-45cbbc2b, address 10.211.141.189, transport_address inet[/10.211.141.189:9310]
19:05:41,845 DEBUG thread-1 search.discovery.ec2: 70 - [Crusader] using dynamic discovery nodes [[#cloud-i-c767d9a9-0][inet[/10.91.25.118:9310]], [#cloud-i-7164da1f-0][inet[/10.122.62.201:9310]], [#cloud-i-5b553335-0][inet[/10.254.91.0:9310]], [#cloud-i-4bcbbc25-0][inet[/10.210.79.34:9310]], [#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]]
19:05:42,205 TRACE thread-2 ery.zen.ping.unicast: 66 - [Crusader] [1] failed to connect to [#cloud-i-c767d9a9-0][inet[/10.91.25.118:9310]]
org.elasticsearch.transport.ConnectTransportException: [][inet[/10.91.25.118:9310]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:514)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:475)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more
19:05:42,207 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00b103dd]]
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:05:42,714 TRACE thread-6 ery.zen.ping.unicast: 66 - [Crusader] [1] failed to connect to [#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]
org.elasticsearch.transport.ConnectTransportException: [][inet[/10.211.141.189:9310]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:514)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:475)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more
19:05:42,791 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x0004b7b1]]
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:05:42,799 DEBUG thread-5 arch.transport.netty: 70 - [Crusader] Connected to node [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]]
19:05:42,800 TRACE thread-5 ery.zen.ping.unicast: 62 - [Crusader] [1] connecting to [Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]], disconnect[false]
19:05:42,879 TRACE ker #1-2 ery.zen.ping.unicast: 62 - [Crusader] [1] received response from [Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]: [ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}]
19:06:03,816 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x017c8f7f]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:03,817 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00ede59e]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:03,817 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00b10190]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:03,817 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x017efd36]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:03,818 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x01845568]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:03,818 TRACE thread-4 ery.zen.ping.unicast: 66 - [Crusader] [1] failed to connect to [#cloud-i-5b553335-0][inet[/10.254.91.0:9310]]
org.elasticsearch.transport.ConnectTransportException: [][inet[/10.254.91.0:9310]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:514)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:475)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more
19:06:03,822 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x010e2610]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:11,538 WARN main sticsearch.discovery: 86 - [Crusader] waited for 30s and no initial state was set by the discovery
19:06:11,538 INFO main sticsearch.discovery: 78 - [Crusader] elasticsearch-dev/NPnZk3_ITg6dn_fhW2Ogsg
19:06:11,538 DEBUG main lasticsearch.gateway: 70 - [Crusader] can't wait on start for (possibly) reading state from gateway, will do it asynchronously
19:06:11,542 INFO main g.elasticsearch.http: 78 - [Crusader] bound_address {inet[/0:0:0:0:0:0:0:0:9311]}, publish_address {inet[/10.210.79.34:9311]}
19:06:11,542 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Registered org.elasticsearch.jmx.ResourceDMBean@93c5e7 under org.elasticsearch:service=transport
19:06:11,542 TRACE main rg.elasticsearch.jmx: 62 - [Crusader] Registered org.elasticsearch.jmx.ResourceDMBean@1a5e4e4 under org.elasticsearch:service=transport,transportType=netty
19:06:11,543 INFO main g.elasticsearch.node: 78 - [Crusader] {elasticsearch/0.16.2}[1106]: started
19:06:12,118 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] building dynamic unicast discovery nodes...
19:06:12,118 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] adding i-c767d9a9, address 10.91.25.118, transport_address inet[/10.91.25.118:9310]
19:06:12,119 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] adding i-7164da1f, address 10.122.62.201, transport_address inet[/10.122.62.201:9310]
19:06:12,119 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] adding i-5b553335, address 10.254.91.0, transport_address inet[/10.254.91.0:9310]
19:06:12,119 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] adding i-4bcbbc25, address 10.210.79.34, transport_address inet[/10.210.79.34:9310]
19:06:12,119 TRACE thread-4 search.discovery.ec2: 62 - [Crusader] adding i-45cbbc2b, address 10.211.141.189, transport_address inet[/10.211.141.189:9310]
19:06:12,119 DEBUG thread-4 search.discovery.ec2: 70 - [Crusader] using dynamic discovery nodes [[#cloud-i-c767d9a9-0][inet[/10.91.25.118:9310]], [#cloud-i-7164da1f-0][inet[/10.122.62.201:9310]], [#cloud-i-5b553335-0][inet[/10.254.91.0:9310]], [#cloud-i-4bcbbc25-0][inet[/10.210.79.34:9310]], [#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]]
19:06:12,119 TRACE thread-4 ery.zen.ping.unicast: 62 - [Crusader] [1] connecting to [Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]], disconnect[false]
19:06:12,120 TRACE ker #1-1 ery.zen.ping.unicast: 62 - [Crusader] [1] received response from [Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]: [ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}]
19:06:24,820 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x012f2b32]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,821 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00cac877]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,821 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00f49ac3]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,821 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x0085097d]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,822 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x00dc904a]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,822 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x01bcfbeb]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,822 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x0172e08e]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,823 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x0032087b]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,824 TRACE thread-3 ery.zen.ping.unicast: 66 - [Crusader] [1] failed to connect to [#cloud-i-7164da1f-0][inet[/10.122.62.201:9310]]
org.elasticsearch.transport.ConnectTransportException: [][inet[/10.122.62.201:9310]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:514)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:475)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
... 3 more
19:06:24,825 TRACE boss #1 arch.transport.netty: 66 - [Crusader] (Ignoring) Exception caught on netty layer [[id: 0x01a3a52c]]
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
19:06:24,827 DEBUG thread-7 arch.transport.netty: 70 - [Crusader] Connected to node [[#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]]
19:06:24,828 TRACE thread-7 ery.zen.ping.unicast: 62 - [Crusader] [1] connecting to [#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]], disconnect[true]
19:06:24,830 TRACE ker #1-2 ery.zen.ping.unicast: 62 - [Crusader] [1] received response from [#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]: [ping_response{target [[Martha Johansson][YeoIUl_RSkCgt2A5wRLZVg][inet[/10.211.141.189:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Martha Johansson][YeoIUl_RSkCgt2A5wRLZVg][inet[/10.211.141.189:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Crusader][NPnZk3_ITg6dn_fhW2Ogsg][inet[/10.210.79.34:9310]]], master [null], cluster_name[elasticsearch-dev]}, ping_response{target [[Martha Johansson][YeoIUl_RSkCgt2A5wRLZVg][inet[/10.211.141.189:9310]]], master [null], cluster_name[elasticsearch-dev]}]
19:06:24,831 DEBUG ker #1-2 arch.transport.netty: 70 - [Crusader] Disconnected from [[#cloud-i-45cbbc2b-0][inet[/10.211.141.189:9310]]]
19:06:41,558 INFO main tListableBeanFactory: 422 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1315415: defining beans [systemPlaceholderConfigurer,configParams,mailSender,elasticSearchConfiguration,esServer,esDataStore,delegatingStore,mapStoreManager,hzServer,backup,baseReflector,baseReflector$child#0,baseReflector$child#1,baseReflector$child#2,baseReflector$child#3,baseReflector$child#4,baseReflector$child#5,baseReflector$child#6,baseReflector$child#7,baseReflector$child#8,baseReflector$child#9,baseReflector$child#10,baseReflector$child#11,baseReflector$child#12,baseReflector$child#13,baseReflector$child#14,cronJobs,myScheduler,org.springframework.scheduling.support.MethodInvokingRunnable#0,org.springframework.scheduling.support.MethodInvokingRunnable#1,org.springframework.scheduling.config.ScheduledTaskRegistrar#0,org.springframework.security.web.PortMapperImpl#0,org.springframework.security.web.context.HttpSessionSecurityContextRepository#0,org.springframework.security.web.authentication.session.SessionFixationProtectionStrategy#0,org.springframework.security.authentication.ProviderManager#0,org.springframework.security.access.vote.AffirmativeBased#0,org.springframework.security.web.access.intercept.FilterSecurityInterceptor#0,org.springframework.security.web.access.DefaultWebInvocationPrivilegeEvaluator#0,org.springframework.security.authentication.AnonymousAuthenticationProvider#0,org.springframework.security.web.savedrequest.HttpSessionRequestCache#0,org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint#0,org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter#0,org.springframework.security.config.http.UserDetailsServiceInjectionBeanPostProcessor#0,org.springframework.security.filterChainProxy,restUserDetailsService,org.springframework.security.authentication.dao.DaoAuthenticationProvider#0,org.springframework.security.authentication.DefaultAuthenticationEventPublisher#0,org.springframework.security.authenticationManager]; root of factory hierarchy
19:06:41,558 ERROR main ontext.ContextLoader: 220 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esServer' defined in class path resource [spring-context.xml]: Invocation of init method failed; nested exception is org.elasticsearch.discovery.MasterNotDiscoveredException:
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1412)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:574)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4135)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4630)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:905)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:740)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:500)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:785)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445)
at org.apache.catalina.core.StandardService.start(StandardService.java:519)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:581)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.elasticsearch.discovery.MasterNotDiscoveredException:
at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:162)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
log4j:WARN No appenders could be found for logger (org.elasticsearch.monitor.jvm).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "qtp17486873-33" java.lang.NoClassDefFoundError: org/eclipse/jetty/io/nio/SelectorManager$SelectSet$2
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:756)
at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:195)
at org.eclipse.jetty.client.SelectConnector.run(SelectConnector.java:180)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.io.nio.SelectorManager$SelectSet$2
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1645)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
... 5 more
Exception in thread "elasticsearch[cached]-pool-1-thread-2" java.lang.NullPointerException
at org.elasticsearch.cluster.node.DiscoveryNodes.iterator(DiscoveryNodes.java:65)
at org.elasticsearch.cluster.node.DiscoveryNodes.iterator(DiscoveryNodes.java:42)
at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:299)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Exception in thread "elasticsearch[cached]-pool-1-thread-1" java.lang.NoClassDefFoundError: org/elasticsearch/discovery/zen/ZenDiscovery$2
at org.elasticsearch.discovery.zen.ZenDiscovery.innterJoinCluster(ZenDiscovery.java:250)
at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:62)
at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.discovery.zen.ZenDiscovery$2
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1645)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
... 6 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment