Skip to content

Instantly share code, notes, and snippets.

@weilliu
Last active August 31, 2015 22:43
Show Gist options
  • Save weilliu/b0a74feb3b95ee8052ea to your computer and use it in GitHub Desktop.
Save weilliu/b0a74feb3b95ee8052ea to your computer and use it in GitHub Desktop.
ServerRestart query node log
============================================================
Running SvcRestart[SingleService]-N1QL
workload=N1QLWorkloadGroup
service/action=kill
service/restore_delay=5
service/name=SYSV_SERVICE
testcase=ServiceFailureScenario
service/count=2
Logging to /root/sdkd-2.0/sdkdclient-ng/log/SDK-SDK/CB-4.0.0-4047-enterprise/SvcRestart[SingleService]-N1QL/08-31-15/081118/f1a48f1bb259d76fd753678b96082f29
To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
--workload N1QLWorkloadGroup --service-action kill \
--service-restore_delay 5 --service-name SYSV_SERVICE --testcase \
ServiceFailureScenario --service-count 2 \
--cluster_node=172.23.106.87:::n1ql \
--cluster_node=172.23.106.88:::index \
--cluster_node=172.23.105.194:::n1ql \
--cluster_node=172.23.105.174:::kv --cluster_ssh-username=root \
--cluster_ssh-password=couchbase -C share/rexec --rexec_path \
/root/sdkd-2.0/run-sdkd-java --rexec_port 8050 --n1ql-nthreads 1 \
--n1ql-index-engine view --n1ql-index-type primary \
--n1ql-prepared false --n1ql-parameterized true --ramp=10
---------------------------------------->8
[1.04 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1.17 DEBUG] (Nodelist <init>:364) Number of active nodes 4
[1.34 INFO] (RunContext run:104) Starting cluster and driver
[1.34 INFO] (CBCluster startCluster:386) Node http://172.23.105.174:8091 is master now
[1.34 INFO] (HostPortDriver launch:32) Invoking SDKD as '/root/sdkd-2.0/run-sdkd-java'
[1.35 DEBUG] (CBCluster startCluster:392) Stopping any existing rebalance operations..
[1.37 INFO] (SDKD log:137) ++ pgrep -f 'java.+-jar.*sdkd-java'
[1.38 INFO] (SDKD log:137) ++ dirname /root/sdkd-2.0/run-sdkd-java
[1.38 INFO] (SDKD log:137) + SDKD_ROOT=/root/sdkd-2.0
[1.38 INFO] (SDKD log:137) + cd /root/sdkd-2.0
[1.38 INFO] (SDKD log:137) + JAR=
[1.38 INFO] (SDKD log:137) + JAR_DEFAULT=
[1.38 INFO] (SDKD log:137) + '[' -z '' ']'
[1.38 INFO] (SDKD log:137) + JAR_DEFAULT=1
[1.38 INFO] (SDKD log:137) + JAR=/root/sdkd-2.0/sdkd_exec.jar
[1.38 INFO] (SDKD log:137) + '[' '!' -e /root/sdkd-2.0/sdkd_exec.jar ']'
[1.39 INFO] (SDKD log:137) + '[' '!' -e /root/sdkd-2.0/sdkd_exec.jar ']'
[1.39 INFO] (SDKD log:137) + java -ea -jar /root/sdkd-2.0/sdkd_exec.jar -l 8050 -shared 100 -sync true
[1.88 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[1.97 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"COMPONENTS":{"Core_VERSION":"couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71)","SDK_VERSION":"couchbase-java-client/2.2.0-SNAPSHOT (git: 2.1.2-71-g9891310) (Linux/2.6.32-358.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17)"},"CAPS":{"VIEWS":true,"OBS":true,"DS_SHARED":true,"CANCEL":true,"CONTINUOUS":true,"TTL":true,"PREAMBLE":false}}
[3.99 DEBUG] (CBCluster clearSingleCluster:142) Failing over existing and ejecting node <URI:172.23.105.194:8091,ns_1@172.23.105.194>
[4.03 DEBUG] (CBCluster clearSingleCluster:142) Failing over existing and ejecting node <URI:172.23.106.87:8091,ns_1@172.23.106.87>
[4.08 DEBUG] (CBCluster clearSingleCluster:142) Failing over existing and ejecting node <URI:172.23.106.88:8091,ns_1@172.23.106.88>
[4.16 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa
[4.16 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.16 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa
[4.16 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.16 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa
[4.16 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.16 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa
[4.16 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.57 INFO] (NodeHost createSSH:160) SSH Initialized for http://172.23.106.87:8091
[4.57 INFO] (NodeHost createSSH:160) SSH Initialized for http://172.23.105.194:8091
[4.58 INFO] (NodeHost createSSH:160) SSH Initialized for http://172.23.105.174:8091
[4.58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.105.194
[4.58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.106.87
[4.58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.105.174
[4.60 INFO] (NodeHost createSSH:160) SSH Initialized for http://172.23.106.88:8091
[4.60 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.106.88
[4.69 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1a5e04fa
[4.69 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@25f4963a
[4.79 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@749fa274
[4.99 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@17bfc999
[4.99 DEBUG] (CBCluster setupNewCluster:290) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@70e7ffe8
[14.06 INFO] (CBCluster tryOnce:307) Adding node with services n1ql
[18.18 INFO] (CBCluster tryOnce:307) Adding node with services n1ql
[22.51 INFO] (CBCluster tryOnce:307) Adding node with services index
[26.64 INFO] (CBCluster setupNewCluster:314) All nodes added. Will rebalance
[26.67 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[27.67 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[27.67 DEBUG] (CBCluster setupServerGroups:241) Not creating any groups
[27.68 INFO] (CBCluster setupMainBucket:228) Creating bucket default
[27.70 INFO] (CBCluster setupMainBucket:230) Bucket creation submitted
[32.30 INFO] (CBCluster waitForBucketReady:222) Bucket creation done
[32.34 INFO] (RunContext run:204) Driver and cluster initialized
[32.34 INFO] (RunContext run:209) Ramp for 10 seconds. Cluster modification: Execute '/etc/init.d/couchbase-server stop' on 2 nodes waiting 1 seconds between each invocation sleep for 5 seconds and restore the nodes. Restoring is peformed by: re-active the services by invoking /etc/init.d/couchbase-server starton each of the nodes. Rebound for 90 seconds.
[32.37 INFO] (RunContext call:260) Running scenario..
[32.37 INFO] (Scenario run:72) Starting RAMP phase
[32.37 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["172.23.105.194",8091],["172.23.106.87",8091],["172.23.106.88",8091]], Username=default, ClusterCertificate=, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=172.23.105.174}
[32.41 INFO] (SDKD log:137) Aug 31, 2015 3:32:29 PM com.couchbase.client.core.CouchbaseCore <init>
[32.42 INFO] (SDKD log:137) INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='cacerts', sslKeystorePassword='couchbase', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, queryTimeout=75000, viewTimeout=15000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
[32.96 INFO] (SDKD log:137) Aug 31, 2015 3:32:29 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[32.96 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.174
[33.37 INFO] (SDKD log:137) Aug 31, 2015 3:32:30 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call
[33.37 INFO] (SDKD log:137) INFO: Opened bucket default
[33.39 INFO] (SDKD log:137) Aug 31, 2015 3:32:30 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[33.39 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87
[33.39 INFO] (SDKD log:137) Aug 31, 2015 3:32:30 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[33.40 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194
[33.43 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[33.43 DEBUG] (Handle sendMessageAsync:183) > CB_N1QL_CREATE_INDEX@101.2 => {NQIndexEngine=view, NQIndexType=primary, TimeRes=1, NQParamValues=n1ql,n1qldoc, NQDefaultIndexName=idx1, NQParam=tag,type}
[33.44 INFO] (SDKD log:137) Aug 31, 2015 3:32:30 PM com.couchbase.sdkd.cbclient.N1QLIndexCommandContext execIter
[33.44 INFO] (SDKD log:137) INFO: n1ql index: null
[38.81 DEBUG] (Handle receiveMessage:158) < CB_N1QL_CREATE_INDEX@101.2 => {"Timings":{"Step":1,"Windows":[{"Errors":{"0":1},"Min":1,"Avg":1,"Max":1,"Count":1}],"Base":1227398},"Summary":{"0":1}}
[38.81 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[38.81 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["172.23.105.194",8091],["172.23.106.87",8091],["172.23.106.88",8091]], Username=default, ClusterCertificate=, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=172.23.105.174}
[38.81 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.sdkd.server.SdkServer shutdown
[38.81 INFO] (SDKD log:137) WARNING: Shutting Down..
[38.82 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.client.core.CouchbaseCore <init>
[38.82 INFO] (SDKD log:137) INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='cacerts', sslKeystorePassword='couchbase', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, queryTimeout=75000, viewTimeout=15000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
[38.86 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[38.87 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.174
[38.90 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call
[38.90 INFO] (SDKD log:137) INFO: Opened bucket default
[38.90 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[38.90 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[38.90 DEBUG] (Handle sendMessageAsync:183) > CB_N1QL_QUERY@102.5 => {NQIndexEngine=view, NQIndexType=primary, NQScanConsistency=request_plus, NQParameterized=true, TimeRes=1, NQParamValues=n1ql,n1qldoc, NQDefaultIndexName=idx1, NQParam=tag,type, NQPrepared=false}
[38.90 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194
[38.91 INFO] (Scenario run:83) RAMP phase started. Waiting for 10 seconds
[38.91 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.sdkd.cbclient.N1QLQueryCommandContext execIter
[38.91 INFO] (SDKD log:137) INFO: Running n1ql query: SELECT * from default WHERE tag = "n1ql" AND type = "n1qldoc";
[38.91 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.sdkd.cbclient.N1QLQueryCommandContext execIter
[38.91 INFO] (SDKD log:137) INFO: n1ql drop_index: DROP INDEX `default`.`#primary`;
[38.91 INFO] (SDKD log:137) Aug 31, 2015 3:32:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[38.91 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87
[48.91 INFO] (Scenario run:86) Starting CHANGE phase
[48.91 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 172.23.105.194
[49.44 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[49.45 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.105.194
[49.45 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractGenericHandler exceptionCaught
[49.45 INFO] (SDKD log:137) INFO: [/172.23.105.194:8093][QueryEndpoint]: Connection reset by peer: Connection reset by peer
[49.46 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.46 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.46 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.46 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[49.46 INFO] (SDKD log:137) at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
[49.46 INFO] (SDKD log:137)
[49.46 INFO] (SDKD log:137) Exception in thread "SDK Handle-102: CMD [CB_N1QL_QUERY] " com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
[49.46 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3
[49.46 INFO] (SDKD log:137) at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:339)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:311)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:137)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
[49.46 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[49.46 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.105.194
[49.46 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.47 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.47 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.47 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.47 INFO] (SDKD log:137)
[49.49 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.49 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.49 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.50 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.50 INFO] (SDKD log:137)
[49.50 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.50 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.50 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.50 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.50 INFO] (SDKD log:137)
[49.53 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.53 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.53 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.53 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.53 INFO] (SDKD log:137)
[49.53 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.53 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.53 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.53 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.53 INFO] (SDKD log:137)
[49.56 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.56 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.56 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.56 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.56 INFO] (SDKD log:137)
[49.57 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.57 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.57 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.57 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.57 INFO] (SDKD log:137)
[49.60 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.60 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.60 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.60 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.60 INFO] (SDKD log:137)
[49.60 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.60 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.60 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.60 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.61 INFO] (SDKD log:137)
[49.63 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.63 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.63 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.63 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.63 INFO] (SDKD log:137)
[49.64 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.64 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[49.64 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.64 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.64 INFO] (SDKD log:137)
[49.67 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.67 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 64 MILLISECONDS:
[49.67 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.67 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.67 INFO] (SDKD log:137)
[49.67 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.67 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 64 MILLISECONDS:
[49.67 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.67 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.67 INFO] (SDKD log:137)
[49.74 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.74 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 128 MILLISECONDS:
[49.74 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.74 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.74 INFO] (SDKD log:137)
[49.74 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.74 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 128 MILLISECONDS:
[49.74 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.74 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.74 INFO] (SDKD log:137)
[49.87 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.87 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 256 MILLISECONDS:
[49.87 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.87 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.87 INFO] (SDKD log:137)
[49.87 INFO] (SDKD log:137) Aug 31, 2015 3:32:46 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[49.87 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 256 MILLISECONDS:
[49.87 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[49.87 INFO] (SDKD log:137) +++ Received exception of ID 0
[49.87 INFO] (SDKD log:137)
[50.13 INFO] (SDKD log:137) Aug 31, 2015 3:32:47 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[50.13 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 512 MILLISECONDS:
[50.13 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[50.13 INFO] (SDKD log:137) +++ Received exception of ID 0
[50.13 INFO] (SDKD log:137)
[50.13 INFO] (SDKD log:137) Aug 31, 2015 3:32:47 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[50.13 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 512 MILLISECONDS:
[50.13 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[50.13 INFO] (SDKD log:137) +++ Received exception of ID 0
[50.13 INFO] (SDKD log:137)
[50.64 INFO] (SDKD log:137) Aug 31, 2015 3:32:47 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[50.65 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 1024 MILLISECONDS:
[50.65 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[50.65 INFO] (SDKD log:137) +++ Received exception of ID 0
[50.65 INFO] (SDKD log:137)
[50.65 INFO] (SDKD log:137) Aug 31, 2015 3:32:47 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[50.65 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 1024 MILLISECONDS:
[50.65 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[50.65 INFO] (SDKD log:137) +++ Received exception of ID 0
[50.65 INFO] (SDKD log:137)
[51.62 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@3bd72e86
[51.67 INFO] (SDKD log:137) Aug 31, 2015 3:32:48 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[51.67 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 2048 MILLISECONDS:
[51.67 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[51.67 INFO] (SDKD log:137) +++ Received exception of ID 0
[51.67 INFO] (SDKD log:137)
[51.67 INFO] (SDKD log:137) Aug 31, 2015 3:32:48 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[51.67 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 2048 MILLISECONDS:
[51.67 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[51.67 INFO] (SDKD log:137) +++ Received exception of ID 0
[51.67 INFO] (SDKD log:137)
[52.62 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 172.23.106.87
[53.15 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[53.16 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.106.87
[53.16 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[53.16 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.106.87
[53.16 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.16 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.16 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.16 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.16 INFO] (SDKD log:137)
[53.17 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.17 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.17 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.17 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.17 INFO] (SDKD log:137)
[53.20 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.20 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.20 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.20 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.20 INFO] (SDKD log:137)
[53.20 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.20 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.20 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.20 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.20 INFO] (SDKD log:137)
[53.23 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.23 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.23 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.23 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.23 INFO] (SDKD log:137)
[53.24 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.24 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.24 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.24 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.24 INFO] (SDKD log:137)
[53.27 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.27 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.27 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.27 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.27 INFO] (SDKD log:137)
[53.27 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.27 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.27 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.27 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.27 INFO] (SDKD log:137)
[53.30 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.30 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.30 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.30 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.30 INFO] (SDKD log:137)
[53.30 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.30 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.30 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.31 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.31 INFO] (SDKD log:137)
[53.34 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.34 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.34 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.34 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.34 INFO] (SDKD log:137)
[53.34 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.34 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS:
[53.34 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.34 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.34 INFO] (SDKD log:137)
[53.37 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.37 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 64 MILLISECONDS:
[53.37 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.37 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.37 INFO] (SDKD log:137)
[53.37 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.37 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 64 MILLISECONDS:
[53.37 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.37 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.37 INFO] (SDKD log:137)
[53.44 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.44 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 128 MILLISECONDS:
[53.44 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.44 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.44 INFO] (SDKD log:137)
[53.44 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.44 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 128 MILLISECONDS:
[53.44 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.44 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.44 INFO] (SDKD log:137)
[53.57 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.57 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 256 MILLISECONDS:
[53.57 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.57 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.57 INFO] (SDKD log:137)
[53.57 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.57 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 256 MILLISECONDS:
[53.57 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.57 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.57 INFO] (SDKD log:137)
[53.72 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.72 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[53.72 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[53.72 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.72 INFO] (SDKD log:137)
[53.72 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.73 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[53.73 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[53.73 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.73 INFO] (SDKD log:137)
[53.83 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.83 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 512 MILLISECONDS:
[53.83 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.83 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.83 INFO] (SDKD log:137)
[53.83 INFO] (SDKD log:137) Aug 31, 2015 3:32:50 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[53.83 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 512 MILLISECONDS:
[53.83 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[53.83 INFO] (SDKD log:137) +++ Received exception of ID 0
[53.83 INFO] (SDKD log:137)
[54.35 INFO] (SDKD log:137) Aug 31, 2015 3:32:51 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[54.35 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 1024 MILLISECONDS:
[54.35 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[54.35 INFO] (SDKD log:137) +++ Received exception of ID 0
[54.35 INFO] (SDKD log:137)
[54.35 INFO] (SDKD log:137) Aug 31, 2015 3:32:51 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[54.35 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 1024 MILLISECONDS:
[54.35 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[54.35 INFO] (SDKD log:137) +++ Received exception of ID 0
[54.35 INFO] (SDKD log:137)
[54.72 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@12b7e09e
[55.37 INFO] (SDKD log:137) Aug 31, 2015 3:32:52 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[55.37 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 2048 MILLISECONDS:
[55.37 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[55.37 INFO] (SDKD log:137) +++ Received exception of ID 0
[55.37 INFO] (SDKD log:137)
[55.37 INFO] (SDKD log:137) Aug 31, 2015 3:32:52 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[55.37 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 2048 MILLISECONDS:
[55.37 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[55.38 INFO] (SDKD log:137) +++ Received exception of ID 0
[55.38 INFO] (SDKD log:137)
[55.73 INFO] (Scenario sleepAndDo:109) Sleeping 5 seconds
[57.42 INFO] (SDKD log:137) Aug 31, 2015 3:32:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[57.42 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[57.42 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[57.42 INFO] (SDKD log:137) +++ Received exception of ID 0
[57.42 INFO] (SDKD log:137)
[57.43 INFO] (SDKD log:137) Aug 31, 2015 3:32:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[57.43 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[57.43 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[57.43 INFO] (SDKD log:137) +++ Received exception of ID 0
[57.43 INFO] (SDKD log:137)
[57.82 INFO] (SDKD log:137) Aug 31, 2015 3:32:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[57.82 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[57.82 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[57.82 INFO] (SDKD log:137) +++ Received exception of ID 0
[57.82 INFO] (SDKD log:137)
[57.82 INFO] (SDKD log:137) Aug 31, 2015 3:32:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[57.83 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[57.83 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[57.83 INFO] (SDKD log:137) +++ Received exception of ID 0
[57.83 INFO] (SDKD log:137)
[60.73 INFO] (Scenario executeChange:134) Turning on services for nodes
[60.73 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 172.23.105.194
[61.52 INFO] (SDKD log:137) Aug 31, 2015 3:32:58 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[61.52 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[61.52 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[61.52 INFO] (SDKD log:137) +++ Received exception of ID 0
[61.52 INFO] (SDKD log:137)
[61.52 INFO] (SDKD log:137) Aug 31, 2015 3:32:58 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[61.53 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[61.53 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[61.53 INFO] (SDKD log:137) +++ Received exception of ID 0
[61.53 INFO] (SDKD log:137)
[61.92 INFO] (SDKD log:137) Aug 31, 2015 3:32:58 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[61.92 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[61.92 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[61.92 INFO] (SDKD log:137) +++ Received exception of ID 0
[61.92 INFO] (SDKD log:137)
[61.92 INFO] (SDKD log:137) Aug 31, 2015 3:32:58 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[61.92 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[61.92 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[61.92 INFO] (SDKD log:137) +++ Received exception of ID 0
[61.92 INFO] (SDKD log:137)
[62.33 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@87a05bf
[63.33 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 172.23.106.87
[64.94 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@61dcdbd
[65.63 INFO] (SDKD log:137) Aug 31, 2015 3:33:02 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[65.63 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[65.63 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[65.63 INFO] (SDKD log:137) +++ Received exception of ID 0
[65.63 INFO] (SDKD log:137)
[65.63 INFO] (SDKD log:137) Aug 31, 2015 3:33:02 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[65.63 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[65.63 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[65.63 INFO] (SDKD log:137) +++ Received exception of ID 0
[65.63 INFO] (SDKD log:137)
[65.94 INFO] (Scenario run:97) CHANGE phase done
[65.94 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[66.02 INFO] (SDKD log:137) Aug 31, 2015 3:33:03 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[66.02 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[66.02 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[66.02 INFO] (SDKD log:137) +++ Received exception of ID 0
[66.02 INFO] (SDKD log:137)
[66.02 INFO] (SDKD log:137) Aug 31, 2015 3:33:03 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[66.02 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[66.02 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[66.02 INFO] (SDKD log:137) +++ Received exception of ID 0
[66.02 INFO] (SDKD log:137)
[69.73 INFO] (SDKD log:137) Aug 31, 2015 3:33:06 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[69.73 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[69.73 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[69.73 INFO] (SDKD log:137) +++ Received exception of ID 0
[69.73 INFO] (SDKD log:137)
[69.73 INFO] (SDKD log:137) Aug 31, 2015 3:33:06 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[69.73 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[69.73 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[69.73 INFO] (SDKD log:137) +++ Received exception of ID 0
[69.73 INFO] (SDKD log:137)
[70.12 INFO] (SDKD log:137) Aug 31, 2015 3:33:07 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[70.12 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[70.12 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[70.12 INFO] (SDKD log:137) +++ Received exception of ID 0
[70.12 INFO] (SDKD log:137)
[70.12 INFO] (SDKD log:137) Aug 31, 2015 3:33:07 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[70.12 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[70.12 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093
[70.13 INFO] (SDKD log:137) +++ Received exception of ID 0
[70.13 INFO] (SDKD log:137)
[73.83 INFO] (SDKD log:137) Aug 31, 2015 3:33:10 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[73.83 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[73.83 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[73.83 INFO] (SDKD log:137) +++ Received exception of ID 0
[73.83 INFO] (SDKD log:137)
[73.83 INFO] (SDKD log:137) Aug 31, 2015 3:33:10 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete
[73.83 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 4096 MILLISECONDS:
[73.83 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093
[73.83 INFO] (SDKD log:137) +++ Received exception of ID 0
[73.83 INFO] (SDKD log:137)
[74.22 INFO] (SDKD log:137) Aug 31, 2015 3:33:11 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[74.22 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194
[74.22 INFO] (SDKD log:137) Aug 31, 2015 3:33:11 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[74.22 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194
[77.93 INFO] (SDKD log:137) Aug 31, 2015 3:33:14 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[77.93 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87
[77.93 INFO] (SDKD log:137) Aug 31, 2015 3:33:14 PM com.couchbase.client.core.node.CouchbaseNode$1 call
[77.93 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87
[155.94 INFO] (Scenario run:105) REBOUND done. Will collect results
[155.94 DEBUG] (Handle sendMessageAsync:183) > CANCEL@102.5
[155.94 DEBUG] (Handle receiveMessage:158) < CANCEL@102.5
[155.95 INFO] (SDKD log:137) Aug 31, 2015 3:34:32 PM com.couchbase.sdkd.server.SdkServer shutdown
[155.95 INFO] (SDKD log:137) WARNING: Shutting Down..
[155.95 INFO] (SDKD log:137) Aug 31, 2015 3:34:32 PM com.couchbase.sdkd.server.SdkServer shutdown
[155.95 INFO] (SDKD log:137) WARNING: Shutting Down..
[155.95 ERROR] (BRun maybePropagate:241) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: java.io.IOException: Connection Closed
at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:115) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:262) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:258) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_25]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_25]
Caused by: java.io.IOException: Connection Closed
at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:154) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.workload.Workload.collect(Workload.java:164) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.workload.WorkloadGroup.collect(WorkloadGroup.java:73) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:113) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
... 8 common frames omitted
[155.95 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[155.96 DEBUG] (Handle sendMessageAsync:183) > GOODBYE@0.7
[155.96 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@100.8
[155.96 INFO] (SDKD log:137) Aug 31, 2015 3:34:32 PM com.couchbase.sdkd.server.ControlServer run
[155.96 INFO] (SDKD log:137) INFO: ioLoop returned false. Returning..
[155.96 INFO] (SDKD log:137) Aug 31, 2015 3:34:32 PM com.couchbase.sdkd.server.SdkServer shutdown
[155.96 INFO] (HostPortDriver stop:123) SDKD is not yet terminated. Terminating..
[155.96 INFO] (SDKD log:137) WARNING: Shutting Down..
[155.96 INFO] (SDKD log:137) Aug 31, 2015 3:34:32 PM com.couchbase.sdkd.server.SdkServer shutdown
[155.96 INFO] (HostPortDriver terminate:44) Waiting for process termination..
[155.96 INFO] (SDKD log:137) WARNING: Shutting Down..
[155.96 ERROR] (LineGobbler err:160) Stream Error
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[na:1.8.0_25]
at java.io.BufferedInputStream.read(BufferedInputStream.java:336) ~[na:1.8.0_25]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[na:1.8.0_25]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[na:1.8.0_25]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[na:1.8.0_25]
at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.8.0_25]
at java.io.BufferedReader.fill(BufferedReader.java:161) ~[na:1.8.0_25]
at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[na:1.8.0_25]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[na:1.8.0_25]
at com.couchbase.sdkdclient.util.LineGobbler.run(LineGobbler.java:76) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
[156.30 ERROR] (HostPortDriver stop:135) SDKD terminated with error 143
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment