Created
August 31, 2015 23:18
-
-
Save weilliu/2e48752e3cc798b3d538 to your computer and use it in GitHub Desktop.
Failover and re adding two query and index node
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ============================================================ | |
| Running ReAdd2-N1QL | |
| failover/next_action=FO_READD_REBALANCE | |
| workload=N1QLWorkloadGroup | |
| failover/count=2 | |
| failover/next_delay=45 | |
| testcase=FailoverScenario | |
| Logging to /root/sdkd-2.0/sdkdclient-ng/log/SDK-SDK/CB-4.0.0-4047-enterprise/ReAdd2-N1QL/08-31-15/083414/7514d21ce28c7ca714b031f6460df3a6 | |
| To re-run the test, copy/paste the following into the shell. | |
| You may also copy/paste (except the first line) into an argfile | |
| 8<---------------------------------------- | |
| ./stester \ | |
| --failover-next_action FO_READD_REBALANCE --workload \ | |
| N1QLWorkloadGroup --failover-count 2 --failover-next_delay 45 \ | |
| --testcase FailoverScenario --cluster_node=172.23.106.87:::index \ | |
| --cluster_node=172.23.106.88:::n1ql \ | |
| --cluster_node=172.23.105.194:::index \ | |
| --cluster_node=172.23.105.174:::kv --cluster_ssh-username=root \ | |
| --cluster_ssh-password=couchbase -C share/rexec --rexec_path \ | |
| /root/sdkd-2.0/run-sdkd-java --rexec_port 8050 --n1ql-nthreads 1 \ | |
| --n1ql-index-engine gsi --n1ql-index-type primary --n1ql-prepared \ | |
| true --n1ql-parameterized false --cluster-2inodes-count 2 \ | |
| --cluster-n1qlnodes-count 2 --ramp=10 | |
| ---------------------------------------->8 | |
| [1.06 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution | |
| [1.18 DEBUG] (Nodelist <init>:357) Number of active nodes 4 | |
| [1.18 DEBUG] (NodelistBuilder reserveForRemoval:74) ret [http://172.23.105.194:8091, http://172.23.106.87:8091] | |
| [1.36 INFO] (RunContext run:102) Ramp for 10 seconds. Cluster modification: Fail over 2 nodes. Then sleep for 45 seconds and readd the nodes to the cluster and rebalance. Rebound for 90 seconds. | |
| [1.38 INFO] (RunContext run:124) Starting cluster and driver | |
| [1.38 INFO] (HostPortDriver launch:32) Invoking SDKD as '/root/sdkd-2.0/run-sdkd-java' | |
| [1.38 INFO] (CBCluster startCluster:428) Node http://172.23.105.174:8091 is master now | |
| [1.38 DEBUG] (CBCluster startCluster:434) Stopping any existing rebalance operations.. | |
| [1.40 INFO] (SDKD log:137) ++ pgrep -f 'java.+-jar.*sdkd-java' | |
| [1.41 INFO] (SDKD log:137) ++ dirname /root/sdkd-2.0/run-sdkd-java | |
| [1.41 INFO] (SDKD log:137) + SDKD_ROOT=/root/sdkd-2.0 | |
| [1.41 INFO] (SDKD log:137) + cd /root/sdkd-2.0 | |
| [1.41 INFO] (SDKD log:137) + JAR= | |
| [1.41 INFO] (SDKD log:137) + JAR_DEFAULT= | |
| [1.41 INFO] (SDKD log:137) + '[' -z '' ']' | |
| [1.41 INFO] (SDKD log:137) + JAR_DEFAULT=1 | |
| [1.41 INFO] (SDKD log:137) + JAR=/root/sdkd-2.0/sdkd_exec.jar | |
| [1.41 INFO] (SDKD log:137) + '[' '!' -e /root/sdkd-2.0/sdkd_exec.jar ']' | |
| [1.41 INFO] (SDKD log:137) + '[' '!' -e /root/sdkd-2.0/sdkd_exec.jar ']' | |
| [1.41 INFO] (SDKD log:137) + java -ea -jar /root/sdkd-2.0/sdkd_exec.jar -l 8050 -shared 100 -sync true | |
| [1.49 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa | |
| [1.50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84] | |
| [1.50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84] | |
| [1.50 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity /root/.ssh/id_rsa | |
| [1.90 INFO] (NodeHost createSSH:157) SSH Initialized for http://172.23.106.87:8091 | |
| [1.90 INFO] (NodeHost createSSH:157) SSH Initialized for http://172.23.105.194:8091 | |
| [1.91 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0 | |
| [1.91 INFO] (NodeHost createSSH:157) SSH Initialized for http://172.23.105.174:8091 | |
| [1.92 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.105.174 | |
| [1.92 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.106.87 | |
| [1.92 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.105.194 | |
| [1.97 INFO] (NodeHost createSSH:157) SSH Initialized for http://172.23.106.88:8091 | |
| [1.97 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 172.23.106.88 | |
| [2.03 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@56634374 | |
| [2.03 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@11fdb4c7 | |
| [2.12 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"COMPONENTS":{"Core_VERSION":"couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71)","SDK_VERSION":"couchbase-java-client/2.2.0-SNAPSHOT (git: 2.1.2-71-g9891310) (Linux/2.6.32-358.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17)"},"CAPS":{"VIEWS":true,"OBS":true,"DS_SHARED":true,"CANCEL":true,"CONTINUOUS":true,"TTL":true,"PREAMBLE":false}} | |
| [2.13 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@15214034 | |
| [2.35 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1f544c19 | |
| [2.35 DEBUG] (CBCluster setupNewCluster:294) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@6bc62575 | |
| [2.39 DEBUG] (CBCluster setupNewCluster:315) Adding node with KV, N1QL and 2i http://172.23.105.194:8091 | |
| [6.28 DEBUG] (CBCluster setupNewCluster:315) Adding node with KV, N1QL and 2i http://172.23.106.87:8091 | |
| [10.44 DEBUG] (CBCluster setupNewCluster:343) Adding node with KV http://172.23.106.88:8091 | |
| [14.68 INFO] (CBCluster setupNewCluster:356) All nodes added. Will rebalance | |
| [14.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [15.72 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete | |
| [15.73 DEBUG] (CBCluster setupServerGroups:241) Not creating any groups | |
| [15.73 INFO] (CBCluster setupMainBucket:228) Creating bucket default | |
| [15.74 INFO] (CBCluster setupMainBucket:230) Bucket creation submitted | |
| [21.39 INFO] (CBCluster waitForBucketReady:222) Bucket creation done | |
| [21.43 INFO] (RunContext run:143) Driver and cluster initialized | |
| [21.44 INFO] (RunContext call:167) Running scenario.. | |
| [21.44 INFO] (Scenario run:72) Starting RAMP phase | |
| [21.44 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["172.23.105.194",8091],["172.23.106.87",8091],["172.23.106.88",8091]], Username=default, ClusterCertificate=, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=172.23.105.194} | |
| [21.48 INFO] (SDKD log:137) Aug 31, 2015 4:10:34 PM com.couchbase.client.core.CouchbaseCore <init> | |
| [21.48 INFO] (SDKD log:137) INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='cacerts', sslKeystorePassword='couchbase', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, queryTimeout=75000, viewTimeout=15000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false} | |
| [22.01 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [22.01 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194 | |
| [22.46 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call | |
| [22.46 INFO] (SDKD log:137) INFO: Opened bucket default | |
| [22.50 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [22.50 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.174 | |
| [22.52 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [22.52 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87 | |
| [22.55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1 | |
| [22.55 DEBUG] (Handle sendMessageAsync:183) > CB_N1QL_CREATE_INDEX@101.2 => {NQIndexEngine=gsi, NQIndexType=primary, TimeRes=1, NQParamValues=n1ql,n1qldoc, NQDefaultIndexName=idx1, NQParam=tag,type} | |
| [22.55 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [22.56 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.88 | |
| [22.57 INFO] (SDKD log:137) Aug 31, 2015 4:10:35 PM com.couchbase.sdkd.cbclient.N1QLIndexCommandContext execIter | |
| [22.57 INFO] (SDKD log:137) INFO: n1ql index: null | |
| [31.06 DEBUG] (Handle receiveMessage:158) < CB_N1QL_CREATE_INDEX@101.2 => {"Timings":{"Step":1,"Windows":[{"Errors":{"0":1},"Min":1,"Avg":1,"Max":1,"Count":1}],"Base":1229686},"Summary":{"0":1}} | |
| [31.06 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3 | |
| [31.06 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["172.23.105.194",8091],["172.23.106.87",8091],["172.23.106.88",8091]], Username=default, ClusterCertificate=, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=172.23.105.194} | |
| [31.06 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.sdkd.server.SdkServer shutdown | |
| [31.06 INFO] (SDKD log:137) WARNING: Shutting Down.. | |
| [31.06 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.CouchbaseCore <init> | |
| [31.06 INFO] (SDKD log:137) INFO: CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='cacerts', sslKeystorePassword='couchbase', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-jvm-core/1.2.0-SNAPSHOT (git: 1.1.2-78-g0d14c71), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, queryTimeout=75000, viewTimeout=15000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false} | |
| [31.09 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [31.09 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194 | |
| [31.13 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.config.DefaultConfigurationProvider$6 call | |
| [31.13 INFO] (SDKD log:137) INFO: Opened bucket default | |
| [31.14 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [31.14 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.174 | |
| [31.14 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4 | |
| [31.14 DEBUG] (Handle sendMessageAsync:183) > CB_N1QL_QUERY@102.5 => {NQIndexEngine=gsi, NQIndexType=primary, NQScanConsistency=request_plus, NQParameterized=false, TimeRes=1, NQParamValues=n1ql,n1qldoc, NQDefaultIndexName=idx1, NQParam=tag,type, NQPrepared=true} | |
| [31.15 INFO] (Scenario run:83) RAMP phase started. Waiting for 10 seconds | |
| [31.15 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.sdkd.cbclient.N1QLQueryCommandContext execIter | |
| [31.15 INFO] (SDKD log:137) INFO: Running n1ql query: SELECT * from default WHERE tag = "n1ql" AND type = "n1qldoc"; | |
| [31.15 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.sdkd.cbclient.N1QLQueryCommandContext execIter | |
| [31.15 INFO] (SDKD log:137) INFO: n1ql drop_index: DROP INDEX `default`.`#primary` using gsi; | |
| [31.15 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [31.15 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.88 | |
| [31.15 INFO] (SDKD log:137) Aug 31, 2015 4:10:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [31.15 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87 | |
| [41.15 INFO] (Scenario run:86) Starting CHANGE phase | |
| [41.17 INFO] (CBCluster failoverNodes:680) Failing over <URI:172.23.105.194:8091,ns_1@172.23.105.194> | |
| [41.51 INFO] (CBCluster failoverNodes:680) Failing over <URI:172.23.106.87:8091,ns_1@172.23.106.87> | |
| [41.60 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.endpoint.AbstractGenericHandler exceptionCaught | |
| [41.60 INFO] (SDKD log:137) INFO: [/172.23.105.194:8093][QueryEndpoint]: Connection reset by peer: Connection reset by peer | |
| [41.60 INFO] (SDKD log:137) Exception in thread "SDK Handle-102: CMD [CB_N1QL_QUERY] " com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight. | |
| [41.61 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1 | |
| [41.61 INFO] (SDKD log:137) at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:339) | |
| at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:311) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) | |
| at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) | |
| at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:137) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) | |
| at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) | |
| at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) | |
| at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:834) | |
| at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87) | |
| at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) | |
| at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) | |
| at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [41.61 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete | |
| [41.61 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: | |
| [41.61 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093 | |
| [41.61 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3 | |
| [41.61 INFO] (SDKD log:137) at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) | |
| at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) | |
| at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) | |
| at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) | |
| at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) | |
| at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) | |
| at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [41.62 INFO] (SDKD log:137) | |
| [41.62 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete | |
| [41.62 INFO] (SDKD log:137) WARNING: [/172.23.105.194:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: | |
| [41.62 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.105.194:8093 | |
| [41.62 INFO] (SDKD log:137) +++ Received exception of ID 2 | |
| [41.62 INFO] (SDKD log:137) | |
| [41.62 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [41.62 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.105.194 | |
| [41.63 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [41.63 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.105.194 | |
| [41.79 INFO] (FailoverAction change:104) Nodes failed over. Requested sleep for 45 seconds | |
| [41.87 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete | |
| [41.87 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: | |
| [41.87 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093 | |
| [41.87 INFO] (SDKD log:137) +++ Received exception of ID 2 | |
| [41.87 INFO] (SDKD log:137) | |
| [41.87 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.endpoint.AbstractEndpoint$2 operationComplete | |
| [41.87 INFO] (SDKD log:137) WARNING: [/172.23.106.87:8093][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: | |
| [41.87 INFO] (SDKD log:137) java.net.ConnectException: Connection refused: /172.23.106.87:8093 | |
| [41.87 INFO] (SDKD log:137) +++ Received exception of ID 2 | |
| [41.87 INFO] (SDKD log:137) | |
| [41.87 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [41.87 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.106.87 | |
| [41.88 INFO] (SDKD log:137) Aug 31, 2015 4:10:54 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [41.88 INFO] (SDKD log:137) INFO: Disconnected from Node 172.23.106.87 | |
| [86.79 INFO] (FailoverAction doNextAction:68) Readding nodes after failover | |
| [86.85 INFO] (FailoverAction doNextAction:85) Rebalancing nodes after failover | |
| [86.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [87.90 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [88.92 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [89.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [90.94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [91.10 INFO] (SDKD log:137) Aug 31, 2015 4:11:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [91.10 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194 | |
| [91.10 INFO] (SDKD log:137) Aug 31, 2015 4:11:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [91.10 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87 | |
| [91.53 INFO] (SDKD log:137) Aug 31, 2015 4:11:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [91.53 INFO] (SDKD log:137) INFO: Connected to Node 172.23.106.87 | |
| [91.53 INFO] (SDKD log:137) Aug 31, 2015 4:11:44 PM com.couchbase.client.core.node.CouchbaseNode$1 call | |
| [91.53 INFO] (SDKD log:137) INFO: Connected to Node 172.23.105.194 | |
| [91.95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [92.97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [94.00 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [95.01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% | |
| [96.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3% | |
| [97.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 6% | |
| [98.04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 11% | |
| [99.06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15% | |
| [100.08 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 19% | |
| [101.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 23% | |
| [102.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 28% | |
| [103.11 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32% | |
| [104.12 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 36% | |
| [105.12 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 41% | |
| [106.14 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 45% | |
| [107.14 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 50% | |
| [108.15 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 54% | |
| [109.16 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 58% | |
| [110.16 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 63% | |
| [111.17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 67% | |
| [112.19 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 69% | |
| [113.20 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 72% | |
| [114.21 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 74% | |
| [115.21 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77% | |
| [116.22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79% | |
| [117.23 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82% | |
| [118.24 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 84% | |
| [119.25 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87% | |
| [120.25 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89% | |
| [121.26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92% | |
| [122.26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 94% | |
| [123.27 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 97% | |
| [124.28 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 99% | |
| [125.28 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete | |
| [125.28 INFO] (Scenario run:97) CHANGE phase done | |
| [125.28 INFO] (Scenario run:103) Starting REBOUND for 90 seconds | |
| [215.28 INFO] (Scenario run:105) REBOUND done. Will collect results | |
| [215.28 DEBUG] (Handle sendMessageAsync:183) > CANCEL@102.5 | |
| [215.29 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.SdkServer shutdown | |
| [215.29 INFO] (SDKD log:137) WARNING: Shutting Down.. | |
| [215.29 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.SdkServer shutdown | |
| [215.29 INFO] (SDKD log:137) WARNING: Shutting Down.. | |
| [215.29 DEBUG] (Handle receiveMessage:158) < CANCEL@102.5 | |
| [215.30 ERROR] (BRun maybePropagate:239) Suppressing driver-related error | |
| com.couchbase.sdkdclient.context.HarnessException: java.io.IOException: Connection Closed | |
| at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:115) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:169) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:165) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_25] | |
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_25] | |
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_25] | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_25] | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_25] | |
| at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_25] | |
| Caused by: java.io.IOException: Connection Closed | |
| at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:154) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.workload.Workload.collect(Workload.java:164) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.workload.WorkloadGroup.collect(WorkloadGroup.java:73) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:113) ~[sdkdclient-1.0-SNAPSHOT.jar:na] | |
| ... 8 common frames omitted | |
| [215.30 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6 | |
| [215.30 DEBUG] (Handle sendMessageAsync:183) > GOODBYE@0.7 | |
| [215.30 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@100.8 | |
| [215.31 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.ControlServer run | |
| [215.31 INFO] (SDKD log:137) INFO: ioLoop returned false. Returning.. | |
| [215.31 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.SdkServer shutdown | |
| [215.31 INFO] (SDKD log:137) WARNING: Shutting Down.. | |
| [215.31 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.SdkServer shutdown | |
| [215.31 INFO] (SDKD log:137) WARNING: Shutting Down.. | |
| [215.31 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.server.ControlServer run | |
| [215.31 INFO] (SDKD log:137) INFO: Cleaned up all children. Now returning for real.. | |
| [215.31 INFO] (SDKD log:137) Aug 31, 2015 4:13:48 PM com.couchbase.sdkd.Daemon main | |
| [215.31 INFO] (SDKD log:137) INFO: ControlServer returned. Exiting program | |
| [215.31 INFO] (HostPortDriver stop:123) SDKD is not yet terminated. Terminating.. | |
| [215.31 INFO] (HostPortDriver terminate:44) Waiting for process termination.. | |
| [215.67 ERROR] (HostPortDriver stop:135) SDKD terminated with error 143 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment