Skip to content

Instantly share code, notes, and snippets.

@tleyden
Created May 22, 2015 16:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tleyden/6b903e40cf87b5dd4ed8 to your computer and use it in GitHub Desktop.
Save tleyden/6b903e40cf87b5dd4ed8 to your computer and use it in GitHub Desktop.
core@ip-10-109-157-208 ~ $ sudo docker run --net=host tleyden5iwx/couchbase-cluster-go update-wrapper couchbase-fleet launch-cbs --version 3.0.1 --num-nodes 2 --userpass "user:passw0rd"
2015/05/22 16:35:55 Update-Wrapper: updating to latest code
github.com/tleyden/couchbase-cluster-go (download)
github.com/coreos/fleet (download)
github.com/coreos/go-systemd (download)
github.com/tleyden/go-etcd (download)
github.com/docopt/docopt-go (download)
github.com/tleyden/couchbase-cluster-go
github.com/tleyden/couchbase-cluster-go/cmd/couchbase-cluster
github.com/tleyden/couchbase-cluster-go/cmd/couchbase-fleet
github.com/tleyden/couchbase-cluster-go/cmd/experiment
github.com/tleyden/couchbase-cluster-go/cmd/sync-gw-cluster
github.com/tleyden/couchbase-cluster-go/cmd/sync-gw-config
2015/05/22 16:36:02 remainingargs: [launch-cbs --version 3.0.1 --num-nodes 2 --userpass user:passw0rd], target: couchbase-fleet
2015/05/22 16:36:02 Invoking target: couchbase-fleet with args: [launch-cbs --version 3.0.1 --num-nodes 2 --userpass user:passw0rd]
2015/05/22 16:36:02 Connect to etcd on localhost
2015/05/22 16:36:02 verifyEnoughMachinesAvailable()
2015/05/22 16:36:02 /verifyEnoughMachinesAvailable()
2015/05/22 16:36:02 Generating node from data/couchbase_node@.service.template with params: {CB_VERSION:community-3.0.1 CONTAINER_TAG:latest}
2015/05/22 16:36:02 Couchbase node fleet unit json: {"options":[{"Section":"Unit","Name":"Description","Value":"couchbase_node"},{"Section":"Unit","Name":"After","Value":"docker.service"},{"Section":"Unit","Name":"Requires","Value":"docker.service"},{"Section":"Unit","Name":"After","Value":"etcd.service"},{"Section":"Unit","Name":"Requires","Value":"etcd.service"},{"Section":"Service","Name":"TimeoutStartSec","Value":"0"},{"Section":"Service","Name":"TimeoutStopSec","Value":"0"},{"Section":"Service","Name":"EnvironmentFile","Value":"/etc/environment"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker kill couchbase"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker rm couchbase"},{"Section":"Service","Name":"ExecStartPre","Value":"/usr/bin/docker pull couchbase/server:community-3.0.1"},{"Section":"Service","Name":"ExecStartPre","Value":"/usr/bin/docker pull tleyden5iwx/couchbase-cluster-go:latest"},{"Section":"Service","Name":"ExecStart","Value":"/bin/bash -c '/usr/bin/docker run --name couchbase -v /opt/couchbase/var:/opt/couchbase/var --net=host couchbase/server:community-3.0.1'"},{"Section":"Service","Name":"ExecStop","Value":"/bin/bash -c '/usr/bin/docker run --net=host tleyden5iwx/couchbase-cluster-go:latest update-wrapper couchbase-cluster remove-and-rebalance --local-ip $COREOS_PRIVATE_IPV4; sudo docker stop couchbase'"},{"Section":"X-Fleet","Name":"Conflicts","Value":"couchbase_node*.service"}],"desiredState":"launched"}
2015/05/22 16:36:02 Launch fleet unit couchbase_node (1)
2015/05/22 16:36:03 response body:
2015/05/22 16:36:03 Generating sidekick from data/couchbase_sidekick@.service.template with params: {CB_VERSION:community-3.0.1 CONTAINER_TAG:latest UNIT_NUMBER:1}
2015/05/22 16:36:03 Couchbase sidekick fleet unit json: {"options":[{"Section":"Unit","Name":"Description","Value":"couchbase_sidekick"},{"Section":"Unit","Name":"After","Value":"docker.service"},{"Section":"Unit","Name":"Requires","Value":"docker.service"},{"Section":"Unit","Name":"After","Value":"etcd.service"},{"Section":"Unit","Name":"Requires","Value":"etcd.service"},{"Section":"Unit","Name":"After","Value":"fleet.service"},{"Section":"Unit","Name":"Requires","Value":"fleet.service"},{"Section":"Unit","Name":"BindsTo","Value":"couchbase_node@1.service"},{"Section":"Unit","Name":"After","Value":"couchbase_node@1.service"},{"Section":"Service","Name":"TimeoutStartSec","Value":"0"},{"Section":"Service","Name":"EnvironmentFile","Value":"/etc/environment"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker kill couchbase-sidekick"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker rm couchbase-sidekick"},{"Section":"Service","Name":"ExecStartPre","Value":"/usr/bin/docker pull tleyden5iwx/couchbase-cluster-go:latest"},{"Section":"Service","Name":"ExecStart","Value":"/bin/bash -c '/usr/bin/docker run --name couchbase-sidekick --net=host tleyden5iwx/couchbase-cluster-go:latest update-wrapper couchbase-cluster start-couchbase-sidekick --local-ip=$COREOS_PRIVATE_IPV4'"},{"Section":"Service","Name":"ExecStop","Value":"/usr/bin/docker stop couchbase-sidekick"},{"Section":"X-Fleet","Name":"MachineOf","Value":"couchbase_node@1.service"}],"desiredState":"launched"}
2015/05/22 16:36:03 Launch fleet unit couchbase_sidekick (1)
2015/05/22 16:36:03 response body:
2015/05/22 16:36:03 Launch fleet unit couchbase_node (2)
2015/05/22 16:36:03 response body:
2015/05/22 16:36:03 Generating sidekick from data/couchbase_sidekick@.service.template with params: {CB_VERSION:community-3.0.1 CONTAINER_TAG:latest UNIT_NUMBER:2}
2015/05/22 16:36:03 Couchbase sidekick fleet unit json: {"options":[{"Section":"Unit","Name":"Description","Value":"couchbase_sidekick"},{"Section":"Unit","Name":"After","Value":"docker.service"},{"Section":"Unit","Name":"Requires","Value":"docker.service"},{"Section":"Unit","Name":"After","Value":"etcd.service"},{"Section":"Unit","Name":"Requires","Value":"etcd.service"},{"Section":"Unit","Name":"After","Value":"fleet.service"},{"Section":"Unit","Name":"Requires","Value":"fleet.service"},{"Section":"Unit","Name":"BindsTo","Value":"couchbase_node@2.service"},{"Section":"Unit","Name":"After","Value":"couchbase_node@2.service"},{"Section":"Service","Name":"TimeoutStartSec","Value":"0"},{"Section":"Service","Name":"EnvironmentFile","Value":"/etc/environment"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker kill couchbase-sidekick"},{"Section":"Service","Name":"ExecStartPre","Value":"-/usr/bin/docker rm couchbase-sidekick"},{"Section":"Service","Name":"ExecStartPre","Value":"/usr/bin/docker pull tleyden5iwx/couchbase-cluster-go:latest"},{"Section":"Service","Name":"ExecStart","Value":"/bin/bash -c '/usr/bin/docker run --name couchbase-sidekick --net=host tleyden5iwx/couchbase-cluster-go:latest update-wrapper couchbase-cluster start-couchbase-sidekick --local-ip=$COREOS_PRIVATE_IPV4'"},{"Section":"Service","Name":"ExecStop","Value":"/usr/bin/docker stop couchbase-sidekick"},{"Section":"X-Fleet","Name":"MachineOf","Value":"couchbase_node@2.service"}],"desiredState":"launched"}
2015/05/22 16:36:03 Launch fleet unit couchbase_sidekick (2)
2015/05/22 16:36:03 response body:
2015/05/22 16:36:03 Waiting for cluster to be up ..
2015/05/22 16:36:03 Connect to etcd on localhost
2015/05/22 16:36:03 FindLiveNode returned err: Error getting key. Err: 100: Key not found (/couchbase.com/couchbase-node-state) [3959] or empty ip
2015/05/22 16:36:03 Sleeping 10 seconds
2015/05/22 16:36:13 FindLiveNode returned err: Error getting key. Err: 100: Key not found (/couchbase.com/couchbase-node-state) [3989] or empty ip
2015/05/22 16:36:13 Sleeping 20 seconds
2015/05/22 16:36:33 FindLiveNode returned err: Error getting key. Err: 100: Key not found (/couchbase.com/couchbase-node-state) [4032] or empty ip
2015/05/22 16:36:33 Sleeping 30 seconds
2015/05/22 16:37:03 Couchbase node ip: 10.203.169.73
2015/05/22 16:37:03 Verifying REST service at http://10.203.169.73:8091/ to be up
2015/05/22 16:37:03 Connecting to liveNodeIp: 10.203.169.73
2015/05/22 16:37:03 CheckNumNodesClusterHealthy()
2015/05/22 16:37:03 GetClusterNodes() called with: 10.203.169.73
2015/05/22 16:37:03 All cluster nodes appear to be healthy
2015/05/22 16:37:03 Connect to etcd on localhost
2015/05/22 16:37:03 Couchbase node ip: 10.203.169.73
2015/05/22 16:37:03 Verifying REST service at http://10.203.169.73:8091/ to be up
2015/05/22 16:37:03 WaitUntilNoRebalanceRunning()
2015/05/22 16:37:03 No rebalance running, sleeping 15s. (1/2)
2015/05/22 16:37:18 WaitUntilNoRebalanceRunning()
2015/05/22 16:37:18 Sleeping 30 seconds
2015/05/22 16:37:48 WaitUntilNoRebalanceRunning()
2015/05/22 16:37:48 No rebalance running, sleeping 15s. (2/2)
2015/05/22 16:38:03 No rebalance running after several checks
2015/05/22 16:38:03 Cluster is up!
$ fleetctl journal -f couchbase_node@1.service
-- Logs begin at Fri 2015-05-22 15:43:04 UTC. --
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: 58fc58bd0b89: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: 76b4cd18cbd3: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: bad22cd0c382: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: ea5434191fb1: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: 4587a814706c: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: 4587a814706c: Already exists
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: Digest: sha256:211ee38f73c6e9ecf94b17d43833657e268b32bfacddf3409b00fdd3a62125d2
May 22 16:36:17 ip-10-109-157-208.ec2.internal docker[6540]: Status: Image is up to date for tleyden5iwx/couchbase-cluster-go:latest
May 22 16:36:17 ip-10-109-157-208.ec2.internal systemd[1]: Started couchbase_node.
May 22 16:36:17 ip-10-109-157-208.ec2.internal bash[6557]: Starting Couchbase Server -- Web UI available at http://<ip>:8091
core@ip-10-109-157-208 ~ $ fleetctl journal -f couchbase_sidekick@1.service
-- Logs begin at Fri 2015-05-22 15:43:04 UTC. --
May 22 16:36:24 ip-10-109-157-208.ec2.internal systemd[1]: Started couchbase_sidekick.
May 22 16:36:24 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:24 Update-Wrapper: updating to latest code
May 22 16:36:25 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go (download)
May 22 16:36:25 ip-10-109-157-208.ec2.internal bash[6679]: github.com/coreos/fleet (download)
May 22 16:36:26 ip-10-109-157-208.ec2.internal bash[6679]: github.com/coreos/go-systemd (download)
May 22 16:36:26 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/go-etcd (download)
May 22 16:36:27 ip-10-109-157-208.ec2.internal bash[6679]: github.com/docopt/docopt-go (download)
May 22 16:36:28 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go
May 22 16:36:28 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go/cmd/couchbase-cluster
May 22 16:36:30 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go/cmd/couchbase-fleet
May 22 16:36:32 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go/cmd/experiment
May 22 16:36:35 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go/cmd/sync-gw-cluster
May 22 16:36:37 ip-10-109-157-208.ec2.internal bash[6679]: github.com/tleyden/couchbase-cluster-go/cmd/sync-gw-config
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 remainingargs: [start-couchbase-sidekick --local-ip=10.109.157.208], target: couchbase-cluster
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 Invoking target: couchbase-cluster with args: [start-couchbase-sidekick --local-ip=10.109.157.208]
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 args: map[remove-and-rebalance:false get-live-node-ip:false --help:false wait-until-running:false --etcd-servers:<nil> start-couchbase-sidekick:true --local-ip:10.109.157.208 --discover-local-ip:false]
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 localIp: 10.109.157.208
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 Connect to etcd on localhost
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 BecomeFirstClusterNode()
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 Key /couchbase.com/couchbase-node-state already exists
May 22 16:36:39 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:39 Got error Get http://10.109.157.208:8091/pools: dial tcp 10.109.157.208:8091: connection refused trying to fetch details. Assume that the cluster is not up yet, sleeping and will retry
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 Version: 3.0.1-1444-rel-community
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 JoinExistingCluster() called
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 Calling FindLiveNode()
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 Couchbase node ip: 10.203.169.73
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 Verifying REST service at http://10.203.169.73:8091/ to be up
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 liveNodeIp: 10.203.169.73
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 JoinLiveNode() called with 10.203.169.73
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 GetClusterNodes() called with: 10.203.169.73
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 No cluster node found for 10.109.157.208. Not retrying
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 WaitUntilInClusterAndHealthy() returned error: Unable to find node with hostname 10.109.157.208 in [map[thisNode:true clusterCompatibility:196608 uptime:27 memoryFree:0 mcdMemoryReserved:0 mcdMemoryAllocated:0 systemStats:map[cpu_utilization_rate:0 swap_total:0 swap_used:0 mem_total:0 mem_free:0] memoryTotal:0 clusterMembership:active recoveryType:none interestingStats:map[] status:warmup hostname:10.203.169.73:8091 version:3.0.1-1444-rel-community ports:map[proxy:11211 direct:11210] couchApiBase:http://10.203.169.73:8092/ otpCookie:ydavpxhjqhgtexdy otpNode:ns_1@127.0.0.1 os:x86_64-unknown-linux-gnu]]. Call AddNodeRetry()
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 AddNode()
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 AddNode posting to http://10.203.169.73:8091/controller/addNode with data: hostname=10.109.157.208&password=passw0rd&user=user
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 Using username/password pulled from etcd
May 22 16:36:49 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:36:49 POST to http://10.203.169.73:8091/controller/addNode
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 WaitUntilNoRebalanceRunning()
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 TriggerRebalance()
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 GetClusterNodes() called with: 10.203.169.73
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 OtpNodeList, otpNode: ns_1@10.109.157.208
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 OtpNodeList, otpNode: ns_1@10.203.169.73
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 TriggerRebalance otpNodeList: [ns_1@10.109.157.208 ns_1@10.203.169.73]
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 TriggerRebalance encoded form value: knownNodes=ns_1%4010.109.157.208%2Cns_1%4010.203.169.73
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 Using username/password pulled from etcd
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 POST to http://10.203.169.73:8091/controller/rebalance
May 22 16:37:07 ip-10-109-157-208.ec2.internal bash[6679]: 2015/05/22 16:37:07 EventLoop()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment