Skip to content

Instantly share code, notes, and snippets.

@EvanK

EvanK/00-deisctl-start-platform.txt Secret

Created October 29, 2014 18:06
Show Gist options
  • Save EvanK/d14303cc0decfe6955f9 to your computer and use it in GitHub Desktop.
Save EvanK/d14303cc0decfe6955f9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
00-deisctl-start-platform.txt
# ran `deisctl stop platform` beforehand, which completed successfully, then (@ 17:47:54 UTC) ran:
$ deisctl start platform
● ▴ ■
■ ● ▴ Starting Deis...
▴ ■ ●
Storage subsystem...
deis-store-monitor.service: activating/start-pre
deis-store-monitor.service: active/running
deis-store-daemon.service: activating/start-pre
deis-store-daemon.service: active/running
deis-store-metadata.service: active/running
deis-store-volume.service: activating/start-pre
deis-store-gateway.service: active/running
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
deis-store-volume.service: activating/auto-restart
deis-store-volume.service: activating/start-pre
^C $
Raw
01-journalctl--deis-store-monitor.txt
core@deis-staging-node1 ~ $ journalctl -u deis-store-monitor
-- Logs begin at Wed 2014-10-29 15:33:07 UTC, end at Wed 2014-10-29 18:03:21 UTC. --
Oct 29 16:22:32 deis-staging-node1 systemd[1]: Starting deis-store-monitor...
Oct 29 16:22:33 deis-staging-node1 sh[1295]: Unable to find image 'ubuntu-debootstrap:14.04' locally
Oct 29 16:22:34 deis-staging-node1 sh[1295]: ubuntu-debootstrap:14.04: The image you are pulling has been verified
Oct 29 16:23:11 deis-staging-node1 sh[1295]: Status: Downloaded newer image for ubuntu-debootstrap:14.04
Oct 29 16:23:12 deis-staging-node1 sh[1399]: Pulling repository deis/store-monitor
Oct 29 16:23:40 deis-staging-node1 sh[1399]: Status: Downloaded newer image for deis/store-monitor:v0.15.0
Oct 29 16:23:40 deis-staging-node1 systemd[1]: Started deis-store-monitor.
Oct 29 16:23:43 deis-staging-node1 sh[1521]: 2014-10-29 16:23:43.522439 7fdc484bb700 0 -- :/1000023 >> 10.60.61.35:6789/0 pipe(0x7fdc440261f0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fdc4402
Oct 29 16:23:43 deis-staging-node1 sh[1521]: 2014-10-29 16:23:43.522439 7fdc484bb700 0 -- :/1000023 >> 10.60.61.35:6789/0 pipe(0x7fdc440261f0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7fdc4402
Oct 29 16:23:47 deis-staging-node1 sh[1521]: got monmap epoch 2
Oct 29 16:23:47 deis-staging-node1 sh[1521]: creating /tmp/ceph.mon.keyring
Oct 29 16:23:47 deis-staging-node1 sh[1521]: importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
Oct 29 16:23:47 deis-staging-node1 sh[1521]: importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
Oct 29 16:23:47 deis-staging-node1 sh[1521]: ceph-mon: set fsid to 7b6294ce-aa3c-4bc5-9b37-06149965eaff
Oct 29 16:23:47 deis-staging-node1 sh[1521]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-deis-staging-node1 for mon.deis-staging-node1
Oct 29 16:23:47 deis-staging-node1 sh[1521]: 2014-10-29 16:23:47.867895 7f85b397c800 0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process ceph-mon, pid 1
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.026384 7f85b397c800 0 mon.deis-staging-node1 does not exist in monmap, will attempt to join an existing cluster
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.026477 7f85b397c800 0 using public_addr 10.60.61.35:6789/0 -> 10.60.61.35:6789/0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.026528 7f85b397c800 0 starting mon.deis-staging-node1 rank -1 at 10.60.61.35:6789/0 mon_data /var/lib/ceph/mon/ceph-
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.026664 7f85b397c800 1 mon.deis-staging-node1@-1(probing) e0 preinit fsid 7b6294ce-aa3c-4bc5-9b37-06149965eaff
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.026709 7f85b397c800 1 mon.deis-staging-node1@-1(probing) e0 initial_members deis-staging-node3, filtering seed monm
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.132871 7f85ad3ad700 1 mon.deis-staging-node1@-1(synchronizing).paxosservice(pgmap 1..2) refresh upgraded, format 0 -
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.132888 7f85ad3ad700 1 mon.deis-staging-node1@-1(synchronizing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.133896 7f85ad3ad700 0 mon.deis-staging-node1@-1(synchronizing).mds e1 print_map
Oct 29 16:23:48 deis-staging-node1 sh[1521]: epoch 1
Oct 29 16:23:48 deis-staging-node1 sh[1521]: flags 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: created 2014-10-29 16:23:35.068188
Oct 29 16:23:48 deis-staging-node1 sh[1521]: modified 2014-10-29 16:23:35.068207
Oct 29 16:23:48 deis-staging-node1 sh[1521]: tableserver 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: root 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: session_timeout 60
Oct 29 16:23:48 deis-staging-node1 sh[1521]: session_autoclose 300
Oct 29 16:23:48 deis-staging-node1 sh[1521]: max_file_size 1099511627776
Oct 29 16:23:48 deis-staging-node1 sh[1521]: last_failure 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: last_failure_osd_epoch 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:23:48 deis-staging-node1 sh[1521]: max_mds 1
Oct 29 16:23:48 deis-staging-node1 sh[1521]: in
Oct 29 16:23:48 deis-staging-node1 sh[1521]: up {}
Oct 29 16:23:48 deis-staging-node1 sh[1521]: failed
Oct 29 16:23:48 deis-staging-node1 sh[1521]: stopped
Oct 29 16:23:48 deis-staging-node1 sh[1521]: data_pools 0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: metadata_pool 1
Oct 29 16:23:48 deis-staging-node1 sh[1521]: inline_data disabled
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.134004 7f85ad3ad700 1 mon.deis-staging-node1@-1(synchronizing).osd e1 e1: 0 osds: 0 up, 0 in
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.168585 7f85ad3ad700 0 mon.deis-staging-node1@-1(synchronizing).osd e1 crush map has features 1107558400, adjusting m
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.168656 7f85ad3ad700 0 mon.deis-staging-node1@-1(synchronizing).osd e1 crush map has features 1107558400, adjusting m
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.168675 7f85ad3ad700 0 mon.deis-staging-node1@-1(synchronizing).osd e1 crush map has features 1107558400, adjusting m
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.168693 7f85ad3ad700 0 mon.deis-staging-node1@-1(synchronizing).osd e1 crush map has features 1107558400, adjusting m
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.169083 7f85ad3ad700 1 mon.deis-staging-node1@-1(synchronizing).paxosservice(auth 1..2) refresh upgraded, format 0 ->
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.265426 7f85ab9a8700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x1cef680 sd=19 :40244 s=2 pgs=5 cs=1 l=0 c=0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.266500 7f85ab8a7700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x1cf0a80 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.266630 7f85ab8a7700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x1cf0a80 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.287045 7f85abaa9700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1ceea00 sd=20 :53419 s=2 pgs=7 cs=1 l=0 c=0
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.288881 7f85ac3ab700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cf0d00 sd=19 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:48 deis-staging-node1 sh[1521]: 2014-10-29 16:23:48.289006 7f85ac3ab700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cf0d00 sd=19 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:50 deis-staging-node1 sh[1521]: 2014-10-29 16:23:50.170332 7f85ad3ad700 0 mon.deis-staging-node1@-1(probing) e3 my rank is now 0 (was -1)
Oct 29 16:23:50 deis-staging-node1 sh[1521]: 2014-10-29 16:23:50.171625 7f85ab8a7700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cee780 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:50 deis-staging-node1 sh[1521]: 2014-10-29 16:23:50.171642 7f85ab8a7700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cee780 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 16:23:50 deis-staging-node1 sh[1521]: 2014-10-29 16:23:50.173200 7f85ad3ad700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 16:23:50 deis-staging-node1 sh[1521]: 2014-10-29 16:23:50.173289 7f85ad3ad700 1 mon.deis-staging-node1@0(electing).elector(1) init, last seen epoch 1
Oct 29 16:23:55 deis-staging-node1 sh[1521]: 2014-10-29 16:23:55.174505 7f85ad3ad700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 16:23:55 deis-staging-node1 sh[1521]: 2014-10-29 16:23:55.174555 7f85ad3ad700 1 mon.deis-staging-node1@0(electing).elector(1) init, last seen epoch 1
Oct 29 16:23:55 deis-staging-node1 sh[1521]: 2014-10-29 16:23:55.590010 7f85ad3ad700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 16:23:55 deis-staging-node1 sh[1521]: 2014-10-29 16:23:55.590066 7f85ad3ad700 1 mon.deis-staging-node1@0(electing).elector(7) init, last seen epoch 7
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.047986 7f85ad3ad700 0 log [INF] : mon.deis-staging-node1@0 won leader election with quorum 0,1,2
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.615450 7f85ad3ad700 0 log [INF] : monmap e3: 3 mons at {deis-staging-node1=10.60.61.35:6789/0,deis-staging-node2=10.
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.615611 7f85ad3ad700 0 log [INF] : pgmap v2: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.615750 7f85ad3ad700 0 log [INF] : mdsmap e1: 0/0/1 up
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.615855 7f85ad3ad700 0 log [INF] : osdmap e1: 0 osds: 0 up, 0 in
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.847011 7f85ad3ad700 0 log [WRN] : message from mon.2 was stamped 0.054715s in the future, clocks not synchronized
Oct 29 16:23:56 deis-staging-node1 sh[1521]: 2014-10-29 16:23:56.847418 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd create"} v 0) v
Oct 29 16:23:57 deis-staging-node1 sh[1521]: 2014-10-29 16:23:57.002200 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd create"} v 0) v
Oct 29 16:23:57 deis-staging-node1 sh[1521]: 2014-10-29 16:23:57.067463 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e2 e2: 1 osds: 0 up, 0 in
Oct 29 16:23:57 deis-staging-node1 sh[1521]: 2014-10-29 16:23:57.103359 7f85ad3ad700 0 log [INF] : osdmap e2: 1 osds: 0 up, 0 in
Oct 29 16:23:57 deis-staging-node1 sh[1521]: 2014-10-29 16:23:57.388715 7f85ad3ad700 0 log [INF] : pgmap v3: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:23:59 deis-staging-node1 sh[1521]: 2014-10-29 16:23:59.553103 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e3 e3: 2 osds: 0 up, 0 in
Oct 29 16:23:59 deis-staging-node1 sh[1521]: 2014-10-29 16:23:59.850283 7f85ad3ad700 0 log [INF] : osdmap e3: 2 osds: 0 up, 0 in
Oct 29 16:24:02 deis-staging-node1 sh[1521]: 2014-10-29 16:24:01.222728 7f85ad3ad700 0 log [INF] : pgmap v4: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:03 deis-staging-node1 sh[1521]: 2014-10-29 16:24:03.332829 7f85ad3ad700 0 log [WRN] : message from mon.2 was stamped 0.052543s in the future, clocks not synchronized
Oct 29 16:24:03 deis-staging-node1 sh[1521]: 2014-10-29 16:24:03.450065 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd create"} v 0) v
Oct 29 16:24:05 deis-staging-node1 sh[1521]: 2014-10-29 16:24:05.334389 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:06 deis-staging-node1 sh[1521]: 2014-10-29 16:24:06.061757 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e4 e4: 3 osds: 0 up, 0 in
Oct 29 16:24:06 deis-staging-node1 sh[1521]: 2014-10-29 16:24:06.148902 7f85ad3ad700 0 log [INF] : osdmap e4: 3 osds: 0 up, 0 in
Oct 29 16:24:06 deis-staging-node1 sh[1521]: 2014-10-29 16:24:06.722921 7f85ab4a3700 0 -- 10.60.61.35:6789/0 >> 10.60.61.35:0/3403772085 pipe(0x1e3ed00 sd=23 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:24:07 deis-staging-node1 sh[1521]: 2014-10-29 16:24:07.390530 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:07 deis-staging-node1 sh[1521]: 2014-10-29 16:24:07.390582 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:08 deis-staging-node1 sh[1521]: 2014-10-29 16:24:08.432932 7f85ad3ad700 0 log [INF] : pgmap v5: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:10 deis-staging-node1 sh[1521]: 2014-10-29 16:24:10.285195 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:10 deis-staging-node1 sh[1521]: 2014-10-29 16:24:10.285322 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:11 deis-staging-node1 sh[1521]: 2014-10-29 16:24:11.431797 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd crush add", "ar
Oct 29 16:24:11 deis-staging-node1 sh[1521]: 2014-10-29 16:24:11.432039 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd crush add", "ar
Oct 29 16:24:12 deis-staging-node1 sh[1521]: 2014-10-29 16:24:12.286854 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e5 e5: 3 osds: 0 up, 0 in
Oct 29 16:24:12 deis-staging-node1 sh[1521]: 2014-10-29 16:24:12.396987 7f85ad3ad700 0 log [INF] : osdmap e5: 3 osds: 0 up, 0 in
Oct 29 16:24:12 deis-staging-node1 sh[1521]: 2014-10-29 16:24:12.697118 7f85ad3ad700 0 log [INF] : pgmap v6: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:13 deis-staging-node1 sh[1521]: 2014-10-29 16:24:13.895746 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:24:15 deis-staging-node1 sh[1521]: 2014-10-29 16:24:15.018327 7f85ad3ad700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "osd crush add", "ar
Oct 29 16:24:17 deis-staging-node1 sh[1521]: 2014-10-29 16:24:17.753563 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e6 e6: 3 osds: 1 up, 1 in
Oct 29 16:24:17 deis-staging-node1 sh[1521]: 2014-10-29 16:24:17.787672 7f85ad3ad700 0 log [INF] : osd.1 10.60.61.36:6801/1 boot
Oct 29 16:24:17 deis-staging-node1 sh[1521]: 2014-10-29 16:24:17.788973 7f85ad3ad700 0 log [INF] : osdmap e6: 3 osds: 1 up, 1 in
Oct 29 16:24:22 deis-staging-node1 sh[1521]: 2014-10-29 16:24:22.061634 7f85ad3ad700 0 log [INF] : pgmap v7: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:23 deis-staging-node1 sh[1521]: 2014-10-29 16:24:23.945022 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e7 e7: 3 osds: 1 up, 1 in
Oct 29 16:24:23 deis-staging-node1 sh[1521]: 2014-10-29 16:24:23.991357 7f85ad3ad700 0 log [INF] : osdmap e7: 3 osds: 1 up, 1 in
Oct 29 16:24:25 deis-staging-node1 sh[1521]: 2014-10-29 16:24:25.478555 7f85ad3ad700 0 log [INF] : pgmap v8: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:27 deis-staging-node1 sh[1521]: 2014-10-29 16:24:27.698172 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e8 e8: 3 osds: 2 up, 2 in
Oct 29 16:24:27 deis-staging-node1 sh[1521]: 2014-10-29 16:24:27.760153 7f85ad3ad700 0 log [INF] : osd.2 10.60.61.35:6801/1 boot
Oct 29 16:24:27 deis-staging-node1 sh[1521]: 2014-10-29 16:24:27.760331 7f85ad3ad700 0 log [INF] : osdmap e8: 3 osds: 2 up, 2 in
Oct 29 16:24:32 deis-staging-node1 sh[1521]: 2014-10-29 16:24:32.810863 7f85ad3ad700 0 log [INF] : pgmap v9: 192 pgs: 192 creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.456608 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).osd e9 e9: 3 osds: 3 up, 3 in
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.481525 7f85ad3ad700 0 log [INF] : osd.0 10.60.61.37:6801/1 boot
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.482014 7f85ad3ad700 0 log [INF] : osdmap e9: 3 osds: 3 up, 3 in
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.718141 7f85ad3ad700 0 mon.deis-staging-node1@0(leader).mds e2 print_map
Oct 29 16:24:34 deis-staging-node1 sh[1521]: epoch 2
Oct 29 16:24:34 deis-staging-node1 sh[1521]: flags 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: created 2014-10-29 16:23:35.068188
Oct 29 16:24:34 deis-staging-node1 sh[1521]: modified 2014-10-29 16:24:32.025187
Oct 29 16:24:34 deis-staging-node1 sh[1521]: tableserver 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: root 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: session_timeout 60
Oct 29 16:24:34 deis-staging-node1 sh[1521]: session_autoclose 300
Oct 29 16:24:34 deis-staging-node1 sh[1521]: max_file_size 1099511627776
Oct 29 16:24:34 deis-staging-node1 sh[1521]: last_failure 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: last_failure_osd_epoch 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:24:34 deis-staging-node1 sh[1521]: max_mds 1
Oct 29 16:24:34 deis-staging-node1 sh[1521]: in
Oct 29 16:24:34 deis-staging-node1 sh[1521]: up {}
Oct 29 16:24:34 deis-staging-node1 sh[1521]: failed
Oct 29 16:24:34 deis-staging-node1 sh[1521]: stopped
Oct 29 16:24:34 deis-staging-node1 sh[1521]: data_pools 0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: metadata_pool 1
Oct 29 16:24:34 deis-staging-node1 sh[1521]: inline_data disabled
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 4215: 10.60.61.37:6800/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.718459 7f85ad3ad700 0 log [INF] : mds.? 10.60.61.37:6800/1 up:boot
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.718586 7f85ad3ad700 1 mon.deis-staging-node1@0(leader).mds e2 adding standby 10.60.61.37:6800/1 as mds.0
Oct 29 16:24:34 deis-staging-node1 sh[1521]: 2014-10-29 16:24:34.718668 7f85ad3ad700 0 log [INF] : mdsmap e2: 0/0/1 up, 1 up:standby
Oct 29 16:24:35 deis-staging-node1 sh[1521]: 2014-10-29 16:24:35.185910 7f85ac3ab700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cefb80 sd=22 :53422 s=2 pgs=13 cs=1 l=0 c=
Oct 29 16:24:35 deis-staging-node1 sh[1521]: 2014-10-29 16:24:35.190618 7f85aae9d700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:0/1677540212 pipe(0x1f95480 sd=26 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:24:41 deis-staging-node1 sh[1521]: 2014-10-29 16:24:41.804210 7f85abaa9700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x1cf1700 sd=20 :40247 s=2 pgs=10 cs=1 l=0 c=
Oct 29 16:24:44 deis-staging-node1 sh[1521]: 2014-10-29 16:24:44.759547 7f85adbae700 1 mon.deis-staging-node1@0(leader).paxos(paxos updating c 1..49) accept timeout, calling fresh e
Oct 29 16:24:44 deis-staging-node1 sh[1521]: 2014-10-29 16:24:44.760683 7f85ab7a6700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x1cefb80 sd=22 :53422 s=1 pgs=13 cs=2 l=0 c=
Oct 29 16:24:44 deis-staging-node1 sh[1521]: 2014-10-29 16:24:44.760757 7f85b3977700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x1cf1700 sd=20 :40247 s=1 pgs=10 cs=2 l=0 c=
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.026964 7f85adbae700 0 mon.deis-staging-node1@0(probing).data_health(8) update_stats avail 2% total 5960684 used 5607
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027046 7f85adbae700 -1 mon.deis-staging-node1@0(probing).data_health(8) reached critical levels of available space on
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027048 7f85adbae700 0 ** Shutdown via Data Health Service **
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027068 7f85ac2aa700 -1 mon.deis-staging-node1@0(probing) e3 *** Got Signal Interrupt ***
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027075 7f85ac2aa700 1 mon.deis-staging-node1@0(probing) e3 shutdown
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027163 7f85ac2aa700 0 quorum service shutdown
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027165 7f85ac2aa700 0 mon.deis-staging-node1@0(shutdown).health(8) HealthMonitor::service_shutdown 1 services
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.027167 7f85ac2aa700 0 quorum service shutdown
Oct 29 16:24:48 deis-staging-node1 sh[1521]: 2014-10-29 16:24:48.034041 7f85aac9b700 0 -- 10.60.61.35:6789/0 >> 10.60.61.35:0/3403772085 pipe(0x2007700 sd=29 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:24:48 deis-staging-node1 docker[2102]: Error response from daemon: No such container: deis-store-monitor
Oct 29 16:24:48 deis-staging-node1 docker[2102]: 2014/10/29 16:24:48 Error: failed to remove one or more containers
Oct 29 16:37:29 deis-staging-node1 systemd[1]: Stopped deis-store-monitor.
Oct 29 16:38:09 deis-staging-node1 systemd[1]: Starting deis-store-monitor...
Oct 29 16:38:10 deis-staging-node1 systemd[1]: Started deis-store-monitor.
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.505670 7f860a96c800 0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process ceph-mon, pid 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.865093 7f860a96c800 0 starting mon.deis-staging-node1 rank 0 at 10.60.61.35:6789/0 mon_data /var/lib/ceph/mon/ceph-d
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.865631 7f860a96c800 1 mon.deis-staging-node1@-1(probing) e3 preinit fsid 7b6294ce-aa3c-4bc5-9b37-06149965eaff
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.866035 7f860a96c800 1 mon.deis-staging-node1@-1(probing).paxosservice(pgmap 1..9) refresh upgraded, format 0 -> 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.866211 7f860a96c800 1 mon.deis-staging-node1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.867532 7f860a96c800 0 mon.deis-staging-node1@-1(probing).mds e2 print_map
Oct 29 16:38:12 deis-staging-node1 sh[3350]: epoch 2
Oct 29 16:38:12 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:12 deis-staging-node1 sh[3350]: modified 2014-10-29 16:24:32.025187
Oct 29 16:38:12 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:12 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:12 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:12 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: last_failure_osd_epoch 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:12 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: in
Oct 29 16:38:12 deis-staging-node1 sh[3350]: up {}
Oct 29 16:38:12 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:12 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:12 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:12 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 4215: 10.60.61.37:6800/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.868159 7f860a96c800 0 mon.deis-staging-node1@-1(probing).osd e9 crush map has features 1107558400, adjusting msgr re
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.868312 7f860a96c800 0 mon.deis-staging-node1@-1(probing).osd e9 crush map has features 1107558400, adjusting msgr re
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.868477 7f860a96c800 0 mon.deis-staging-node1@-1(probing).osd e9 crush map has features 1107558400, adjusting msgr re
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.868626 7f860a96c800 0 mon.deis-staging-node1@-1(probing).osd e9 crush map has features 1107558400, adjusting msgr re
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.869163 7f860a96c800 1 mon.deis-staging-node1@-1(probing).paxosservice(auth 1..10) refresh upgraded, format 0 -> 1
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.870441 7f860a96c800 0 mon.deis-staging-node1@-1(probing) e3 my rank is now 0 (was -1)
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.874436 7f8604395700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.874479 7f8604395700 1 mon.deis-staging-node1@0(electing).elector(8) init, last seen epoch 8
Oct 29 16:38:12 deis-staging-node1 sh[3350]: 2014-10-29 16:38:12.919035 7f8604395700 0 log [INF] : mon.deis-staging-node1@0 won leader election with quorum 0,1,2
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.185883 7f8604395700 0 log [INF] : monmap e3: 3 mons at {deis-staging-node1=10.60.61.35:6789/0,deis-staging-node2=10.
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.234094 7f8604395700 0 log [INF] : pgmap v10: 192 pgs: 192 creating; 0 bytes data, 5475 MB used, 151 MB / 5820 MB ava
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.234154 7f8604395700 1 mon.deis-staging-node1@0(leader).mds e2 adding standby 10.60.61.37:6800/1 as mds.0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.234194 7f8604395700 0 log [INF] : mdsmap e2: 0/0/1 up, 1 up:standby
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.234242 7f8604395700 0 log [INF] : osdmap e9: 3 osds: 3 up, 3 in
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.323425 7f8604395700 0 log [INF] : pgmap v11: 192 pgs: 192 creating; 0 bytes data, 5475 MB used, 151 MB / 5820 MB ava
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.418622 7f8604395700 0 mon.deis-staging-node1@0(leader).mds e3 print_map
Oct 29 16:38:13 deis-staging-node1 sh[3350]: epoch 3
Oct 29 16:38:13 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:13 deis-staging-node1 sh[3350]: modified 2014-10-29 16:38:13.234171
Oct 29 16:38:13 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:13 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:13 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:13 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: last_failure_osd_epoch 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:13 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:13 deis-staging-node1 sh[3350]: in 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: up {0=4215}
Oct 29 16:38:13 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:13 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:13 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:13 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:13 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 4215: 10.60.61.37:6800/1 'deis-staging-node3' mds.0.1 up:creating seq 1
Oct 29 16:38:13 deis-staging-node1 sh[3350]: 2014-10-29 16:38:13.418747 7f8604395700 0 log [INF] : mdsmap e3: 1/1/1 up {0=deis-staging-node3=up:creating}
Oct 29 16:38:19 deis-staging-node1 sh[3350]: 2014-10-29 16:38:19.200657 7f860288f700 0 -- 10.60.61.35:6789/0 >> 10.60.61.35:0/3532184913 pipe(0x263c280 sd=21 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:38:34 deis-staging-node1 sh[3350]: 2014-10-29 16:38:34.809188 7f8604395700 0 mon.deis-staging-node1@0(leader).mds e4 print_map
Oct 29 16:38:35 deis-staging-node1 sh[3350]: epoch 4
Oct 29 16:38:35 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:35 deis-staging-node1 sh[3350]: modified 2014-10-29 16:38:32.933718
Oct 29 16:38:35 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:35 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:35 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:35 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: last_failure_osd_epoch 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:35 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:35 deis-staging-node1 sh[3350]: in 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: up {0=4215}
Oct 29 16:38:35 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:35 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:35 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:35 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:35 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:35 deis-staging-node1 sh[3350]: 4215: 10.60.61.37:6800/1 'deis-staging-node3' mds.0.1 up:creating seq 1 laggy since 2014-10-29 16:38:32.933668
Oct 29 16:38:35 deis-staging-node1 sh[3350]: 2014-10-29 16:38:34.809591 7f8604395700 0 log [INF] : mdsmap e4: 1/1/1 up {0=deis-staging-node3=up:creating(laggy or crashed)}
Oct 29 16:38:35 deis-staging-node1 sh[3350]: 2014-10-29 16:38:35.688609 7f8604395700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:38:35 deis-staging-node1 sh[3350]: 2014-10-29 16:38:35.688759 7f8604395700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:38:36 deis-staging-node1 sh[3350]: 2014-10-29 16:38:36.500702 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e10 e10: 3 osds: 2 up, 3 in
Oct 29 16:38:36 deis-staging-node1 sh[3350]: 2014-10-29 16:38:36.632153 7f8604395700 0 log [INF] : osdmap e10: 3 osds: 2 up, 3 in
Oct 29 16:38:36 deis-staging-node1 sh[3350]: 2014-10-29 16:38:36.728566 7f8604395700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:38:36 deis-staging-node1 sh[3350]: 2014-10-29 16:38:36.930794 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e11 e11: 3 osds: 3 up, 3 in
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.150341 7f8604395700 0 log [INF] : osd.0 10.60.61.37:6800/1 boot
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.150470 7f8604395700 0 log [INF] : osdmap e11: 3 osds: 3 up, 3 in
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.451794 7f8604395700 0 log [INF] : pgmap v12: 192 pgs: 192 creating; 0 bytes data, 5475 MB used, 151 MB / 5820 MB ava
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.648064 7f8604395700 0 mon.deis-staging-node1@0(leader).mds e5 print_map
Oct 29 16:38:37 deis-staging-node1 sh[3350]: epoch 5
Oct 29 16:38:37 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:37 deis-staging-node1 sh[3350]: modified 2014-10-29 16:38:37.226582
Oct 29 16:38:37 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:37 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:37 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:37 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: last_failure_osd_epoch 11
Oct 29 16:38:37 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:37 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:37 deis-staging-node1 sh[3350]: in
Oct 29 16:38:37 deis-staging-node1 sh[3350]: up {}
Oct 29 16:38:37 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:37 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:37 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:37 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 4308: 10.60.61.37:6805/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 4314: 10.60.61.36:6805/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.648416 7f8604395700 0 log [INF] : mds.? 10.60.61.36:6805/1 up:boot
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.648440 7f8604395700 0 log [INF] : mds.? 10.60.61.37:6805/1 up:boot
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.648496 7f8604395700 1 mon.deis-staging-node1@0(leader).mds e5 adding standby 10.60.61.36:6805/1 as mds.0
Oct 29 16:38:37 deis-staging-node1 sh[3350]: 2014-10-29 16:38:37.648538 7f8604395700 0 log [INF] : mdsmap e5: 0/0/1 up, 2 up:standby
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.194970 7f8604395700 0 log [INF] : pgmap v13: 192 pgs: 192 creating; 0 bytes data, 5475 MB used, 151 MB / 5820 MB ava
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.783535 7f8601a81700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:0/3437251985 pipe(0x263f980 sd=28 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.788980 7f8604395700 0 mon.deis-staging-node1@0(leader).mds e6 print_map
Oct 29 16:38:38 deis-staging-node1 sh[3350]: epoch 6
Oct 29 16:38:38 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:38 deis-staging-node1 sh[3350]: modified 2014-10-29 16:38:37.648512
Oct 29 16:38:38 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:38 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:38 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:38 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: last_failure_osd_epoch 11
Oct 29 16:38:38 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:38 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:38 deis-staging-node1 sh[3350]: in 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: up {0=4314}
Oct 29 16:38:38 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:38 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:38 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:38 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 4314: 10.60.61.36:6805/1 'deis-staging-node2' mds.0.2 up:creating seq 1
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 4308: 10.60.61.37:6805/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.789191 7f8604395700 0 log [INF] : mdsmap e6: 1/1/1 up {0=deis-staging-node2=up:creating}, 1 up:standby
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.798057 7f860187f700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:0/897590953 pipe(0x263f700 sd=29 :6789 s=0 pgs=0 cs=0 l=0
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.915090 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e12 e12: 3 osds: 1 up, 3 in
Oct 29 16:38:38 deis-staging-node1 sh[3350]: 2014-10-29 16:38:38.997908 7f8604395700 0 log [INF] : osdmap e12: 3 osds: 1 up, 3 in
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 2014-10-29 16:38:40.061818 7f8604395700 0 log [INF] : pgmap v14: 192 pgs: 192 creating; 0 bytes data, 5475 MB used, 151 MB / 5820 MB ava
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 2014-10-29 16:38:40.721121 7f8604395700 0 mon.deis-staging-node1@0(leader).mds e7 print_map
Oct 29 16:38:40 deis-staging-node1 sh[3350]: epoch 7
Oct 29 16:38:40 deis-staging-node1 sh[3350]: flags 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: created 2014-10-29 16:23:35.068188
Oct 29 16:38:40 deis-staging-node1 sh[3350]: modified 2014-10-29 16:38:39.749445
Oct 29 16:38:40 deis-staging-node1 sh[3350]: tableserver 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: root 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: session_timeout 60
Oct 29 16:38:40 deis-staging-node1 sh[3350]: session_autoclose 300
Oct 29 16:38:40 deis-staging-node1 sh[3350]: max_file_size 1099511627776
Oct 29 16:38:40 deis-staging-node1 sh[3350]: last_failure 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: last_failure_osd_epoch 11
Oct 29 16:38:40 deis-staging-node1 sh[3350]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:38:40 deis-staging-node1 sh[3350]: max_mds 1
Oct 29 16:38:40 deis-staging-node1 sh[3350]: in 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: up {0=4314}
Oct 29 16:38:40 deis-staging-node1 sh[3350]: failed
Oct 29 16:38:40 deis-staging-node1 sh[3350]: stopped
Oct 29 16:38:40 deis-staging-node1 sh[3350]: data_pools 0
Oct 29 16:38:40 deis-staging-node1 sh[3350]: metadata_pool 1
Oct 29 16:38:40 deis-staging-node1 sh[3350]: inline_data disabled
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 4314: 10.60.61.36:6805/1 'deis-staging-node2' mds.0.2 up:creating seq 1
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 4308: 10.60.61.37:6805/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 4317: 10.60.61.35:6805/1 'deis-staging-node1' mds.-1.0 up:standby seq 1
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 2014-10-29 16:38:40.724762 7f8604395700 0 log [INF] : mds.? 10.60.61.35:6805/1 up:boot
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 2014-10-29 16:38:40.724902 7f8604395700 0 log [INF] : mdsmap e7: 1/1/1 up {0=deis-staging-node2=up:creating}, 2 up:standby
Oct 29 16:38:40 deis-staging-node1 sh[3350]: 2014-10-29 16:38:40.929994 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e13 e13: 3 osds: 3 up, 3 in
Oct 29 16:38:41 deis-staging-node1 sh[3350]: 2014-10-29 16:38:41.256298 7f8604395700 0 log [INF] : osd.2 10.60.61.35:6800/1 boot
Oct 29 16:38:41 deis-staging-node1 sh[3350]: 2014-10-29 16:38:41.256550 7f8604395700 0 log [INF] : osd.1 10.60.61.36:6800/1 boot
Oct 29 16:38:41 deis-staging-node1 sh[3350]: 2014-10-29 16:38:41.256828 7f8604395700 0 log [INF] : osdmap e13: 3 osds: 3 up, 3 in
Oct 29 16:38:42 deis-staging-node1 sh[3350]: 2014-10-29 16:38:42.107756 7f8604395700 0 log [INF] : pgmap v15: 192 pgs: 129 creating, 12 peering, 42 down+peering, 9 creating+down+pee
Oct 29 16:38:42 deis-staging-node1 sh[3350]: 2014-10-29 16:38:42.563758 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e14 e14: 3 osds: 3 up, 3 in
Oct 29 16:38:42 deis-staging-node1 sh[3350]: 2014-10-29 16:38:42.629711 7f8604395700 0 log [INF] : osdmap e14: 3 osds: 3 up, 3 in
Oct 29 16:38:43 deis-staging-node1 sh[3350]: 2014-10-29 16:38:43.809675 7f8604395700 0 log [INF] : pgmap v16: 192 pgs: 118 peering, 23 creating+peering, 42 down+peering, 9 creating+
Oct 29 16:38:44 deis-staging-node1 sh[3350]: 2014-10-29 16:38:44.145423 7f8604b96700 1 mon.deis-staging-node1@0(leader).osd e14 New setting for CEPH_OSDMAP_FULL -- doing propose
Oct 29 16:38:45 deis-staging-node1 sh[3350]: 2014-10-29 16:38:45.990136 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e15 e15: 3 osds: 3 up, 3 in full
Oct 29 16:38:46 deis-staging-node1 sh[3350]: 2014-10-29 16:38:46.160624 7f8604395700 0 log [INF] : osdmap e15: 3 osds: 3 up, 3 in full
Oct 29 16:38:47 deis-staging-node1 sh[3350]: 2014-10-29 16:38:47.480145 7f8604395700 0 log [INF] : pgmap v17: 192 pgs: 118 peering, 23 creating+peering, 42 down+peering, 9 creating+
Oct 29 16:38:48 deis-staging-node1 sh[3350]: 2014-10-29 16:38:48.582507 7f8604395700 0 log [INF] : pgmap v18: 192 pgs: 55 inactive, 11 creating, 105 peering, 21 creating+peering; 0
Oct 29 16:38:49 deis-staging-node1 sh[3350]: 2014-10-29 16:38:49.864778 7f8604395700 0 log [INF] : pgmap v19: 192 pgs: 55 inactive, 11 creating, 105 peering, 21 creating+peering; 0
Oct 29 16:38:54 deis-staging-node1 sh[3350]: 2014-10-29 16:38:54.155968 7f8604395700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth add", "entity"
Oct 29 16:38:54 deis-staging-node1 sh[3350]: 2014-10-29 16:38:54.828101 7f8604395700 0 log [INF] : pgmap v20: 192 pgs: 55 inactive, 11 creating, 105 peering, 21 creating+peering; 0
Oct 29 16:38:56 deis-staging-node1 sh[3350]: 2014-10-29 16:38:56.302002 7f8604395700 1 mon.deis-staging-node1@0(leader).osd e16 e16: 3 osds: 3 up, 3 in full
Oct 29 16:38:56 deis-staging-node1 sh[3350]: 2014-10-29 16:38:56.439043 7f8604395700 0 log [INF] : osdmap e16: 3 osds: 3 up, 3 in full
Oct 29 16:38:56 deis-staging-node1 sh[3350]: 2014-10-29 16:38:56.737019 7f8604395700 0 log [INF] : pgmap v21: 192 pgs: 106 inactive, 23 creating, 54 peering, 9 creating+peering; 0 b
Oct 29 16:38:57 deis-staging-node1 sh[3350]: 2014-10-29 16:38:57.011851 7f8604395700 0 log [INF] : pgmap v22: 320 pgs: 106 inactive, 151 creating, 54 peering, 9 creating+peering; 0
Oct 29 16:38:58 deis-staging-node1 sh[3350]: 2014-10-29 16:38:58.633190 7f8604395700 0 log [INF] : pgmap v23: 320 pgs: 102 inactive, 151 creating, 54 peering, 9 creating+peering, 4
Oct 29 16:39:01 deis-staging-node1 sh[3350]: 2014-10-29 16:39:01.082064 7f8604395700 0 log [INF] : pgmap v24: 320 pgs: 156 inactive, 122 creating, 38 peering, 4 active+clean; 0 byte
Oct 29 16:39:03 deis-staging-node1 sh[3350]: 2014-10-29 16:39:03.448140 7f8604395700 0 log [INF] : pgmap v25: 320 pgs: 136 inactive, 71 creating, 7 active, 89 peering, 17 active+cle
Oct 29 16:39:05 deis-staging-node1 sh[3350]: 2014-10-29 16:39:05.407881 7f8604395700 0 log [INF] : pgmap v26: 320 pgs: 136 inactive, 32 creating, 7 active, 128 peering, 17 active+cl
Oct 29 16:39:08 deis-staging-node1 sh[3350]: 2014-10-29 16:39:08.109749 7f8604395700 0 log [INF] : pgmap v27: 320 pgs: 136 inactive, 32 creating, 128 peering, 24 active+clean; 570 b
Oct 29 16:39:12 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.411665 7f8604395700 0 log [INF] : pgmap v28: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:39:12 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.569221 7f8602990700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x23e8f00 sd=19 :56325 s=2 pgs=7 cs=1 l=0 c=0
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.660493 7f8602a91700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x23e9400 sd=20 :43152 s=2 pgs=5 cs=1 l=0 c=0
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870238 7f8604b96700 0 mon.deis-staging-node1@0(leader).data_health(10) update_stats avail 2% total 5960684 used 5611
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870393 7f8604b96700 -1 mon.deis-staging-node1@0(leader).data_health(10) reached critical levels of available space on
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870400 7f8604b96700 0 ** Shutdown via Data Health Service **
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870419 7f8603292700 -1 mon.deis-staging-node1@0(leader) e3 *** Got Signal Interrupt ***
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870426 7f8603292700 1 mon.deis-staging-node1@0(leader) e3 shutdown
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870475 7f8603292700 0 quorum service shutdown
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870480 7f8603292700 0 mon.deis-staging-node1@0(shutdown).health(10) HealthMonitor::service_shutdown 1 services
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870482 7f8603292700 0 quorum service shutdown
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870880 7f8603393700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x23e8f00 sd=19 :56325 s=1 pgs=7 cs=2 l=0 c=0
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.870910 7f860a94f700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x23e9400 sd=20 :43152 s=1 pgs=5 cs=2 l=0 c=0
Oct 29 16:39:13 deis-staging-node1 sh[3350]: 2014-10-29 16:39:12.876176 7f8601a81700 0 -- 10.60.61.35:6789/0 >> 10.60.61.35:0/3532184913 pipe(0x2659900 sd=31 :6789 s=0 pgs=0 cs=0 l=
Oct 29 16:39:13 deis-staging-node1 docker[3767]: Error response from daemon: No such container: deis-store-monitor
Oct 29 16:39:13 deis-staging-node1 docker[3767]: 2014/10/29 16:39:13 Error: failed to remove one or more containers
Oct 29 16:44:36 deis-staging-node1 systemd[1]: Stopped deis-store-monitor.
Oct 29 16:44:41 deis-staging-node1 systemd[1]: Starting deis-store-monitor...
Oct 29 16:44:42 deis-staging-node1 systemd[1]: Started deis-store-monitor.
Oct 29 16:44:44 deis-staging-node1 sh[4196]: 2014-10-29 16:44:44.740223 7f93dcb73800 0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process ceph-mon, pid 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.314284 7f93dcb73800 0 starting mon.deis-staging-node1 rank 0 at 10.60.61.35:6789/0 mon_data /var/lib/ceph/mon/ceph-d
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.314437 7f93dcb73800 1 mon.deis-staging-node1@-1(probing) e3 preinit fsid 7b6294ce-aa3c-4bc5-9b37-06149965eaff
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.315504 7f93dcb73800 1 mon.deis-staging-node1@-1(probing).paxosservice(pgmap 1..28) refresh upgraded, format 0 -> 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.315524 7f93dcb73800 1 mon.deis-staging-node1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.318240 7f93dcb73800 0 mon.deis-staging-node1@-1(probing).mds e7 print_map
Oct 29 16:44:45 deis-staging-node1 sh[4196]: epoch 7
Oct 29 16:44:45 deis-staging-node1 sh[4196]: flags 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: created 2014-10-29 16:23:35.068188
Oct 29 16:44:45 deis-staging-node1 sh[4196]: modified 2014-10-29 16:38:39.749445
Oct 29 16:44:45 deis-staging-node1 sh[4196]: tableserver 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: root 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: session_timeout 60
Oct 29 16:44:45 deis-staging-node1 sh[4196]: session_autoclose 300
Oct 29 16:44:45 deis-staging-node1 sh[4196]: max_file_size 1099511627776
Oct 29 16:44:45 deis-staging-node1 sh[4196]: last_failure 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: last_failure_osd_epoch 11
Oct 29 16:44:45 deis-staging-node1 sh[4196]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:44:45 deis-staging-node1 sh[4196]: max_mds 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: in 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: up {0=4314}
Oct 29 16:44:45 deis-staging-node1 sh[4196]: failed
Oct 29 16:44:45 deis-staging-node1 sh[4196]: stopped
Oct 29 16:44:45 deis-staging-node1 sh[4196]: data_pools 0
Oct 29 16:44:45 deis-staging-node1 sh[4196]: metadata_pool 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: inline_data disabled
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 4314: 10.60.61.36:6805/1 'deis-staging-node2' mds.0.2 up:creating seq 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 4308: 10.60.61.37:6805/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 4317: 10.60.61.35:6805/1 'deis-staging-node1' mds.-1.0 up:standby seq 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.318881 7f93dcb73800 0 mon.deis-staging-node1@-1(probing).osd e16 crush map has features 1107558400, adjusting msgr r
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.319032 7f93dcb73800 0 mon.deis-staging-node1@-1(probing).osd e16 crush map has features 1107558400, adjusting msgr r
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.319208 7f93dcb73800 0 mon.deis-staging-node1@-1(probing).osd e16 crush map has features 1107558400, adjusting msgr r
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.319370 7f93dcb73800 0 mon.deis-staging-node1@-1(probing).osd e16 crush map has features 1107558400, adjusting msgr r
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.320030 7f93dcb73800 1 mon.deis-staging-node1@-1(probing).paxosservice(auth 1..14) refresh upgraded, format 0 -> 1
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.320981 7f93dcb73800 0 mon.deis-staging-node1@-1(probing) e3 my rank is now 0 (was -1)
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.327132 7f93d65a5700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.327177 7f93d65a5700 1 mon.deis-staging-node1@0(electing).elector(10) init, last seen epoch 10
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.407681 7f93d65a5700 0 log [INF] : mon.deis-staging-node1@0 won leader election with quorum 0,1,2
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.671237 7f93d65a5700 0 log [INF] : monmap e3: 3 mons at {deis-staging-node1=10.60.61.35:6789/0,deis-staging-node2=10.
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.671319 7f93d65a5700 0 log [INF] : pgmap v28: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.671369 7f93d65a5700 0 log [INF] : mdsmap e7: 1/1/1 up {0=deis-staging-node2=up:creating}, 2 up:standby
Oct 29 16:44:45 deis-staging-node1 sh[4196]: 2014-10-29 16:44:45.671417 7f93d65a5700 0 log [INF] : osdmap e16: 3 osds: 3 up, 3 in full
Oct 29 16:45:07 deis-staging-node1 sh[4196]: 2014-10-29 16:45:07.987663 7f93d65a5700 1 mon.deis-staging-node1@0(leader).osd e17 e17: 3 osds: 3 up, 3 in full
Oct 29 16:45:08 deis-staging-node1 sh[4196]: 2014-10-29 16:45:08.259461 7f93d65a5700 0 log [INF] : osdmap e17: 3 osds: 3 up, 3 in full
Oct 29 16:45:09 deis-staging-node1 sh[4196]: 2014-10-29 16:45:09.247447 7f93d65a5700 0 mon.deis-staging-node1@0(leader).mds e8 print_map
Oct 29 16:45:09 deis-staging-node1 sh[4196]: epoch 8
Oct 29 16:45:09 deis-staging-node1 sh[4196]: flags 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: created 2014-10-29 16:23:35.068188
Oct 29 16:45:09 deis-staging-node1 sh[4196]: modified 2014-10-29 16:45:06.386200
Oct 29 16:45:09 deis-staging-node1 sh[4196]: tableserver 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: root 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: session_timeout 60
Oct 29 16:45:09 deis-staging-node1 sh[4196]: session_autoclose 300
Oct 29 16:45:09 deis-staging-node1 sh[4196]: max_file_size 1099511627776
Oct 29 16:45:09 deis-staging-node1 sh[4196]: last_failure 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: last_failure_osd_epoch 17
Oct 29 16:45:09 deis-staging-node1 sh[4196]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:45:09 deis-staging-node1 sh[4196]: max_mds 1
Oct 29 16:45:09 deis-staging-node1 sh[4196]: in 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: up {0=4317}
Oct 29 16:45:09 deis-staging-node1 sh[4196]: failed
Oct 29 16:45:09 deis-staging-node1 sh[4196]: stopped
Oct 29 16:45:09 deis-staging-node1 sh[4196]: data_pools 0
Oct 29 16:45:09 deis-staging-node1 sh[4196]: metadata_pool 1
Oct 29 16:45:09 deis-staging-node1 sh[4196]: inline_data disabled
Oct 29 16:45:09 deis-staging-node1 sh[4196]: 4317: 10.60.61.35:6805/1 'deis-staging-node1' mds.0.3 up:creating seq 1 laggy since 2014-10-29 16:45:05.368711
Oct 29 16:45:09 deis-staging-node1 sh[4196]: 2014-10-29 16:45:09.247610 7f93d65a5700 0 log [INF] : mdsmap e8: 1/1/1 up {0=deis-staging-node1=up:creating(laggy or crashed)}
Oct 29 16:45:10 deis-staging-node1 sh[4196]: 2014-10-29 16:45:10.196815 7f93d65a5700 0 log [INF] : pgmap v29: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:45:10 deis-staging-node1 sh[4196]: 2014-10-29 16:45:10.750355 7f93d65a5700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:45:11 deis-staging-node1 sh[4196]: 2014-10-29 16:45:11.171660 7f93d65a5700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:45:11 deis-staging-node1 sh[4196]: 2014-10-29 16:45:11.368930 7f93d65a5700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 16:45:12 deis-staging-node1 sh[4196]: 2014-10-29 16:45:12.883672 7f93d65a5700 1 mon.deis-staging-node1@0(leader).osd e18 e18: 3 osds: 3 up, 3 in full
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:12.990538 7f93d65a5700 0 log [INF] : osdmap e18: 3 osds: 3 up, 3 in full
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.751620 7f93d65a5700 0 mon.deis-staging-node1@0(leader).mds e9 print_map
Oct 29 16:45:13 deis-staging-node1 sh[4196]: epoch 9
Oct 29 16:45:13 deis-staging-node1 sh[4196]: flags 0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: created 2014-10-29 16:23:35.068188
Oct 29 16:45:13 deis-staging-node1 sh[4196]: modified 2014-10-29 16:45:12.105892
Oct 29 16:45:13 deis-staging-node1 sh[4196]: tableserver 0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: root 0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: session_timeout 60
Oct 29 16:45:13 deis-staging-node1 sh[4196]: session_autoclose 300
Oct 29 16:45:13 deis-staging-node1 sh[4196]: max_file_size 1099511627776
Oct 29 16:45:13 deis-staging-node1 sh[4196]: last_failure 0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: last_failure_osd_epoch 18
Oct 29 16:45:13 deis-staging-node1 sh[4196]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:45:13 deis-staging-node1 sh[4196]: max_mds 1
Oct 29 16:45:13 deis-staging-node1 sh[4196]: in
Oct 29 16:45:13 deis-staging-node1 sh[4196]: up {}
Oct 29 16:45:13 deis-staging-node1 sh[4196]: failed
Oct 29 16:45:13 deis-staging-node1 sh[4196]: stopped
Oct 29 16:45:13 deis-staging-node1 sh[4196]: data_pools 0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: metadata_pool 1
Oct 29 16:45:13 deis-staging-node1 sh[4196]: inline_data disabled
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 4503: 10.60.61.36:6808/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 4506: 10.60.61.37:6808/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 4509: 10.60.61.35:6805/1 'deis-staging-node1' mds.-1.0 up:standby seq 1
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.752022 7f93d65a5700 0 log [INF] : mds.? 10.60.61.36:6808/1 up:boot
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.752099 7f93d65a5700 0 log [INF] : mds.? 10.60.61.37:6808/1 up:boot
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.752167 7f93d65a5700 0 log [INF] : mds.? 10.60.61.35:6805/1 up:boot
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.752246 7f93d65a5700 1 mon.deis-staging-node1@0(leader).mds e9 adding standby 10.60.61.35:6805/1 as mds.0
Oct 29 16:45:13 deis-staging-node1 sh[4196]: 2014-10-29 16:45:13.752338 7f93d65a5700 0 log [INF] : mdsmap e9: 0/0/1 up, 3 up:standby
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 2014-10-29 16:45:14.323470 7f93d65a5700 0 log [INF] : pgmap v30: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 2014-10-29 16:45:14.619111 7f93d65a5700 0 mon.deis-staging-node1@0(leader).mds e10 print_map
Oct 29 16:45:14 deis-staging-node1 sh[4196]: epoch 10
Oct 29 16:45:14 deis-staging-node1 sh[4196]: flags 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: created 2014-10-29 16:23:35.068188
Oct 29 16:45:14 deis-staging-node1 sh[4196]: modified 2014-10-29 16:45:13.752298
Oct 29 16:45:14 deis-staging-node1 sh[4196]: tableserver 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: root 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: session_timeout 60
Oct 29 16:45:14 deis-staging-node1 sh[4196]: session_autoclose 300
Oct 29 16:45:14 deis-staging-node1 sh[4196]: max_file_size 1099511627776
Oct 29 16:45:14 deis-staging-node1 sh[4196]: last_failure 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: last_failure_osd_epoch 18
Oct 29 16:45:14 deis-staging-node1 sh[4196]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 16:45:14 deis-staging-node1 sh[4196]: max_mds 1
Oct 29 16:45:14 deis-staging-node1 sh[4196]: in 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: up {0=4509}
Oct 29 16:45:14 deis-staging-node1 sh[4196]: failed
Oct 29 16:45:14 deis-staging-node1 sh[4196]: stopped
Oct 29 16:45:14 deis-staging-node1 sh[4196]: data_pools 0
Oct 29 16:45:14 deis-staging-node1 sh[4196]: metadata_pool 1
Oct 29 16:45:14 deis-staging-node1 sh[4196]: inline_data disabled
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 4509: 10.60.61.35:6805/1 'deis-staging-node1' mds.0.4 up:creating seq 1
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 4503: 10.60.61.36:6808/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 4506: 10.60.61.37:6808/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 2014-10-29 16:45:14.619928 7f93d65a5700 0 log [INF] : mdsmap e10: 1/1/1 up {0=deis-staging-node1=up:creating}, 2 up:standby
Oct 29 16:45:14 deis-staging-node1 sh[4196]: 2014-10-29 16:45:14.904722 7f93d65a5700 1 mon.deis-staging-node1@0(leader).osd e19 e19: 3 osds: 3 up, 3 in full
Oct 29 16:45:15 deis-staging-node1 sh[4196]: 2014-10-29 16:45:15.006127 7f93d65a5700 0 log [INF] : osdmap e19: 3 osds: 3 up, 3 in full
Oct 29 16:45:15 deis-staging-node1 sh[4196]: 2014-10-29 16:45:15.302846 7f93d65a5700 0 log [INF] : pgmap v31: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:45:16 deis-staging-node1 sh[4196]: 2014-10-29 16:45:16.551100 7f93d65a5700 1 mon.deis-staging-node1@0(leader).osd e20 e20: 3 osds: 3 up, 3 in full
Oct 29 16:45:16 deis-staging-node1 sh[4196]: 2014-10-29 16:45:16.594705 7f93d65a5700 0 log [INF] : osdmap e20: 3 osds: 3 up, 3 in full
Oct 29 16:45:17 deis-staging-node1 sh[4196]: 2014-10-29 16:45:17.424314 7f93d65a5700 0 log [INF] : pgmap v32: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.186578 7f93d4ba0700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x2509180 sd=19 :57871 s=2 pgs=7 cs=1 l=0 c=0
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.193410 7f93d4ca1700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x2508280 sd=20 :44698 s=2 pgs=4 cs=1 l=0 c=0
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.320810 7f93d6da6700 0 mon.deis-staging-node1@0(leader).data_health(12) update_stats avail 2% total 5960684 used 5612
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.320956 7f93d6da6700 -1 mon.deis-staging-node1@0(leader).data_health(12) reached critical levels of available space on
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.320960 7f93d6da6700 0 ** Shutdown via Data Health Service **
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.320987 7f93d54a2700 -1 mon.deis-staging-node1@0(leader) e3 *** Got Signal Interrupt ***
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.321009 7f93d54a2700 1 mon.deis-staging-node1@0(leader) e3 shutdown
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.321035 7f93d54a2700 0 quorum service shutdown
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.321037 7f93d54a2700 0 mon.deis-staging-node1@0(shutdown).health(12) HealthMonitor::service_shutdown 1 services
Oct 29 16:45:45 deis-staging-node1 sh[4196]: 2014-10-29 16:45:45.321039 7f93d54a2700 0 quorum service shutdown
Oct 29 16:45:45 deis-staging-node1 docker[4611]: Error response from daemon: No such container: deis-store-monitor
Oct 29 16:45:45 deis-staging-node1 docker[4611]: 2014/10/29 16:45:45 Error: failed to remove one or more containers
Oct 29 17:47:36 deis-staging-node1 systemd[1]: Stopped deis-store-monitor.
Oct 29 17:47:47 deis-staging-node1 systemd[1]: Starting deis-store-monitor...
Oct 29 17:47:48 deis-staging-node1 systemd[1]: Started deis-store-monitor.
Oct 29 17:47:50 deis-staging-node1 sh[9812]: 2014-10-29 17:47:50.820271 7fbcc1725800 0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process ceph-mon, pid 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.178235 7fbcc1725800 0 starting mon.deis-staging-node1 rank 0 at 10.60.61.35:6789/0 mon_data /var/lib/ceph/mon/ceph-d
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.178458 7fbcc1725800 1 mon.deis-staging-node1@-1(probing) e3 preinit fsid 7b6294ce-aa3c-4bc5-9b37-06149965eaff
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.179501 7fbcc1725800 1 mon.deis-staging-node1@-1(probing).paxosservice(pgmap 1..32) refresh upgraded, format 0 -> 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.179536 7fbcc1725800 1 mon.deis-staging-node1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.182529 7fbcc1725800 0 mon.deis-staging-node1@-1(probing).mds e10 print_map
Oct 29 17:47:51 deis-staging-node1 sh[9812]: epoch 10
Oct 29 17:47:51 deis-staging-node1 sh[9812]: flags 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: created 2014-10-29 16:23:35.068188
Oct 29 17:47:51 deis-staging-node1 sh[9812]: modified 2014-10-29 16:45:13.752298
Oct 29 17:47:51 deis-staging-node1 sh[9812]: tableserver 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: root 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: session_timeout 60
Oct 29 17:47:51 deis-staging-node1 sh[9812]: session_autoclose 300
Oct 29 17:47:51 deis-staging-node1 sh[9812]: max_file_size 1099511627776
Oct 29 17:47:51 deis-staging-node1 sh[9812]: last_failure 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: last_failure_osd_epoch 18
Oct 29 17:47:51 deis-staging-node1 sh[9812]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 17:47:51 deis-staging-node1 sh[9812]: max_mds 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: in 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: up {0=4509}
Oct 29 17:47:51 deis-staging-node1 sh[9812]: failed
Oct 29 17:47:51 deis-staging-node1 sh[9812]: stopped
Oct 29 17:47:51 deis-staging-node1 sh[9812]: data_pools 0
Oct 29 17:47:51 deis-staging-node1 sh[9812]: metadata_pool 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: inline_data disabled
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 4509: 10.60.61.35:6805/1 'deis-staging-node1' mds.0.4 up:creating seq 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 4503: 10.60.61.36:6808/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 4506: 10.60.61.37:6808/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.182813 7fbcc1725800 0 mon.deis-staging-node1@-1(probing).osd e20 crush map has features 1107558400, adjusting msgr r
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.182839 7fbcc1725800 0 mon.deis-staging-node1@-1(probing).osd e20 crush map has features 1107558400, adjusting msgr r
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.182857 7fbcc1725800 0 mon.deis-staging-node1@-1(probing).osd e20 crush map has features 1107558400, adjusting msgr r
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.182890 7fbcc1725800 0 mon.deis-staging-node1@-1(probing).osd e20 crush map has features 1107558400, adjusting msgr r
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.185152 7fbcc1725800 1 mon.deis-staging-node1@-1(probing).paxosservice(auth 1..19) refresh upgraded, format 0 -> 1
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.185868 7fbcc1725800 0 mon.deis-staging-node1@-1(probing) e3 my rank is now 0 (was -1)
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.186605 7fbcb9952700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x35e5180 sd=20 :0 s=1 pgs=0 cs=0 l=0 c=0x35b
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.186981 7fbcc16c7700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x35e4280 sd=21 :0 s=1 pgs=0 cs=0 l=0 c=0x35a
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.296931 7fbcb9050700 0 -- 10.60.61.35:6789/0 >> 10.60.61.36:6789/0 pipe(0x35e5680 sd=22 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.301686 7fbcba954700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.301752 7fbcba954700 1 mon.deis-staging-node1@0(electing).elector(12) init, last seen epoch 12
Oct 29 17:47:51 deis-staging-node1 sh[9812]: 2014-10-29 17:47:51.312177 7fbcb8f4f700 0 -- 10.60.61.35:6789/0 >> 10.60.61.37:6789/0 pipe(0x35e6300 sd=22 :6789 s=0 pgs=0 cs=0 l=0 c=0x
Oct 29 17:47:56 deis-staging-node1 sh[9812]: 2014-10-29 17:47:56.405464 7fbcbb155700 0 log [INF] : mon.deis-staging-node1@0 won leader election with quorum 0,1
Oct 29 17:47:56 deis-staging-node1 sh[9812]: 2014-10-29 17:47:56.520414 7fbcba954700 0 log [INF] : monmap e3: 3 mons at {deis-staging-node1=10.60.61.35:6789/0,deis-staging-node2=10.
Oct 29 17:47:56 deis-staging-node1 sh[9812]: 2014-10-29 17:47:56.520461 7fbcba954700 0 log [INF] : pgmap v32: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 17:47:56 deis-staging-node1 sh[9812]: 2014-10-29 17:47:56.520508 7fbcba954700 0 log [INF] : mdsmap e10: 1/1/1 up {0=deis-staging-node1=up:creating}, 2 up:standby
Oct 29 17:47:56 deis-staging-node1 sh[9812]: 2014-10-29 17:47:56.520548 7fbcba954700 0 log [INF] : osdmap e20: 3 osds: 3 up, 3 in full
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.453371 7fbcba954700 0 log [INF] : mon.deis-staging-node1 calling new monitor election
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.453493 7fbcba954700 1 mon.deis-staging-node1@0(electing).elector(14) init, last seen epoch 14
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.576208 7fbcba954700 0 log [INF] : mon.deis-staging-node1@0 won leader election with quorum 0,1,2
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.658639 7fbcba954700 0 log [INF] : monmap e3: 3 mons at {deis-staging-node1=10.60.61.35:6789/0,deis-staging-node2=10.
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.658885 7fbcba954700 0 log [INF] : pgmap v32: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.659063 7fbcba954700 0 log [INF] : mdsmap e10: 1/1/1 up {0=deis-staging-node1=up:creating}, 2 up:standby
Oct 29 17:47:57 deis-staging-node1 sh[9812]: 2014-10-29 17:47:57.659231 7fbcba954700 0 log [INF] : osdmap e20: 3 osds: 3 up, 3 in full
Oct 29 17:48:01 deis-staging-node1 sh[9812]: 2014-10-29 17:48:01.296217 7fbcba954700 1 mon.deis-staging-node1@0(leader).osd e21 e21: 3 osds: 3 up, 3 in full
Oct 29 17:48:01 deis-staging-node1 sh[9812]: 2014-10-29 17:48:01.343668 7fbcba954700 0 log [INF] : osdmap e21: 3 osds: 3 up, 3 in full
Oct 29 17:48:01 deis-staging-node1 sh[9812]: 2014-10-29 17:48:01.570579 7fbcba954700 0 log [INF] : pgmap v33: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 17:48:15 deis-staging-node1 sh[9812]: 2014-10-29 17:48:15.250548 7fbcb8a4a700 0 -- 10.60.61.35:6789/0 >> 10.60.61.35:0/809728751 pipe(0x3a02300 sd=23 :6789 s=0 pgs=0 cs=0 l=0
Oct 29 17:48:15 deis-staging-node1 sh[9812]: 2014-10-29 17:48:15.879246 7fbcba954700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 17:48:15 deis-staging-node1 sh[9812]: 2014-10-29 17:48:15.892213 7fbcba954700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 17:48:16 deis-staging-node1 sh[9812]: 2014-10-29 17:48:16.331366 7fbcba954700 0 mon.deis-staging-node1@0(leader) e3 handle_command mon_command({"prefix": "auth get-or-create"
Oct 29 17:48:16 deis-staging-node1 sh[9812]: 2014-10-29 17:48:16.812058 7fbcba954700 1 mon.deis-staging-node1@0(leader).osd e22 e22: 3 osds: 3 up, 3 in full
Oct 29 17:48:16 deis-staging-node1 sh[9812]: 2014-10-29 17:48:16.888831 7fbcba954700 0 log [INF] : osdmap e22: 3 osds: 3 up, 3 in full
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.459567 7fbcba954700 1 mon.deis-staging-node1@0(leader).osd e23 e23: 3 osds: 3 up, 3 in full
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.525194 7fbcba954700 0 log [INF] : osdmap e23: 3 osds: 3 up, 3 in full
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.862706 7fbcba954700 0 mon.deis-staging-node1@0(leader).mds e11 print_map
Oct 29 17:48:17 deis-staging-node1 sh[9812]: epoch 11
Oct 29 17:48:17 deis-staging-node1 sh[9812]: flags 0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: created 2014-10-29 16:23:35.068188
Oct 29 17:48:17 deis-staging-node1 sh[9812]: modified 2014-10-29 17:48:16.912595
Oct 29 17:48:17 deis-staging-node1 sh[9812]: tableserver 0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: root 0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: session_timeout 60
Oct 29 17:48:17 deis-staging-node1 sh[9812]: session_autoclose 300
Oct 29 17:48:17 deis-staging-node1 sh[9812]: max_file_size 1099511627776
Oct 29 17:48:17 deis-staging-node1 sh[9812]: last_failure 0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: last_failure_osd_epoch 23
Oct 29 17:48:17 deis-staging-node1 sh[9812]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 17:48:17 deis-staging-node1 sh[9812]: max_mds 1
Oct 29 17:48:17 deis-staging-node1 sh[9812]: in
Oct 29 17:48:17 deis-staging-node1 sh[9812]: up {}
Oct 29 17:48:17 deis-staging-node1 sh[9812]: failed
Oct 29 17:48:17 deis-staging-node1 sh[9812]: stopped
Oct 29 17:48:17 deis-staging-node1 sh[9812]: data_pools 0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: metadata_pool 1
Oct 29 17:48:17 deis-staging-node1 sh[9812]: inline_data disabled
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 4506: 10.60.61.37:6808/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 4917: 10.60.61.36:6805/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 4920: 10.60.61.35:6805/1 'deis-staging-node1' mds.-1.0 up:standby seq 1
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.863312 7fbcba954700 0 log [INF] : mds.? 10.60.61.36:6805/1 up:boot
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.863462 7fbcba954700 0 log [INF] : mds.? 10.60.61.35:6805/1 up:boot
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.863608 7fbcba954700 1 mon.deis-staging-node1@0(leader).mds e11 adding standby 10.60.61.35:6805/1 as mds.0
Oct 29 17:48:17 deis-staging-node1 sh[9812]: 2014-10-29 17:48:17.863766 7fbcba954700 0 log [INF] : mdsmap e11: 0/0/1 up, 3 up:standby
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 2014-10-29 17:48:18.072006 7fbcba954700 0 log [INF] : pgmap v34: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 2014-10-29 17:48:18.220429 7fbcba954700 0 mon.deis-staging-node1@0(leader).mds e12 print_map
Oct 29 17:48:18 deis-staging-node1 sh[9812]: epoch 12
Oct 29 17:48:18 deis-staging-node1 sh[9812]: flags 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: created 2014-10-29 16:23:35.068188
Oct 29 17:48:18 deis-staging-node1 sh[9812]: modified 2014-10-29 17:48:17.863714
Oct 29 17:48:18 deis-staging-node1 sh[9812]: tableserver 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: root 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: session_timeout 60
Oct 29 17:48:18 deis-staging-node1 sh[9812]: session_autoclose 300
Oct 29 17:48:18 deis-staging-node1 sh[9812]: max_file_size 1099511627776
Oct 29 17:48:18 deis-staging-node1 sh[9812]: last_failure 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: last_failure_osd_epoch 23
Oct 29 17:48:18 deis-staging-node1 sh[9812]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 17:48:18 deis-staging-node1 sh[9812]: max_mds 1
Oct 29 17:48:18 deis-staging-node1 sh[9812]: in 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: up {0=4920}
Oct 29 17:48:18 deis-staging-node1 sh[9812]: failed
Oct 29 17:48:18 deis-staging-node1 sh[9812]: stopped
Oct 29 17:48:18 deis-staging-node1 sh[9812]: data_pools 0
Oct 29 17:48:18 deis-staging-node1 sh[9812]: metadata_pool 1
Oct 29 17:48:18 deis-staging-node1 sh[9812]: inline_data disabled
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 4920: 10.60.61.35:6805/1 'deis-staging-node1' mds.0.5 up:creating seq 1
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 4917: 10.60.61.36:6805/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 2014-10-29 17:48:18.220828 7fbcba954700 0 log [INF] : mdsmap e12: 1/1/1 up {0=deis-staging-node1=up:creating}, 1 up:standby
Oct 29 17:48:18 deis-staging-node1 sh[9812]: 2014-10-29 17:48:18.339397 7fbcba954700 0 log [INF] : pgmap v35: 320 pgs: 130 inactive, 32 creating, 128 peering, 30 active+clean; 570 b
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 2014-10-29 17:48:19.288833 7fbcba954700 0 mon.deis-staging-node1@0(leader).mds e13 print_map
Oct 29 17:48:19 deis-staging-node1 sh[9812]: epoch 13
Oct 29 17:48:19 deis-staging-node1 sh[9812]: flags 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: created 2014-10-29 16:23:35.068188
Oct 29 17:48:19 deis-staging-node1 sh[9812]: modified 2014-10-29 17:48:19.191174
Oct 29 17:48:19 deis-staging-node1 sh[9812]: tableserver 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: root 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: session_timeout 60
Oct 29 17:48:19 deis-staging-node1 sh[9812]: session_autoclose 300
Oct 29 17:48:19 deis-staging-node1 sh[9812]: max_file_size 1099511627776
Oct 29 17:48:19 deis-staging-node1 sh[9812]: last_failure 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: last_failure_osd_epoch 23
Oct 29 17:48:19 deis-staging-node1 sh[9812]: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separa
Oct 29 17:48:19 deis-staging-node1 sh[9812]: max_mds 1
Oct 29 17:48:19 deis-staging-node1 sh[9812]: in 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: up {0=4920}
Oct 29 17:48:19 deis-staging-node1 sh[9812]: failed
Oct 29 17:48:19 deis-staging-node1 sh[9812]: stopped
Oct 29 17:48:19 deis-staging-node1 sh[9812]: data_pools 0
Oct 29 17:48:19 deis-staging-node1 sh[9812]: metadata_pool 1
Oct 29 17:48:19 deis-staging-node1 sh[9812]: inline_data disabled
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 4920: 10.60.61.35:6805/1 'deis-staging-node1' mds.0.5 up:creating seq 1
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 4917: 10.60.61.36:6805/1 'deis-staging-node2' mds.-1.0 up:standby seq 1
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 4923: 10.60.61.37:6805/1 'deis-staging-node3' mds.-1.0 up:standby seq 1
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 2014-10-29 17:48:19.289519 7fbcba954700 0 log [INF] : mds.? 10.60.61.37:6805/1 up:boot
Oct 29 17:48:19 deis-staging-node1 sh[9812]: 2014-10-29 17:48:19.289600 7fbcba954700 0 log [INF] : mdsmap e13: 1/1/1 up {0=deis-staging-node1=up:creating}, 2 up:standby
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.185754 7fbcbb155700 0 mon.deis-staging-node1@0(leader).data_health(16) update_stats avail 2% total 5960684 used 5613
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.185881 7fbcbb155700 -1 mon.deis-staging-node1@0(leader).data_health(16) reached critical levels of available space on
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.185883 7fbcbb155700 0 ** Shutdown via Data Health Service **
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.185974 7fbcb9851700 -1 mon.deis-staging-node1@0(leader) e3 *** Got Signal Interrupt ***
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.185981 7fbcb9851700 1 mon.deis-staging-node1@0(leader) e3 shutdown
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.186003 7fbcb9851700 0 quorum service shutdown
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.186005 7fbcb9851700 0 mon.deis-staging-node1@0(shutdown).health(16) HealthMonitor::service_shutdown 1 services
Oct 29 17:48:51 deis-staging-node1 sh[9812]: 2014-10-29 17:48:51.186007 7fbcb9851700 0 quorum service shutdown
Oct 29 17:48:51 deis-staging-node1 docker[10231]: Error response from daemon: No such container: deis-store-monitor
Oct 29 17:48:51 deis-staging-node1 docker[10231]: 2014/10/29 17:48:51 Error: failed to remove one or more containers
core@deis-staging-node1 ~ $
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment