Skip to content

Instantly share code, notes, and snippets.

View gvilarino's full-sized avatar

Guido Vilariño gvilarino

380 followers · 87 following
View GitHub Profile
@gvilarino
gvilarino / out.json
Created March 3, 2015 01:23
Output of `curl -L http://127.0.0.1:4001/v2/stats/leader` in a 5-vm cluster on Azure when on the same affinity group
{
"leader": "c8a198fbca084485bffa492dbff5431d",
"followers": {
"32b14a46cbb044de8a8ece67c38efa7c": {
"latency": {
"current": 41.5293,
"average": 59.301658470459,
"standardDeviation": 78.3051592183,
"minimum": 0.8503,
"maximum": 1759.5405
@gvilarino
gvilarino / gist:481be3733c7df93ce770
Created March 6, 2015 23:30
core@dos-deis-coreos-0 ~ $ nse deis-store-daemon
root@dos-deis-coreos-0:/# ceph -s
cluster 65ec094b-bd7d-41f9-bb71-4e0cde49eba9
health HEALTH_WARN 782 pgs peering; 782 pgs stuck inactive; 782 pgs stuck unclean; 3 requests are blocked > 32 sec; mds cluster is degraded; clock skew detected on mon.dos-deis-coreos-2, mon.dos-deis-coreos-4, mon.dos-deis-coreos-3, mon.dos-deis-coreos-1
monmap e5: 5 mons at {dos-deis-coreos-0=100.72.44.5:6789/0,dos-deis-coreos-1=100.78.148.77:6789/0,dos-deis-coreos-2=100.72.56.19:6789/0,dos-deis-coreos-3=100.72.142.131:6789/0,dos-deis-coreos-4=100.72.92.22:6789/0}, election epoch 414, quorum 0,1,2,3,4 dos-deis-coreos-0,dos-deis-coreos-2,dos-deis-coreos-4,dos-deis-coreos-3,dos-deis-coreos-1
mdsmap e75: 1/1/1 up {0=dos-deis-coreos-0=up:replay}, 4 up:standby
osdmap e64: 5 osds: 5 up, 5 in
pgmap v158112: 1536 pgs, 12 pools, 1457 MB data, 2490 objects
43159 MB used, 451 GB / 500 GB avail
782 peering
@gvilarino
gvilarino / gist:de3ffa39af3a9c77c151
Created March 6, 2015 23:41
6 Mar 23:33:53 ntpdate[38098]: step time server 97.107.128.58 offset 11.356867 sec
6 Mar 23:34:08 ntpdate[6608]: step time server 97.107.128.58 offset 12.610266 sec
6 Mar 23:34:24 ntpdate[29965]: step time server 97.107.128.58 offset 6.312251 sec
6 Mar 23:34:46 ntpdate[63914]: step time server 66.228.59.187 offset 14.479774 sec
6 Mar 23:35:00 ntpdate[39373]: step time server 200.160.7.186 offset 8.707206 sec
@gvilarino
gvilarino / gist:3ba85ea8c73a307e07a0
Created March 6, 2015 23:46
root@dos-deis-coreos-0:/# ceph -w
cluster 65ec094b-bd7d-41f9-bb71-4e0cde49eba9
health HEALTH_WARN 782 pgs peering; 782 pgs stuck inactive; 782 pgs stuck unclean; 4 requests are blocked > 32 sec; mds cluster is degraded
monmap e5: 5 mons at {dos-deis-coreos-0=100.72.44.5:6789/0,dos-deis-coreos-1=100.78.148.77:6789/0,dos-deis-coreos-2=100.72.56.19:6789/0,dos-deis-coreos-3=100.72.142.131:6789/0,dos-deis-coreos-4=100.72.92.22:6789/0}, election epoch 420, quorum 0,1,2,3,4 dos-deis-coreos-0,dos-deis-coreos-2,dos-deis-coreos-4,dos-deis-coreos-3,dos-deis-coreos-1
mdsmap e75: 1/1/1 up {0=dos-deis-coreos-0=up:replay}, 4 up:standby
osdmap e66: 5 osds: 5 up, 5 in
pgmap v158135: 1536 pgs, 12 pools, 1457 MB data, 2490 objects
43173 MB used, 451 GB / 500 GB avail
782 peering
754 active+clean
@gvilarino
gvilarino / gist:2e7ebb7076000d59b4a5
Created March 6, 2015 23:52
core@dos-deis-coreos-0 ~ $ nse deis-store-daemon
root@dos-deis-coreos-0:/# ceph -w
cluster 65ec094b-bd7d-41f9-bb71-4e0cde49eba9
health HEALTH_WARN 782 pgs peering; 782 pgs stuck inactive; 782 pgs stuck unclean; 4 requests are blocked > 32 sec; mds cluster is degraded
monmap e5: 5 mons at {dos-deis-coreos-0=100.72.44.5:6789/0,dos-deis-coreos-1=100.78.148.77:6789/0,dos-deis-coreos-2=100.72.56.19:6789/0,dos-deis-coreos-3=100.72.142.131:6789/0,dos-deis-coreos-4=100.72.92.22:6789/0}, election epoch 424, quorum 0,1,2,3,4 dos-deis-coreos-0,dos-deis-coreos-2,dos-deis-coreos-4,dos-deis-coreos-3,dos-deis-coreos-1
mdsmap e75: 1/1/1 up {0=dos-deis-coreos-0=up:replay}, 4 up:standby
osdmap e66: 5 osds: 5 up, 5 in
pgmap v158141: 1536 pgs, 12 pools, 1457 MB data, 2490 objects
43155 MB used, 451 GB / 500 GB avail
782 peering
@gvilarino
gvilarino / gist:49571937cac0a317cd32
Created March 6, 2015 23:59
root@dos-deis-coreos-0:/# ceph health detail
HEALTH_WARN 829 pgs peering; 829 pgs stuck inactive; 829 pgs stuck unclean; 3 requests are blocked > 32 sec; 1 osds have slow requests; mds cluster is degraded
pg 7.78 is stuck inactive for 133195.697270, current state peering, last acting [3,1,4]
pg 5.7a is stuck inactive for 146531.391652, current state peering, last acting [2,3,4]
pg 10.75 is stuck inactive for 3088.445485, current state peering, last acting [2,0,3]
pg 1.7e is stuck inactive for 78252.769665, current state peering, last acting [1,2,4]
pg 0.7f is stuck inactive for 212578.439306, current state peering, last acting [3,0,1]
pg 3.7c is stuck inactive for 73054.532305, current state peering, last acting [1,3,0]
pg 3.7d is stuck inactive for 73050.601483, current state peering, last acting [1,2,0]
pg 0.7e is stuck inactive for 24956.433239, current state peering, last acting [1,0,4]
@gvilarino
gvilarino / gist:8fd9015f50ffad48aa9b
Created March 7, 2015 00:01
root@dos-deis-coreos-0:/# ceph osd tree
# id weight type name up/down reweight
-1 5 root default
-2 1 host dos-deis-coreos-2
0 1 osd.0 up 1
-3 1 host dos-deis-coreos-4
2 1 osd.2 up 1
-4 1 host dos-deis-coreos-0
1 1 osd.1 up 1
-5 1 host dos-deis-coreos-3
@gvilarino
gvilarino / gist:0d237d728ecfcdf4a20a
Created March 7, 2015 00:04
core@dos-deis-coreos-3 ~ $ nse deis-store-daemon
root@dos-deis-coreos-3:/# ceph osd out 1
marked out osd.1.
root@dos-deis-coreos-3:/# ceph health detail
HEALTH_WARN 829 pgs peering; 829 pgs stuck inactive; 829 pgs stuck unclean; mds cluster is degraded
pg 7.78 is stuck inactive for 133508.123561, current state peering, last acting [3,1,4]
pg 5.7a is stuck inactive for 146843.817944, current state peering, last acting [2,3,4]
pg 10.75 is stuck inactive for 3400.871778, current state peering, last acting [2,0,3]
pg 1.7e is stuck inactive for 78565.195958, current state peering, last acting [1,2,4]
pg 0.7f is stuck inactive for 212890.865599, current state peering, last acting [3,0,1]
@gvilarino
gvilarino / gist:75e6595f038e317548fb
Created March 7, 2015 00:17
core@dos-deis-coreos-0 ~ $ nse deis-store-daemon
root@dos-deis-coreos-0:/# ceph -w
cluster 65ec094b-bd7d-41f9-bb71-4e0cde49eba9
health HEALTH_WARN 829 pgs peering; 353 pgs stale; 829 pgs stuck inactive; 829 pgs stuck unclean; mds cluster is degraded; 1/4 in osds are down; 1 mons down, quorum 0,1,2,4 dos-deis-coreos-0,dos-deis-coreos-2,dos-deis-coreos-4,dos-deis-coreos-1
monmap e5: 5 mons at {dos-deis-coreos-0=100.72.44.5:6789/0,dos-deis-coreos-1=100.78.148.77:6789/0,dos-deis-coreos-2=100.72.56.19:6789/0,dos-deis-coreos-3=100.72.142.131:6789/0,dos-deis-coreos-4=100.72.92.22:6789/0}, election epoch 426, quorum 0,1,2,4 dos-deis-coreos-0,dos-deis-coreos-2,dos-deis-coreos-4,dos-deis-coreos-1
mdsmap e76: 1/1/1 up {0=dos-deis-coreos-0=up:replay}, 3 up:standby
osdmap e74: 5 osds: 4 up, 4 in
pgmap v158163: 1536 pgs, 12 pools, 1457 MB data, 2490 objects
32548 MB used, 363 GB / 400 GB avail
282 stale+peering
@gvilarino
gvilarino / gist:a3388bdac9c1d8408ec7
Created March 7, 2015 02:13
-- Logs begin at Sat 2015-03-07 01:09:39 UTC. --
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.auth
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.contenttypes
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.humanize
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.messages
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.sessions
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.sites
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django.contrib.staticfiles
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > django_fsm
Mar 07 01:30:37 dos-deis-coreos-4 sh[2859]: > json_field