ceph health detail

[root@setup-42-4 ~]# ceph health detail
HEALTH_WARN 128 pgs stuck unclean
pg 14.65 is stuck unclean for 3246.280851, current state active+remapped, last acting [5,8]
pg 14.64 is stuck unclean for 3246.280856, current state active+remapped, last acting [5,8]
pg 14.67 is stuck unclean for 3246.080916, current state active+remapped, last acting [5,1]
pg 14.66 is stuck unclean for 3246.280863, current state active+remapped, last acting [5,3]
pg 14.69 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.68 is stuck unclean for 3225.902057, current state active+remapped, last acting [6,3]
pg 14.6b is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.6a is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.6d is stuck unclean for 3245.549714, current state active+remapped, last acting [5,0]
pg 14.6c is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.6f is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.6e is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.51 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.50 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.53 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.52 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.55 is stuck unclean since forever, current state active+remapped, last acting [5,3]
pg 14.54 is stuck unclean for 3245.997829, current state active+remapped, last acting [5,7]
pg 14.57 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.56 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.59 is stuck unclean since forever, current state active+remapped, last acting [5,7]
pg 14.58 is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.5b is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.5a is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.5d is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.5c is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.5f is stuck unclean since forever, current state active+remapped, last acting [5,3]
pg 14.5e is stuck unclean for 3246.280974, current state active+remapped, last acting [5,2]
pg 14.41 is stuck unclean since forever, current state active+remapped, last acting [5,3]
pg 14.40 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.43 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.42 is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.45 is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.44 is stuck unclean for 3245.997892, current state active+remapped, last acting [5,7]
pg 14.47 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.46 is stuck unclean for 3230.903666, current state active+remapped, last acting [6,8]
pg 14.49 is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.48 is stuck unclean for 3228.902720, current state active+remapped, last acting [6,4]
pg 14.4b is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.4a is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.4d is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.4c is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.4f is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.4e is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.31 is stuck unclean for 3246.281033, current state active+remapped, last acting [5,3]
pg 14.30 is stuck unclean since forever, current state active+remapped, last acting [5,6]
pg 14.33 is stuck unclean since forever, current state active+remapped, last acting [5,7]
pg 14.32 is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.35 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.34 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.37 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.36 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.39 is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.38 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.3b is stuck unclean for 3246.206933, current state active+remapped, last acting [5,1]
pg 14.3a is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.3d is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.3c is stuck unclean for 3246.206904, current state active+remapped, last acting [5,1]
pg 14.3f is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.3e is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.21 is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.20 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.23 is stuck unclean since forever, current state active+remapped, last acting [5,3]
pg 14.22 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.25 is stuck unclean for 3246.908307, current state active+remapped, last acting [6,2]
pg 14.24 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.27 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.26 is stuck unclean since forever, current state active+remapped, last acting [6,5]
pg 14.29 is stuck unclean since forever, current state active+remapped, last acting [5,3]
pg 14.28 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.2b is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.2a is stuck unclean for 3234.904201, current state active+remapped, last acting [6,4]
pg 14.2d is stuck unclean for 3246.281154, current state active+remapped, last acting [5,8]
pg 14.2c is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.2f is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.2e is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.11 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.10 is stuck unclean for 3244.907175, current state active+remapped, last acting [6,0]
pg 14.13 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.12 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.15 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.14 is stuck unclean since forever, current state active+remapped, last acting [6,4]
pg 14.17 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.16 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.19 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.18 is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.1b is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.1a is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.1d is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.1c is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.1f is stuck unclean for 3236.905244, current state active+remapped, last acting [6,3]
pg 14.1e is stuck unclean for 3246.281267, current state active+remapped, last acting [5,8]
pg 14.1 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.0 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.3 is stuck unclean for 3246.281283, current state active+remapped, last acting [5,4]
pg 14.2 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.5 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.4 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.7 is stuck unclean since forever, current state active+remapped, last acting [6,0]
pg 14.6 is stuck unclean since forever, current state active+remapped, last acting [6,2]
pg 14.9 is stuck unclean for 3246.281312, current state active+remapped, last acting [5,2]
pg 14.8 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.b is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.a is stuck unclean since forever, current state active+remapped, last acting [6,5]
pg 14.d is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.c is stuck unclean for 3246.557869, current state active+remapped, last acting [5,2]
pg 14.f is stuck unclean since forever, current state active+remapped, last acting [5,6]
pg 14.e is stuck unclean since forever, current state active+remapped, last acting [6,5]
pg 14.71 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.70 is stuck unclean since forever, current state active+remapped, last acting [5,1]
pg 14.73 is stuck unclean for 3224.902368, current state active+remapped, last acting [6,8]
pg 14.72 is stuck unclean since forever, current state active+remapped, last acting [5,2]
pg 14.75 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.74 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.77 is stuck unclean for 3246.156882, current state active+remapped, last acting [6,1]
pg 14.76 is stuck unclean for 3246.281363, current state active+remapped, last acting [5,4]
pg 14.79 is stuck unclean since forever, current state active+remapped, last acting [5,0]
pg 14.78 is stuck unclean since forever, current state active+remapped, last acting [5,4]
pg 14.7b is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.7a is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.7d is stuck unclean for 3245.647539, current state active+remapped, last acting [5,0]
pg 14.7c is stuck unclean since forever, current state active+remapped, last acting [5,8]
pg 14.7f is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.7e is stuck unclean for 3245.550221, current state active+remapped, last acting [5,0]
pg 14.61 is stuck unclean for 3246.081453, current state active+remapped, last acting [5,1]
pg 14.60 is stuck unclean since forever, current state active+remapped, last acting [6,3]
pg 14.63 is stuck unclean since forever, current state active+remapped, last acting [6,1]
pg 14.62 is stuck unclean since forever, current state active+remapped, last acting [6,3]

ceph.conf

[root@setup-42-4 ~]# cat /etc/ceph/ceph.conf

[global]
        auth supported = cephx
        keyring = /etc/ceph/keyring
        public network = 10.2.0.0/24
        cluster network = 10.2.0.0/24
        osd_heartbeat_interval = 1
        osd_heartbeat_grace = 5
        paxos propose interval = 5

[client.radosgw.gateway]
        host = gateway
        keyring = /etc/ceph/keyring.radosgw.gateway
        rgw socket path = /tmp/radosgw.sock

[mon]
        mon data = /etc/ceph/mon.$id
        keyring = /etc/ceph/keyring.$name

[mon.a]
        host = compute-0-0
        mon addr = 10.2.0.253:6789

[mon.b]
        host = compute-0-1
        mon addr = 10.2.0.252:6789

[mon.c]
        host = compute-0-2
        mon addr = 10.2.0.251:6789

[mds]
        keyring = /etc/ceph/keyring.$name


[mds.setup-42-4]
        host = setup-42-4

[mds.compute-0-0]
        host = compute-0-0

[mds.compute-0-1]
        host = compute-0-1

[mds.compute-0-2]
        host = compute-0-2

[mds.vm-manager-0-0]
        host = vm-manager-0-0


[osd]
        osd journal = $osd_data/journal
        osd journal size = 4000
        keyring = /etc/ceph/keyring.$name


[osd.0]
        host = setup-42-4
        osd data = /ceph/osd.0

[osd.1]
        host = compute-0-0
        osd data = /state/partition1/osd.1

[osd.2]
        host = compute-0-0
        osd data = /ceph/osd.2

[osd.3]
        host = compute-0-1
        osd data = /state/partition1/osd.3

[osd.4]
        host = compute-0-1
        osd data = /ceph/osd.4

[osd.5]
        host = compute-0-2
        osd data = /state/partition1/osd.5

[osd.6]
        host = compute-0-2
        osd data = /ceph/osd.6

[osd.7]
        host = vm-manager-0-0
        osd data = /state/partition1/osd.7

[osd.8]
        host = vm-manager-0-0
        osd data = /ceph/osd.8

crush map

# begin crush map
# Generated by CloudNode script

# devices
device 6 osd.6
device 3 osd.3
device 7 osd.7
device 2 osd.2
device 8 osd.8
device 1 osd.1
device 4 osd.4
device 0 osd.0
device 5 osd.5

# types
type 6 root
type 1 host
type 4 room
type 0 osd
type 3 row
type 2 rack
type 5 datacenter


# buckets
host prim-zone04-sec-zone04-compute-0-2 {
        id -16
        alg straw
        hash 0
        item osd.5 weight 1
        item osd.6 weight 1
}
host sec-zone02-compute-0-0 {
        id -22
        alg straw
        hash 0
        item osd.1 weight 1
        item osd.2 weight 1
}
host mgmt-vm-manager-0-0 {
        id -10
        alg straw
        hash 0
        item osd.7 weight 1.000
}
host obj-zone03-compute-0-1 {
        id -36
        alg straw
        hash 0
        item osd.3 weight 1
}
host prim-zone02-compute-0-0 {
        id -25
        alg straw
        hash 0
        item osd.1 weight 1
        item osd.2 weight 1
}
host mgmt-setup-42-4 {
        id -9
        alg straw
        hash 0
        item osd.0 weight 1.000
}
host prim-zone03-compute-0-1 {
        id -34
        alg straw
        hash 0
        item osd.3 weight 1
        item osd.4 weight 1
}
host obj-zone03-prim-zone03-compute-0-1 {
        id -37
        alg straw
        hash 0
        item osd.4 weight 1
}
host sec-zone04-compute-0-2 {
        id -13
        alg straw
        hash 0
        item osd.5 weight 1
        item osd.6 weight 1
}
host setup-42-4 {
        id -2
        alg straw
        hash 0
        item osd.0 weight 1.000
}
host compute-0-0 {
        id -4
        alg straw
        hash 0
        item osd.1 weight 1.000
        item osd.2 weight 1.000
}
host obj-zone04-prim-zone04-sec-zone04-compute-0-2 {
        id -19
        alg straw
        hash 0
        item osd.5 weight 1
        item osd.6 weight 1
}
host compute-0-1 {
        id -5
        alg straw
        hash 0
        item osd.3 weight 1.000
        item osd.4 weight 1.000
}
host vm-manager-0-0 {
        id -7
        alg straw
        hash 0
        item osd.7 weight 1.000
        item osd.8 weight 1.000
}
host obj-zone02-compute-0-0 {
        id -28
        alg straw
        hash 0
        item osd.1 weight 1
        item osd.2 weight 1
}
host compute-0-2 {
        id -6
        alg straw
        hash 0
        item osd.5 weight 1.000
        item osd.6 weight 1.000
}
host sec-zone03-compute-0-1 {
        id -31
        alg straw
        hash 0
        item osd.3 weight 1
        item osd.4 weight 1
}

# racks
rack unknownrack {
        id -3
        alg straw
        hash 0
        item setup-42-4 weight 1.000
        item compute-0-0 weight 2.000
        item compute-0-1 weight 2.000
        item compute-0-2 weight 2.000
        item vm-manager-0-0 weight 2.000
}




# roots
root sec-zone02 {
        id -20
        alg straw
        hash 0
        item sec-zone02-compute-0-0 weight 2.000
}
root prim-zone04 {
        id -14
        alg straw
        hash 0
        item prim-zone04-sec-zone04-compute-0-2 weight 2.000
}
root obj-zone02 {
        id -26
        alg straw
        hash 0
        item obj-zone02-compute-0-0 weight 2.000
}
root default {
        id -1
        alg straw
        hash 0
        item unknownrack weight 9.000
}
root sec-zone04 {
        id -11
        alg straw
        hash 0
        item sec-zone04-compute-0-2 weight 2.000
}
root obj-zone03 {
        id -35
        alg straw
        hash 0
        item obj-zone03-compute-0-1 weight 1.000
        item obj-zone03-prim-zone03-compute-0-1 weight 1.000
}
root mgmt {
        id -8
        alg straw
        hash 0
        item mgmt-setup-42-4 weight 1.000
        item mgmt-vm-manager-0-0 weight 1.000
}
root prim-zone02 {
        id -23
        alg straw
        hash 0
        item prim-zone02-compute-0-0 weight 2.000
}
root obj-zone04 {
        id -17
        alg straw
        hash 0
        item obj-zone04-prim-zone04-sec-zone04-compute-0-2 weight 2.000
}
root sec-zone03 {
        id -29
        alg straw
        hash 0
        item sec-zone03-compute-0-1 weight 2.000
}
root prim-zone03 {
        id -32
        alg straw
        hash 0
        item prim-zone03-compute-0-1 weight 2.000
}

# rules
rule sec-zone02 {
        ruleset 7
        type replicated
        min_size 1
        max_size 10
        step take sec-zone02
        step chooseleaf firstn 0 type host
        step emit
}
rule prim-zone04 {
        ruleset 5
        type replicated
        min_size 1
        max_size 10
        step take prim-zone04
        step chooseleaf firstn 0 type host
        step emit
}
rule obj-zone02 {
        ruleset 9
        type replicated
        min_size 1
        max_size 10
        step take obj-zone02
        step chooseleaf firstn 0 type host
        step emit
}
rule data {
        ruleset 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule sec-zone04 {
        ruleset 4
        type replicated
        min_size 1
        max_size 10
        step take sec-zone04
        step chooseleaf firstn 0 type host
        step emit
}
rule obj-zone03 {
        ruleset 12
        type replicated
        min_size 1
        max_size 10
        step take obj-zone03
        step chooseleaf firstn 0 type host
        step emit
}
rule mgmt {
        ruleset 3
        type replicated
        min_size 1
        max_size 10
        step take mgmt
        step chooseleaf firstn 0 type host
        step emit
}
rule prim-zone02 {
        ruleset 8
        type replicated
        min_size 1
        max_size 10
        step take prim-zone02
        step chooseleaf firstn 0 type host
        step emit
}
rule obj-zone04 {
        ruleset 6
        type replicated
        min_size 1
        max_size 10
        step take obj-zone04
        step chooseleaf firstn 0 type host
        step emit
}
rule sec-zone03 {
        ruleset 10
        type replicated
        min_size 1
        max_size 10
        step take sec-zone03
        step chooseleaf firstn 0 type host
        step emit
}
rule metadata {
        ruleset 1
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule rbd {
        ruleset 2
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule prim-zone03 {
        ruleset 11
        type replicated
        min_size 1
        max_size 10
        step take prim-zone03
        step chooseleaf firstn 0 type host
        step emit
}

# end crush map

osd_tree

# id    weight  type name       up/down reweight
-35     2       root obj-zone03
-36     1               host obj-zone03-compute-0-1
3       1                       osd.3   up      1
-37     1               host obj-zone03-prim-zone03-compute-0-1
4       1                       osd.4   up      1
-32     2       root prim-zone03
-34     2               host prim-zone03-compute-0-1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-29     2       root sec-zone03
-31     2               host sec-zone03-compute-0-1
3       1                       osd.3   up      1
4       1                       osd.4   up      1
-26     2       root obj-zone02
-28     2               host obj-zone02-compute-0-0
1       1                       osd.1   up      1
2       1                       osd.2   up      1
-23     2       root prim-zone02
-25     2               host prim-zone02-compute-0-0
1       1                       osd.1   up      1
2       1                       osd.2   up      1
-20     2       root sec-zone02
-22     2               host sec-zone02-compute-0-0
1       1                       osd.1   up      1
2       1                       osd.2   up      1
-17     2       root obj-zone04
-19     2               host obj-zone04-prim-zone04-sec-zone04-compute-0-2
5       1                       osd.5   up      1
6       1                       osd.6   up      1
-14     2       root prim-zone04
-16     2               host prim-zone04-sec-zone04-compute-0-2
5       1                       osd.5   up      1
6       1                       osd.6   up      1
-11     2       root sec-zone04
-13     2               host sec-zone04-compute-0-2
5       1                       osd.5   up      1
6       1                       osd.6   up      1
-8      2       root mgmt
-9      1               host mgmt-setup-42-4
0       1                       osd.0   up      1
-10     1               host mgmt-vm-manager-0-0
7       1                       osd.7   up      1
-1      9       root default
-3      9               rack unknownrack
-2      1                       host setup-42-4
0       1                               osd.0   up      1
-4      2                       host compute-0-0
1       1                               osd.1   up      1
2       1                               osd.2   up      1
-5      2                       host compute-0-1
3       1                               osd.3   up      1
4       1                               osd.4   up      1
-6      2                       host compute-0-2
5       1                               osd.5   up      1
6       1                               osd.6   up      1
-7      2                       host vm-manager-0-0
7       1                               osd.7   up      1
8       1                               osd.8   up      1