bzub

The osd crush location config option is no longer supported. Please update your ceph.conf to use the crush location option instead.

cephfs: tool/cephfs: displaying “list” in journal event mode (pr#11236, huanwen ren)
cephfs: tools/cephfs: add pg_files command (issue#17249, pr#11026, John Spray)
cephfs: tools/cephfs: add scan_links command which fixes linkages errors (pr#11446, Yan, Zheng)
core,tools: Added append functionality to rados tool. (pr#11036, Tomy Cheru)
core,tools: vstart: decrease pool size if <3 OSDs (pr#11528, John Spray)
mon: if crushtool config is empty use internal crush test (pr#11765, Bassam Tabbara)
performance,tools: rados: add hints to rados bench (pr#12169, Sage Weil)
rbd,tools: rbd : make option –stripe-unit w/ B/K/M work (pr#12407, Jianpeng Ma)

bzub

Rook On Kubernetes (The Hard Way)

Why?

This only exists to help me learn about the inner workings of [Rook][rook]/[Ceph][ceph] and to test design ideas for Rook and [ceph-docker][ceph-docker] as I spin the wheels.

You should use [rook-operator][rook-operator] to try Rook on Kubernetes. Not this.

Status

Ideas I've implemented or want to implement.

bzub

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: rook
    role: osd
    rook_cluster: rookcluster
  name: osd
spec:
  selector:

bzub

TODO

• Define defaults and override rules.
  • Should there be a "default" DataPolicy that is assumed unless overridden?
  • The DataPolicies, DataEndpoints, and StorageNodes lists are ordered. How can we use that ordering to make predictable results from selections?
    • Proposal for StorageNodes: The last group of selectors (all must match) to match a host is what applies.
    • Proposal for DataPolicies: The last policy to match a device or directory is what applies.
    • Proposal for DataEndpoints: Inclusive match. If any one rule describes a device or directory then it's added to the group.
• Seems like it could be possible to combine DataPolicies and DataEndpoints into one resource type.

Hopefully there are few flaws in the logic and this proposal is a good compromise between simplicity and power.

bzub

# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "http://kube-etcd:2379"

bzub

Container Linux Config / Ignition

Here's the relevant bits that configure IPs on loopback via systemd-networkd. They are the same on all my nodes, so no variables.

networkd:
  units:
    - name: 00-vip-lo.network
      contents: |
        [Match]
        Name=lo
        [Network]

bzub

This workaround solves many issues where software in a Kubernetes pod expects a static, rarely changed IP address. Since we bind to a Kubernetes Service IP, it comes with many benefits such as automatic DNS entries within the cluster, health checking automation, and abstraction of network details. In fact these features of Services are used in the pod examples here to avoid hard-coding IP addresses and ports into the pod definitions.

TODO: More details.

Results

    cluster 9ece8946-1b9d-4bb5-8d51-34c968119795

bzub

Docs and manifests are now in the kube-router repository.

bzub

###
### Stage 00-new-cluster complete. Logging to /tmp/np-test-logs/00-new-cluster.\{iptables,ipset\}
###
networkpolicy "test-network-policy" created
###
### Stage 01-np-created complete. Logging to /tmp/np-test-logs/01-np-created.\{iptables,ipset\}
###
deployment "test-np-server" created
service "test-np-server" exposed
###

bzub

#!/usr/bin/env sh

set -e

[ -z "${KUBECONFIG}" ] && KUBECONFIG="${PWD}/assets/auth/kubeconfig"

SUDO="sudo UNIFIED_CGROUP_HIERARCHY=0"

${SUDO} rkt --insecure-options=image run \
  --interactive \