bh7cw

INFO: Attempting infra node pod rebalancing...
INFO: Number of AZs 1
INFO: Number of INFRA NODES 2
INFO: Rebalancing prometheus pods...
INFO: prometheus pods needing rebalance: 0
INFO: Rebalancing alertmanager pods...
INFO: alertmanager pods needing rebalance: 0
INFO: Rebalancing splunk-heavy-forwarder pods...
INFO: splunk-heavy-forwarder pods needing rebalance: 1
INFO: Deleting pod splunkforwarder-deployment-765bfc8cb-v7wx6

bh7cw

#!/bin/bash
# set -x

echo "INFO: start putting the cluster into an imbalanced state"

# target workloads to rebalance
TARGET_PODS="alertmanager-main|prometheus-k8s|prometheus-adapter|splunkforwarder-deployment"

# check all the target pods on each infra node
for NODE in $(oc get node -l node-role.kubernetes.io/infra= --no-headers | awk '{print $1}'); do

bh7cw

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sre-1270-rebalancing-infra-nodes
  namespace: openshift-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:

bh7cw

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sre-1270-rebalancing-infra-nodes-openshift-monitoring
  namespace: openshift-monitoring
rules:
- apiGroups:
  - ""
  resources:

bh7cw

[    0.000000] Linux version 5.8.10-200.fc32.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1), GNU ld version 2.34-4.fc32) #1 SMP Thu Sep 17 16:48:25 UTC 2020
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-8bc8c9d5d7019af4e917dbd1cfc2a042027d5f7f48809a4108a4bdbb6d7f2072/vmlinuz-5.8.10-200.fc32.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.firstboot ostree=/ostree/boot.1/fedora-coreos/8bc8c9d5d7019af4e917dbd1cfc2a042027d5f7f48809a4108a4bdbb6d7f2072/0 ignition.platform.id=azurestack
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: 

bh7cw

{
  "ignition": {
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "gecos": "CoreOS Admin",
        "groups": [

bh7cw

[    0.000000] Linux version 5.8.4-200.fc32.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1), GNU ld version 2.34-4.fc32) #1 SMP Wed Aug 26 22:28:08 UTC 2020
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-b1ac8e031d8ccbd7de7e44f03e46fc62a6989c74d1375e6532e73ae08027c249/vmlinuz-5.8.4-200.fc32.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.firstboot ostree=/ostree/boot.1/fedora-coreos/b1ac8e031d8ccbd7de7e44f03e46fc62a6989c74d1375e6532e73ae08027c249/0 ignition.platform.id=aws
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[  

bh7cw

Tested the new fcos image with the ignition config file https://gist.github.com/bh7cw/6ac27653d19815304332277ff2aacdb8

[core@ip-172-31-14-2 system]$ ls afterburn-sshkeys.target.requires/
afterburn-sshkeys@core.service  afterburn-sshkeys@test.service
[core@ip-172-31-14-2 system]$ systemctl list-dependencies afterburn-sshkeys.target
afterburn-sshkeys.target
● ├─afterburn-sshkeys@core.service
● └─afterburn-sshkeys@test.service
[core@ip-172-31-14-2 system]$ sudo systemctl start afterburn-sshkeys.target
[core@ip-172-31-14-2 system]$ sudo journalctl -u 'afterburn-sshkeys@*.service'

bh7cw

{
  "ignition": {
    "version": "3.1.0"
  },
  "passwd": {
    "users": [
      {
        "groups": [
          "sudo"
        ],

bh7cw

Refer to https://github.com/coreos/afterburn/pull/481#issuecomment-675054667

sudo systemctl start afterburn-sshkeys@test.service
sudo systemctl start afterburn-sshkeys@jing.service

[core@ip-172-31-13-182 system]$ systemctl list-dependencies afterburn-sshkeys.target
afterburn-sshkeys.target
● └─afterburn-sshkeys@core.service
--------------------------------------
Different from test before, which shows all the instances, not just core