NodeNetworkConfigurationPolicy (NNCP) example

nncp-node1.yaml

# https://docs.openshift.com/container-platform/4.11/networking/k8s_nmstate/k8s-nmstate-updating-node-network-config.html

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: node1
spec:
  nodeSelector:
    kubernetes.io/hostname: node1.example.com
  desiredState:
    dns-resolver:
      config:
        server:
          - 10.15.168.26
        search:
          - example.com

    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 10.15.168.254
          next-hop-interface: bridge-corpnet

    interfaces:
      - name: bridge-corpnet
        description: Bridge to example.com network (10.15.168.0/24) and default router (internet)
        state: up
        type: linux-bridge
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: enp1s0f0
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 10.15.168.21
              prefix-length: 24

      - name: enp1s0f0
        description: Bridge member (bridge-corpnet)
        state: up
        type: ethernet
        lldp:
          enabled: true


      - name: bridge-data
        description: Bridge to 172.16.1.0/24 and 172.31.255.0/24(VLAN 999)
        state: up
        type: linux-bridge
        mtu: 9000
        bridge:
          options:
            stp:
              enabled: true
          port:
            - name: bond-data
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 172.16.1.21
              prefix-length: 24
  
      - name: bond-data
        description: LACP bond to switch1 which provides 172.16.1.0/24 and 172.31.255.0/24(VLAN 999)
        state: up
        type: bond
        mtu: 9000
        link-aggregation:
          mode: 802.3ad
          port:
            - ens1f0
            - ens1f1
        ipv4:
          enabled: false
        ipv6:
          enabled: false
  
      - name: ens1f0
        description: LACP bond member (bond-data)
        state: up
        type: ethernet
        mtu: 9000
        lldp:
          enabled: true
  
      - name: ens1f1
        description: LACP bond member (bond-data)
        state: up
        type: ethernet
        mtu: 9000
        lldp:
          enabled: true


      - name: bridge-priv
        description: Bridge to 172.16.2.0/24
        state: up
        type: linux-bridge
        mtu: 9000
        bridge:
          options:
            stp:
              enabled: true
          port:
            - name: bond-priv
        ipv4:
          enabled: true
          dhcp: false
          address:
            - ip: 172.16.2.21
              prefix-length: 24
  
      - name: bond-priv
        description: LACP bond to switch2 which provides 172.16.2.0/24
        state: up
        type: bond
        mtu: 9000
        link-aggregation:
          mode: 802.3ad
          port:
            - ens12f0
            - ens12f1
        ipv4:
          enabled: false
        ipv6:
          enabled: false
  
      - name: ens12f0
        description: LACP bond member (bond-priv)
        state: up
        type: ethernet
        mtu: 9000
        lldp:
          enabled: true
  
      - name: ens12f1
        description: LACP bond member (bond-priv)
        state: up
        type: ethernet
        mtu: 9000
        lldp:
          enabled: true


# The docs say to use NAD's (NetworkAttachmentDefinition) instead of creating bridges on VLANs
# https://docs.openshift.com/container-platform/4.10/virt/virtual_machines/vm_networking/virt-attaching-vm-multiple-networks.html#attaching-to-multiple-networks
#      - name: bond-data.999
#        description: airgap
#        state: up 
#        type: vlan 
#        mtu: 9000
#        vlan:
#          base-iface: bond-data 
#          id: 999 
#        ipv4:
#          enabled: true
#          dhcp: false
#          address:
#            - ip: 172.31.255.13
#              prefix-length: 24

nncp-node1_UNDO.yaml

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: node1-UNDO
spec:
  nodeSelector:
    kubernetes.io/hostname: node1.example.com
  desiredState:
    interfaces:
    - name: bridge-data
      state: absent
      type: linux-bridge

    - name: bond-data
      state: absent
      type: bond

    - name: ens1f0
      state: absent
      type: ethernet

    - name: ens1f1
      state: absent
      type: ethernet


    - name: bridge-priv
      state: absent
      type: linux-bridge

    - name: bond-priv
      state: absent
      type: bond

    - name: ens12f0
      state: absent
      type: ethernet

    - name: ens12f1
      state: absent
      type: ethernet


# Removing the primary interface without creating a replacement will leave the node inaccessible!
#    - name: bridge-corpnet
#      state: absent
#      type: linux-bridge

# The docs say to use NAD's (NetworkAttachmentDefinition) instead of creating bridges on VLANs
#    - name: bond-data.999
#      state: absent 
#      type: vlan 

Here's a simple diagram of the interfaces and networks I'm trying to connect to (different view))

node1 has 5 network interfaces. The first (top) interface connects to my "corpnet" and can route out to internet resources. The second and third interfaces connect to my "data" network and are bonded. The fourth and fifth interfaces connect to my "private" network, when packets are untagged, and also connects to my "vlan999" network if the packets get the vlan999 tag added by node1.

                             +---+
                             |   +----+
                          +--+        |
  +-----------+           |           +--+
  |           +---------->|   internet   |
  | corpnet   |           +-+          +-+
  +-----------+             +-----+    |
     ^                            +----+
     |                  +--+
     |               +--+  +-+
+----+----+         ++       +-+
|         +-------->|  "data"  +----+
|         +-------> ++   network    |
|         |          +---+       +--+
|  node1  |              +-------+
|         |
|         |              +---+
|         |        +-----+   +-+
|         |      +-|           |
|         |      |    "priv"   +---+
|         +----->|     network     |
|         +----> +-+    (untagged) +--+
|         |        |      &           |
|         |        |   vlan 999     +-+
|         |        +---+  network  +-+
|         |            |           |
+---------+            +-----------+