tripleo-dpdk-user-data-kernel-args.yaml

heat_template_version: 2014-10-16

description: >
  This is an example showing how you can do firstboot configuration
  of the nodes via cloud-init.  To enable this, replace the default
  mapping of OS::TripleO::NodeUserData in ../overcloud_resource_registry*

parameters:
  ComputeKernelArgs:
    description: >
      Space seprated list of Kernel args to be update to grub.
      The given args will be appended to existing args of GRUB_CMDLINE_LINUX in file /etc/default/grub
      Example: "intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=1"
    type: string
    default: ""
  ComputeDpdkHostnameFormat:
    type: string
    default: ""

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: compute_kernel_args}

  # Verify the logs on /var/log/cloud-init.log on the overcloud node
  compute_kernel_args:
    type: OS::Heat::SoftwareConfig
    properties:
      config:
        str_replace:
          template: |
            #!/bin/bash
            set -x
            FORMAT=$COMPUTE_HOSTNAME_FORMAT
            if [[ -z $FORMAT ]] ; then
              FORMAT="dpdk" ;
            else
              # Assumption: only %index% and %stackname% are the variables in Host name format
              FORMAT=$(echo $FORMAT | sed  's/\%index\%//g' | sed 's/\%stackname\%//g') ;
            fi
            if [[ $(hostname) == *$FORMAT* ]] ; then
              sed 's/^\(GRUB_CMDLINE_LINUX=".*\)"/\1 $KERNEL_ARGS"/g' -i /etc/default/grub ;
              grub2-mkconfig -o /etc/grub2.cfg

              # Sometimes, the IP of the provisioning network is not acquired at the time of
              # cloud-init, which will fail the metadata query, looping for 10seconds to ensure
              # the network connectivity is ready.
              i=0
              while [ $i -lt 5 ]; do
                NETWORK=$(curl -m 10 http://169.254.169.254/openstack/latest/network_data.json)
                if [ $? -eq 0 ]; then
                  # Rebooting without running os-net-config will have default ifcfg scripts which
                  # is boot the interfaces in alphanumberic order and if DHCPDISCOVER of an interface\
                  # fails, then the network.service gets failed and does not try others.
                  # Here we are identifying the provisioning network interface and keep the
                  # BOOTPROTO as dpch and for all other interfaces make it as none. So that
                  # network.service will invoke dhcp only on the provisioning network.
                  # We are identifying the provisioning network using the meta data of the node,
                  # which will provide the mac address of the provisioning network interface.
                  # NOTE: Only one provisioning network interface is supported with this script

                  MAC=$(echo $NETWORK | jq -r ".links[0].ethernet_mac_address")
                  IFACE=$(ip a | grep $MAC -B1 | awk 'NR==1{print $2;}' | cut -d ":" -f1)
                  find /etc/sysconfig/network-scripts/ -name 'ifcfg-*' ! -name 'ifcfg-'$IFACE -type f -exec sed 's/^BOOTPROTO=.*/BOOTPROTO=none/g' -i {} +

                  reboot
                  break
                fi
                sleep 2
                i=`expr $i + 1`
              done

            fi
          params:
            $KERNEL_ARGS: {get_param: ComputeKernelArgs}
            $COMPUTE_HOSTNAME_FORMAT: {get_param: ComputeDpdkHostnameFormat}

outputs:
  # This means get_resource from the parent template will get the userdata, see:
  # http://docs.openstack.org/developer/heat/template_guide/composition.html#making-your-template-resource-more-transparent
  # Note this is new-for-kilo, an alternative is returning a value then using
  # get_attr in the parent template instead.
  OS::stack_id:
    value: {get_resource: userdata}

Hi Saravanan,

I saw your question(https://openstack.nimeyo.com/91474/openstack-dev-tripleo-setting-kernel-args-overcloud-nodes) and solution(here),
I am also facing the same issue, while configuring ovs-dpdk with SRIOV on compute node.

Issues:

1. passing required kernel parameters like huge-pages, and reboot, since ironic has no support now in newton release (ROSP 10)
2. rebooting the node after configuring some dpdk bridges.

How you are doing reboot in OS::Heat::SoftwareConfig, how heat is transferring control after reboot.

I saw you are working on something interesting here https://github.com/krsacme/tht-dpdk, can you please explain a bit.

Thanks

Also I see some of your changes in newton release related to DPDK and OVS

Hi Nobin,

As discussed in the irc, the reboot is done be the bash script which will be run by cloud-init. After reboot, cloud-init will continue where it has let off. Heat will not even know about this reboot. After reboot, configuration of the node will resume.

tht-dpdk repo contains the THT and environment files for deploying newton. It is for my experiment of using dpdk as composable role like controller, compute, compute-dpdk. It is still under development/testing, not mature yet. If you are interested, i will update you once things are in good shape.

Thanks.

Hi Saravanan,

Is it possible that above template can be expanded to apply different KERNEL_ARGS parameters to overcloud compute nodes ?
for example, assuming there are two compute nodes in one openstack instance, we'd like to configure 32G hugepages on compute node 1 and 64G hugepages on compute node 2.

Thanks!
zenghui