Skip to content

Instantly share code, notes, and snippets.

@jodh-intel

jodh-intel/kata-full-fat-kernel.md

Last active January 11, 2022 15:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jodh-intel/b9e0d241b6bb1700bc0074dadc3c99b8 to your computer and use it in GitHub Desktop.
Save jodh-intel/b9e0d241b6bb1700bc0074dadc3c99b8 to your computer and use it in GitHub Desktop.
Download ZIP
Idea for a "full fat" Kata Containers guest kernel
Raw
kata-full-fat-kernel.md

Full fat kernel

Background

  • Originally, the guest kernel was as minimal as possible.
  • Booted extremely fast, but didn't support all uses-cases.
  • Over time, we have added lots of extra kernel CONFIG_* options...
    • For new kernel options
    • For new hardware features

Problems

  • "Death by 1000 cuts": we are slowly bloating the kernel over time.
  • Kernel now much bigger/slower than the original.
  • Every time we do this, we need to adjust the metrics tests to take account of the changes.

Idea

Why not just enable all kernel options for all architectures?

Rationale

For the common case:

  • The guest environment should be "transparent" to the workload.

  • It should be atleast as functional as the host environment.

    (to be as equivalent as possible to a "traditional Linux container").

Impacts (page 1)

  • "boot speed" and "memory density" will probably be impacted.

  • But is that a problem?

    • The default Kata Containers configuration is NOT optimised.
    • Building "full fat" emphasises that, which is a good thing.

Impacts (page 2)

  • Difficult / impossible to test "exotic" features in a public CI:

    • CI's don't provide all hardware features

      (often environment already virtualised).

    • CI's don't always support all architectures.

    • However, we are already facing these challenges.

      A full fat kernel provides an opportunity to overcome this though, without custom kernels.

Impacts (page 3)

  • Enabling all config option provides a larger attack surface.
    • Is this a concern?
    • Maybe, but current default kernel may not be ideal for production use anyway.

Benefits (page 1)

  • Enabling all options is the distro approach for kernels.

  • Ensures all use cases "just work" (TM)

    Helpful to newcomers (Principle of least astonishment).

    Examples of currently problematic use-cases, all of which require rebuilding guest kernels: QAT and GPU passthrough (1, 2).

Benefits (page 2)

  • In some ways, a slow(er) boot or larger memory footprint is helpful: it emphasises that: Kata is NOT optimised "out of the box"!. It can't be!

  • Simplifies packaging

    Rather than packaging (and maintaining!) multiple set of assets for features like QAT or GPU passthrough, the default kernels would work.

Considerations (page 1)

  • This change would (have to) alter the meaning of the current kernel "fragments".

    • Rather than defining a "small-ish kernel", they could become the true minimal configurations.

    • This implies the CI would need some extra jobs:

      kernel type test workloads
      "full fat" any
      minimal metrics

Considerations (page 2)

  • Need ability to package "custom configurations" (kernel, rootfs, config files).

  • Configurations could be use-case or architecture specific.

  • For example, we might want to offer example configurations for:

    • "fastest boot"
    • "smallest (memory resident) kernel" / maximum memory density.

Questions

  • If we do this, should we build the kernel monolithically or use modules?

  • How do we package custom configurations? (GitHub, Snap, OBS, other, ...?)

  • Where would such packages be hosted?

  • How would such packages be tested?

Summary

Status Quo:

  • We currently have a slightly "chubby kernel" which offers most features.

  • Difficult for users to explore newer hardware features.

  • A full fat kernel would:

    • Help usability
    • Allow us to focus more aggressively on other "profiles"

References

@jodh-intel
Copy link
Author

jodh-intel commented Dec 14, 2021 •
edited

Waiting to be presented to the Kata Containers Architecture Committee meeting.

To view as HTML:

$ infile="kata-full-fat-kernel.md"
$ outfile="/tmp/full-fat-kernel.html"
$ pandoc -s -f markdown -t revealjs -V revealjs-url="https://unpkg.com/reveal.js" --metadata title="Kata Containers"--metadata subtitle="A \"full fat\" guest kernel?" --metadata author="James O. D. Hunt" --metadata date="14 December 2021" --incremental -o "$outfile" "$infile"
$ "$BROWSER" "$outfile"

@jodh-intel
Copy link
Author

jodh-intel commented Dec 15, 2021 •
edited

See also:

@jodh-intel
Copy link
Author

Presented at the AC meeting today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment