grahamwhaley

How to config k8s to have many pods (on a node)

I had a need (understand this is for some testing, not for a real deployment ;-) ) to run a lot of pods (like >=1k of them) on a single k8s node. Now, I had the hw available - 88cores and 377Gb of RAM - but, k8s has some inbuilt limits by default that will not let you launch more than 110 pods, and if you get past that, you'll hit a network limit at about 250 pods... so, before I forget, here is how to configure to run more...

kubeadm

In your kubeadm init file, something like:

sboeuf

Global concepts

Addressable space

Depends on the processor capabilities, which can be around 36~40 bits for recent Intel processors. If taking 39 bits as an example, this means the total addressable space will be 1 TiB of available addresses. The guest RAM is part of this addressable space, same as devices, PCI holes, ...

BAR (Base Address Register)

A base address register is part of the PCI configuration space of each PCI device, and it declares an extra memory region related to the device that can be found at this specific base address.

smoser

backdoor-image

Description

backdoor-image can be used to easily add user with passwordless sudo access to a image or a root filesystem.

Operating on an image requires the 'mount-image-callback' tool from cloud-utils. That can be installed on ubuntu via apt-get install -qy cloud-image-utils.

mcastelino

#!/bin/bash
# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
# ex: ts=8 sw=4 sts=4 et filetype=sh

VMN=${VMN:=1}

NEMU=~/build-x86_64/x86_64-softmmu/qemu-system-x86_64

sudo $NEMU \
 -trace events=/tmp/events \

extremecoders-re

Setting up Qemu with a tap interface

There are two parts to networking within QEMU:

• The virtual network device that is provided to the guest (e.g. a PCI network card).
• The network backend that interacts with the emulated NIC (e.g. puts packets onto the host's network).

Example: User mode network

mcastelino

Connecting a veth device to tap

• veth device from CNI/CNM plugin: eth0
• tap device that connects to the VM: tap0

Redirecting traffic between the two devices

tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev tap0

mcastelino

Tracing QEMU-KVM Interactions

But default in linux you can figure out how many times and for what reasons there is a VM Exit from a VM into the kvm kernel module. However given the ubiquity of vhost and the ability of kvm to emulate most device models directly in the kernel, most of those VM exits do not result in a transition from host kernel into the QEMU. The transitions from VM -> kvm -> QEMU are typically the most expensive.

Here we try to figure out how many of the VM Exits result in the invocation of QEMU.

Tracking VM-KVM Interactions

This can be done very simply with perf

egernst

Using a Docker CNM plugin to play with SRIO-V

This gist describes the setup necessary for testing SRIO-V based connectivity between two physical boxes which are each setup as described here, and directly connected via their respective SRIO-V enabled NICs.

Setup host system's packages

For this scenario, I'm setting up two Ubuntu 16.04 systems which have a SRIO-V enabled interface as well as a second port for accessing the SUT. To setup:

mcastelino

How docker swarm load balanced traffic flows (on a given host).

How the traffic that get to a host is outside the scope of this.

Create a simple service

docker service create --name testswarm --replicas 1 --publish 8080:80 nginx /bin/bash -c "hostname > /usr/share/nginx/html/hostname; nginx -g \"daemon off;\""

nerdalert

Docker Macvlan and Ipvlan Experimental Driver Examples

• The build will be vendored into github.com/docker/docker in the next few days. In the meantime here is the binary that will be getting vendored. docker-1.11.0-dev.zip
• Ipvlan L2 mode network with multiple subnets without a parent specified
• For a long test that will create 54 networks and 120+ containers, then delete them all and recreate them again try ipvlan-macvlan-it.sh Instructions here Docker Macvlan and Ipvlan Manual IT Test
• FYI Note: When the parent is empty or the --internal flag is used, a linux type dummy interface is dynamically created by Libnetwork to act as the parent. This network is completely isolated and is the equivalent to a --internal flag. This is a good mode for demoing.
• The first test requires an interface