Test even loadbalancing

README.md

Create the deploymentn with 100 backends and one Service

kubectl apply -f backends.yaml

Run a pod to test the Service using the httptest program, that allows to specify different parameters of the http connections

./httptest -h
Usage: httpget [options] [url]

  -duration int
        Duration in seconds (default 5)
  -keepalives
        http keepalives (default true)
  -qps int
        queries per second (default 3)
  -requests int
        Number of requests (default 5)
  -workers int
        Number of workers (default 2)

The output is easy to parse with the common cli tools like sort, uniq, awk, ...

kubectl run -it test --privileged --image  registry.k8s.io/e2e-test-images/agnhost:2.39 --command -- ash
$ wget https://storage.googleapis.com/aojea-misc/httptest && chmod +x httptest
$ ./httptest --keepalives=false -requests 200 -workers 10 -duration 200 -qps 10 http://10.128.0.12/hostname > test.log
$ ./httptest --keepalives=true -requests 200 -workers 10 -duration 200 -qps 10 http://10.128.0.12/hostname > test_keepalives.log

Cilium

Using http keepalives the connection is reused and the traffic is directed to the same backend because the client reuses the TCP connection

cut -d\  -f 5 test_keepalives.log | sort | uniq  -c | sort -r
     40 backends-7cbf886b86-jg27f
     40 backends-7cbf886b86-fsgk8
     20 backends-7cbf886b86-wgrc7
     20 backends-7cbf886b86-mbc74
     20 backends-7cbf886b86-dhd2m
     20 backends-7cbf886b86-ch8rj
     20 backends-7cbf886b86-b5l72
     20 backends-7cbf886b86-7rvds

without http keepalives the traffic is sent to different backends

cut -d\  -f 5 test.log | sort | uniq  -c | sort -r
      6 backends-7cbf886b86-zv99t
      6 backends-7cbf886b86-wpwz7
      6 backends-7cbf886b86-pvs5s
      5 backends-7cbf886b86-sgcg8
      5 backends-7cbf886b86-9mvd4
      4 backends-7cbf886b86-x9rxc
      4 backends-7cbf886b86-v6498
      4 backends-7cbf886b86-jqtjh
      4 backends-7cbf886b86-d7mvr
      4 backends-7cbf886b86-7sgrg
      4 backends-7cbf886b86-5b2tc
      4 backends-7cbf886b86-4pwwg
      3 backends-7cbf886b86-w4s9z
      3 backends-7cbf886b86-tn5x9
      3 backends-7cbf886b86-tfsmz
      3 backends-7cbf886b86-t4knn
      3 backends-7cbf886b86-ssj9t
      3 backends-7cbf886b86-r778l
      3 backends-7cbf886b86-qmlmz
      3 backends-7cbf886b86-qlfvv
      3 backends-7cbf886b86-pw9jd
      3 backends-7cbf886b86-pn7ts
      3 backends-7cbf886b86-lfj9v
      3 backends-7cbf886b86-hdvfd
      3 backends-7cbf886b86-g6nc8
      3 backends-7cbf886b86-c7pp2
      3 backends-7cbf886b86-c7n5c
      3 backends-7cbf886b86-7gxd2
      3 backends-7cbf886b86-5qh9d
      3 backends-7cbf886b86-4tklz
      3 backends-7cbf886b86-27s6f
      2 backends-7cbf886b86-z9jhd
      2 backends-7cbf886b86-xf6vw
      2 backends-7cbf886b86-xbmrm
      2 backends-7cbf886b86-wgrc7
      2 backends-7cbf886b86-vvbgw
      2 backends-7cbf886b86-rr7dc
      2 backends-7cbf886b86-r8bz5
      2 backends-7cbf886b86-mbc74
      2 backends-7cbf886b86-lwctj
      2 backends-7cbf886b86-lvxm4
      2 backends-7cbf886b86-knz85
      2 backends-7cbf886b86-jg27f
      2 backends-7cbf886b86-hrsqx
      2 backends-7cbf886b86-h8zvz
      2 backends-7cbf886b86-gjr8m
      2 backends-7cbf886b86-fsgk8
      2 backends-7cbf886b86-fpq4k
      2 backends-7cbf886b86-fm2b8
      2 backends-7cbf886b86-dm566
      2 backends-7cbf886b86-b9cxf
      2 backends-7cbf886b86-b72dv
      2 backends-7cbf886b86-b6dng
      2 backends-7cbf886b86-8lqmn
      2 backends-7cbf886b86-7rvds
      2 backends-7cbf886b86-7cz9g
      2 backends-7cbf886b86-2wg6b
      2 backends-7cbf886b86-2nxs7
      2 backends-7cbf886b86-2nmhr
      2 backends-7cbf886b86-294wq
      1 backends-7cbf886b86-z9sdz
      1 backends-7cbf886b86-z4rjx
      1 backends-7cbf886b86-xlwjs
      1 backends-7cbf886b86-xhkn9
      1 backends-7cbf886b86-x8dms
      1 backends-7cbf886b86-x295h
      1 backends-7cbf886b86-vxxzg
      1 backends-7cbf886b86-vhq9j
      1 backends-7cbf886b86-tm4wt
      1 backends-7cbf886b86-swpx4
      1 backends-7cbf886b86-qvrvp
      1 backends-7cbf886b86-phz87
      1 backends-7cbf886b86-phcp2
      1 backends-7cbf886b86-ndd28
      1 backends-7cbf886b86-mztxk
      1 backends-7cbf886b86-mr42c
      1 backends-7cbf886b86-l9lrv
      1 backends-7cbf886b86-kc4df
      1 backends-7cbf886b86-dkmmh
      1 backends-7cbf886b86-dhd2m
      1 backends-7cbf886b86-ch8rj
      1 backends-7cbf886b86-c45ct
      1 backends-7cbf886b86-b5l72
      1 backends-7cbf886b86-9gttp
      1 backends-7cbf886b86-94wqh
      1 backends-7cbf886b86-8bqm6
      1 backends-7cbf886b86-8bc55
      1 backends-7cbf886b86-5lhgc
      1 backends-7cbf886b86-2fgdj

the distribution does not seem even, but let's see if there is statistically difference running much more requests

/ # ./httptest --keepalives=false -requests 50000 -workers 30 -duration 20000 -qps 1000 http://10.128.0.12/hostname > test2.log
/ # cut -d\  -f 5 test2.log | sort | uniq  -c | sort -r
    620 backends-7cbf886b86-tfsmz
    591 backends-7cbf886b86-7cz9g
    590 backends-7cbf886b86-5lhgc
    588 backends-7cbf886b86-knz85
    580 backends-7cbf886b86-mlrnb
    577 backends-7cbf886b86-xbmrm
    575 backends-7cbf886b86-phz87
    571 backends-7cbf886b86-jqtjh
    570 backends-7cbf886b86-vvbgw
    568 backends-7cbf886b86-27s6f
    567 backends-7cbf886b86-pn7ts
    567 backends-7cbf886b86-hdvfd
    566 backends-7cbf886b86-txbn6
    565 backends-7cbf886b86-rsdm6
    564 backends-7cbf886b86-2nxs7
    562 backends-7cbf886b86-dhd2m
    559 backends-7cbf886b86-w4s9z
    553 backends-7cbf886b86-vhq9j
    548 backends-7cbf886b86-9mvd4
    546 backends-7cbf886b86-7rvds
    545 backends-7cbf886b86-hrsqx
    542 backends-7cbf886b86-c7pp2
    540 backends-7cbf886b86-2wg6b
    538 backends-7cbf886b86-fpq4k
    536 backends-7cbf886b86-t4knn
    534 backends-7cbf886b86-2fgdj
    531 backends-7cbf886b86-ppz2g
    529 backends-7cbf886b86-jzck4
    525 backends-7cbf886b86-v6498
    520 backends-7cbf886b86-tn5x9
    517 backends-7cbf886b86-xf6vw
    517 backends-7cbf886b86-x8dms
    517 backends-7cbf886b86-ssj9t
    517 backends-7cbf886b86-8bqm6
    515 backends-7cbf886b86-d7mvr
    514 backends-7cbf886b86-phcp2
    509 backends-7cbf886b86-wgrc7
    509 backends-7cbf886b86-r778l
    507 backends-7cbf886b86-swpx4
    506 backends-7cbf886b86-kc4df
    506 backends-7cbf886b86-g6nc8
    503 backends-7cbf886b86-z4rjx
    503 backends-7cbf886b86-ndd28
    503 backends-7cbf886b86-mztxk
    501 backends-7cbf886b86-b9cxf
    500 backends-7cbf886b86-x9rxc
    500 backends-7cbf886b86-wpwz7
    498 backends-7cbf886b86-bprgg
    497 backends-7cbf886b86-jjvjz
    497 backends-7cbf886b86-gjr8m
    497 backends-7cbf886b86-c7n5c
    493 backends-7cbf886b86-lfj9v
    492 backends-7cbf886b86-qvrvp
    492 backends-7cbf886b86-h8zvz
    492 backends-7cbf886b86-7sgrg
    492 backends-7cbf886b86-4pwwg
    490 backends-7cbf886b86-l9lrv
    490 backends-7cbf886b86-5b2tc
    489 backends-7cbf886b86-zv99t
    488 backends-7cbf886b86-x295h
    487 backends-7cbf886b86-c45ct
    486 backends-7cbf886b86-z9jhd
    486 backends-7cbf886b86-rr7dc
    486 backends-7cbf886b86-dkmmh
    483 backends-7cbf886b86-5qh9d
    482 backends-7cbf886b86-pw9jd
    481 backends-7cbf886b86-94wqh
    481 backends-7cbf886b86-8bc55
    479 backends-7cbf886b86-mbc74
    478 backends-7cbf886b86-vxxzg
    473 backends-7cbf886b86-qlfvv
    470 backends-7cbf886b86-z9sdz
    470 backends-7cbf886b86-qmlmz
    470 backends-7cbf886b86-b72dv
    470 backends-7cbf886b86-9z5nr
    470 backends-7cbf886b86-2nmhr
    470 backends-7cbf886b86-294wq
    469 backends-7cbf886b86-b6dng
    467 backends-7cbf886b86-mr42c
    462 backends-7cbf886b86-b5l72
    460 backends-7cbf886b86-xlwjs
    460 backends-7cbf886b86-xhkn9
    458 backends-7cbf886b86-tqs7b
    457 backends-7cbf886b86-r8bz5
    456 backends-7cbf886b86-pvs5s
    455 backends-7cbf886b86-4tklz
    448 backends-7cbf886b86-9gttp
    446 backends-7cbf886b86-ch8rj
    445 backends-7cbf886b86-jg27f
    443 backends-7cbf886b86-2sb69
    441 backends-7cbf886b86-fsgk8
    440 backends-7cbf886b86-8lqmn
    436 backends-7cbf886b86-fcjd9
    430 backends-7cbf886b86-dm566
    425 backends-7cbf886b86-sgcg8
    408 backends-7cbf886b86-lvxm4
    394 backends-7cbf886b86-fm2b8
    390 backends-7cbf886b86-tm4wt
    390 backends-7cbf886b86-7gxd2
    380 backends-7cbf886b86-lwctj

the number of conntrack entries is checking the PodIP is similra to the number of requests and we didn't exhaused the ephemeral port range on this environment, so it is not likely that we are reusing something

cilium bpf ct list global -d | grep 10.48.0.115 | wc
level=info msg="Initializing Google metrics" subsys=metrics
  39308 1021869 11781540

It seems cilium uses this code to select the backend

* Backend slot 0 is always reserved for the service frontend. */
#if LB_SELECTION == LB_SELECTION_RANDOM
static __always_inline __u32
lb4_select_backend_id(struct __ctx_buff *ctx,
		      struct lb4_key *key,
		      const struct ipv4_ct_tuple *tuple __maybe_unused,
		      const struct lb4_service *svc)
{
	__u16 slot = (get_prandom_u32() % svc->count) + 1;
	struct lb4_service *be = lb4_lookup_backend_slot(ctx, key, slot);

	return be ? be->backend_id : 0;
}

kube-proxy

kube-proxy shows similar behavior

 ./httptest --keepalives=false -requests 50000 -workers 30 -duration 20000 -qps 1000  http://10.76.6.103/hostname > test2.log

 cut -d\  -f 5 test2.log | sort | uniq  -c | sort -r
    552 backends-99fc4ffd8-tnx77
    545 backends-99fc4ffd8-m8k58
    543 backends-99fc4ffd8-lpxmf
    543 backends-99fc4ffd8-bpfxv
    540 backends-99fc4ffd8-bzxgq
    538 backends-99fc4ffd8-jc8vt
    533 backends-99fc4ffd8-tcgg6
    532 backends-99fc4ffd8-7bfll
    530 backends-99fc4ffd8-mtwfs
    530 backends-99fc4ffd8-dsx68
    528 backends-99fc4ffd8-477kx
    526 backends-99fc4ffd8-gd226
    526 backends-99fc4ffd8-flpbv
    525 backends-99fc4ffd8-2gfws
    524 backends-99fc4ffd8-h2crh
    523 backends-99fc4ffd8-rhpvx
    522 backends-99fc4ffd8-9mg44
    521 backends-99fc4ffd8-x2kng
    521 backends-99fc4ffd8-p8rsf
    521 backends-99fc4ffd8-dtz5w
    520 backends-99fc4ffd8-7bbhn
    519 backends-99fc4ffd8-27j9l
    518 backends-99fc4ffd8-p8h8c
    518 backends-99fc4ffd8-mkqbs
    518 backends-99fc4ffd8-7n4qq
    517 backends-99fc4ffd8-9hq8c
    516 backends-99fc4ffd8-s627l
    516 backends-99fc4ffd8-98qhk
    516 backends-99fc4ffd8-7vl86
    514 backends-99fc4ffd8-vbj5t
    513 backends-99fc4ffd8-tmsjd
    513 backends-99fc4ffd8-84rb7
    512 backends-99fc4ffd8-zkb6j
    512 backends-99fc4ffd8-xvztf
    512 backends-99fc4ffd8-7gvfs
    511 backends-99fc4ffd8-cf7s8
    510 backends-99fc4ffd8-xsgvm
    509 backends-99fc4ffd8-fq2s2
    508 backends-99fc4ffd8-sx5zd
    508 backends-99fc4ffd8-pchnx
    507 backends-99fc4ffd8-hpvmd
    506 backends-99fc4ffd8-kdzg4
    505 backends-99fc4ffd8-b5wfg
    504 backends-99fc4ffd8-vs2sb
    504 backends-99fc4ffd8-brzg6
    504 backends-99fc4ffd8-6fx6r
    502 backends-99fc4ffd8-8mmdd
    502 backends-99fc4ffd8-648f4
    500 backends-99fc4ffd8-p4v85
    500 backends-99fc4ffd8-8rt52
    499 backends-99fc4ffd8-txhtx
    496 backends-99fc4ffd8-hcmcw
    495 backends-99fc4ffd8-t7rp8
    494 backends-99fc4ffd8-vldmv
    494 backends-99fc4ffd8-s7fld
    494 backends-99fc4ffd8-rkbfh
    494 backends-99fc4ffd8-q9hqq
    494 backends-99fc4ffd8-l5r7p
    494 backends-99fc4ffd8-7sqzh
    494 backends-99fc4ffd8-6nznr
    493 backends-99fc4ffd8-w8jfk
    493 backends-99fc4ffd8-vmmbc
    492 backends-99fc4ffd8-zvj46
    492 backends-99fc4ffd8-pmlz4
    491 backends-99fc4ffd8-kjc7k
    491 backends-99fc4ffd8-7kp2l
    489 backends-99fc4ffd8-r72lh
    489 backends-99fc4ffd8-hhdjb
    489 backends-99fc4ffd8-gdfxv
    489 backends-99fc4ffd8-4nwkr
    488 backends-99fc4ffd8-48p8c
    487 backends-99fc4ffd8-rhdt5
    487 backends-99fc4ffd8-n2xsx
    487 backends-99fc4ffd8-5hrcf
    486 backends-99fc4ffd8-2dzsn
    485 backends-99fc4ffd8-k8ncv
    484 backends-99fc4ffd8-s6s6m
    484 backends-99fc4ffd8-dnhm4
    484 backends-99fc4ffd8-dfdl2
    481 backends-99fc4ffd8-xwdb7
    481 backends-99fc4ffd8-k9hsd
    480 backends-99fc4ffd8-vsm7x
    480 backends-99fc4ffd8-t42cw
    480 backends-99fc4ffd8-cvtrn
    480 backends-99fc4ffd8-9fzpx
    476 backends-99fc4ffd8-rbz7c
    476 backends-99fc4ffd8-n7wnz
    475 backends-99fc4ffd8-wbgc9
    475 backends-99fc4ffd8-dpxp8
    473 backends-99fc4ffd8-x5zr5
    473 backends-99fc4ffd8-25zpz
    472 backends-99fc4ffd8-mqcwn
    468 backends-99fc4ffd8-nvk6d
    464 backends-99fc4ffd8-zk66c
    463 backends-99fc4ffd8-fwdzl
    459 backends-99fc4ffd8-8x569
    457 backends-99fc4ffd8-h2r8f
    451 backends-99fc4ffd8-7fd7g
    448 backends-99fc4ffd8-fzqjp
    423 backends-99fc4ffd8-x7zh9

kube-proxy uses the statistics mode that use the same function get_random_u32

static bool
statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
{
	const struct xt_statistic_info *info = par->matchinfo;
	bool ret = info->flags & XT_STATISTIC_INVERT;
	int nval, oval;

	switch (info->mode) {
	case XT_STATISTIC_MODE_RANDOM:
		if ((get_random_u32() & 0x7FFFFFFF) < info->u.random.probability)
			ret = !ret;
		break;
	case XT_STATISTIC_MODE_NTH:
		do {
			oval = atomic_read(&info->master->count);
			nval = (oval == info->u.nth.every) ? 0 : oval + 1;
		} while (atomic_cmpxchg(&info->master->count, oval, nval) != oval);
		if (nval == 0)
			ret = !ret;
		break;
	}

	return ret;
}

However, kube-proxy iptables uses the random function differently

kubernetes/kubernetes#37932 (comment)

backends.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backends
  labels:
    app: backends
spec:
  replicas: 100
  selector:
    matchLabels:
      app: backends
  template:
    metadata:
      labels:
        app: backends
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: agnhost
        image: k8s.gcr.io/e2e-test-images/agnhost:2.39
        args:
          - netexec
          - --http-port=80
          - --delay-shutdown=30
---
apiVersion: v1
kind: Service
metadata:
  name: backends
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: backends
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

httptest.go

// Copyright 2023 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"context"
	"crypto/tls"
	"flag"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/http/httptrace"
	"net/url"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"

	"golang.org/x/time/rate"
)

var (
	numRequests int
	duration    int
	concurrency int
	qps         int
	keepalive   bool
)

func init() {
	flag.IntVar(&numRequests, "requests", 5, "Number of requests")
	flag.IntVar(&duration, "duration", 5, "Duration in seconds")
	flag.IntVar(&concurrency, "workers", 2, "Number of workers")
	flag.IntVar(&qps, "qps", 3, "queries per second")
	flag.BoolVar(&keepalive, "keepalives", true, "http keepalives")

	flag.Usage = func() {
		fmt.Fprint(os.Stderr, "Usage: httpget [options] [url]\n\n")
		flag.PrintDefaults()
	}
}

func main() {
	// Parse command line flags and arguments
	flag.Parse()
	args := flag.Args()

	// trap Ctrl+C and call cancel on the context
	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(duration)*time.Second)

	// Enable signal handler
	signalCh := make(chan os.Signal, 2)
	defer func() {
		close(signalCh)
		cancel()
	}()

	signal.Notify(signalCh, os.Interrupt, syscall.SIGINT)
	go func() {
		select {
		case <-signalCh:
			log.Printf("Exiting: received signal")
			cancel()
		case <-ctx.Done():
		}
	}()

	if len(args) != 1 {
		flag.Usage()
		os.Exit(1)
	}

	u, err := url.ParseRequestURI(args[0])
	if err != nil {
		log.Fatalf("error parsing destination url %s : %v", args[0], err)
	}

	requests := make(chan string, numRequests)
	results := make(chan *requestInfo, numRequests)
	defer close(results)
	// create the workers
	for i := 0; i < concurrency; i++ {
		go worker(ctx, requests, results)
	}
	// feed the workers
	limiter := rate.NewLimiter(rate.Limit(qps), 1)
	for i := 0; i < numRequests; i++ {
		err = limiter.Wait(ctx)
		if err != nil {
			log.Printf("error waiting for the rate limiter: %v", err)
		} else {
			requests <- u.String()
		}
	}

	fmt.Println("local", "URL", "body", "status", "DNSLookup", "Dialing", "ServerProcessing", "Duration")
	for i := 0; i < numRequests; i++ {
		select {
		case r, ok := <-results:
			if !ok { //ch is closed //immediately return err
				break
			}

			body := r.ResponseBody
			if len(body) > 64 {
				body = body[:64]
			}
			if r.ResponseErr != nil {
				body = "error"
			}
			fmt.Println(r.RequestStart, r.RequestLocalAddr, r.RequestURL, body, r.ResponseStatus, r.DNSLookup, r.Dialing, r.ServerProcessing, r.Duration)
		case <-ctx.Done():
			log.Fatal("context cancelled")
		}
	}

	os.Exit(0)
}

func worker(ctx context.Context, requests chan string, results chan *requestInfo) {
	transport := http.Transport{
		Proxy:               http.ProxyFromEnvironment,
		DialContext:         dialer().DialContext,
		TLSHandshakeTimeout: 10 * time.Second,
		DisableKeepAlives:   !keepalive,
	}

	client := http.Client{
		Transport: &transport,
		Timeout:   5 * time.Second,
	}

	for {
		var req *http.Request
		var err error
		select {
		case <-ctx.Done():
			return
		case uri := <-requests:
			req, err = http.NewRequest("GET", uri, nil)
			if err != nil {
				log.Printf("error creating request %s : %v", uri, err)
			}
			req.Header.Set("User-Agent", "Test/1.0")
		}
		startTime := time.Now()

		reqInfo := &requestInfo{
			RequestURL:     req.URL.String(),
			RequestVerb:    req.Method,
			RequestHeaders: req.Header,
			RequestStart:   startTime.Format("2006-01-02 15:04:05"),
		}
		var getConn, dnsStart, dialStart, tlsStart, serverStart time.Time
		trace := &httptrace.ClientTrace{
			// DNS
			DNSStart: func(info httptrace.DNSStartInfo) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				dnsStart = time.Now()
			},
			DNSDone: func(info httptrace.DNSDoneInfo) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				reqInfo.DNSLookup = time.Since(dnsStart)
			},
			// Dial
			ConnectStart: func(network, addr string) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				dialStart = time.Now()
			},
			ConnectDone: func(network, addr string, err error) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				reqInfo.Dialing = time.Since(dialStart)
			},
			// TLS
			TLSHandshakeStart: func() {
				tlsStart = time.Now()
			},
			TLSHandshakeDone: func(_ tls.ConnectionState, _ error) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				reqInfo.TLSHandshake = time.Since(tlsStart)
			},
			// Connection (it can be DNS + Dial or just the time to get one from the connection pool)
			GetConn: func(hostPort string) {
				getConn = time.Now()
			},
			GotConn: func(info httptrace.GotConnInfo) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				reqInfo.GetConnection = time.Since(getConn)
				reqInfo.ConnectionReused = info.Reused
				reqInfo.RequestLocalAddr = info.Conn.LocalAddr().String()
			},
			// Server Processing (time since we wrote the request until first byte is received)
			WroteRequest: func(info httptrace.WroteRequestInfo) {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				serverStart = time.Now()
			},
			GotFirstResponseByte: func() {
				reqInfo.muTrace.Lock()
				defer reqInfo.muTrace.Unlock()
				reqInfo.ServerProcessing = time.Since(serverStart)
			},
		}
		req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
		resp, err := client.Do(req)
		if err != nil {
			log.Printf("http request error: %v", err)
			reqInfo.ResponseErr = err
			results <- reqInfo
			continue
		}
		reqInfo.Duration = time.Since(startTime)
		reqInfo.ResponseStatus = resp.Status
		// Read now to capture time to read full response, not just headers
		data, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			log.Printf("http request error: %v", err)
			reqInfo.ResponseErr = err
			results <- reqInfo
			continue
		}
		defer resp.Body.Close()
		reqInfo.ResponseBody = string(data)
		results <- reqInfo
	}
}

// The dialer reduces the TIME-WAIT period to 1 seconds instead of the OS default of 60 seconds.
// Using 1 second instead of 0 because SO_LINGER socket option to 0 causes pending data to be
// discarded and the connection to be aborted with an RST rather than for the pending data to be
// transmitted and the connection closed cleanly with a FIN.
func dialer() *net.Dialer {
	dialer := &net.Dialer{
		Control: func(network, address string, c syscall.RawConn) error {
			return c.Control(func(fd uintptr) {
				syscall.SetsockoptLinger(int(fd), syscall.SOL_SOCKET, syscall.SO_LINGER, &syscall.Linger{Onoff: 1, Linger: 1})
			})
		},
	}
	return dialer
}

// requestInfo keeps track of information about a request/response combination
type requestInfo struct {
	RequestHeaders   http.Header
	RequestVerb      string
	RequestURL       string
	RequestLocalAddr string
	RequestStart     string

	ResponseStatus  string
	ResponseBody    string
	ResponseHeaders http.Header
	ResponseErr     error

	muTrace          sync.Mutex // Protect trace fields
	DNSLookup        time.Duration
	Dialing          time.Duration
	GetConnection    time.Duration
	TLSHandshake     time.Duration
	ServerProcessing time.Duration
	ConnectionReused bool

	Duration time.Duration
}

@thockin share a nice repro to show that this is working as intended https://go.dev/play/p/2Q-q4EVNBfm

"Working as designed" may not be the same as "working as desired". :)

at least we know what to expect, that is the first step, now we need to figure out how to move to the desired state

Sorry, I wasn't aware of this until yesterday, but what is the problem, and what is the desired state?

I will do some tests myself for the fun of it, e.g. with proxy-mode=nftables, and maybe with proxy-mode=ipvs and maglev hashing (mh).

and what is the desired state?

this depends on the end-user, you can see IPVS has a considerable numbers of algorithms but we don't expose those on the Service API.

What we can do is to document this, I had to do the test myself because I didn't have it very clear, right now the algorithm is:

• same connection goes to same backend
• different connections are randomly balanced