gistfile1.txt

Create 2 instances of this pod:
```
apiVersion: v1
kind: Pod
metadata:
  name: netserver-4
  namespace: default
spec:
  containers:
  - command:
    - /netexec
    - --http-port=8080
    - --udp-port=8081
    image: bprashanth/netexec:0.0
    imagePullPolicy: Always
    name: webserver
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    - containerPort: 8081
      name: udp
      protocol: UDP
    resources:
      requests:
        cpu: 100m
```

Get into one and run:
```
$ kubectl exec -it netserver /bin/bash
$ ip addr

container $ tcpdump -i eth0 -vv 
```
Or more simply, this script:
```
#! /bin/bash
kubectl delete pod netserver-4 --grace-period=0; kubectl create -f ~/rtmp/kubeproxytest/netserver.yaml; sleep 2;
kubectl exec -it netserver-4 -- tcpdump -i eth0 -n -vv
```

Bad case:
```
$ kubectl exec -it test-container /bin/bash
container $ echo hostName | timeout 1 nc -u 10.245.1.5 8081
container $
```

Back at the ranch:
```
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:37:42.474291 IP (tos 0x0, ttl 62, id 12309, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.50198 > netserver-4.8081: [udp sum ok] UDP, length 9
19:37:42.890640 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.1 tell netserver-4, length 28
19:37:42.890653 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.1 tell netserver-4, length 28
19:37:42.890667 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.245.1.1 is-at 52:17:41:2a:8b:a4 (oui Unknown), length 28
19:37:42.906727 IP (tos 0x0, ttl 64, id 27436, offset 0, flags [DF], proto UDP (17), length 69)
    netserver-4.59746 > 10.0.0.10.domain: [bad udp cksum 0x1646 -> 0xf85a!] 20097+ PTR? 6.2.245.10.in-addr.arpa. (41)
19:37:42.911169 IP (tos 0x0, ttl 62, id 35120, offset 0, flags [DF], proto UDP (17), length 69)
    10.0.0.10.domain > netserver-4.59746: [udp sum ok] 20097 NXDomain q: PTR? 6.2.245.10.in-addr.arpa. 0/0/0 (41)
19:37:43.912402 IP (tos 0x0, ttl 64, id 27610, offset 0, flags [DF], proto UDP (17), length 69)
    netserver-4.50756 > 10.0.0.10.domain: [bad udp cksum 0x1646 -> 0xb4bd!] 46402+ PTR? 1.1.245.10.in-addr.arpa. (41)
19:37:43.918185 IP (tos 0x0, ttl 62, id 35352, offset 0, flags [DF], proto UDP (17), length 69)
    10.0.0.10.domain > netserver-4.50756: [udp sum ok] 46402 NXDomain q: PTR? 1.1.245.10.in-addr.arpa. 0/0/0 (41)
19:37:43.918374 IP (tos 0x0, ttl 64, id 27611, offset 0, flags [DF], proto UDP (17), length 68)
    netserver-4.55724 > 10.0.0.10.domain: [bad udp cksum 0x1645 -> 0xf561!] 17044+ PTR? 10.0.0.10.in-addr.arpa. (40)
19:37:43.925790 IP (tos 0x0, ttl 62, id 35353, offset 0, flags [DF], proto UDP (17), length 68)
    10.0.0.10.domain > netserver-4.55724: [udp sum ok] 17044 NXDomain q: PTR? 10.0.0.10.in-addr.arpa. 0/0/0 (40)
19:37:47.914966 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has netserver-4 tell 10.245.1.1, length 28
19:37:47.914982 ARP, Ethernet (len 6), IPv4 (len 4), Reply netserver-4 is-at 82:d2:ff:0b:cd:c8 (oui Unknown), length 28
```

Good case:
```
$ kubectl exec -it test-container /bin/bash
container $ echo hostName | timeout 1 nc -u 10.245.1.5 8081
netserver
container $
```
And:
```
19:53:24.798827 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has netserver-4 tell 10.245.1.1, length 28
19:53:24.798852 ARP, Ethernet (len 6), IPv4 (len 4), Reply netserver-4 is-at ce:83:c6:cc:e1:85 (oui Unknown), length 28
19:53:24.798863 IP (tos 0x0, ttl 62, id 55536, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.36636 > netserver-4.8081: [udp sum ok] UDP, length 9
19:53:24.799087 IP (tos 0x0, ttl 64, id 45644, offset 0, flags [DF], proto UDP (17), length 39)
    netserver-4.8081 > 10.245.2.6.36636: [bad udp cksum 0x1919 -> 0xd347!] UDP, length 11
19:53:25.134343 IP (tos 0x0, ttl 64, id 42537, offset 0, flags [DF], proto UDP (17), length 69)
    netserver-4.42663 > 10.0.0.10.domain: [bad udp cksum 0x1646 -> 0x26fa!] 25251+ PTR? 1.1.245.10.in-addr.arpa. (41)
19:53:25.137237 IP (tos 0x0, ttl 62, id 43848, offset 0, flags [DF], proto UDP (17), length 69)
    10.0.0.10.domain > netserver-4.42663: [udp sum ok] 25251 NXDomain q: PTR? 1.1.245.10.in-addr.arpa. 0/0/0 (41)
19:53:25.137519 IP (tos 0x0, ttl 64, id 42538, offset 0, flags [DF], proto UDP (17), length 69)
    netserver-4.58424 > 10.0.0.10.domain: [bad udp cksum 0x1646 -> 0xebdb!] 24618+ PTR? 6.2.245.10.in-addr.arpa. (41)
19:53:25.139888 IP (tos 0x0, ttl 62, id 43849, offset 0, flags [DF], proto UDP (17), length 69)
    10.0.0.10.domain > netserver-4.58424: [udp sum ok] 24618 NXDomain q: PTR? 6.2.245.10.in-addr.arpa. 0/0/0 (41)
19:53:26.141141 IP (tos 0x0, ttl 64, id 42645, offset 0, flags [DF], proto UDP (17), length 68)
    netserver-4.38378 > 10.0.0.10.domain: [bad udp cksum 0x1645 -> 0x8916!] 62113+ PTR? 10.0.0.10.in-addr.arpa. (40)
19:53:26.144167 IP (tos 0x0, ttl 62, id 44088, offset 0, flags [DF], proto UDP (17), length 68)
    10.0.0.10.domain > netserver-4.38378: [udp sum ok] 62113 NXDomain q: PTR? 10.0.0.10.in-addr.arpa. 0/0/0 (40)
19:53:29.802950 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.1 tell netserver-4, length 28
19:53:29.802985 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.245.1.1 is-at 4e:eb:7d:ae:d9:04 (oui Unknown), length 28
```

If it doesn't reproduce kill the 241.1.5 pod and retry.
Note the curious ARP request, before the failing request:
```
netserver-4 $ ip neighbor
netserver-4 $ ip neighbor
```
But after the success:
```
netserver-4 $ ip neighbor
10.245.1.1 dev eth0 lladdr 52:17:41:2a:8b:a4 DELAY
```
Where 10.245.1.1 is cbr0.

All at once:
```
# nc -u 10.245.1.5 8081
hostName
hostName
hostName
hostName
hostName
netserver-4

```
```
pod "netserver-4" deleted
pod "netserver-4" created
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:14:43.751130 IP (tos 0x0, ttl 62, id 33611, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.60741 > 10.245.1.5.8081: [udp sum ok] UDP, length 9
23:14:48.762972 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:14:49.762972 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:14:50.762968 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:14:55.759570 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:14:55.759601 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.245.1.5 is-at f6:f8:8f:3f:57:56, length 28
23:14:55.759617 IP (tos 0x0, ttl 62, id 35222, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.60741 > 10.245.1.5.8081: [udp sum ok] UDP, length 9
23:14:55.759907 IP (tos 0x0, ttl 64, id 42195, offset 0, flags [DF], proto UDP (17), length 39)
    10.245.1.5.8081 > 10.245.2.6.60741: [bad udp cksum 0x1919 -> 0x751e!] UDP, length 11



15:16:17-beeps~/goproj/src/k8s.io/kubernetes] (kubelet_plugins)$ ./repro.sh 
pod "netserver-4" deleted
pod "netserver-4" created
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:16:34.978072 IP (tos 0x0, ttl 62, id 42388, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.35824 > 10.245.1.5.8081: [udp sum ok] UDP, length 9
23:16:41.778665 IP (tos 0x0, ttl 62, id 42414, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.35824 > 10.245.1.5.8081: [udp sum ok] UDP, length 9
23:16:46.928938 IP (tos 0x0, ttl 62, id 43415, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.35824 > 10.245.1.5.8081: [udp sum ok] UDP, length 9


-----3 failures




23:18:03.738968 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:18:04.738960 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:18:05.738956 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:18:07.873561 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.5 tell 10.245.1.1, length 28
23:18:07.873586 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.245.1.5 is-at 02:1a:0e:6b:ba:36, length 28

------no another one 

23:18:07.873600 IP (tos 0x0, ttl 62, id 53565, offset 0, flags [DF], proto UDP (17), length 37)
    10.245.2.6.35824 > 10.245.1.5.8081: [udp sum ok] UDP, length 9
23:18:07.873811 IP (tos 0x0, ttl 64, id 6569, offset 0, flags [DF], proto UDP (17), length 39)
    10.245.1.5.8081 > 10.245.2.6.35824: [bad udp cksum 0x1919 -> 0xd673!] UDP, length 11
23:18:12.874950 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.245.1.1 tell 10.245.1.5, length 28
23:18:12.874980 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.245.1.1 is-at 52:17:41:2a:8b:a4, length 28

```