r7vme

---
apiVersion: v1
kind: Pod
metadata:
  name: iscsipd
spec:
  containers:
  - name: iscsipd-rw
    image: kubernetes/pause
    volumeMounts:

r7vme

tmp2|dev⚡ ⇒ terraform plan           
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

azurerm_resource_group.test: Refreshing state... (ID: /subscriptions/1be3b2e6-497b-45b9-915f-...ae23c6a/resourceGroups/roman-test-vmss)
azurerm_virtual_network.test: Refreshing state... (ID: /subscriptions/1be3b2e6-497b-45b9-915f-...crosoft.Network/virtualNetworks/acctvn)
azurerm_public_ip.test: Refreshing state... (ID: /subscriptions/1be3b2e6-497b-45b9-915f-...crosoft.Network/publicIPAddresses/test)
azurerm_lb.test: Refreshing state... (ID: /subscriptions/1be3b2e6-497b-45b9-915f-...s/Microsoft.Network/loadBalancers/test)
azurerm_subnet.test: Refreshing state... (ID: /subscriptions/1be3b2e6-497b-45b9-915f-...virtualNetworks/acctvn/subnets/acctsub)

r7vme

2017-11-29 17:38:32.959 [INFO][98] ipsets.go 297: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=1.36662ms
2017-11-29 17:38:32.959 [INFO][98] int_dataplane.go 705: Finished applying updates to dataplane. msecToApply=1.8547310000000001
2017-11-29 17:38:33.178 [INFO][98] health.go 121: Overall health summary=&health.HealthReport{Live:true, Ready:true}
2017-11-29 17:38:33.495 [INFO][98] health.go 121: Overall health summary=&health.HealthReport{Live:true, Ready:true}
2017-11-29 17:38:43.177 [INFO][98] health.go 121: Overall health summary=&health.HealthReport{Live:true, Ready:true}
2017-11-29 17:38:43.336 [INFO][98] int_dataplane.go 690: Applying dataplane updates
2017-11-29 17:38:43.336 [INFO][98] ipsets.go 224: Asked to resync with the dataplane on next update. family="inet"
2017-11-29 17:38:43.336 [INFO][98] ipsets.go 255: Resyncing ipsets with dataplane. family="inet"
2017-11-29 17:38:43.337 [INFO][98] ipsets.go 297: Finished resync family="inet" numInconsistenciesFound=0 resyncDurati

r7vme

2017-11-30 09:14:12.492980 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Skipping datastore connection test
Using autodetected IPv4 address on interface eth0: 172.23.2.38/30
No AS number configured on node resource, using global value
Using node name: master-w3p6j-3696049930-sqgh9
time="2017-11-30T09:14:13Z" level=info msg="Loading config from environment" 
2017-11-30 09:14:13.155767 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
time="2017-11-30T09:14:13Z" level=info msg="Auto-assign 1 ipv4, 0 ipv6 addrs for host 'master-w3p6j-3696049930-sqgh9'" 
time="2017-11-30T09:14:13Z" level=info msg="Ran out of existing affine blocks for host 'master-w3p6j-3696049930-sqgh9'" 
time="2017-11-30T09:14:13Z" level=info msg="Need to allocate 1 more addresses - allocate another block" 

r7vme

# I'm deleting pod behind this socket 172.31.53.156:443

/ # netstat -nptu
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.20.191.42:46256     172.31.53.156:443       ESTABLISHED 1/kvm-operator-node
tcp        0      0 172.20.191.42:38938     172.31.0.1:443          ESTABLISHED 1/kvm-operator-node


/ # tcpdump -i eth0 -nn host 172.31.53.156

r7vme

# Connection before pod killed
tcp        0      0 172.20.191.42:55406     172.31.53.156:443       ESTABLISHED 1/kvm-operator-node

# Connection after pod killed (in this state for a while)
tcp        0     45 172.20.191.42:55406     172.31.53.156:443       ESTABLISHED 1/kvm-operator-node

# TCP dump right before pod killed

/ # tcpdump -i eth0 -nn host 172.31.53.156
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

r7vme

~|⇒ kubectl logs node-controller-599372825-mb87w -n mo3ti -f
{"caller":"kvm-operator-node-controller/controller.go:137","info":"starting kvm-operator node controller","time":"17-12-08 15:10:45.288"}
{"caller":"github.com/giantswarm/kvm-operator-node-controller/provider/cloud.go:61","info":"trying to use in-cluster config","time":"17-12-08 15:10:45.300"}
{"caller":"kvm-operator-node-controller/controller.go:179","info":"checking node status","node":"worker-320uw-3021243511-wzlns","time":"17-12-08 15:10:45.323"}
{"caller":"kvm-operator-node-controller/controller.go:243","info":"node state","node":"worker-320uw-3021243511-wzlns","state":"True","time":"17-12-08 15:10:45.323"}
{"caller":"kvm-operator-node-controller/controller.go:179","info":"checking node status","node":"master-yq9wi-144519471-3dv37","time":"17-12-08 15:10:45.323"}
{"caller":"kvm-operator-node-controller/controller.go:243","info":"node state","node":"master-yq9wi-144519471-3dv37","state":"True","time":"17-12-08 15:10:45.323"}

{"caller":"github.c

r7vme

/ # netstat -nptu
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0     87 172.20.125.137:60708    172.31.53.156:443       ESTABLISHED 1/kvm-operator-node
tcp        0      0 172.20.125.137:33122    172.31.0.1:443          ESTABLISHED 1/kvm-operator-node
/ # netstat -nptu
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0    132 172.20.125.137:60708    172.31.53.156:443       ESTABLISHED 1/kvm-operator-node
tcp        0      0 172.20.125.137:33122    172.31.0.1:443          ESTABLISHED 1/kvm-operator-node

r7vme

opsctl-linux-amd64|⇒ opsctl ssh centaur master1
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xefabdc]

goroutine 1 [running]:
github.com/giantswarm/opsctl/service/ssh.(*Service).SSH(0xc4203fce30, 0x7ffe1c73f4f0, 0x7, 0x0, 0x0)
        /go/src/github.com/giantswarm/opsctl/service/ssh/service.go:72 +0xfc
github.com/giantswarm/opsctl/command/ssh.(*Command).execute(0xc42022d480, 0x0, 0x0)
        /go/src/github.com/giantswarm/opsctl/command/ssh/command.go:211 +0xa65
github.com/giantswarm/opsctl/command/ssh.(*Command).Execute(0xc42022d480, 0xc420426240, 0xc42022d640, 0x2, 0x2)

r7vme

  1 15:05:16.348441 IP 172.18.7.250.43990 > 10.141.5.158.1521: Flags [S], seq 1338878575, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale     7], length 0
  2 15:05:16.349755 IP 10.141.5.158.1521 > 172.18.7.250.43990: Flags [S.], seq 3998758358, ack 1338878576, win 29200, options [mss 1460,nop,nop,s    ackOK,nop,wscale 7], length 0
  3 15:05:16.349780 IP 172.18.7.250.43990 > 10.141.5.158.1521: Flags [.], ack 1, win 229, length 0
  4 15:05:16.364374 IP 172.18.7.250.43990 > 10.141.5.158.1521: Flags [P.], seq 1:229, ack 1, win 229, length 228
  5 15:05:16.365308 IP 10.141.5.158.1521 > 172.18.7.250.43990: Flags [.], ack 229, win 237, length 0
  6 15:05:16.377677 IP 10.141.5.158.1521 > 172.18.7.250.43990: Flags [P.], seq 1:9, ack 229, win 237, length 8
  7 15:05:16.377713 IP 172.18.7.250.43990 > 10.141.5.158.1521: Flags [.], ack 9, win 229, length 0
  8 15:05:16.378232 IP 172.18.7.250.43990 > 10.141.5.158.1521: Flags [P.], seq 229:457, ack 9, win 229, length 228
  9 15:05:16.381622 IP 10.141.5.158.1521 > 17