Feilong Wang openstacker

## gist:3d65f95e8d7d30cb118ffdf4ad7b3160

Member
SuperQ commented on 23 Jul 2018 •
One additional debugging option, get a dump of the goroutines with http://localhost:9100/debug/pprof/goroutine?debug=1. Normally it would be only a few, like this.

@juliusv

Member
juliusv commented on 23 Jul 2018
@SuperQ Yeah, I would recommend listing the active goroutines, as we already know it's connected to a goroutine leak. But prefer debug=2 over debug=1, so: http://localhost:9100/debug/pprof/goroutine?debug=2

## gist:1c29260894cc89e9c2e72e2968958490
For anyone who hits this, it's caused by api-services that no longer have backends running...

In my case it was KEDA, but there are a number of different services that install aggregated API servers.

To fix it:

kubectl get apiservice
Look for ones the AVAILABLE is False

If you don't need those APIs any more, delete them:

## gist:b959eda152d2f3844e50ab98d07814f2
echo "HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Server: netcat!

<!doctype html>
<html><body><h1>`hostname`</h1></body></html>" > index.http && while true; do cat index.http | nc -l -p 80; done

## gist:509e67ddb526e9b85499330a3a190338
If you want installable .deb package instead of putting everything to /usr/local, do this:

sudo apt-get build-dep curl
sudo apt-get install libc-ares-dev build-essential
apt-get source curl
cd curl-*
This will download curl sources with Debian/Ubuntu build files and patches.

Edit file debian/control: add line libc-ares-dev to Build-Depends

## build k8s hyperkube
$ export DOCKER_REGISTRY=flwang
$ export HYPERKUBE_VERSION=v1.16.9
$ export KUBE_GIT_TREE_STATE="clean"
$ bash build/run.sh make cross KUBE_FASTBUILD=true ARCH=amd64 && cd cluster/images/hyperkube/ && make VERSION=$HYPERKUBE_VERSION ARCH=amd64 REGISTRY=$DOCKER_REGISTRY && docker push $DOCKER_REGISTRY/hyperkube-amd64:$HYPERKUBE_VERSION


## gist:63f1dd4de96ef0dbdccefd6ede634e7b
openstack coe cluster template list --fields public,hidden,network_driver -f json | jq -c '.[] | select(.public==true and .hidden==false)'

glance --os-image-api-version 1 image-create --disk-format raw --container-format bare --file fedora-coreos-31.20200505.3.0-openstack.x86_64.raw --min-disk 10 --size 10737418240 --name fedora-coreos-31-x86_64 --is-public=True

## devstack k8s template
openstack coe cluster template create cc-v1.17.4 \
--keypair feilong \
--flavor ds1G \
--master-flavor ds2G \
--coe kubernetes \
--external-network e0943db8-4e34-4ef0-8973-93ef72a735e4 \
--network-driver calico \
--docker-storage-driver=overlay2 \
--image=0529c960-92cf-4913-b079-4b7b2984ef4a \
--volume-driver=cinder \

## gist:29bb95ec57f134fbe547dbb159bbe635
kubectl port-forward prometheus-prometheus-prometheus-0 -n kube-system 9090:9090

kubectl port-forward prometheus-operator-grafana-597f5fdffc-q5c7m -n kube-system 3000:3000

kubectl delete apiservice v1beta1.custom.metrics.k8s.io

## set terminal title
set-title(){
  ORIG=$PS1
  TITLE="\e]2;$@\a"
  PS1=${ORIG}${TITLE}
}

## test ds
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: debug-app
  namespace: kube-system
  labels:
    k8s-app: debug-app
spec:
  selector:

	Member
	SuperQ commented on 23 Jul 2018 •
	One additional debugging option, get a dump of the goroutines with http://localhost:9100/debug/pprof/goroutine?debug=1. Normally it would be only a few, like this.

	@juliusv

	Member
	juliusv commented on 23 Jul 2018
	@SuperQ Yeah, I would recommend listing the active goroutines, as we already know it's connected to a goroutine leak. But prefer debug=2 over debug=1, so: http://localhost:9100/debug/pprof/goroutine?debug=2
	For anyone who hits this, it's caused by api-services that no longer have backends running...

	In my case it was KEDA, but there are a number of different services that install aggregated API servers.

	To fix it:

	kubectl get apiservice
	Look for ones the AVAILABLE is False

	If you don't need those APIs any more, delete them:
	echo "HTTP/1.1 200 OK
	Content-Type: text/html; charset=UTF-8
	Server: netcat!

	<!doctype html>
	<html><body><h1>`hostname`</h1></body></html>" > index.http && while true; do cat index.http \| nc -l -p 80; done
	If you want installable .deb package instead of putting everything to /usr/local, do this:

	sudo apt-get build-dep curl
	sudo apt-get install libc-ares-dev build-essential
	apt-get source curl
	cd curl-*
	This will download curl sources with Debian/Ubuntu build files and patches.

	Edit file debian/control: add line libc-ares-dev to Build-Depends
	$ export DOCKER_REGISTRY=flwang
	$ export HYPERKUBE_VERSION=v1.16.9
	$ export KUBE_GIT_TREE_STATE="clean"
	$ bash build/run.sh make cross KUBE_FASTBUILD=true ARCH=amd64 && cd cluster/images/hyperkube/ && make VERSION=$HYPERKUBE_VERSION ARCH=amd64 REGISTRY=$DOCKER_REGISTRY && docker push $DOCKER_REGISTRY/hyperkube-amd64:$HYPERKUBE_VERSION
	openstack coe cluster template list --fields public,hidden,network_driver -f json \| jq -c '.[] \| select(.public==true and .hidden==false)'

	glance --os-image-api-version 1 image-create --disk-format raw --container-format bare --file fedora-coreos-31.20200505.3.0-openstack.x86_64.raw --min-disk 10 --size 10737418240 --name fedora-coreos-31-x86_64 --is-public=True
	openstack coe cluster template create cc-v1.17.4 \
	--keypair feilong \
	--flavor ds1G \
	--master-flavor ds2G \
	--coe kubernetes \
	--external-network e0943db8-4e34-4ef0-8973-93ef72a735e4 \
	--network-driver calico \
	--docker-storage-driver=overlay2 \
	--image=0529c960-92cf-4913-b079-4b7b2984ef4a \
	--volume-driver=cinder \
	kubectl port-forward prometheus-prometheus-prometheus-0 -n kube-system 9090:9090

	kubectl port-forward prometheus-operator-grafana-597f5fdffc-q5c7m -n kube-system 3000:3000

	kubectl delete apiservice v1beta1.custom.metrics.k8s.io
	---
	apiVersion: apps/v1
	kind: DaemonSet
	metadata:
	name: debug-app
	namespace: kube-system
	labels:
	k8s-app: debug-app
	spec:
	selector: