I have two Kubernetes clusters created, GKE clusters in the us-east1 region specifically.
We want to be certain that we're setting up Consul with federation enabled. For this, I'm following the WAN Federation Between Multiple Kubernetes Clusters Through Mesh Gateways guide.
Notably, we want to install Consul using v0.48.0
in our primary datacenter and v0.49.0
in our secondary datacenter. For context, I'm using kubectx + kubens to manage my K8s contexts and namespaces.
$ kctx dc1
Switched to context "dc1".
$ kubectl apply --kustomize "github.com/hashicorp/consul-api-gateway/config/crd?ref=v0.4.0"
customresourcedefinition.apiextensions.k8s.io/gatewayclassconfigs.api-gateway.consul.hashicorp.com created
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
...
$ helm upgrade --install --values ./values-dc1.yaml consul hashicorp/consul --version="0.48.0" --create-namespace --namespace consul
Release "consul" does not exist. Installing it now.
W1013 15:39:51.233871 98549 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1013 15:39:52.640457 98549 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: consul
LAST DEPLOYED: Thu Oct 13 15:39:30 2022
NAMESPACE: consul
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!
Your release is named consul.
...
$ kns consul
Context "dc1" modified.
Active namespace is "consul".
$ kubectl apply -f ./proxydefaults.yaml
proxydefaults.consul.hashicorp.com/global created
$ kubectl get secret consul-federation --namespace consul --output yaml > consul-federation-secret.yaml
First, get the API URL and paste it into values-dc2.yaml
.
$ kctx dc2
Switched to context "dc2".
# Copy the API URL for use in values-dc2.yaml
$ export CLUSTER=$(kubectl config view -o jsonpath="{.contexts[?(@.name == \"$(kubectl config current-context)\")].context.cluster}")
$ kubectl config view -o jsonpath="{.clusters[?(@.name == \"$CLUSTER\")].cluster.server}" | pbcopy
# <PASTE API URL INTO values-dc2.yaml>
$ kubectl create namespace consul
namespace/consul created
$ kns consul
Context "dc2" modified.
Active namespace is "consul".
$ kubectl apply -f ./consul-federation-secret.yaml
secret/consul-federation created
$ kubectl apply --kustomize "github.com/hashicorp/consul-api-gateway/config/crd?ref=v0.4.0"
customresourcedefinition.apiextensions.k8s.io/gatewayclassconfigs.api-gateway.consul.hashicorp.com created
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
...
$ helm upgrade --install --values ./values-dc2.yaml consul hashicorp/consul --version="0.49.0" --create-namespace --namespace consul
Release "consul" does not exist. Installing it now.
W1013 15:52:53.029229 99674 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1013 15:52:54.302580 99674 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: consul
LAST DEPLOYED: Thu Oct 13 15:52:30 2022
NAMESPACE: consul
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!
Your release is named consul.
...
$ kubectl apply -f proxydefaults.yaml
proxydefaults.consul.hashicorp.com/global created
# Verify federation was successful
$ kubectl exec statefulset/consul-server --namespace consul -- consul members -wan
$ kubectl apply -f gateway.yaml
gateway.gateway.networking.k8s.io/api-gateway created
$ kubectl apply -f service.yaml
servicedefaults.consul.hashicorp.com/echo-1 created
service/echo-1 created
serviceaccount/echo-1 created
deployment.apps/echo-1 created
referencegrant.gateway.networking.k8s.io/example-reference-grant created
$ kubectl apply -f route.yaml
httproute.gateway.networking.k8s.io/echo-1 created
$ export GATEWAY_IP=$(kubectl get gateway api-gateway -o json | jq -r '.status.addresses | first | .value')
$ curl --insecure https://$GATEWAY_IP:8443
{
"path": "/",
"host": "34.75.90.220:8443",
"method": "GET",
...
"namespace": "default",
"ingress": "",
"service": "echo-1",
"pod": "echo-1-bcbf544f-6n724"
}
$ kubectl get deployment consul-api-gateway-controller -o yaml | yq '.spec.template.spec.initContainers[1]'
command:
- /bin/sh
- -ec
- |
consul-k8s-control-plane acl-init \
-component-name=api-gateway-controller \
-acl-auth-method=consul-k8s-component-auth-method-dc2 \
-primary-datacenter=dc1 \
-consul-api-timeout=5s \
-log-level=info \
-log-json=false
env:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: CONSUL_CACERT
value: /consul/tls/ca/tls.crt
- name: CONSUL_HTTP_ADDR
value: https://$(HOST_IP):8501
image: hashicorp/consul-k8s-control-plane:0.49.0
imagePullPolicy: IfNotPresent
name: api-gateway-controller-acl-init
resources:
limits:
cpu: 50m
memory: 25Mi
requests:
cpu: 50m
memory: 25Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /consul/login
name: consul-data
- mountPath: /consul/tls/ca
name: consul-ca-cert
readOnly: true
$ kubectl get deployment consul-api-gateway-controller -o yaml | yq '.spec.template.spec.containers[0]'
command:
- /bin/sh
- -ec
- |
consul-api-gateway server \
-sds-server-host consul-api-gateway-controller.consul.svc \
-k8s-namespace consul \
-primary-datacenter=dc1 \
-log-level info \
-log-json=false
env:
- name: CONSUL_CACERT
value: /consul/tls/ca/tls.crt
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: CONSUL_HTTP_TOKEN_FILE
value: /consul/login/acl-token
- name: CONSUL_HTTP_ADDR
value: https://$(HOST_IP):8501
image: hashicorppreview/consul-api-gateway:0.5-dev
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -ec
- /consul-bin/consul logout
name: api-gateway-controller
ports:
- containerPort: 9090
name: sds
protocol: TCP
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /consul-bin
name: consul-bin
- mountPath: /consul/tls/ca
name: consul-ca-cert
readOnly: true
- mountPath: /consul/login
name: consul-data
readOnly: true
@nathancoleman , I've finally had time to test this, everything appears to be working correctly all status details appear fine, the api gateway works in the first data center, but when trying to curl to the api gateway in the second data center I get the following:
If I telnet on the first data center I can connect to the server, on the second data center:
The network configurations are the same for both data centers, so I'm really not sure where the problem is. There's nothing useful in the logs for the API gateway.