FailoverPriorityReadme.md

Demonstration of Failover Priority with Istio Multicluster

Sample Environment

• Four Clusters: {alvin,simon,theodore,dave}
• Ingress Gateway
• East-West Gateway
• Istiod
• Istio Echo Application

Preparation

1. Create VirtualDestination

apiVersion: networking.gloo.solo.io/v2
kind: VirtualDestination
metadata:
  name: echo
  namespace: default
spec:
  hosts:
  - echo.global
  ports:
  - number: 8080
    protocol: HTTP
    targetPort:
      number: 8080
  - number: 9080
    protocol: GRPC
    targetPort:
      number: 9080
  services:
  - name: backend
    namespace: default

2. Create OutlierDetectionPolicy

apiVersion: resilience.policy.gloo.solo.io/v2
kind: OutlierDetectionPolicy
metadata:
  name: outlier-detection
  namespace: default
spec:
  applyToDestinations:
  - kind: VIRTUAL_DESTINATION
    selector:
      name: echo
  config:
    baseEjectionTime: 30s
    consecutiveErrors: 2
    interval: 1s
    maxEjectionPercent: 100

3. Scale down Gloo Mesh Agents

4. Update Destination Rules with following patch:

    portLevelSettings:
    - loadBalancer:
        localityLbSetting:
          enabled: true
          failoverPriority:
          - tier
      outlierDetection:

5. Label Workloads(Pods/Templates) with tier: primary|secondary

6. Label WorkloadEntries with tier: primary|secondary

Test

1. Scale the client cluster's backend workloads to zero.

2. Send a few dozen requests:

for i in {1..36}; do client http://echo.global:8080 | grep Cluster; done

The results should look like:

[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>
[0 body] Cluster=<<Name of Cluster Matching Tier>>

3. Relabel the client workload to the other tier, redeploy, and regenerate requests.

The results should look like:

[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>
[0 body] Cluster=<<Other Cluster>>

Verification

istioctl lets you look at the endpoints being loadbalanced with the following command:

istioctl pc endpoints deployment/frontend-v1 --cluster "outbound|8080||echo.global" -o yaml

The field that indicates if loadbalancing is working is priority. (Note that priority: 1 is actually the second priority)