3scale migration from templates to operator (in HA mode)

README.adoc

https://access.redhat.com/documentation/en-us/red_hat_3scale_api_management/2.8/html-single/operating_3scale/index#threescale-backup-restore

Migration

Preparation

Export variables that will be used in the instructions.

export OLD_DB_HOST=<old DB host>

export DB_USERNAME=<DB username>
export DB_DATABASE=<DB database>
export DB_HOST=<current DB host>

export WILDCARD_DOMAIN=<new cluster wildcard domain>

Back up

1. Backup PostgreSQL

   pg_dump -U ${DB_USERNAME} -W -h ${OLD_DB_HOST} -F c -v ${DB_DATABASE} -f system-database.dump

2. Backing up system-storage

   oc rsync $(oc get pods -l 'deploymentConfig=system-app' --template="{{ (index .items 0).metadata.name }}"):/opt/system/public/system ./local/dir

3. Backing up backend-redis

   oc cp $(oc get pods -l 'deploymentConfig=backend-redis' --template="{{ (index .items 0).metadata.name }}"):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb

4. Backing up system-redis

   oc cp $(oc get pods -l 'deploymentConfig=system-redis' --template="{{ (index .items 0).metadata.name }}"):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb

   (skipping restoring Zync database)

5. Backing up OpenShift secrets and ConfigMaps

   oc get secrets system-smtp -o json > system-smtp.json
   oc get secrets system-seed -o json > system-seed.json
   oc get secrets system-database -o json > system-database.json
   oc get secrets backend-internal-api -o json > backend-internal-api.json
   oc get secrets system-events-hook -o json > system-events-hook.json
   oc get secrets system-app -o json > system-app.json
   oc get secrets system-recaptcha -o json > system-recaptcha.json
   oc get secrets system-redis -o json > system-redis.json
   oc get secrets zync -o json > zync.json
   oc get secrets system-master-apicast -o json > system-master-apicast.json

   oc get configmaps system-environment -o json > system-environment.json
   oc get configmaps apicast-environment -o json > apicast-environment.json

Restore

1. Create a new project

   oc new-project threescale-apim

2. Create a secret for registry.redhat.io

   Follow instructions in https://access.redhat.com/RegistryAuthentication

   oc create secret generic <pull_secret_name> \
       --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
       --type=kubernetes.io/dockerconfigjson>
   
   oc secrets link default <pull_secret_name> --for=pull

3. Install 3scale operator

4. Restore secrets

   (skipping system-database, system-redis and backend-redis secrets)

   Note	if the namespace name has changed, update to the new 3scale namespace in the new cluster.

   oc apply -f system-smtp.json
   oc apply -f system-seed.json
   oc apply -f backend-internal-api.json
   oc apply -f system-events-hook.json
   oc apply -f system-app.json
   oc apply -f system-recaptcha.json
   oc apply -f zync.json
   oc apply -f system-master-apicast.json

5. Restore configmaps

   oc apply -f system-environment.json
   oc apply -f apicast-environment.json

6. Create database secret

   cat << EOF > system-database-secret.yml
   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     labels:
       app: 3scale-database
     name: system-database
   stringData:
     DB_PASSWORD: ${DB_PASSWORD}
     DB_USER: ${DB_USERNAME}
     URL: postgresql://${DB_USERNAME}:${DB_PASSWORD}@${DB_HOST}/${DB_DATABASE}
   EOF

7. Create Cluster Role for Azure File

   cat << EOF > azure-cloud-provider-secrets.yml
   
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: system:azure-cloud-provider-secrets
   rules:
   - apiGroups:
     - ""
     resources:
     - secrets
     verbs:
     - get
     - create
   EOF
   
   oc apply -f azure-cloud-provider-secrets.yml

   See https://docs.openshift.com/container-platform/4.8/storage/dynamic-provisioning.html#azure-file-definition_dynamic-provisioning.

8. Assign role to service account

   oc adm policy add-cluster-role-to-user system:azure-cloud-provider-secrets system:serviceaccount:kube-system:persistent-volume-binder

9. Get UID range on the 3scale namespace and save it in a variable

   oc describe project threescale-apim | grep uid-range
   export OPENSHIFT_UID=<UID>

10. Create RWX Storage Class for 3scale system-storage

    cat << EOF > azure-file-3scale.yml
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: azure-file-3scale
    mountOptions:
    - uid=${OPENSHIFT_UID}
    - gid=0
    - mfsymlinks
    - cache=strict
    parameters:
      skuName: Standard_LRS
    provisioner: kubernetes.io/azure-file
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    EOF
    
    oc create -f azure-file-3scale.yml

11. Create temporary Redis instances

    oc new-app -f template-only-redis.yml

    This template has the following changes with regards to the standard one: * all resources not related to Redis are removed * backend-redis storage increased from 1G to 10G * APP_LABEL (the app label on OpenShift resources) changed from 3scale-api-management to 3scale-redis * removed serviceAccount amp from resources definition

Note	add MESSAGE_BUS_URL value

1. Restore Redis databases

   Follow all the steps in https://access.redhat.com/documentation/en-us/red_hat_3scale_api_management/2.11/html/operating_3scale/threescale-backup-restore-using-templates#ensuring-information-consistency

2. Restore PostgreSQL database (the database should exist)

   pg_restore -d ${DATABASE_NAME} -U ${DATABASE_USERNAME} system-database.dump

3. Create APIManager

   cat << EOF > apimanager.yml
   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager
   spec:
     wildcardDomain: ${WILDCARD_DOMAIN}
     resourceRequirementsEnabled: true
     highAvailability:
       enabled: true
     system:
       fileStorage:
         persistentVolumeClaim:
           storageClassName: azure-file-3scale
   EOF
   
   oc apply -f apimanager.yml

4. Update domains to adjust to the new Wildcard Domain

   See solution How to change the Public domain (Developer Portal Domain), the Admin domain (Admin Portal Domain) and the Master Portal domain in 3scale

To do a batch rewrite of the domains, use the following steps:

bundle exec rails console

Print the current values for the domains

Provider.all.to_a.each do |p|
  puts p.domain
  puts p.self_domain
end

Replace the wildcard domain with the new value

Provider.all.to_a.each do |p|
  p.domain = p.domain.gsub("apps.CURRENT", "apps.NEW")
  p.self_domain = p.self_domain.gsub("apps.CURRENT", "apps.NEW")
  p.save!
end

Verify

Get admin username and password

oc get secret system-seed --template="{{.data.ADMIN_USER}}" | base64 -d
oc get secret system-seed --template="{{.data.ADMIN_PASSWORD}}" | base64 -d

Verify RH-SSO connections

Upgrade 3scale

Use Operator to upgrade 3scale, one version at a time: 2.8 to 2.9 2.9 to 2.10 2.10 to 2.11

Update number of replicas on the APIManager resource

In the APIManager instance, change all replicas from replicas: 1 to replicas: 2, except APIcast. For APIcast it is required to have staging gateway to pull the policies, but the production gateway can be disabled if not used.

spec:
  imageStreamTagImportInsecure: false
  resourceRequirementsEnabled: true
  system:
    appSpec:
      replicas: 2
    fileStorage:
      persistentVolumeClaim:
        storageClassName: azure-file-3scale
    sidekiqSpec:
      replicas: 2
    sphinxSpec: {}
  appLabel: 3scale-api-management
  zync:
    appSpec:
      replicas: 2
    queSpec:
      replicas: 2
  backend:
    cronSpec:
      replicas: 2
    listenerSpec:
      replicas: 2
    workerSpec:
      replicas: 2
  tenantName: 3scale
  apicast:
    managementAPI: status
    openSSLVerify: false
    productionSpec:
      replicas: 0
    registryURL: 'http://apicast-staging:8090/policies'
    responseCodes: true
    stagingSpec:
      replicas: 1
  highAvailability:
    enabled: true
  wildcardDomain: <DOMAIN>

Restore system-storage

oc rsync ./local/dir/system/ $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system

APIcast.adoc

Create a secret for connecting with admin portal

oc create secret generic apicast-admin-portal-secret --from-literal=AdminPortalURL=https://<ACESS_TOKEN>@<PORTAL_URL>

Create secret for TLS certificates

oc create secret tls router-certs --cert=server.crt --key=server.key

Production APIcast

apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: 3scale-apicast-production
spec:
  adminPortalCredentialsRef:
    name: apicast-configuration-url-secret
  replicas: 4
  deploymentEnvironment: production
  configurationLoadMode: boot
  logLevel: warn
  pathRoutingEnabled: true
  responseCodesIncluded: true
  cacheConfigurationSeconds: 120
  managementAPIScope: status
  openSSLPeerVerificationEnabled: false
  httpsPort: 8443
  httpsCertificateSecretRef:
    name: router-certs
  extendedMetrics: true
  resources:
    limits:
      cpu: '2'
      memory: 1Gi
    requests:
      cpu: '1'
      memory: 500Mi

Staging APIcast

apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: 3scale-apicast-staging
spec:
  adminPortalCredentialsRef:
    name: apicast-configuration-url-secret
  replicas: 2
  deploymentEnvironment: staging
  configurationLoadMode: lazy
  logLevel: info
  pathRoutingEnabled: true
  responseCodesIncluded: true
  cacheConfigurationSeconds: 0
  managementAPIScope: status
  openSSLPeerVerificationEnabled: false
  httpsPort: 8443
  httpsCertificateSecretRef:
    name: router-certs
  extendedMetrics: true
  resources:
    limits:
      cpu: '2'
      memory: 500Mi
    requests:
      cpu: '1'
      memory: 128Mi

template-only-redis.yml

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: 3scale-api-management-redis
objects:
- apiVersion: apps.openshift.io/v1
  kind: DeploymentConfig
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: backend
      threescale_component_element: redis
    name: backend-redis
  spec:
    replicas: 1
    selector:
      deploymentConfig: backend-redis
    strategy:
      resources: {}
      type: Recreate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: ${APP_LABEL}
          deploymentConfig: backend-redis
          threescale_component: backend
          threescale_component_element: redis
      spec:
        containers:
        - args:
          - /etc/redis.d/redis.conf
          - --daemonize
          - "no"
          command:
          - /opt/rh/rh-redis5/root/usr/bin/redis-server
          image: backend-redis:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            initialDelaySeconds: 10
            periodSeconds: 10
            tcpSocket:
              port: 6379
          name: backend-redis
          readinessProbe:
            exec:
              command:
              - container-entrypoint
              - bash
              - -c
              - redis-cli set liveness-probe "`date`" | grep OK
            initialDelaySeconds: 10
            periodSeconds: 30
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "2"
              memory: 32Gi
            requests:
              cpu: "1"
              memory: 1Gi
          volumeMounts:
          - mountPath: /var/lib/redis/data
            name: backend-redis-storage
          - mountPath: /etc/redis.d/
            name: redis-config
        volumes:
        - name: backend-redis-storage
          persistentVolumeClaim:
            claimName: backend-redis-storage
        - configMap:
            items:
            - key: redis.conf
              path: redis.conf
            name: redis-config
          name: redis-config
    test: false
    triggers:
    - type: ConfigChange
    - imageChangeParams:
        automatic: true
        containerNames:
        - backend-redis
        from:
          kind: ImageStreamTag
          name: backend-redis:${AMP_RELEASE}
      type: ImageChange
  status:
    availableReplicas: 0
    latestVersion: 0
    observedGeneration: 0
    replicas: 0
    unavailableReplicas: 0
    updatedReplicas: 0
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: backend
      threescale_component_element: redis
    name: backend-redis
  spec:
    ports:
    - port: 6379
      protocol: TCP
      targetPort: 6379
    selector:
      deploymentConfig: backend-redis
  status:
    loadBalancer: {}
- apiVersion: v1
  data:
    redis.conf: |
      protected-mode no

      port 6379

      timeout 0
      tcp-keepalive 300

      daemonize no
      supervised no

      loglevel notice

      databases 16

      save 900 1
      save 300 10
      save 60 10000

      stop-writes-on-bgsave-error yes

      rdbcompression yes
      rdbchecksum yes

      dbfilename dump.rdb

      slave-serve-stale-data yes
      slave-read-only yes

      repl-diskless-sync no
      repl-disable-tcp-nodelay no

      appendonly yes
      appendfilename "appendonly.aof"
      appendfsync everysec
      no-appendfsync-on-rewrite no
      auto-aof-rewrite-percentage 100
      auto-aof-rewrite-min-size 64mb
      aof-load-truncated yes

      lua-time-limit 5000

      activerehashing no

      aof-rewrite-incremental-fsync yes
      dir /var/lib/redis/data
  kind: ConfigMap
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
      threescale_component_element: redis
    name: redis-config
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: backend
      threescale_component_element: redis
    name: backend-redis-storage
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
  status: {}
- apiVersion: image.openshift.io/v1
  kind: ImageStream
  metadata:
    annotations:
      openshift.io/display-name: Backend Redis
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: backend
    name: backend-redis
  spec:
    lookupPolicy:
      local: false
    tags:
    - annotations:
        openshift.io/display-name: Backend ${AMP_RELEASE} Redis
      from:
        kind: DockerImage
        name: ${REDIS_IMAGE}
      generation: null
      importPolicy:
        insecure: ${{IMAGESTREAM_TAG_IMPORT_INSECURE}}
      name: ${AMP_RELEASE}
      referencePolicy:
        type: ""
  status:
    dockerImageRepository: ""
- apiVersion: v1
  kind: Secret
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: backend
    name: backend-redis
  stringData:
    REDIS_QUEUES_SENTINEL_HOSTS: ""
    REDIS_QUEUES_SENTINEL_ROLE: ""
    REDIS_QUEUES_URL: redis://backend-redis:6379/1
    REDIS_STORAGE_SENTINEL_HOSTS: ""
    REDIS_STORAGE_SENTINEL_ROLE: ""
    REDIS_STORAGE_URL: redis://backend-redis:6379/0
  type: Opaque
- apiVersion: v1
  kind: Secret
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
    name: system-redis
  stringData:
    MESSAGE_BUS_NAMESPACE: ${SYSTEM_MESSAGE_BUS_REDIS_NAMESPACE}
    MESSAGE_BUS_SENTINEL_HOSTS: ""
    MESSAGE_BUS_SENTINEL_ROLE: ""
    MESSAGE_BUS_URL: ${SYSTEM_MESSAGE_BUS_REDIS_URL}
    NAMESPACE: ${SYSTEM_REDIS_NAMESPACE}
    SENTINEL_HOSTS: ""
    SENTINEL_ROLE: ""
    URL: ${SYSTEM_REDIS_URL}
  type: Opaque
- apiVersion: apps.openshift.io/v1
  kind: DeploymentConfig
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
      threescale_component_element: redis
    name: system-redis
  spec:
    replicas: 1
    selector:
      deploymentConfig: system-redis
    strategy:
      resources: {}
      type: Recreate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: ${APP_LABEL}
          deploymentConfig: system-redis
          threescale_component: system
          threescale_component_element: redis
      spec:
        containers:
        - args:
          - /etc/redis.d/redis.conf
          - --daemonize
          - "no"
          command:
          - /opt/rh/rh-redis5/root/usr/bin/redis-server
          image: system-redis:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            initialDelaySeconds: 10
            periodSeconds: 5
            tcpSocket:
              port: 6379
          name: system-redis
          readinessProbe:
            exec:
              command:
              - container-entrypoint
              - bash
              - -c
              - redis-cli set liveness-probe "`date`" | grep OK
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
          resources:
            limits:
              cpu: 500m
              memory: 32Gi
            requests:
              cpu: 150m
              memory: 256Mi
          terminationMessagePath: /dev/termination-log
          volumeMounts:
          - mountPath: /var/lib/redis/data
            name: system-redis-storage
          - mountPath: /etc/redis.d/
            name: redis-config
        volumes:
        - name: system-redis-storage
          persistentVolumeClaim:
            claimName: system-redis-storage
        - configMap:
            items:
            - key: redis.conf
              path: redis.conf
            name: redis-config
          name: redis-config
    test: false
    triggers:
    - type: ConfigChange
    - imageChangeParams:
        automatic: true
        containerNames:
        - system-redis
        from:
          kind: ImageStreamTag
          name: system-redis:${AMP_RELEASE}
      type: ImageChange
  status:
    availableReplicas: 0
    latestVersion: 0
    observedGeneration: 0
    replicas: 0
    unavailableReplicas: 0
    updatedReplicas: 0
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
      threescale_component_element: redis
    name: system-redis-storage
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  status: {}
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
      threescale_component_element: redis
    name: system-redis
  spec:
    ports:
    - name: redis
      port: 6379
      protocol: TCP
      targetPort: 6379
    selector:
      deploymentConfig: system-redis
  status:
    loadBalancer: {}
- apiVersion: image.openshift.io/v1
  kind: ImageStream
  metadata:
    annotations:
      openshift.io/display-name: System Redis
    creationTimestamp: null
    labels:
      app: ${APP_LABEL}
      threescale_component: system
    name: system-redis
  spec:
    lookupPolicy:
      local: false
    tags:
    - annotations:
        openshift.io/display-name: System ${AMP_RELEASE} Redis
      from:
        kind: DockerImage
        name: ${REDIS_IMAGE}
      generation: null
      importPolicy:
        insecure: ${{IMAGESTREAM_TAG_IMPORT_INSECURE}}
      name: ${AMP_RELEASE}
      referencePolicy:
        type: ""
  status:
    dockerImageRepository: ""
parameters:
- description: AMP release tag.
  name: AMP_RELEASE
  required: true
  value: "2.11"
- description: Used for object app labels
  name: APP_LABEL
  required: true
  value: 3scale-redis
- description: Set to true if the server may bypass certificate verification or connect directly over HTTP during image import.
  name: IMAGESTREAM_TAG_IMPORT_INSECURE
  required: true
  value: "false"
- description: Redis image to use
  name: REDIS_IMAGE
  required: true
  value: registry.redhat.io/rhscl/redis-5-rhel7:5
- description: Define the external system-redis to connect to
  name: SYSTEM_REDIS_URL
  required: true
  value: redis://system-redis:6379/1
- description: Define the external system-redis message bus to connect to. By default the same value as SYSTEM_REDIS_URL but with the logical database incremented by 1 and the result applied mod 16
  name: SYSTEM_MESSAGE_BUS_REDIS_URL
- description: Define the namespace to be used by System's Redis Database. The empty value means not namespaced
  name: SYSTEM_REDIS_NAMESPACE
- description: Define the namespace to be used by System's Message Bus Redis Database. The empty value means not namespaced
  name: SYSTEM_MESSAGE_BUS_REDIS_NAMESPACE