dzintars/project-quay-setup.md

## project-quay-setup.md

      
    Raw
  

              project-quay-setup.md
            
          
    [UPDATE] by Dzintars Klavins @ 2023-05-15
These instructions is no more quite accurate. Current setup is much more simpler. Probably I will update these instructions at some point.
[WIP] by Dzintars Klavins @ 2021-08-28
Project Quay Setup

These instructions is written for myself as an reminder guide.
It is higly possible that they are wrong, outdated, un-maintained.
I do not carry any responsibility about following these instructions. Use at your own risk!
If you have any suggestions, i will be happy to improve this document.
This setup utilizes podman play kube tool on Fedora 34 WorkStation.
Intention of this setup is to use it for Kubernetes (Minikube) and Bazel (build system) for temporary image storage.
The alternatives is to use Minikubes built-in simple registry or some of the public offerings.
I don't like public services as I like to save some money and tinker with the new stuff.
General TODO


 Make it rootless
 Handle volume persistance corectly
 Automate?

Prerequisites


Podman
Local storage (ideally would be neat to use Minio S3 from my oswee.ansible.minio role, but I left this for later)
Clean up the mounted volume directories if starting over. Don't forget to sudo ranger because files in those
directories belongs to sudo user.

Bootstrap

Firewall

Get the active firewall zone
sudo firewall-cmd --get-active-zones
See which zone is used for your network interface. Most likely it will be public zone.
Open the ports required for this setup
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --permanent --zone=public --add-port=9981/tcp
sudo firewall-cmd --reload
Add the image registry domain to the /etc/hosts file
echo '192.168.0.2 registry.domain.local' | sudo tee -a /etc/hosts
Or if you willing to add to the specific line then
sudo sed -i "2i192.168.0.2 registry.domain.local" /etc/hosts
where 2i prefix is the line nuber in which you want that record to appear.
Create the ~/quay/config-pod.yaml file with the content:
---
apiVersion: v1
kind: Pod
metadata:
  name: quay
  labels:
    app: quay
spec:
  restartPolicy: Always
  containers:
    - name: postgres
      image: docker.io/postgres:latest
      env:
        - name: POSTGRES_USER
          value: quayuser
        - name: POSTGRES_PASSWORD
          value: quaypass
        - name: POSTGRES_DB
          value: quay
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/postgresql/data:Z
          name: postgres-data

    - name: redis
      image: registry.redhat.io/rhel8/redis-5:1
      env:
        - name: REDIS_PASSWORD
          value: strongpassword
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/redis/data:Z
          name: redis-data

    - name: config
      image: quay.io/projectquay/quay:latest
      ports:
        - containerPort: 8080
          hostPort: 9981
          protocol: TCP
      args:
        - 'config'
        - 'secret'

  volumes:
    - name: postgres-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
        type: Directory
    - name: redis-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
        type: Directory
Make sure the volume directories exists
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
mkdir -p /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage
TODO: The volume location could be more conventional/generic
Run the sudo podaman play kube ~/quay/config-pod.yaml
Enable TRGM module
sudo podman exec -it quay-postgres /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm" | psql -d quay -U quayuser'
At this point you should have healthy Quay-Config, Redis and Postgresql containers running inside single quay pod.
Now it's time to generate configuration file for the Quay registry.
Access the configuration page at http://localhost:9981
Because I am using HAProxy with TLS termination (my proxy encrypts all traffic with walid TLS certificates) in this setup
i will skip any TLS part. Thou... in general it is bad idea to run unencripted traffic inside of the perimeter and probably
i will fix that some day soon.
For the Redis and Postgresql hosts use 127.0.0.1 IP address because that is how containers inside the Pod
can comunicate.
For "Database Server:" field shold be 127.0.0.1:5432.
You will instantly see the "Please specify a non-localhost hostname" warning. We will deal with that later.
For the "Redis Hostname:" field should be 127.0.0.1
Also add your username you will be using for Quay registry to the "Super Users:" field at the bottom of the page.
Now, the tricky part:
In order for get the "Validate Configuration Changes" button available, we need temporary to replace 127.0.0.1 IP address
for Redis and Postgresql with any valid domain, like example.com
Then we need to open Chrome or Firefox Developer Tools and to select the "Validate Configuration Changes" button.
In the source code we need to find the lines which looks like
<button class="btn btn-warning ng-scope ng-hide" ng-click="checkValidateAndSave()" ng-show="!configform.$valid">
  <i class="fa fa-lg fa-sort"></i>
  <!-- ngIf: configform.$error['required'].length -->
  <!-- ngIf: !configform.$error['required'].length --><span ng-if="!configform.$error['required'].length" class="ng-scope">
    Invalid configuration field
  </span><!-- end ngIf: !configform.$error['required'].length -->
</button>
and we need to change ng-show="!configform.$valid" to ng-show="configform.$valid"
This will prevent from hiding the "Validate Configuration Changes" button when we use 127.0.0.1 IP address as databases host addresses.
Once validation is successful you need to download the config tar.
Export environment variable
export QUAY=/home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data
Move the tar to the Quay config directory
sudo tar -xvf ~/Downloads/quay-config.tar.gz -C $QUAY/config
Remove the Quay Config container from the pod
sudo podman stop quay-config
This should stop and remove the config container form the pod but leave the pod networking intact.
So, now inject the real registry container in the same pod
  sudo podman run --pod quay -d \
  --name=registry \
  -v $QUAY/config:/quay-registry/conf/stack:Z \
  -v $QUAY/storage:/datastorage:Z \
  quay.io/projectquay/quay:latest
As you see, I do not expose any ports there, because those ports already was exposed by Config container.
We are lucky that Config and Registry containers uses 8080 port inside. :)
Check the logs
sudo podman logs quay-redis
If you see something like
1:M 29 Aug 2021 00:12:03.113 # Background saving error
1:M 29 Aug 2021 00:12:09.028 * 1 changes in 900 seconds. Saving...
1:M 29 Aug 2021 00:12:09.029 * Background saving started by pid 514
514:C 29 Aug 2021 00:12:09.029 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis/data) for saving: Permission denied
this means we have some problem with volume mount permissions.
Most likely we need to use podman unshare and sudo setfacl -m u:26:-wx /some/volume/dir to set the right permissions. (UNVALIDATED/TODO)
Why all this magic?

My main intention was to expose as less ports as possible and make the setup more "portable" if i could say so.
Managing single Pod is much more easier thand 3 or 4 separate conatiners. And I do really like play kube :)
Restore

To restore the existing setup we need another Pod manifest because we will use real Quay Registry image there.
All other paramaters we leave intact.
Create new file
touch ~/quay/registry-pod.yaml
with the content of
---
apiVersion: v1
kind: Pod
metadata:
  name: quay
  labels:
    app: quay
spec:
  restartPolicy: Always
  containers:
    - name: postgres
      image: docker.io/postgres:latest
      env:
        - name: POSTGRES_USER
          value: quayuser
        - name: POSTGRES_PASSWORD
          value: quaypass
        - name: POSTGRES_DB
          value: quay
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/postgresql/data:Z
          name: postgres-data

    - name: redis
      image: registry.redhat.io/rhel8/redis-5:1
      env:
        - name: REDIS_PASSWORD
          value: strongpassword
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /var/lib/redis/data:Z
          name: redis-data

    - name: registry
      image: quay.io/projectquay/quay:latest
      ports:
        - containerPort: 8080
          hostPort: 9981
          protocol: TCP
      securityContext:
        allowPrivilegeEscalation: false
        # privileged: true
        readOnlyRootFilesystem: false
      volumeMounts:
        - mountPath: /quay-registry/conf/stack:Z
          name: quay-config
        - mountPath: /datastorage:Z
          name: quay-storage

  volumes:
    - name: postgres-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/postgres/data
        type: Directory
    - name: redis-data
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/redis/data
        type: Directory
    - name: quay-config
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/config
        type: Directory
    - name: quay-storage
      hostPath:
        path: /home/dzintars/containers/github.com/dzintars/quay/volumes/quay/data/storage
        type: Directory
Now, because we used volume mounts, all our data was persisted in those directories and new Pod will use the same configs
and data.
In theory... I haven't checked it yet. Writing this while i am doing all this. :)
References


Floating Kwaaaay with Podman and systemd 2019-03-08
Deploy Project Quay for proof-of-concept (non-production) purposes