Skip to content

Instantly share code, notes, and snippets.

@dmccuk
Last active July 28, 2023 01:53
Show Gist options
  • Star 13 You must be signed in to star a gist
  • Fork 11 You must be signed in to fork a gist
  • Save dmccuk/098fccc488c8a5aeaa5f859855660018 to your computer and use it in GitHub Desktop.
Save dmccuk/098fccc488c8a5aeaa5f859855660018 to your computer and use it in GitHub Desktop.

This is a walk through of this github repo: https://github.com/kurokobo/awx-on-k3s

Subscribe To Me On YouTube: https://bit.ly/lon_sub

Thanks to @kurokobo for making these steps. Check his github on the link above. If this video helped you get the AWX-operator working, please give the video a like and subscribe to my channel. I appreciate it!

Links: Rancher: https://rancher.com/docs/k3s/latest/en/ YouTube Video: https://youtu.be/xVOnBDvpbK0

Here I add the missing sections so less expereinces people can achieve installing and setting up AWX on K3s with HTTPS.

My setup:

  • Instance Type: t3a.xlarge
  • Centos 8 - AMI: ami-0e1b0ca392c9f2523
  • Region: eu-west-2
  • My user has root privilege via sudo
  • For some reason, the CentOS iage started with no space in /
  • Now login and grow XFS for / on your server
sudo yum -y install cloud-utils-growpart gdisk
lsblk
sudo growpart /dev/nvme0n1 2
sudo xfs_growfs /
exit #(back to the ec2-user)

KUBECTL install

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl kubectl version --client

INSTALL K3s

sudo systemctl disable firewalld --now curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 k3s kubectl cluster-info k3s kubectl config view

Install awx-operator

k3s kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.13.0/deploy/awx-operator.yaml git clone https://github.com/kurokobo/awx-on-k3s.git

cd awx-on-k3s/ AWX_HOST="ec2-18-130-175-220.eu-west-2.compute.amazonaws.com" openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./base/tls.crt -keyout ./base/tls.key -subj "/CN=${AWX_HOST}/O=${AWX_HOST}" -addext "subjectAltName = DNS:${AWX_HOST}" vi base/awx.yaml

  admin_user: admin
  admin_password_secret: awx-admin-password

  ingress_type: ingress
  ingress_tls_secret: awx-secret-tls
  hostname: ec2-3-8-114-43.eu-west-2.compute.amazonaws.com  <--- your name

modify password if required: vi base/kustomization.yaml


  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=awx-postgres
      - port=5432
      - database=awx
      - username=awx
      - password=Ansible123!  <--- HERE
      - type=managed

  - name: awx-admin-password
    type: Opaque
    literals:
      - password=Ansible123!  <--- AND HERE

sudo mkdir -p /data/postgres sudo mkdir -p /data/projects sudo chown 1000:0 /data/projects k3s kubectl apply -k base

$ k3s kubectl apply -k base
namespace/awx created
secret/awx-admin-password created
secret/awx-postgres-configuration created
secret/awx-secret-tls created
persistentvolume/awx-postgres-volume created
persistentvolume/awx-projects-volume created
persistentvolumeclaim/awx-projects-claim created
awx.awx.ansible.com/awx created

k3s kubectl logs -f deployment/awx-operator (process takes maybe 5-7 minutes)

  • Depending on your setup, you may need to open the FW for https/443.

Once it's complete, you should see this:

--------------------------- Ansible Task Status Event StdOut  -----------------

PLAY RECAP *********************************************************************
localhost                  : ok=54   changed=0    unreachable=0    failed=0    skipped=37   rescued=0    ignored=0

Check everything is up.

k3s kubectl -n awx get awx,all,ingress,secrets

NAME                      AGE
awx.awx.ansible.com/awx   13m

NAME                      READY   STATUS    RESTARTS   AGE
pod/awx-postgres-0        1/1     Running   0          12m
pod/awx-59ff55b5b-shhb5   4/4     Running   0          12m

NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None          <none>        5432/TCP   12m
service/awx-service    ClusterIP   10.43.83.81   <none>        80/TCP     12m

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx   1/1     1            1           12m

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-59ff55b5b   1         1         1       12m

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     12m

NAME                                    CLASS    HOSTS                                            ADDRESS        PORTS     AGE
ingress.networking.k8s.io/awx-ingress   <none>   ec2-3-8-114-43.eu-west-2.compute.amazonaws.com   172.31.31.20   80, 443   12m

NAME                                TYPE                                  DATA   AGE
secret/awx-admin-password           Opaque                                1      13m
secret/default-token-vkgx4          kubernetes.io/service-account-token   3      13m
secret/awx-postgres-configuration   Opaque                                6      13m
secret/awx-secret-tls               kubernetes.io/tls                     2      13m
secret/awx-app-credentials          Opaque                                3      12m
secret/awx-token-h55dk              kubernetes.io/service-account-token   3      12m
secret/awx-secret-key               Opaque                                1      12m
secret/awx-broadcast-websocket      Opaque                                1      12m

Now login to AWX

https://ec2-18-130-175-220.eu-west-2.compute.amazonaws.com

Default user: admin password: Ansible123!

Enjoy!

@marcpope
Copy link

I am getting 1 failed event.

localhost                  : ok=32   changed=0    unreachable=0    failed=1    skipped=25   rescued=0    ignored=0   

-------------------------------------------------------------------------------
{"level":"error","ts":1634307528.2201762,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"awx-controller","request":"awx/awx","error":"event runner on failed","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

The deployment.apps/awx is not ready:

[root@ansible awx-on-k3s]# k3s kubectl -n awx get awx,all,ingress,secrets
NAME                      AGE
awx.awx.ansible.com/awx   48m

NAME                      READY   STATUS    RESTARTS   AGE
pod/awx-59ff55b5b-m2r8t   0/4     Pending   0          47m
pod/awx-postgres-0        1/1     Running   0          47m

NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None           <none>        5432/TCP   47m
service/awx-service    ClusterIP   10.43.27.165   <none>        80/TCP     47m

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx   0/1     1            0           47m

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-59ff55b5b   1         1         0       47m

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     47m

NAME                                    CLASS    HOSTS                        ADDRESS          PORTS     AGE
ingress.networking.k8s.io/awx-ingress   <none>   ansible.mydomain.net   XXX.XXX.158.21   80, 443   47m

NAME                                TYPE                                  DATA   AGE
secret/default-token-8vvnc          kubernetes.io/service-account-token   3      48m
secret/awx-admin-password           Opaque                                1      48m
secret/awx-postgres-configuration   Opaque                                6      48m
secret/awx-secret-tls               kubernetes.io/tls                     2      48m
secret/awx-secret-key               Opaque                                1      48m
secret/awx-broadcast-websocket      Opaque                                1      48m
secret/awx-app-credentials          Opaque                                3      47m
secret/awx-token-n5cn2              kubernetes.io/service-account-token   3      47m

@dmccuk
Copy link
Author

dmccuk commented Oct 15, 2021

Hi Marc, It looks to be the controller that's erroring. Did you make any other updates to the awx.yaml file? What cpu and memory do you have available in the VM your using?

@marcpope
Copy link

marcpope commented Oct 15, 2021

I followed everything in your guide exactly other than having to expand the volume I didn't need to do that.

CentOS 8 Stream
Running 8GB Ram, 2 Core CPU - from a Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz

my awx.yaml file changes were just:

  admin_user: falconinet
  admin_password_secret: somepassword

  ingress_type: ingress
  ingress_tls_secret: somepassword
  hostname: ansible.mydomain.net

@marcpope
Copy link

So I think the problem was I had 2 Core 1 CPU instead of 2 CPU, 1 Core Each. It needs 2 CPU Sockets. Also, if you follow https://github.com/kurokobo/awx-on-k3s, it is updated to install .14 instead of .13

@baclaeys
Copy link

I was trying to build this on Rocky Linux 8.5 and I came across the following error when I tried running the following command.

[root@awx ~]# curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644
[INFO] Finding release for channel stable
[INFO] Using v1.21.5+k3s2 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s2/sha256sum-amd64.txt
[INFO] Skipping binary downloaded, installed k3s matches hash
Rancher K3s Common (stable) 6.6 kB/s | 2.9 kB 00:00
Error:
Problem: package k3s-selinux-0.4-1.el8.noarch requires container-selinux < 2:2.164.2, but none of the providers can be installed

  • cannot install the best candidate for the job
  • package container-selinux-2:2.124.0-1.gitf958d0c.module+el8.5.0+681+c9a1951f.noarch is filtered out by modular filtering
  • package container-selinux-2:2.130.0-1.module+el8.5.0+708+6758137d.noarch is filtered out by modular filtering
    (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Adding --skip-broken and/or --no best did not change the result. To get around this used https://access.redhat.com/solutions/5588101 for reference.

The commands I ran were as follows:
yum module disable container-tools
yum module enable container-tools:2.0

After that the command ran successfully.

@dmccuk
Copy link
Author

dmccuk commented Nov 16, 2021

Thanks for that @baclaeys. I'll add a section on ISSUES in the main gist in case others people have a similar issue.

@Pzharyuk
Copy link

Any idea why I get this permissions error?

When I view the logs via "kubectl --namespace='awx' logs awx-posgres-0" I get this error "mkdir: cannot create directory '/var/lib/posgressql/data': Permission denied"

Shouldn't it be using "/data/postgres"?

@dmccuk
Copy link
Author

dmccuk commented Jan 1, 2022

Hi @Pzharyuk

I don't have a K3s cluster running atm for me to check. I created this based on the kurokobo github page so it might be worth checking their issues page on the github repo. It's possible others have had this issue and there is a fix?

Sorry I can't be more helpful right now.

@sbilque
Copy link

sbilque commented Jan 3, 2022

Hi @Pzharyuk
I've resolved this issue with the method described in CrashLoopBackOff on awx-postgres-0 pod :

# Replace `/data/postgres` with your directory path.
POSTGRES_PATH=/data/postgres
sudo mkdir -p $POSTGRES_PATH/data
sudo chmod 755 $POSTGRES_PATH $POSTGRES_PATH/data
sudo chown 0:0 $POSTGRES_PATH $POSTGRES_PATH/data

@Pzharyuk
Copy link

Pzharyuk commented Jan 4, 2022

Thank you @sbilque I will give this a try!

@bgaber
Copy link

bgaber commented Apr 21, 2022

I am attempting to install on Ubuntu 18.04 and Ubuntu 20.04. On both the k3s kubectl apply -k base command does not successfully complete and produces error:
{"level":"error","ts":1650555844.9798474,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"awx-controller","request":"awx/awx","error":"event runner on failed","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

@nasirnj
Copy link

nasirnj commented Apr 21, 2022

Dear All - I have followed exact steps, but it seems like its a never ending install, and im getting "404 page not found" can't. Please guide when possible. Thank you.

[root@awx ~]# k3s kubectl -n awx get awx,all,ingress,secrets
NAME AGE
awx.awx.ansible.com/awx 38m

NAME READY STATUS RESTARTS AGE
pod/awx-postgres-0 1/1 Running 0 37m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-postgres ClusterIP None 5432/TCP 37m
service/awx-service ClusterIP 10.43.112.100 80/TCP 37m

NAME READY AGE
statefulset.apps/awx-postgres 1/1 37m

NAME TYPE DATA AGE
secret/awx-admin-password Opaque 1 38m
secret/default-token-dtqpk kubernetes.io/service-account-token 3 38m
secret/awx-postgres-configuration Opaque 6 38m
secret/awx-secret-tls kubernetes.io/tls 2 38m
secret/awx-secret-key Opaque 1 38m
secret/awx-broadcast-websocket Opaque 1 38m
secret/awx-app-credentials Opaque 3 37m

[root@awx awx-on-k3s]# cat base/
awx.yaml kustomization.yaml namespace.yaml pvc.yaml pv.yaml tls.crt tls.key
[root@awx awx-on-k3s]# cat base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: awx

generatorOptions:
disableNameSuffixHash: true

secretGenerator:

  • name: awx-secret-tls
    type: kubernetes.io/tls
    files:

    • tls.crt
    • tls.key
  • name: awx-postgres-configuration
    type: Opaque
    literals:

    • host=awx-postgres
    • port=5432
    • database=awx
    • username=awx
    • password=redhat
    • type=managed
  • name: awx-admin-password
    type: Opaque
    literals:

    • password=redhat

resources:

  • namespace.yaml
  • pv.yaml
  • pvc.yaml
  • awx.yaml
    [root@awx awx-on-k3s]# cat base/awx.yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:

These parameters are designed for use with:

- AWX Operator: 0.20.0

https://github.com/ansible/awx-operator/blob/0.20.0/README.md

- AWX: 20.1.0

https://github.com/ansible/awx/blob/20.1.0/INSTALL.md

admin_user: admin
admin_password_secret: awx-admin-password

ingress_type: ingress
ingress_tls_secret: awx-secret-tls
hostname: 10.0.0.218

postgres_configuration_secret: awx-postgres-configuration

postgres_storage_class: awx-postgres-volume
postgres_storage_requirements:
requests:
storage: 8Gi

projects_persistence: true
projects_existing_claim: awx-projects-claim

web_resource_requirements: {}
task_resource_requirements: {}
ee_resource_requirements: {}

@TheBigBear
Copy link

TheBigBear commented Jun 24, 2022

I followed everything in your guide exactly other than having to expand the volume I didn't need to do that.

CentOS 8 Stream Running 8GB Ram, 2 Core CPU - from a Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz

my awx.yaml file changes were just:

  admin_user: falconinet
  admin_password_secret: somepassword

  ingress_type: ingress
  ingress_tls_secret: somepassword
  hostname: ansible.mydomain.net

For any others stumbling over this. The admin, ingress and Postgres passwords or secrets should only be changed in the 'base/kustomize.yaml' file as shown. the entries in the 'base/awx.yaml' they are placeholders that ere being referred to and used in other places .. the only thing to be changed in the 'base/awx.yaml' file is the hostname as shown in the gist.

@TheBigBear
Copy link

TheBigBear commented Jun 26, 2022

Any idea what might be wrong? I followed the article on my almalinux 8 ( centos 8 compatible ) and it ends up with:

PLAY RECAP *********************************************************************
localhost                  : ok=66   changed=2    unreachable=0    failed=0    skipped=43   rescued=0    ignored=0

Which looks correct to me?
But if I try to connect to my k3s node with https://awx.example.com I get

Bad gateway

in my browser.

Here is my ingress info:

k3s kubectl describe ingress awx-ingress -n awx
Name:             awx-ingress
Labels:           app.kubernetes.io/component=awx
                  app.kubernetes.io/managed-by=awx-operator
                  app.kubernetes.io/name=awx
                  app.kubernetes.io/operator-version=0.23.0
                  app.kubernetes.io/part-of=awx
Namespace:        awx
Address:          10.10.97.50
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  awx-secret-tls terminates awx.example.com
Rules:
  Host            Path  Backends
  ----            ----  --------
  awx.ict.om.org
                  /   awx-service:80 (10.42.0.12:8052)
Annotations:      <none>
Events:           <none>

Where 10.10.97.50 is the correct physical IP of my k3s node.
How do I debug this any further?
A google article suggested the firewalld had to be off on my centos 8 ( almalinux 8 ) but that made no difference even after a reboot.

@TheBigBear
Copy link

Found some more info, in case it helps?

kubectl -n kube-system describe services/traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-10.19.300
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.216.35
IPs:                      10.43.216.35
LoadBalancer Ingress:     10.10.97.50
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  31724/TCP
Endpoints:                10.42.0.8:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  32558/TCP
Endpoints:                10.42.0.8:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason            Age                From           Message
  ----    ------            ----               ----           -------
  Normal  UpdatedIngressIP  34m                svccontroller  LoadBalancer Ingress IP addresses updated: 10.10.97.50
  Normal  AppliedDaemonSet  34m (x6 over 34m)  svccontroller  Applied LoadBalancer DaemonSet kube-system/svclb-traefik

@TheBigBear
Copy link

I have since also switched on hostNetwork to true and added the three extra lines for extra_settings mentioned in the @kurokobo's k3s awx troubleshooting page, but I still get a Bad Gateway message.
How can I best debug this any further?

@rknupfer
Copy link

I keep running into the following when running the "k3s kubectl apply -k base":

$ k3s kubectl apply -k base
persistentvolume/awx-postgres-13-volume unchanged
persistentvolume/awx-projects-volume unchanged
error validating "base": error validating data: ValidationError(AWX.spec): unknown field "postgres_init_container_resource_requirements" in com.ansible.awx.v1beta1.AWX.spec; if you choose to ignore these errors, turn validation off with --validate=false
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found

@triff77
Copy link

triff77 commented Sep 28, 2022

to save anyone pain, using the korokobo doc worked for me on rocky8 https://github.com/kurokobo/awx-on-k3s , but these instructions failed with the following, had to rebuild and start clean:
k3s kubectl apply -k base
persistentvolume/awx-postgres-13-volume unchanged
persistentvolume/awx-projects-volume unchanged
error validating "base": error validating data: ValidationError(AWX.spec): unknown field "postgres_init_container_resource_requirements" in com.ansible.awx.v1beta1.AWX.spec; if you choose to ignore these errors, turn validation off with --validate=false
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment