Skip to content

Instantly share code, notes, and snippets.

@dmccuk
Last active July 28, 2023 01:53
Show Gist options
  • Save dmccuk/098fccc488c8a5aeaa5f859855660018 to your computer and use it in GitHub Desktop.
Save dmccuk/098fccc488c8a5aeaa5f859855660018 to your computer and use it in GitHub Desktop.

This is a walk through of this github repo: https://github.com/kurokobo/awx-on-k3s

Subscribe To Me On YouTube: https://bit.ly/lon_sub

Thanks to @kurokobo for making these steps. Check his github on the link above. If this video helped you get the AWX-operator working, please give the video a like and subscribe to my channel. I appreciate it!

Links: Rancher: https://rancher.com/docs/k3s/latest/en/ YouTube Video: https://youtu.be/xVOnBDvpbK0

Here I add the missing sections so less expereinces people can achieve installing and setting up AWX on K3s with HTTPS.

My setup:

  • Instance Type: t3a.xlarge
  • Centos 8 - AMI: ami-0e1b0ca392c9f2523
  • Region: eu-west-2
  • My user has root privilege via sudo
  • For some reason, the CentOS iage started with no space in /
  • Now login and grow XFS for / on your server
sudo yum -y install cloud-utils-growpart gdisk
lsblk
sudo growpart /dev/nvme0n1 2
sudo xfs_growfs /
exit #(back to the ec2-user)

KUBECTL install

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl kubectl version --client

INSTALL K3s

sudo systemctl disable firewalld --now curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 k3s kubectl cluster-info k3s kubectl config view

Install awx-operator

k3s kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.13.0/deploy/awx-operator.yaml git clone https://github.com/kurokobo/awx-on-k3s.git

cd awx-on-k3s/ AWX_HOST="ec2-18-130-175-220.eu-west-2.compute.amazonaws.com" openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./base/tls.crt -keyout ./base/tls.key -subj "/CN=${AWX_HOST}/O=${AWX_HOST}" -addext "subjectAltName = DNS:${AWX_HOST}" vi base/awx.yaml

  admin_user: admin
  admin_password_secret: awx-admin-password

  ingress_type: ingress
  ingress_tls_secret: awx-secret-tls
  hostname: ec2-3-8-114-43.eu-west-2.compute.amazonaws.com  <--- your name

modify password if required: vi base/kustomization.yaml


  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=awx-postgres
      - port=5432
      - database=awx
      - username=awx
      - password=Ansible123!  <--- HERE
      - type=managed

  - name: awx-admin-password
    type: Opaque
    literals:
      - password=Ansible123!  <--- AND HERE

sudo mkdir -p /data/postgres sudo mkdir -p /data/projects sudo chown 1000:0 /data/projects k3s kubectl apply -k base

$ k3s kubectl apply -k base
namespace/awx created
secret/awx-admin-password created
secret/awx-postgres-configuration created
secret/awx-secret-tls created
persistentvolume/awx-postgres-volume created
persistentvolume/awx-projects-volume created
persistentvolumeclaim/awx-projects-claim created
awx.awx.ansible.com/awx created

k3s kubectl logs -f deployment/awx-operator (process takes maybe 5-7 minutes)

  • Depending on your setup, you may need to open the FW for https/443.

Once it's complete, you should see this:

--------------------------- Ansible Task Status Event StdOut  -----------------

PLAY RECAP *********************************************************************
localhost                  : ok=54   changed=0    unreachable=0    failed=0    skipped=37   rescued=0    ignored=0

Check everything is up.

k3s kubectl -n awx get awx,all,ingress,secrets

NAME                      AGE
awx.awx.ansible.com/awx   13m

NAME                      READY   STATUS    RESTARTS   AGE
pod/awx-postgres-0        1/1     Running   0          12m
pod/awx-59ff55b5b-shhb5   4/4     Running   0          12m

NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None          <none>        5432/TCP   12m
service/awx-service    ClusterIP   10.43.83.81   <none>        80/TCP     12m

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx   1/1     1            1           12m

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-59ff55b5b   1         1         1       12m

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     12m

NAME                                    CLASS    HOSTS                                            ADDRESS        PORTS     AGE
ingress.networking.k8s.io/awx-ingress   <none>   ec2-3-8-114-43.eu-west-2.compute.amazonaws.com   172.31.31.20   80, 443   12m

NAME                                TYPE                                  DATA   AGE
secret/awx-admin-password           Opaque                                1      13m
secret/default-token-vkgx4          kubernetes.io/service-account-token   3      13m
secret/awx-postgres-configuration   Opaque                                6      13m
secret/awx-secret-tls               kubernetes.io/tls                     2      13m
secret/awx-app-credentials          Opaque                                3      12m
secret/awx-token-h55dk              kubernetes.io/service-account-token   3      12m
secret/awx-secret-key               Opaque                                1      12m
secret/awx-broadcast-websocket      Opaque                                1      12m

Now login to AWX

https://ec2-18-130-175-220.eu-west-2.compute.amazonaws.com

Default user: admin password: Ansible123!

Enjoy!

@nasirnj
Copy link

nasirnj commented Apr 21, 2022

Dear All - I have followed exact steps, but it seems like its a never ending install, and im getting "404 page not found" can't. Please guide when possible. Thank you.

[root@awx ~]# k3s kubectl -n awx get awx,all,ingress,secrets
NAME AGE
awx.awx.ansible.com/awx 38m

NAME READY STATUS RESTARTS AGE
pod/awx-postgres-0 1/1 Running 0 37m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-postgres ClusterIP None 5432/TCP 37m
service/awx-service ClusterIP 10.43.112.100 80/TCP 37m

NAME READY AGE
statefulset.apps/awx-postgres 1/1 37m

NAME TYPE DATA AGE
secret/awx-admin-password Opaque 1 38m
secret/default-token-dtqpk kubernetes.io/service-account-token 3 38m
secret/awx-postgres-configuration Opaque 6 38m
secret/awx-secret-tls kubernetes.io/tls 2 38m
secret/awx-secret-key Opaque 1 38m
secret/awx-broadcast-websocket Opaque 1 38m
secret/awx-app-credentials Opaque 3 37m

[root@awx awx-on-k3s]# cat base/
awx.yaml kustomization.yaml namespace.yaml pvc.yaml pv.yaml tls.crt tls.key
[root@awx awx-on-k3s]# cat base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: awx

generatorOptions:
disableNameSuffixHash: true

secretGenerator:

  • name: awx-secret-tls
    type: kubernetes.io/tls
    files:

    • tls.crt
    • tls.key
  • name: awx-postgres-configuration
    type: Opaque
    literals:

    • host=awx-postgres
    • port=5432
    • database=awx
    • username=awx
    • password=redhat
    • type=managed
  • name: awx-admin-password
    type: Opaque
    literals:

    • password=redhat

resources:

  • namespace.yaml
  • pv.yaml
  • pvc.yaml
  • awx.yaml
    [root@awx awx-on-k3s]# cat base/awx.yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:

These parameters are designed for use with:

- AWX Operator: 0.20.0

https://github.com/ansible/awx-operator/blob/0.20.0/README.md

- AWX: 20.1.0

https://github.com/ansible/awx/blob/20.1.0/INSTALL.md

admin_user: admin
admin_password_secret: awx-admin-password

ingress_type: ingress
ingress_tls_secret: awx-secret-tls
hostname: 10.0.0.218

postgres_configuration_secret: awx-postgres-configuration

postgres_storage_class: awx-postgres-volume
postgres_storage_requirements:
requests:
storage: 8Gi

projects_persistence: true
projects_existing_claim: awx-projects-claim

web_resource_requirements: {}
task_resource_requirements: {}
ee_resource_requirements: {}

@TheBigBear
Copy link

TheBigBear commented Jun 24, 2022

I followed everything in your guide exactly other than having to expand the volume I didn't need to do that.

CentOS 8 Stream Running 8GB Ram, 2 Core CPU - from a Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz

my awx.yaml file changes were just:

  admin_user: falconinet
  admin_password_secret: somepassword

  ingress_type: ingress
  ingress_tls_secret: somepassword
  hostname: ansible.mydomain.net

For any others stumbling over this. The admin, ingress and Postgres passwords or secrets should only be changed in the 'base/kustomize.yaml' file as shown. the entries in the 'base/awx.yaml' they are placeholders that ere being referred to and used in other places .. the only thing to be changed in the 'base/awx.yaml' file is the hostname as shown in the gist.

@TheBigBear
Copy link

TheBigBear commented Jun 26, 2022

Any idea what might be wrong? I followed the article on my almalinux 8 ( centos 8 compatible ) and it ends up with:

PLAY RECAP *********************************************************************
localhost                  : ok=66   changed=2    unreachable=0    failed=0    skipped=43   rescued=0    ignored=0

Which looks correct to me?
But if I try to connect to my k3s node with https://awx.example.com I get

Bad gateway

in my browser.

Here is my ingress info:

k3s kubectl describe ingress awx-ingress -n awx
Name:             awx-ingress
Labels:           app.kubernetes.io/component=awx
                  app.kubernetes.io/managed-by=awx-operator
                  app.kubernetes.io/name=awx
                  app.kubernetes.io/operator-version=0.23.0
                  app.kubernetes.io/part-of=awx
Namespace:        awx
Address:          10.10.97.50
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  awx-secret-tls terminates awx.example.com
Rules:
  Host            Path  Backends
  ----            ----  --------
  awx.ict.om.org
                  /   awx-service:80 (10.42.0.12:8052)
Annotations:      <none>
Events:           <none>

Where 10.10.97.50 is the correct physical IP of my k3s node.
How do I debug this any further?
A google article suggested the firewalld had to be off on my centos 8 ( almalinux 8 ) but that made no difference even after a reboot.

@TheBigBear
Copy link

Found some more info, in case it helps?

kubectl -n kube-system describe services/traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-10.19.300
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.216.35
IPs:                      10.43.216.35
LoadBalancer Ingress:     10.10.97.50
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  31724/TCP
Endpoints:                10.42.0.8:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  32558/TCP
Endpoints:                10.42.0.8:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason            Age                From           Message
  ----    ------            ----               ----           -------
  Normal  UpdatedIngressIP  34m                svccontroller  LoadBalancer Ingress IP addresses updated: 10.10.97.50
  Normal  AppliedDaemonSet  34m (x6 over 34m)  svccontroller  Applied LoadBalancer DaemonSet kube-system/svclb-traefik

@TheBigBear
Copy link

I have since also switched on hostNetwork to true and added the three extra lines for extra_settings mentioned in the @kurokobo's k3s awx troubleshooting page, but I still get a Bad Gateway message.
How can I best debug this any further?

@rknupfer
Copy link

I keep running into the following when running the "k3s kubectl apply -k base":

$ k3s kubectl apply -k base
persistentvolume/awx-postgres-13-volume unchanged
persistentvolume/awx-projects-volume unchanged
error validating "base": error validating data: ValidationError(AWX.spec): unknown field "postgres_init_container_resource_requirements" in com.ansible.awx.v1beta1.AWX.spec; if you choose to ignore these errors, turn validation off with --validate=false
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found

@triff77
Copy link

triff77 commented Sep 28, 2022

to save anyone pain, using the korokobo doc worked for me on rocky8 https://github.com/kurokobo/awx-on-k3s , but these instructions failed with the following, had to rebuild and start clean:
k3s kubectl apply -k base
persistentvolume/awx-postgres-13-volume unchanged
persistentvolume/awx-projects-volume unchanged
error validating "base": error validating data: ValidationError(AWX.spec): unknown field "postgres_init_container_resource_requirements" in com.ansible.awx.v1beta1.AWX.spec; if you choose to ignore these errors, turn validation off with --validate=false
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found
Error from server (NotFound): error when creating "base": namespaces "awx" not found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment