Skip to content

Instantly share code, notes, and snippets.

@dmccuk
Last active October 30, 2022 10:59
Show Gist options
  • Save dmccuk/93db22e9b30d1963b8fca0de96fc82f0 to your computer and use it in GitHub Desktop.
Save dmccuk/93db22e9b30d1963b8fca0de96fc82f0 to your computer and use it in GitHub Desktop.

Install AWX-operator on Ubuntu 20 using Minikube

I’ve had a few requests for this demo as It seems like a lot of people are having issues setting up the new version of AWX. Before v18, (v17) you could simply install AWX on a server with enough resources. Now, the preferred way to install AWX is via the AWX-operator. So you need a Kubernetes or OpenShift cluster. For this demo, I’m using Minikube.

Subscribe To Me On YouTube: https://bit.ly/lon_sub

(if you get any errors, check the bottom of this page for the fix)

I’m using minikube because it’s a single node cluster and it keeps the price down. I did try this on a t2.medium, but there just weren’t enough resources to get it working.

This demo will cover the following:

  • Install and setup kubectl and docker.
  • Installing and setting up Minikube in EC2 [t3a.xlarge – 4 x cpu / 16gb memory]
    • ami-0194c3e07668a7e36
    • You might be able to get away with less resources)
  • Setup the AWX-operator (latest version – 0.10.0) and deploy.
  • Get the AWX admin password.
  • Forward the service port so we can access it from the internet and use minikube tunnel.
  • Login to AWX and take a quick tour.
  • All commands will be added into the description.

Make sure you have enough FREE Disk space. 10GB+ please

Links:

https://github.com/ansible/awx-operator https://github.com/ansible/awx-operator/tags

If you like the demo, hit subscribe for more videos like this and give it a like.

Commands:

setup the repo's:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version --client
sudo apt-get update -y &&  sudo apt-get install -y docker.io

Install minikube:

There is an issue with the latest version (1.22) Please install 1.21 for now.

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/```

Use this one below for the time being:

curl -Lo minikube https://github.com/kubernetes/minikube/releases/download/v1.21.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
sudo usermod -aG docker $USER
LOGOUT & BACK IN AGAIN
groups $USER

minikube start

minikube start --addons=ingress --cpus=2 --install-addons=true --kubernetes-version=stable --memory=6g

Check its working:

kubectl get nodes
kubectl get pods
kubectl get pods -A

Install the AWX Operator:

kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.10.0/deploy/awx-operator.yaml
kubectl get pods

Create the deployment file:

vi awx-demo.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  ingress_type: none
  hostname: awx-demo.example.com

Run the deployment:

kubectl apply -f awx-demo.yml
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"

WAIT A FEW MINS...

Get the Admin user password:

kubectl get secrets
kubectl get secret awx-demo-admin-password -o jsonpath="{.data.password}" | base64 --decode

Expose the deployment:

kubectl expose deployment awx-demo --type=LoadBalancer --port=8080

Minikube tunnel

On a new session, start the minikube tunnel:

minikube tunnel

Enable AWX to be access via the Internet:

kubectl port-forward svc/awx-demo-service --address 0.0.0.0 30886:80

Now visit https://your_ip:high_port

You may need to update your FW rules to be able to connect to the AWX login screen


--- ISSUES SECTION ---

  1. Starting minikube tunnel - Exiting due to GUEST_STATUS: state: unknown state "minikube": docker container inspect minikube --format=: exit status 1 ANS: sudo chmod 666 /var/run/docker.sock ; sudo usermod -aG docker ${USER}

  2. If you see the message: ImagePullBackOff or ErrImagePull when you run kubectl get pods, run the following command to see what the issue is:

kubectl describe pods <my-pod> # Output from "kubectl get pods" command for the pod with the issue.

I've seen a few issues where the server runs out of space so it should be easy to fix.

  1. If you have PENDING resources, try running a describe of the resource. If you see this at the bottom you are lacking in resources: ( kubectl describe pods <my-pod> # Output from kubectl get pods )
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x19 over 23m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

This link has some more information: https://containersolutions.github.io/runbooks/posts/kubernetes/0-nodes-available-insufficient/

@dreaddesign
Copy link

this helped me out greatly so thank you so much. I was wondering if there was any chance you'd want to help extend this for using an external postgres DB as well as configuring it on a proper subdomain?

ive done it successfully in a docker container with AWX v14 but the switch to kubernetes really throwing me off.

@nicolaibaralmueller
Copy link

nicolaibaralmueller commented Aug 26, 2021

Hi

Followed your guide on a fresh Ubuntu installation and got the below errors. Access obviously doesn't work from outside.

Also: How do I enable https/443 access with a certificate so awx is accessible from https://awx.mydomain.local and not awx.mydomain.local:30884?

kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
awx            LoadBalancer   10.107.197.210   <pending>     8080:30384/TCP   14m
awx-postgres   ClusterIP      None             <none>        5432/TCP         23m
awx-service    NodePort       10.98.175.86     <none>        80:32671/TCP     22m
minikube tunnel
Status:	
	machine: minikube
	pid: 17293
	route: 10.96.0.0/12 -> 192.168.49.2
	minikube: Running
	services: [awx]
    errors: 
		minikube: no errors
		router: no errors
		loadbalancer emulator: no errors

Had to correct port 30886 to 30884. When accessing the host remotely only the awx logo was briefly shown and then below error.

kubectl port-forward svc/awx-service --address 0.0.0.0 30884:80
Forwarding from 0.0.0.0:30884 -> 8052
Handling connection for 30884

ERR_SSL_PROTOCOL_ERROR

Thanks

@dmccuk
Copy link
Author

dmccuk commented Aug 26, 2021

Hi

Followed your guide on a fresh Ubuntu installation and got the below errors. Access obviously doesn't work from outside.

Also: How do I enable https/443 access with a certificate?

minikube tunnel
Status:	
	machine: minikube
	pid: 17293
	route: 10.96.0.0/12 -> 192.168.49.2
	minikube: Running
	services: [awx]
    errors: 
		minikube: no errors
		router: no errors
		loadbalancer emulator: no errors
kubectl port-forward svc/awx-service --address 0.0.0.0 30886:80
Forwarding from 0.0.0.0:30886 -> 8052
Handling connection for 30886
E0826 12:37:58.184875   17615 portforward.go:400] an error occurred forwarding 30886 -> 8052: error forwarding port 8052 to pod 949dbf535784f42ae169a0b31a933ba5f331015450f380e5c9873e8641ba93d0, uid : exit status 1: 2021/08/26 12:37:58 socat[14263] E connect(5, AF=2 127.0.0.1:8052, 16): Connection refused

Thanks

Hi,

I'm sorry that your port-forward isn't connecting. There are normally 3 reason why you're getting connection refused:

(1) No service is listening to the port.
(2) Problem in the port-forwarding rules: the traffic isn't getting there.
(3) Firewall is blocking the traffic on either end.

Can you run a kubectl get all and make sure everything is up and running in the deployment.

For SSL/TLS try this link from the AWX page:

https://github.com/ansible/awx-operator#network-and-tls-configuration

@dmccuk
Copy link
Author

dmccuk commented Aug 26, 2021

this helped me out greatly so thank you so much. I was wondering if there was any chance you'd want to help extend this for using an external postgres DB as well as configuring it on a proper subdomain?

ive done it successfully in a docker container with AWX v14 but the switch to kubernetes really throwing me off.

@dreaddesign I'd like to expand on what I've done here as I've had requests for both an external PostgreSQL and SSL/TLS, but right now I've just got too much on at work with people off for holidays. If I can, I'd like to come back to this in the future and make a how to for that.

@nicolaibaralmueller
Copy link

I startet over and this time it actually worked so thanks for the guide. Now I just need to figure out how to use port 443 with certificate and backup/restore of the database.

@dreaddesign
Copy link

this helped me out greatly so thank you so much. I was wondering if there was any chance you'd want to help extend this for using an external postgres DB as well as configuring it on a proper subdomain?
ive done it successfully in a docker container with AWX v14 but the switch to kubernetes really throwing me off.

@dreaddesign I'd like to expand on what I've done here as I've had requests for both an external PostgreSQL and SSL/TLS, but right now I've just got too much on at work with people off for holidays. If I can, I'd like to come back to this in the future and make a how to for that.

sounds great!

@dmccuk
Copy link
Author

dmccuk commented Aug 29, 2021

I've just posted a YouTube video of installing and setting up the AWX-operator on Cento 8 using K3s + https. I hope this is a step up from the minikube demo:

https://gist.github.com/dmccuk/098fccc488c8a5aeaa5f859855660018
https://youtu.be/xVOnBDvpbK0

@nicolaibaralmueller
Copy link

I've just posted a YouTube video of installing and setting up the AWX-operator on Cento 8 using K3s + https. I hope this is a step up from the minikube demo:

https://gist.github.com/dmccuk/098fccc488c8a5aeaa5f859855660018
https://youtu.be/xVOnBDvpbK0

Just wanted to let you know that your code was successful on centos 8. First attempt failed but wasn't executing as root. Thank you.

Question: awx.yaml contains postgres storage requirements with 2Gi as default. Is this the only parameter to change in order to set database max size?

pv.yaml and pv.yaml also contain storage parameters.

@shrutebattlestargalactica
Copy link

Hello @dmccuk - this guide has helped me tremendously thank you so much for taking the time to create this! I am new to Kubernetes, and with my deployment I used eksctl for an AWS EKS Cluster. I am stuck at the part where you are setting up the minikube tunnel and doing the port forwarding. Is there some difference in the commands to run the tunnel for EKS? Also - when you grabbed the ip address + port for the URL to the AWX Tower GUI - how did you know which IP address to grab for the URL access?

**_`$ ./aws-runas.exe AWX_POC kubectl get all -l "app.kubernetes.io/managed-by=awx-operator"
NAME READY STATUS RESTARTS AGE
pod/awx-poc-5cd69c578c-vdnlk 4/4 Running 0 136m
pod/awx-poc-postgres-0 1/1 Running 0 136m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-poc LoadBalancer 172.20.xx.45 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.ap-southeast-1.elb.amazonaws.com 8080:31689/TCP 126m
service/awx-poc-postgres ClusterIP None 5432/TCP 136m
service/awx-poc-service NodePort 172.20.xxx.168 80:32765/TCP 136m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx-poc 1/1 1 1 136m

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-poc-5cd69c578c 1 1 1 136m

NAME READY AGE
statefulset.apps/awx-poc-postgres 1/1 136m`_**

@dmccuk
Copy link
Author

dmccuk commented Aug 31, 2021

Hello - this guide has helped me tremendously thank you so much! I am new to Kubernetes, and with my deployment I used eksctl for an EKS Cluster. I am stuck at the part where you are setting up the minikube tunnel and doing the port forwarding. Is there some difference in commands to run the tunnel for EKS? Also - when you grabbed the ip address + port for the URL to the AWX Tower GUI - how did you know which IP address to grab?

If you're using EKS, you don't need to do the [minikube] tunnel or the port-forwarding. The awx-operator has everything it needs to deploy and expose the service. If your new to Kubernetes, I'd go through this course: https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/
It really helped me and it's very reasonably priced. They include labs within the course so you learn everything without having to use your own cluster and EKS could get expensive!

I do have another demo on line that might be more useful [Installing and setting up the AWX-operator on Cento 8 using K3s + https] but definitely try the course out.

Demo: https://youtu.be/xVOnBDvpbK0
Commands: https://gist.github.com/dmccuk/098fccc488c8a5aeaa5f859855660018

@shrutebattlestargalactica

Hello - this guide has helped me tremendously thank you so much! I am new to Kubernetes, and with my deployment I used eksctl for an EKS Cluster. I am stuck at the part where you are setting up the minikube tunnel and doing the port forwarding. Is there some difference in commands to run the tunnel for EKS? Also - when you grabbed the ip address + port for the URL to the AWX Tower GUI - how did you know which IP address to grab?

If you're using EKS, you don't need to do the [minikube] tunnel or the port-forwarding. The awx-operator has everything it needs to deploy and expose the service. If your new to Kubernetes, I'd go through this course: https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/
It really helped me and it's very reasonably priced. They include labs within the course so you learn everything without having to use your own cluster and EKS could get expensive!

I do have another demo on line that might be more useful [Installing and setting up the AWX-operator on Cento 8 using K3s + https] but definitely try the course out.

Demo: https://youtu.be/xVOnBDvpbK0
Commands: https://gist.github.com/dmccuk/098fccc488c8a5aeaa5f859855660018

@dmccuk thanks for the great information. I guess one piece I am still stuck on is the URL address details. I also watched your other deployment video and I guess what I’m trying to determine is if you are using the public ip + service port for the AWX-demo-service? Since you mentioned that AWX-operator has everything it needs and no port forwarding or tunnel, I am just unsure of the address+port information to use to hit the GUI

@shrutebattlestargalactica

Hello - this guide has helped me tremendously thank you so much! I am new to Kubernetes, and with my deployment I used eksctl for an EKS Cluster. I am stuck at the part where you are setting up the minikube tunnel and doing the port forwarding. Is there some difference in commands to run the tunnel for EKS? Also - when you grabbed the ip address + port for the URL to the AWX Tower GUI - how did you know which IP address to grab?

If you're using EKS, you don't need to do the [minikube] tunnel or the port-forwarding. The awx-operator has everything it needs to deploy and expose the service. If your new to Kubernetes, I'd go through this course: https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/
It really helped me and it's very reasonably priced. They include labs within the course so you learn everything without having to use your own cluster and EKS could get expensive!
I do have another demo on line that might be more useful [Installing and setting up the AWX-operator on Cento 8 using K3s + https] but definitely try the course out.
Demo: https://youtu.be/xVOnBDvpbK0
Commands: https://gist.github.com/dmccuk/098fccc488c8a5aeaa5f859855660018

@dmccuk thanks for the great information. I guess one piece I am still stuck on is the URL address details. I also watched your other deployment video and I guess what I’m trying to determine is if you are using the public ip + service port for the AWX-demo-service? Since you mentioned that AWX-operator has everything it needs and no port forwarding or tunnel, I am just unsure of the address+port information to use to hit the GUI

@dmccuk are you advising with an EKS that these steps are not necessary:

-- Expose the deployment:
kubectl expose deployment awx-demo --type=LoadBalancer --port=8080

-- minikube tunnel
On a new session, start the minikube tunnel:

minikube tunnel

-- Enable AWX to be access via the Internet:
kubectl port-forward svc/awx-demo-service --address 0.0.0.0 30886:80

Now visit https://your_ip:high_port

^^ this is what I am trying to figure out how to access Now visit https://your_ip:high_port

@dmccuk
Copy link
Author

dmccuk commented Sep 1, 2021

You're using EKS, I'm using minikube. They are not the same.

The port forwarding and minikube tunnel are only required for the minikube setup. You won't be able to just take my walkthrough for minikube and apply it to EKS. You need to run the steps for EKS. I'd need to go through the EKS setup to know the steps as I've never used it (I don't have any plans to do that).

Why do you need to use EKS? if you're learning, just build a single server (with enough resources) and use minikube? or follow my other K3s demo. Both will setup the awx-operator and give you a login.

@shrutebattlestargalactica

Hi again. Thanks for the quick replies. So this seems to be a common question that gets asked to me "Why are you using an EKS cluster?"

Maybe need to take a step back like I believe you and others are mentioning and scrapping this design. Certainly feel like this EKS cluster may be overkill for a production instance let alone a POC.

@CarlosMonteroTech
Copy link

hi there,
Thanks for the instructions, it was very helpful. I opened the firewall but the connection is still refused. Any help would be much appreciated.

@olegsidokhmetov
Copy link

olegsidokhmetov commented Oct 7, 2021

Hello!

After command minikube tunnel I tried to execute command
$ kubectl port-forward svc/awx-demo-service --address 0.0.0.0 30886:80

but had error
error: unable to forward port because pod is not running. Current status=Pending

Here is all logs

$ kubectl get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/awx-demo-7bbb564887-52llc       0/4     Pending   0          15m
pod/awx-demo-postgres-0             1/1     Running   0          15m
pod/awx-operator-849b7d7d9d-k4vqq   1/1     Running   0          18m

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)             AGE
service/awx-demo               LoadBalancer   10.110.63.96     10.110.63.96   8080:30977/TCP      13m
service/awx-demo-postgres      ClusterIP      None             <none>         5432/TCP            15m
service/awx-demo-service       NodePort       10.99.78.134     <none>         80:30736/TCP        15m
service/awx-operator-metrics   ClusterIP      10.107.212.178   <none>         8383/TCP,8686/TCP   17m
service/kubernetes             ClusterIP      10.96.0.1        <none>         443/TCP             49m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx-demo       0/1     1            0           15m
deployment.apps/awx-operator   1/1     1            1           18m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-demo-7bbb564887       1         1         0       15m
replicaset.apps/awx-operator-849b7d7d9d   1         1         1       18m

NAME                                 READY   AGE
statefulset.apps/awx-demo-postgres   1/1     15m

describe pods awx-demo-7bbb564887-52llc


$ kubectl describe pods awx-demo-7bbb564887-52llc
Name:           awx-demo-7bbb564887-52llc
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/component=awx
                app.kubernetes.io/managed-by=awx-operator
                app.kubernetes.io/name=awx-demo
                app.kubernetes.io/part-of=awx-demo
                app.kubernetes.io/version=19.3.0
                pod-template-hash=7bbb564887
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/awx-demo-7bbb564887
Containers:
  redis:
    Image:      docker.io/redis:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      redis-server
      /etc/redis.conf
    Environment:  <none>
    Mounts:
      /data from awx-demo-redis-data (rw)
      /etc/redis.conf from awx-demo-redis-config (ro,path="redis.conf")
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-pj2tm (ro)
  awx-demo-web:
    Image:      quay.io/ansible/awx:19.3.0
    Port:       8052/TCP
    Host Port:  0/TCP
    Requests:
      cpu:     1
      memory:  2Gi
    Environment:
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /etc/nginx/nginx.conf from awx-demo-nginx-conf (ro,path="nginx.conf")
      /etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
      /etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
      /etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
      /etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-pj2tm (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-demo-task:
    Image:      quay.io/ansible/awx:19.3.0
    Port:       <none>
    Host Port:  <none>
    Args:
      /usr/bin/launch_awx_task.sh
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      SUPERVISOR_WEB_CONFIG_PATH:  /etc/supervisord.conf
      AWX_SKIP_MIGRATIONS:         1
      MY_POD_UID:                   (v1:metadata.uid)
      MY_POD_IP:                    (v1:status.podIP)
      MY_POD_NAMESPACE:            default (v1:metadata.namespace)
    Mounts:
      /etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
      /etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
      /etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
      /etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
      /etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/lib/awx/rsyslog from rsyslog-dir (rw)
      /var/run/awx-rsyslog from rsyslog-socket (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/redis from awx-demo-redis-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-pj2tm (ro)
      /var/run/supervisor from supervisor-socket (rw)
  awx-demo-ee:
    Image:      quay.io/ansible/awx-ee:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      receptor
      --config
      /etc/receptor.conf
    Requests:
      cpu:        500m
      memory:     1Gi
    Environment:  <none>
    Mounts:
      /etc/receptor.conf from awx-demo-receptor-config (ro,path="receptor.conf")
      /var/lib/awx/projects from awx-demo-projects (rw)
      /var/run/receptor from receptor-socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-pj2tm (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  awx-demo-application-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-demo-app-credentials
    Optional:    false
  awx-demo-secret-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-demo-secret-key
    Optional:    false
  awx-demo-settings:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-nginx-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-redis-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-redis-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  awx-demo-redis-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  supervisor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rsyslog-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  receptor-socket:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  rsyslog-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  awx-demo-receptor-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      awx-demo-awx-configmap
    Optional:  false
  awx-demo-projects:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  awx-demo-token-pj2tm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  awx-demo-token-pj2tm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x19 over 23m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

@dmccuk
Copy link
Author

dmccuk commented Oct 7, 2021

Hi Oleg,

It looks like the server you're using needs more resources. How much CPUs and memory do you have?

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x19 over 23m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

This link has some more information:
https://containersolutions.github.io/runbooks/posts/kubernetes/0-nodes-available-insufficient/

I hope it helps.

Dennis

@olegsidokhmetov
Copy link

Hi Oleg,

It looks like the server you're using needs more resources. How much CPUs and memory do you have?

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x19 over 23m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

This link has some more information: https://containersolutions.github.io/runbooks/posts/kubernetes/0-nodes-available-insufficient/

I hope it helps.

Dennis

Thank you for your fast answer. I use VM with 2 CPU and 2Gb RAM.

@dmccuk
Copy link
Author

dmccuk commented Oct 7, 2021

In my demo, I used a t3a.xlarge – 4 x cpu / 16gb memory in AWS. You will need at least 4 CPUs and 8GB memory for it to work properly.

I did try to get a T2.medium working (2CPU X 4GB) but it wasn't enough.

Sorry but with the AWX operator you need more resources.

@CarlosMonteroTech
Copy link

I reinstall it again and not luck. These are my logs:

NAME READY STATUS RESTARTS AGE
pod/awx-demo-7744b86597-d45q4 4/4 Running 0 13m
pod/awx-demo-postgres-0 1/1 Running 0 13m
pod/awx-operator-5dd757f594-7x98q 1/1 Running 0 13m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-demo LoadBalancer 10.99.22.95 10.99.22.95 8080:32194/TCP 11m
service/awx-demo-postgres ClusterIP None 5432/TCP 28m
service/awx-demo-service NodePort 10.100.237.74 80:30717/TCP 27m
service/awx-operator-metrics ClusterIP 10.105.183.10 8383/TCP,8686/TCP 13m
service/kubernetes ClusterIP 10.96.0.1 443/TCP 28m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx-demo 1/1 1 1 2d3h
deployment.apps/awx-operator 1/1 1 1 2d3h

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-demo-7744b86597 1 1 1 64m
replicaset.apps/awx-demo-7bbb564887 0 0 0 2d3h
replicaset.apps/awx-operator-5dd757f594 1 1 1 65m
replicaset.apps/awx-operator-849b7d7d9d 0 0 0 2d3h

NAME READY AGE
statefulset.apps/awx-demo-postgres 1/1 2d3h

test:~$ kubectl describe pods awx-demo-7744b86597-d45q4
Name: awx-demo-7744b86597-d45q4
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Fri, 08 Oct 2021 13:42:20 +1000
Labels: app.kubernetes.io/component=awx
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=awx-demo
app.kubernetes.io/part-of=awx-demo
app.kubernetes.io/version=19.2.0
pod-template-hash=7744b86597
Annotations:
Status: Running
IP: 172.17.0.2
IPs:
IP: 172.17.0.2
Controlled By: ReplicaSet/awx-demo-7744b86597
Containers:
redis:
Container ID: docker://d6ed2245db5ea3c7ad124155d02cdcb258ce232606a75f4c9e721048b7a3699a
Image: docker.io/redis:latest
Image ID: docker-pullable://redis@sha256:2d692e20ba38bf5382d497ee975f0b3389abee73e030cc3315d78c180bf19d66
Port:
Host Port:
Args:
redis-server
/etc/redis.conf
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Environment:
Mounts:
/data from awx-demo-redis-data (rw)
/etc/redis.conf from awx-demo-redis-config (ro,path="redis.conf")
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
awx-demo-web:
Container ID: docker://8ef9d812e55379dfdc0bf2812f11aaf0dbcbf4dc4ed398ba3e9bf6614b3a69ef
Image: quay.io/ansible/awx:19.2.0
Image ID: docker-pullable://quay.io/ansible/awx@sha256:f7cdabee0da2ea195e3dab8a8b39f3f5f1f32f0d2ee3d0ac561ec7d640d7042d
Port: 8052/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Requests:
cpu: 1
memory: 2Gi
Environment:
MY_POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/etc/nginx/nginx.conf from awx-demo-nginx-conf (ro,path="nginx.conf")
/etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
/etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
/etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
/etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
/etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/lib/awx/rsyslog from rsyslog-dir (rw)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
/var/run/supervisor from supervisor-socket (rw)
awx-demo-task:
Container ID: docker://19710ce432b9f9cbc9a4d1ddac026554e502a68ebdd58a3325d787a0fa13da12
Image: quay.io/ansible/awx:19.2.0
Image ID: docker-pullable://quay.io/ansible/awx@sha256:f7cdabee0da2ea195e3dab8a8b39f3f5f1f32f0d2ee3d0ac561ec7d640d7042d
Port:
Host Port:
Args:
/usr/bin/launch_awx_task.sh
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Requests:
cpu: 500m
memory: 1Gi
Environment:
SUPERVISOR_WEB_CONFIG_PATH: /etc/supervisord.conf
AWX_SKIP_MIGRATIONS: 1
MY_POD_UID: (v1:metadata.uid)
MY_POD_IP: (v1:status.podIP)
MY_POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
/etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
/etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
/etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
/etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/lib/awx/rsyslog from rsyslog-dir (rw)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/receptor from receptor-socket (rw)
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
/var/run/supervisor from supervisor-socket (rw)
awx-demo-ee:
Container ID: docker://b90510fcb1ba2af2d1fc4f455bbd1e336f3882674eeed33109cdf892ecbba638
Image: quay.io/ansible/awx-ee:0.3.0
Image ID: docker-pullable://quay.io/ansible/awx-ee@sha256:885facada773ef85bfd4fc952a268f3d6e4331d5d134e79c54bb2bb201f81968
Port:
Host Port:
Args:
receptor
--config
/etc/receptor.conf
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Environment:
Mounts:
/etc/receptor.conf from awx-demo-receptor-config (ro,path="receptor.conf")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/run/receptor from receptor-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
awx-demo-application-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-app-credentials
Optional: false
awx-demo-secret-key:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-secret-key
Optional: false
awx-demo-settings:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-redis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-redis-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-redis-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
supervisor-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
rsyslog-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
receptor-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
rsyslog-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-receptor-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-projects:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-token-4b6tl:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-token-4b6tl
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message


Normal Scheduled 14m default-scheduler Successfully assigned default/awx-demo-7744b86597-d45q4 to minikube
Normal Pulled 14m kubelet Container image "docker.io/redis:latest" already present on machine
Normal Started 14m kubelet Started container awx-demo-web
Normal Started 14m kubelet Started container redis
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx:19.2.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-web
Normal Created 14m kubelet Created container redis
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx:19.2.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-task
Normal Started 14m kubelet Started container awx-demo-task
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx-ee:0.3.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-ee
Normal Started 14m kubelet Started container awx-demo-ee

Firewall
test:~$ sudo ufw status
Status: active

To Action From


30710:30720/tcp ALLOW Anywhere

@dmccuk
Copy link
Author

dmccuk commented Oct 8, 2021

Did you create a larger VM to install AWX on [more CPU and more memory]? I would advise you delete the original VM, create a new one with enough resources and then follow the video again.

@day4me
Copy link

day4me commented Oct 15, 2021

hi, trying to use callback to provision my ec2 instance curl -H 'Content-Type: application/json' --data '{"host_config_key": "key"}' http://IP:30877/api/v2/job_templates/13/callback/ but I get this error: No matching host could be found!. I added HTTP_X_FORWARDED_FOR to configuration but it still doesn't work.

@max18-100
Copy link

Followed your instruction but only the awx-demo-postgres comes up. But no error message.
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME READY STATUS RESTARTS AGE
awx-demo-postgres-0 1/1 Running 0 35m

@nicolaibaralmueller
Copy link

Followed your instruction but only the awx-demo-postgres comes up. But no error message. kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" NAME READY STATUS RESTARTS AGE awx-demo-postgres-0 1/1 Running 0 35m

Using this one now. Very good.
https://github.com/kurokobo/awx-on-k3s

@dmccuk
Copy link
Author

dmccuk commented Jun 23, 2022

Followed your instruction but only the awx-demo-postgres comes up. But no error message. kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" NAME READY STATUS RESTARTS AGE awx-demo-postgres-0 1/1 Running 0 35m

Hi Max18-100. I've not looked at this for a long time! The thing that came up many times before was the VM not having enough resources. In my demo, I used a t3a.xlarge – 4 x cpu / 16gb memory in AWS. You will need at least 4 CPUs and 8GB memory for it to work properly.

I did try to get a T2.medium working (2CPU X 4GB) but it wasn't enough.

Does your VM have enough resources? Alternatively try the K3 install.: https://github.com/kurokobo/awx-on-k3s. I have another video with a demo on my channel here: https://www.youtube.com/watch?v=xVOnBDvpbK0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment