Creating a custom EE for AWX
This is a quick and rough guide to creating and consuming a custom execution environment (EE) in AWX.
What You'll Need [ what I use ]
- Somewhere to build the EE [ Mac Big Sur ]
- Docker installed and running locally [ Docker Desktop - docker 20.10.5 ]
- python3 [ 3.9.2 ]
- An Image Registry [ free quay.io account ]
- AWX [ 19.0.0 on minikube 1.19.0 ]
Setup ansible-builder
Create a new clean python venv:
mkdir ~/ansible-builder && cd ~/ansible-builder python -m venv builder source builder/bin/activate
Install the components:
pip install ansible pip install ansible-builder==1.0.0.0a1
Create the config:
cat execution-environment.yml
---
version: 1
dependencies:
galaxy: requirements.yml
python: requirements.txt
system: bindep.txt
additional_build_steps:
append:
- RUN alternatives --set python /usr/bin/python3
- COPY --from=quay.io/project-receptor/receptor:0.9.7 /usr/bin/receptor /usr/bin/receptor
- RUN mkdir -p /var/run/receptor
- ADD run.sh /run.sh
- CMD /run.sh
- USER 1000
- RUN git lfs install
cat requirements.yml --- collections: - community.general
cat requirements.txt urllib3 git+https://github.com/ansible/ansible-builder.git@devel#egg=ansible-builder
cat bindep.txt python38-devel [platform:rpm compile] subversion [platform:rpm] subversion [platform:dpkg] git-lfs [platform:rpm]
cat context/run.sh #! /bin/bash ansible-runner worker --private-data-dir=/runner
chmod +x context/run.sh
Build the EE Image
ansible-builder build --tag quay.io/philgriffiths/awx-custom-ee:latest --context ./context --container-runtime docker Running command: docker build -f ./context/Dockerfile -t quay.io/philgriffiths/awx-custom-ee:latest ./context Running command: docker run --rm -v /Users/pgriffit/ansible-builder/builder/lib/python3.9/site-packages/ansible_builder:/ansible_builder_mount:Z quay.io/philgriffiths/awx-custom-ee:latest python3 /ansible_builder_mount/introspect.py Running command: docker build -f ./context/Dockerfile -t quay.io/philgriffiths/awx-custom-ee:latest ./context Complete! The build context can be found at: /Users/pgriffit/ansible-builder/context
We should now have a container image:
docker images | grep awx-custom-ee quay.io/philgriffiths/awx-custom-ee latest 94cb5bb21c61 33 minutes ago 701MB
Push this to quay.io:
docker push quay.io/philgriffiths/awx-custom-ee:latest The push refers to repository [quay.io/philgriffiths/awx-custom-ee] ...
Setting Up AWX Environment
Login to AWX, as per the installation instructions:
open `minikube service awx-service --url`
login as ---> admin
password ---> minikube kubectl -- get secret awx-admin-password -o jsonpath='{.data.password}' | base64 --decode
Add a new EE under Administration -> Execution Environments:
Now we can consume it via a Job Template:
See what's happening through the normal Jobs output:
Hope that helps!
Troubleshooting Issues
Some tips on what to check if things aren't working quite as they should!
minikube kubectl get events
This will often show you why an image pull hasn't worked. Might be user error in setting up (!), permissions on the registry:
22m Warning Failed pod/awx-job-35-9f7pg Failed to pull image "quay.io/ansible/awx-ee:1.0.1": rpc error: code = Unknown desc = Error response from daemon: manifest for quay.io/ansible/awx-ee:1.0.1 not found: manifest unknown: manifest unknown 6m19s Warning Failed pod/awx-job-37-hvgqk Failed to pull image "quay.io/ansible/awx-custom-ee:1.0.1": rpc error: code = Unknown desc = Error response from daemon: unauthorized: access to the requested resource is not authorized
You want to see this:
2m58s Normal Scheduled pod/awx-job-39-m5dhd Successfully assigned default/awx-job-39-m5dhd to minikube 2m58s Normal Pulling pod/awx-job-39-m5dhd Pulling image "quay.io/philgriffiths/awx-custom-ee:1.0.1" 2m52s Normal Pulled pod/awx-job-39-m5dhd Successfully pulled image "quay.io/philgriffiths/awx-custom-ee:1.0.1" in 5.815670096s 2m52s Normal Created pod/awx-job-39-m5dhd Created container worker 2m51s Normal Started pod/awx-job-39-m5dhd Started container worker 2m47s Normal Killing pod/awx-job-39-m5dhd Stopping container worker
If the job is just hanging/pending/running forever in the AWX console, then use the above in the first instance.
To check overall status, use:
minikube kubectl get all NAME READY STATUS RESTARTS AGE pod/awx-b5f6cf4d4-fjnj6 4/4 Running 28 13d pod/awx-operator-f768499d-267tg 1/1 Running 8 13d pod/awx-postgres-0 1/1 Running 7 13d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-operator-metrics ClusterIP 10.105.141.181 <none> 8383/TCP,8686/TCP 13d service/awx-postgres ClusterIP None <none> 5432/TCP 13d service/awx-service NodePort 10.100.36.62 <none> 80:32556/TCP 13d service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 13d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx 1/1 1 1 13d deployment.apps/awx-operator 1/1 1 1 13d NAME DESIRED CURRENT READY AGE replicaset.apps/awx-b5f6cf4d4 1 1 1 13d replicaset.apps/awx-operator-f768499d 1 1 1 13d NAME READY AGE statefulset.apps/awx-postgres 1/1 13d
Your pod names will be different! But you should see the first 3 running. If not, then there is something wrong with your deployment. One is for the database, another the operator. Note that the 'main' worker is covered by 'pod/awx-b5f6cf4d4-fjnj6' above and is 4/4. This is a replica set and contains 4 containers - redis, awx-web, awx-task, and awx-ee.
To look at the logs for each container, use:
minikube kubectl logs awx-b5f6cf4d4-fjnj6 awx-ee <-- replace with whatever container you want to see, awx-web, awx-task, awx-ee
This is only really for deeper dive technical runtime type issues.
infrastructure project manager
1 年Phil Griffiths we need awx installation using k8s hassel free on redhat, using awx operater still get issues and need one click installation also separate postgres container as service after awx install is a big headache, it's not upgrade on current version, can you pls take this feedback and improve awx upcoming version ? Instead of brain storming on playbooks we spend time to install and upgrade awx on different environment and it's killing our time
Senior software engineer at Intuit
1 年Thankyou for the document! i spun up awx with default latest images fromm quay and i made my custom awx-ee image following this doc though whenn i try to do docker run with my custom image , i was getting error like below {"status": "error", "job_explanation": "Failed to JSON parse a line from transmit stream."} {"eof": true} some one already mentioned this is normal so update directly in tower GUI execution environment which i did but here.my qus is my container inside pod is stil holding default awx-ee image not my custom one, my req is i want to able to update /etc/resolv.cong pointing to my internal DNS NS and fetch inventory scrpts and templates from internal github which im not able to do from default image as i dont have access sudo access or not able to customize default image. anyone's help on this would be greatly appreciated. Thanskyou!
System Administrator - Team Cymru
2 年Has anyone used this procedure with jfrog instead of quay.io?
Technical Services Engineer | BSc in Computer Science
2 年Hello! Your tutorial is terrific, thanks a lot! Just, if you want to make it better, add the "ANSIBLE_RUNNER_IMAGE" environment var to the execution-environment.yml file, so we can use de Ansible build of our choice when creating the EE. For example, to create an AWX EE for Ansible 2.9 (latest), we can add: #cat execution-environment.yml --- version: 1 build_arg_defaults: ?ANSIBLE_RUNNER_IMAGE: 'quay.io/ansible/ansible-runner:stable-2.9-latest' dependencies: ?galaxy: requirements.yml ?python: requirements.txt ?system: bindep.txt additional_build_steps: ?append: ??- RUN alternatives --set python /usr/bin/python3 ??- COPY --from=quay.io/project-receptor/receptor:0.9.7 /usr/bin/receptor /usr/bin/receptor ??- RUN mkdir -p /var/run/receptor ??- ADD run.sh /run.sh ??- CMD /run.sh ??- USER 1000? ??- RUN git lfs install All the images versions of ansible-runner are are available at quay.io and properly tagged: https://quay.io/repository/ansible/ansible-runner?tab=tags&tag=latest Cheers!
GCP ACE, Redhat Ansible, IBM Cloud
2 年Hello Phil Griffiths : THANKS A LOT for this article. I was trying to run a playbook on a docker based awx that uses the json_query module from collection community.general. But this collection was not part of the default AWX execution engine container. I was searching for a method to build a custom execution engine container image that would contain the collection community.general and the package "jmespath". I was able to create a custom execution engine taking reference from your article that worked perfectly for me. THANKs Again :)