This post contains code and commands you can use to deploy Prefect agents to Google Cloud’s Google Kubernetes Engine. The agents stand ready to execute workflows triggered by Prefect projects. One agent can run tasks from multiple projects.
The example here demonstrates how to create a single agent with minimal customization. It is configured with a Dockerfile
, which installs necessary dependencies, and a k8s.cfg
file, which connects the system to a Prefect account.
Agents are deployed via the gcloud
command-line utility and its kubectl
extension. Proper permissions within a project on Google Cloud are required.
Before you can begin you will need to have several tools installed and configured.
If you don't have a project on Google Cloud, you'll need to start there. You will need its unique identifer before you can continue.
Also, you should install the gcloud
command-line tool by following Google's instructions. Then initialize the tool and connect it with your Google Cloud account. When asked, set your project's id as the default.
gcloud init
Install the kubectl
extensions that allows you to communicate with Google Kubernetes Engine.
gcloud components install kubectl
Install the Prefect Python library with the Kubernetes and Google extensions.
pipenv install "prefect[kubernetes,google]"
Install docker and docker-compose.
Set the region where the cluster will be made. I'll use the one closest to me. You can pick another from Google's roster, if you'd like.
export COMPUTE_REGION=us-west2
Come up with a name for your cluster.
export CLUSTER=prefect-cluster
Create the cluster.
gcloud container clusters create-auto $CLUSTER \
--region=$COMPUTE_REGION;
After the creation process completes, ask for credentials to access it.
gcloud container clusters get-credentials $CLUSTER \
--region $COMPUTE_REGION;
Verify it exists.
kubectl get nodes;
Agents are created by booting and configuring an image within a cluster. The contents of the image are configured by a Dockerfile. Its connection with Prefect is configured by a separate file that is then applied to the machine.
The first step is to come up with a slug for your agent that we'll use in naming conventions. For this example, I will use etl
, which is a common acronym for the extract, transform and load processes that Prefect specializes in supporting.
Create a Dockerfile
in the current directory. At a minimum, it should start by inheriting Prefect’s default image. Nothing else is necessary to get started. But if you agent will require serverside dependencies beyond the basics, this is where you include them.
FROM prefecthq/prefect:latest
The Dockerfile
is then built for Google Artifact Registry. First thing to do is login there.
gcloud auth configure-docker us-west2-docker.pkg.dev
Before you can push the image, you should create a new Docker repository in the artifact registry with a name like prefect-agents
.
gcloud artifacts repositories create prefect-agents \
--repository-format=docker \
--location=us-west2
--description="Prefect agents"
You need to come up with another name for the image. I recommend adding the slug of your directory as a suffix to prefect-agents-
. Be sure to put in your project id. The first release should be tagged as 0.0.1
.
docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.1 .
Push the image.
docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.1
The image should also be aliased to the latest
tag.
docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest .
Push that too.
docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest
You should now create a Kubernetes configuration file that will be applied to the image. It will need include a Prefect API key that allows it to speak to your cloud dashboard. If you don't have a key already, go make one inside your Prefect account.
export API_KEY=your-api-key-here
pipenv run prefect agent kubernetes install --rbac -k $API_KEY > k8s.cfg
Open the configuration file. In the metadata section, give your agent a slugged name. I again recommend using your project slug as a suffix.
metadata:
labels:
app: prefect-agent
name: prefect-agent-etl
Your agent will need labels. These are used to mark your agent as available for the Prefect workflows that carry the same tag. Again I recommend using your agent's slug.
- name: PREFECT__CLOUD__AGENT__LABELS
value: '["etl"]'
Your API key will need to be included.
- name: PREFECT__CLOUD__API_KEY
value: your-api-key-here
Point the configuration file to the latest version of your Docker image in Google Artifact registry.
image: us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest
Save those edits. Now ask kubectl
to connect to our cluster where you would like to deploy this agent.
gcloud container clusters get-credentials prefect-cluster --zone $COMPUTE_REGION;
Ask for a new deployment in your Kubernetes cluster using the configuration file.
kubectl apply -f k8s.cfg
Once created you should be able to see it running in your cluster.
kubectl get deploy
Check the Agents page in your Prefect account. After a few minutes have gone by the new agent should appear.
Deployment of updates is done with the following steps
Up the version number on the the Dockerfile and push it to Google Artifact Registry. Here I'm moving it up to version number 0.0.2
. You want to iterate the image number up with each release.
docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.2 .
Connect to Google Artifact Registry.
gcloud auth configure-docker us-west2-docker.pkg.dev
Push the image.
docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:0.0.2
We also need to overwrite the latest
tag with your update.
docker build -t us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest .
Push it too.
docker push us-west2-docker.pkg.dev/your-project-id-here/prefect-agents/prefect-agents-etl:latest
Save any changes to the configuration file, then apply it.
kubectl apply -f k8s.cfg
Wait a few minutes and the agent should update.