I'm writing these notes because I spent a significant amount of time on a situation related with how docker and docker-compose work, specifically in integration tests. I've learned a few things about how docker that might be useful to remember at the next occasion.
If you use docker-compose for your integration tests, you might end up in a situation where you have erratic tests because,
while tests are run, some containers are not yet doing what they are supposed to do.
For example, you might have a docker-compose.yml that sets up kafka, zookeeper and postgres and then you want to run your application
that connects to these systems. Likely your build pipeline will follow these steps:
docker-compose up- Run the tests, if you are doing scala you'll likely have something like
sbt it:test
When a docker container starts it goes typically through the following states:
- Starting
- Started/Up (the container entry point is running and didn't exit)
- If a healthcheck is setup it can be
Up (healthy: starting),Up (healthy)orUp (unhealthy)
When docker-compose up runs, it spins up the containers, in the order possibly prescribed by the depends_on clauses used in your docker-compose.yaml.
Note that this only ensures that the required containers are at least in a Started state. This means that if a container has a healthcheck setup, the fact that docker-compose up ran successfully does not mean that the container us up and doing what it is supposed to do.
This doesn't mean that the healthcheck clause is useless. Au contraire, I prefer to use it rather than checking the status
of the container from an external point (e.g. doing a curl localhost:<container-exposed-port>), because the tools I can access
for healthcheck are all those specific to the technology the container is about. E.g. in a healthcheck on a postgres container
I can use pg_isready which is not necessarily available on my machine or on my build agent.
So the actions I suggest to do are:
- Specify wherever possible a healthcheck for each container you spin up
- After docker-compose up check the status of each container until they are all healthy:
docker inspect $CONTAINER_NAME --format "{{.State.Health.Status}}" ) == "healthy" - Run your tests
As an alternative solutions, you can also use a fully programmatical approach such as testcontainers