How Kamal runs your migrations

I realized something clever that Kamal does the other day when I was writing the detailed overview of running a kamal setup command. Well, it’s not clever but just simplifies something that can easily complicate the deployment flow.

Database Migrations.

When deploying web apps you’ll often need to change your database schema as you release a new feature or update an old one. In the Kubernetes world, this is done with a sidecar container or a migration service that you have to run in your K8s mess. In Heroku and other PAAS offerings,> you often have a bin/release command that runs your migrations just before your app goes live.

Part of Kamal’s approach is the conditional in the default Rails entry point command. Here’s the default docker-entrypoint that ships with Rails now:

#!/bin/bash -e

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

A few important things to point out here:

1. Migrations by default only run(rails db:prepare) when attempting to boot the rails server, not your job container, or any other role. But what about when you have multiple web containers or roles? I’ll get to that shortly.
2. The last exec “${@}" line just passes through whatever you sent to the bin/docker-entrypoint script. So if you call bin/docker-entrypoint bin/rails c , ,for instance, it’s just going to run that as exec bin/rails c .

Back to point #1(multiple web containers). With Kamal,labeled there are two healthchecks that end up happening to get your application deployed. When you look at the output initially it can be a bit confusing but it’s actually pretty simple what’s going on and it helps speed up the blue/green deployment process.

1. The very first healthcheck output you’ll see boots your freshly built container and you’ll see it labeled as healthcheck-service-destination-sha or healthcheck-hey-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836. This healthcheck runs so that Kamal can confirm that your container can get to a healthy state before doing the rolling deploy. As part of this though since it’s running the default CMD of CMD ["./bin/rails", "server"] this is actually when your migrations run. From our entrypoint script, if the command passed in is to run the server, it'll prepare the db first and then run the server.

  INFO [83f25502] Running docker run --detach --name healthcheck-hey-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836 --publish 3999:3000 --label service=healthcheck-hey-production -e KAMAL_CONTAINER_NAME="healthcheck-hey-production" --env-file .kamal/env/roles/hey-web-production.env --health-cmd "curl -f http://localhost:3000/up || exit 1" --health-interval "1s" ghcr.io/nickhammond/hey:e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836 on 111.222.333.444

  INFO [e5ac6e4a] Running docker container ls --all --filter name=^healthcheck-hey-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 111.222.333.444
  INFO [e5ac6e4a] Finished in 0.156 seconds with exit status 0 (successful).
  INFO container not ready (unhealthy), retrying in 1s (attempt 1/7)...

2. The second healthcheck output is for the actual application container that’ll be running once the Kamal deployment finishes, notice it's named as you would expect hey-web-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836. This second healthcheck is to prep for the rolling deploy and you’ll see it checking for an unhealthy container and a healthy container at the same time. This is the rolling deploy in action and ensures that traffic to traefik can finish delivering traffic to the old container until all requests in transit have been served and promote the new container.

  INFO [0f3c00c7] Running docker run --detach --restart unless-stopped --name hey-web-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836 ghcr.io/nickhammond/hey:e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836 on 111.222.333.444
  INFO [0f3c00c7] Finished in 0.625 seconds with exit status 0 (successful).
  
  INFO [450c443e] Running docker container ls --all --filter name=^hey-web-production-e24f45d1d35fdb393b4f4a8feb5ae92b8a52b836$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 111.222.333.444
  INFO [450c443e] Finished in 0.222 seconds with exit status 0 (successful).
  INFO container not ready (starting), retrying in 1s (attempt 1/7)...

Another thing to point out with this is because Kamal expects one application container to be built for a deploy even if you do have multiple web roles, it’s still the same container that it's building and starting. There’s still that initial healthcheck container based on your application so that when it gets to releasing any additional roles your migrations already ran early on before your application goes live in the first healthcheck.

Another issue that comes up with migrations is if you have a large migration to run as part of a deploy, sometimes it’s going to make sense to do this out of band. For instance, maybe you’re adding an index to a million row table and it’s going to take a few minutes. You don’t really want a deploy hanging and waiting for a long migration to finish, you can but it’s pretty simple to do this directly with your database with Rails.

1. Create the migration locally
2. Run the migration
3. Grab the SQL for the migration, save it for later
4. Commit the updated schema
5. Run the SQL from #3, and wait for it to finish
6. Grab the schema version for this migration and insert it in your database within the schema_migrations table in your destination
7. Deploy your application, it’ll skip the migration since you added the version to the schema_migrations table.

I suppose Kamal could’ve introduced another hook to accomplish the migration process, similar to Heroku but using the simple entry point approach removes an additional step. This approach was also merged into Rails quite a while ago as the default via templates, thanks to @nickjj.