Using Kamal to deploy and manage your services

Kamal is great for deploying containerized apps but you can also use it as a really lightweight services manager. For instance, say you want to host your own services(Redis, Elasticsearch, Docker registry, etc.) instead of using a cloud offering, getting them running with Kamal is as easy as adding the accessory and then telling Kamal to boot them all. Kamal also allows you to easily restart, tail the logs, run commands within that service, and more.

Say for example you have a new Ubuntu box ready to host some of these services on. We'll start by initializing Kamal within a new local directory, we’ll initialize the git repository as well so that Kamal has a SHA for naming the containers.

mkdir services && cd services
git init
kamal init

After running kamal init we’ll have our sample config/deploy.yml and a new .env file to utilize. The first thing is to update your .env file with an actual KAMAL_REGISTRY_PASSWORD, we won’t be pushing anything to the registry but Kamal logs into the registry at the beginning of some of the commands we’ll be running.

Then within your config/deploy.yml we can start adding our accessories and updating a few additional things. I’m not planning on utilizing this Kamal configuration for deploying an actual application, this is just for the various services that multiple apps will use.

At a minimum you’ll need to set:

• service - Set this to something useful because all of your containers will be named after this, service-redis for instance.
• image - You can leave this with the default value, we won’t actually be building an image but Kamal expects this to be set.
• servers - Set at least one server here, we’ll utilize this setting to bootstrap the server with Docker in a bit.
• registry - When we go to boot our services Kamal is going to authenticate with the registry settings you define here.
• accessories - Now we can start building these out. We’ll spin up the redis, memcached, and registry containers from Docker hub.

service: ftk

image: user/my-app

servers:
  - 123.456.789.0

registry:
  username: nick
  password:
    - KAMAL_REGISTRY_PASSWORD

accessories:
  redis:
    image: redis:7.0
    host: 123.456.789.0
    port: 6379
    directories:
      - data:/data
  memcached:
    image: memcached:1.6.24
    host: 123.456.789.0
    port: 11211
    directories:
      - data:/data
  registry:
    image: registry:2.8.3
    host: 123.456.789.0
    port: 5000
    directories:
      - data:/data

With those in place we can go ahead and bootstrap the server with docker via kamal.

kamal server bootstrap

With Docker now running on our server we next have to push up our env files, Kamal boots our accessories by pointing to these files. It doesn’t matter if it’s empty, it just needs to be there and will only contain the mapped secret environment variables you’ve set if any. Since the example above doesn’t actually map any secret environment variables it’ll be empty for them all anyways.

kamal env push

You’ll see that Kamal creates a .env file for all of our accessories as well as the default web and traefik containers.

$ kamal env push
  INFO [854d33cf] Running /usr/bin/env mkdir -p .kamal on 123.456.789.0
  INFO [854d33cf] Finished in 1.279 seconds with exit status 0 (successful).
  INFO [31a2bdb8] Running /usr/bin/env mkdir -p .kamal/locks on 123.456.789.0
  INFO [31a2bdb8] Finished in 0.139 seconds with exit status 0 (successful).
Acquiring the deploy lock...
  INFO [4d499ad2] Running /usr/bin/env mkdir -p .kamal/env/roles on 123.456.789.0
  INFO [4d499ad2] Finished in 0.145 seconds with exit status 0 (successful).
  INFO Uploading .kamal/env/roles/heyo-web.env 100.0%
  INFO [af60ea99] Running /usr/bin/env mkdir -p .kamal/env/traefik on 123.456.789.0
  INFO [af60ea99] Finished in 0.138 seconds with exit status 0 (successful).
  INFO Uploading .kamal/env/traefik/traefik.env 100.0%
  INFO [25a2c91b] Running /usr/bin/env mkdir -p .kamal/env/accessories on 123.456.789.0
  INFO [25a2c91b] Finished in 0.134 seconds with exit status 0 (successful).
  INFO Uploading .kamal/env/accessories/heyo-redis.env 100.0%
  INFO [2fff3572] Running /usr/bin/env mkdir -p .kamal/env/accessories on 123.456.789.0
  INFO [2fff3572] Finished in 0.137 seconds with exit status 0 (successful).
  INFO Uploading .kamal/env/accessories/heyo-memcached.env 100.0%
  INFO [08203368] Running /usr/bin/env mkdir -p .kamal/env/accessories on 123.456.789.0
  INFO [08203368] Finished in 0.137 seconds with exit status 0 (successful).
  INFO Uploading .kamal/env/accessories/heyo-registry.env 100.0%

With those .env files in place for everything we can now boot up all of our accessories.

$ kamal accessory boot all
  INFO [2906020a] Running /usr/bin/env mkdir -p .kamal on 123.456.789.0
  INFO [2906020a] Finished in 0.697 seconds with exit status 0 (successful).
  INFO [a3b3c265] Running /usr/bin/env mkdir -p .kamal/locks on 123.456.789.0
  INFO [a3b3c265] Finished in 0.137 seconds with exit status 0 (successful).
Acquiring the deploy lock...
  INFO [9c865eec] Running /usr/bin/env mkdir -p $PWD/ftk-redis/data on 123.456.789.0
  INFO [9c865eec] Finished in 0.135 seconds with exit status 0 (successful).
  INFO [d094a1d5] Running docker login -u [REDACTED] -p [REDACTED] on 123.456.789.0
  INFO [d094a1d5] Finished in 0.796 seconds with exit status 0 (successful).
  INFO [ac565d9f] Running docker run --name ftk-redis --detach --restart unless-stopped --log-opt max-size="10m" --publish 6379:6379 --env-file .kamal/env/accessories/ftk-redis.env --volume $PWD/ftk-redis/data:/data --label service="ftk-redis" redis:7.0 on 123.456.789.0
  INFO [ac565d9f] Finished in 5.626 seconds with exit status 0 (successful).
  INFO [9fd8d34a] Running /usr/bin/env mkdir -p $PWD/ftk-memcached/data on 123.456.789.0
  INFO [9fd8d34a] Finished in 0.155 seconds with exit status 0 (successful).
  INFO [59bffdf6] Running docker login -u [REDACTED] -p [REDACTED] on 123.456.789.0
  INFO [59bffdf6] Finished in 1.975 seconds with exit status 0 (successful).
  INFO [f6e9415d] Running docker run --name ftk-memcached --detach --restart unless-stopped --log-opt max-size="10m" --publish 11211:11211 --env-file .kamal/env/accessories/ftk-memcached.env --volume $PWD/ftk-memcached/data:/data --label service="ftk-memcached" memcached:1.6.24 on 123.456.789.0
  INFO [f6e9415d] Finished in 5.343 seconds with exit status 0 (successful).
  INFO [c76675fb] Running /usr/bin/env mkdir -p $PWD/ftk-registry/data on 123.456.789.0
  INFO [c76675fb] Finished in 0.139 seconds with exit status 0 (successful).
  INFO [c7867413] Running docker login -u [REDACTED] -p [REDACTED] on 123.456.789.0
  INFO [c7867413] Finished in 0.873 seconds with exit status 0 (successful).
  INFO [9d4784ed] Running docker run --name ftk-registry --detach --restart unless-stopped --log-opt max-size="10m" --publish 5000:5000 --env-file .kamal/env/accessories/ftk-registry.env --volume $PWD/ftk-registry/data:/data --label service="ftk-registry" registry:2.8.3 on 123.456.789.0
  INFO [9d4784ed] Finished in 2.729 seconds with exit status 0 (successful).
Releasing the deploy lock...

You'll notice the 3 docker run commands in the output with our port mappings and referencing those .env files that we pushed up.

  INFO [ac565d9f] Running docker run --name ftk-redis --detach --restart unless-stopped --log-opt max-size="10m" --publish 6379:6379 --env-file .kamal/env/accessories/ftk-redis.env --volume $PWD/ftk-redis/data:/data --label service="ftk-redis" redis:7.0 on 123.456.789.0

Now we can hop onto our server and verify that those services are indeed up and running with telnet.

$ ssh user@123.456.789.0
$ telnet localhost 5000
Trying ::1...
Connected to localhost.

Yep, our new Docker registry is listening on port 5000 which is what we mapped it to in our deploy.yml. You don’t have to manually SSH onto the server to check this though, Kamal also has some helpful commands built in to manage the lifecycle of your accessories.

The details command is a great start which runs a docker ps specifically for our accessories on the server.

$ kamal accessory details all
  INFO [86c4d405] Running docker ps --filter label=service=ftk-redis on 123.456.789.0
  INFO [86c4d405] Finished in 1.162 seconds with exit status 0 (successful).
Accessory redis Host: 123.456.789.0
CONTAINER ID   IMAGE       COMMAND                  CREATED         STATUS         PORTS                                       NAMES
cea88cafd063   redis:7.0   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   ftk-redis

  INFO [1ee80675] Running docker ps --filter label=service=ftk-memcached on 123.456.789.0
  INFO [1ee80675] Finished in 0.158 seconds with exit status 0 (successful).
Accessory memcached Host: 123.456.789.0
CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS         PORTS                                           NAMES
a3ee7cb50f57   memcached:1.6.24   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:11211->11211/tcp, :::11211->11211/tcp   ftk-memcached

  INFO [b8cce31e] Running docker ps --filter label=service=ftk-registry on 123.456.789.0
  INFO [b8cce31e] Finished in 0.157 seconds with exit status 0 (successful).
Accessory registry Host: 123.456.789.0
CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                                       NAMES
a5b0759fc5ed   registry:2.8.3   "/entrypoint.sh /etc…"   3 minutes ago   Up 3 minutes   0.0.0.0:5000->5000/tcp, :::5000->5000/tcp   ftk-registry

From here you can restart, upgrade, tail the logs, exec into the container, and more with Kamal for all of these accessories.

Here's a full list of what you can currently do:

$ kamal accessory
  Commands:
    kamal accessory boot [NAME]      
    kamal accessory details [NAME]   
    kamal accessory exec [NAME] [CMD]
    kamal accessory help [COMMAND]   
    kamal accessory logs [NAME]      
    kamal accessory reboot [NAME]    
    kamal accessory remove [NAME]    
    kamal accessory restart [NAME]   
    kamal accessory start [NAME]     
    kamal accessory stop [NAME]      

Our deploy.yml file is a whopping 31 lines and we’re running three popular services that we can now leverage. These services could have been started on different servers of course too or perhaps aide in the migration of some of these services from one cloud to another.

While Kamal is a great deployment tool it’s gotta be the fastest and easiest possible way to orchestrate running containers on your server(s).