How to Scale End-to-End Observability in AWS Environments

How to use Docker Swarm secrets to store and rotate your SSL certificates with Nginx

Docker Swarm has an excellent feature out of the box — Docker Swarm secrets. Using it you can easily keep your sensitive data like credentials, TLS certificates, etc.

In terms of Docker Swarm services, a secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or in your application’s source code. You can use Docker secrets to centrally manage this data and securely transmit it to only those containers that need access to it.

So, if we want to use it to store our certificates, at first we need a certificate. Here we have two options:

Use self-signed certificate
Buy SSL certificate

We will use self-signed:

~$ mkdir certs && sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./certs/nginx.key -out ./certs/nginx.crt

Command above generates certificate which expiration date is 1 year and place it in ./certs/ directory.

Now we have key and crt files and we already can use it. But besides it, we should always monitor the certificate expiration date. Sure there are a few ways to do it, but it is out of scope topic. Just keep in mind that you can use alerts (Prometheus + Blackbox exporter) of certificate expiration date to trigger your script which in its turn updates the secret with the certificate.

Now we need to create an nginx docker service with our certificate. Here is my docker-compose file with secrets section:

version: '3.4'

services:
  ingress_nginx:
    image: ingress_nginx
    build:
      context: .
    ports:
      - "80:80"
      - "443:443"
    networks:
      - network
    deploy:
      mode: global
      restart_policy:
        condition: any
        delay: 5s
      update_config:
        delay: 30s
        parallelism: 1
        failure_action: rollback
    secrets:
      - source: nginx_key
        target: /etc/nginx/nginx.key
      - source: nginx_cert
        target: /etc/nginx/nginx.crt

secrets:
  nginx_key:
    file: ./certs/nginx.key
  nginx_cert:
    file: ./certs/nginx.crt

networks:
  network:
    driver: overlay

You should keep in mind that you cannot update docker secrets on the fly. It means that you should create a dummy secret every time and replace the old secret with a dummy secret. This is an example script how to update an existing secret:

# Create dummy secret
echo "<<< Creating dummy secret >>>" 
docker secret create dummy_key nginx.key
docker secret create dummy_crt nginx.crt

# Delete old certificate and key from docker secret and replace them with dummy
echo "<<< Delete old certificate and key from service and replace them with dummy >>>" 
docker service update \
    --secret-rm ${stack}_nginx_key \
    --secret-rm ${stack}_nginx_cert \
    --secret-add source=dummy_key,target=/etc/nginx/nginx.key \
    --secret-add source=dummy_crt,target=/etc/nginx/nginx.crt \
    ${stack}_ingress_nginx
    
echo "<<< Delete old certificate from secrets >>>"
docker secret rm  ${stack}_nginx_key
docker secret rm  ${stack}_nginx_cert

# Deploy service with new secrets
echo "<<< Create secret with new certificate and update service >>>"
docker stack deploy --compose-file docker-compose.yml $stack

# Delete dummy secrets
echo "<<< Delete dummy certificate >>>"
docker secret rm dummy_key
docker secret rm dummy_crt

After script execution, we have updated the certificate inside nginx container.