Securely Connecting Watchtower to Docker¶

Overview¶

Watchtower supports secure TLS connections to Docker hosts through its usage of Docker's Go SDK to create a Docker client.

It is highly recommended to review Docker's documentation:

Note

Docker has retired the Docker Machine project that was previously noted in Watchtower's documentation.

Configuration¶

TLS Verification¶

            Argument: --tlsverify
Environment Variable: DOCKER_TLS_VERIFY
                Type: Boolean
             Default: false

http:// and unix:// schemes are incompatible with TLS verification.

When TLS verification is enabled

The use of tcp:// or https:// schemes for the Docker host URL is required.
tcp:// is converted to https:// when TLS verification is enabled.

TLS Certificate Path¶

            Argument: --cert-path
Environment Variable: DOCKER_CERT_PATH
                Type: String
             Default: /etc/ssl/docker

Notes

This specifies the directory where Watchtower's Docker client should find the certificate files within the Watchtower container.
Docker expects the following filenames:
- ca.pem
- cert.pem
- key.pem

Docker Host URL¶

            Argument: --host
Environment Variable: DOCKER_HOST
                Type: String
             Default: unix:///var/run/docker.sock

Specifying multiple connections, such as both the local socket (i.e. /var/run/docker.sock) and a remote host is not supported.

Notes

This is required for connections to any Docker host.
The use of tcp:// or https:// schemes is required when using a TLS connection.
tcp:// is internally converted to https:// when TLS verification is enabled.

Docker API Version¶

            Argument: --api-version
Environment Variable: DOCKER_API_VERSION
                Type: String
             Default: <Auto-negotiated>

Notes

This provides the ability to manually specify the Docker API version.
The default version autonegotiation should be sufficient for normal use cases.

Examples¶

Note

Replace remote-host with your actual Docker host address and /path/to/certs with the path to your certificate directory.

Docker CLIDocker Compose

docker run -d \
  --name watchtower \
  -v /path/to/certs:/etc/ssl/docker:ro \
  ghcr.io/sidneyojr/watchtower --host tcp://remote-host:2376 --cert-path /etc/ssl/docker --tlsverify

Parameter	Description
`--name watchtower`	Assigns the name "watchtower" to the container for easy identification and management.
`-v /path/to/certs:/etc/ssl/docker:ro`	Mounts the local certificate directory to the container's SSL directory as read-only.
`ghcr.io/sidneyojr/watchtower`	Specifies the Docker image to run, which is the Watchtower container image.
`--host tcp://remote-host:2376`	Sets the Docker host to connect to via TCP on port 2376.
`--cert-path /etc/ssl/docker`	Defines the path inside the container where TLS certificates are located.
`--tlsverify`	Enables TLS verification for secure connections to the Docker host.

Tip

If using -e flags to pass environment variables, then remember to place them before the ghcr.io/sidneyojr/watchtower image reference.

services:
    watchtower:
        image: ghcr.io/sidneyojr/watchtower
        environment:
            - DOCKER_HOST=tcp://remote-host:2376
            - DOCKER_CERT_PATH=/etc/ssl/docker
            - DOCKER_TLS_VERIFY=1
        volumes:
            - /path/to/certs:/etc/ssl/docker:ro
        restart: unless-stopped

Parameter	Description
`image: ghcr.io/sidneyojr/watchtower`	Specifies the Docker image for the Watchtower service.
`- DOCKER_HOST=tcp://remote-host:2376`	Sets the Docker host connection URL.
`- DOCKER_CERT_PATH=/etc/ssl/docker`	Specifies the path to the directory containing TLS certificates.
`- DOCKER_TLS_VERIFY=1`	Enables TLS verification for secure connections.
`/path/to/certs:/etc/ssl/docker:ro`	Mounts the local certificate directory to the container's SSL directory as read-only.
`restart: unless-stopped`	Configures the container to restart automatically unless it was manually stopped.

Basic Tutorial¶

This is not a comprehensive guide and is merely a simple tutorial to illustrate a basic test deployment.

The following tutorial is intended to provide a basic walkthrough for manually setting up Watchtower to perform container updates on a Docker host that has enabled access to the Docker daemon using TLS.

This largely follow's Docker's guide for setting up TLS on the Docker host.

Warning

Configuring Docker to accept network connections has critical security implications that can leave you vulnerable to unauthorized access. You are highly encouraged to perform your own due diligence to mitigate these risks.

Tutorial Overview¶

In order for Watchtower to connect via TLS to a Docker daemon, the Docker daemon must be setup to accept remote connections using TLS.

Setting up TLS for Docker involves several key steps:

Certificate Generation:
- Create a Certificate Authority (CA), server certificate, and client certificates using OpenSSL or similar tools.
Server certificates must include the Docker host's IP or DNS name in the Subject Alternative Name (SAN) field.
Daemon Configuration:

Start the Docker daemon with TLS options:
- --tlsverify: Enable TLS verification
- --tlscacert: Path to CA certificate
- --tlscert: Path to server certificate
- --tlskey: Path to server private key
Client Setup:
- Prepare client certificates (cert.pem and key.pem) for authentication.
Environment Variables:
- Set DOCKER_HOST to the secure endpoint (e.g., tcp://host:2376)
- Set DOCKER_CERT_PATH to the directory containing client certificates
- Set DOCKER_TLS_VERIFY=1 to enable verification

For detailed instructions, refer to the Docker documentation on protecting the Docker daemon socket.

Certificate Generation¶

Generate self-signed certificates for testing (replace with proper certificates for production):

Create a CA Key and Certificate¶

Generate a 4096-bit RSA private key for the Certificate Authority and save it to ca-key.pem:
```
openssl genrsa -aes256 -out ca-key.pem 4096
```
Make sure to take note of the passphrase
Create a self-signed X.509 certificate for the Certificate Authority (valid for 365 days) using the private key and save it to ca.pem:
```
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=US/ST=State/L=City/O=Org/CN=ca"
```

Create a Server Key and Certificate¶

Generate a 4096-bit RSA private key for the server certificate and save it to server-key.pem:
```
openssl genrsa -out server-key.pem 4096
```
Generate a certificate signing request (CSR) for the server with common name "localhost" and save it to server.csr:
```
openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr
```
Sign the server CSR with the CA certificate, creating a server certificate valid for 365 days with the specified extensions, and save it to server-cert.pem:
```
echo "subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1" > extfile-server.cnf
```
IP:10.10.10.20 is an example IP. Replace with your host's actual IP.

$HOST typically resolves to the hostname. Change this as necessary (i.e. localhost for testing)
```
echo "extendedKeyUsage = serverAuth" >> extfile-server.cnf
```
This sets the Docker daemon key's extended usage attributes to be used only for server authentication.
```
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile-server.cnf
```

Create a Client Key and Certificate¶

Generate a 4096-bit RSA private key for the client certificate and save it to key.pem:
```
openssl genrsa -out key.pem 4096
```
Generate a certificate signing request (CSR) for the client with common name "client" and save it to client.csr:
```
openssl req -subj '/CN=client' -new -key key.pem -out client.csr
```

Sign the client CSR with the CA certificate, creating a client certificate valid for 365 days with the client authentication extension, and save it to cert.pem:

echo "extendedKeyUsage = clientAuth" > extfile-client.cnf

This makes the key suitable for client authentication.

openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile-client.cnf

Key and Certificate Management¶

Remove both the certificate signing requests (client.csr and server.csr) and extensions config files (extfile-server.cnf and extfile-client.cnf) after generating the server (server-cert.pem) and client cert.pem certificates:
```
rm -v client.csr server.csr extfile-server.cnf extfile-client.cnf
```
Update the file permissions of the ca-key.pem, server-key.pem, and key.pem secret keys:
```
sudo chmod -v 0400 ca-key.pem server-key.pem key.pem
```
Remove ca.pem, server-cert.pem, and cert.pem file write access:
```
sudo chmod -v 0444 ca.pem server-cert.pem cert.pem
```
Create the /etc/docker/certs directory if it doesn't exist:
```
sudo mkdir -p /etc/docker/certs
```

Copy the server-specific files:

sudo cp ca.pem ca-key.pem server-cert.pem server-key.pem /etc/docker/certs/

(Optional): Copy the client-specific files:

sudo cp cert.pem key.pem /etc/docker/certs/

Ensure root ownership:

sudo chown root:root /etc/docker/certs/*

Set directory-level access:
```
sudo chmod 0755 /etc/docker/certs/
```
Verify the files:
```
ls -la /etc/docker/certs/
```
Output should show files with -r-------- for keys and -r--r--r-- for certs.
(Optional) Remove the original files (in the original directory, not /etc/docker/certs):
```
rm -v {ca,ca-key,server-cert,server-key,cert,key}.pem
```

Notes

The server files (ca.pem, server-key.pem, server-cert.pem) stay on the daemon host.
The client's files (ca.pem, key.pem, and cert.pem) can be moved the client's Docker directory (e.g. ~/.docker/).

Setup the Docker Daemon with TLS¶

This assumes Docker is already installed on the host system.

Edit or create /etc/docker/daemon.json to enable TLS and TCP listening:
```
{
  "tls": true,
  "tlscacert": "/etc/docker/certs/ca.pem",
  "tlscert": "/etc/docker/certs/server-cert.pem",
  "tlskey": "/etc/docker/certs/server-key.pem",
  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"]
}
```
Notes
- "tls": true enables TLS verification.
- "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"] adds TCP listening on port 2376 and retains the Unix socket for local access.
- Alternatively, use dockerd flags in /etc/systemd/system/docker.service.d/override.conf for overrides without editing daemon.json.
Restart the daemon:
```
sudo systemctl restart docker
```
Verify the daemon is listening:
```
sudo netstat -tlnp | grep 2376
```

Watchtower Configuration¶

Docker ComposeDocker CLI

services:
    watchtower:
        image: ghcr.io/sidneyojr/watchtower
        environment:
            - DOCKER_HOST=tcp://remote-host:2376
            - DOCKER_CERT_PATH=/etc/ssl/docker
            - DOCKER_TLS_VERIFY=1
        volumes:
            - /path/to/certs:/etc/ssl/docker:ro
        restart: unless-stopped

docker run -d \
  --name watchtower \
  -v /path/to/certs:/etc/ssl/docker:ro \
  ghcr.io/sidneyojr/watchtower --host tcp://remote-host:2376 --cert-path /etc/ssl/docker --tlsverify

remote-host is used, but can be replaced with localhost for local testing.

Troubleshooting¶

Insecure Scheme with TLS Verification¶

When TLS verification is enabled and the Docker host URL uses http://, Watchtower logs the following warning:

TLS verification is enabled but DOCKER_HOST uses insecure scheme 'http://'. Consider using 'https://' or disable TLS verification.

Possible Solutions:

If using a secure connection, then use https://.
If using http://, then disable TLS verification.

Local Socket with TLS Verification¶

When TLS verification is enabled and the Docker host URL is not configured or uses unix://, Watchtower logs the following warning:

TLS verification is enabled but DOCKER_HOST uses local socket 'unix://'. TLS is not applicable for local sockets; consider disabling TLS verification.

Possible Solutions:

If the Docker host URL is not configured, then the default unix:///var/run/docker.sock is used.
If using a local socket (i.e. unix://), then disable TLS verification.

Missing TLS Certificates¶

In order for Watchtower's Docker client to connect to the Docker daemon via TLS, the certificates must be provided to the Watchtower container.

If the certificates are available via the host filesystem of the Watchtower container's Docker host, then this can be accomplished using bind mounts. Other options, such as building a custom Watchtower image with the certificates, are outside the scope of this documentation.

Refer to the documentation for the TLS Certificate Path and the examples.

Other Common Mistakes¶

Using tcp:// without --tlsverify: This disables TLS, potentially allowing insecure connections.
Mismatched certificate paths: Ensure DOCKER_CERT_PATH points to the correct directory containing cert.pem and key.pem.
Expired or invalid certificates: Check certificate validity and SAN fields matching the host.
Firewall blocking TLS port: Ensure port 2376 is open for remote connections.

Certificate Management¶

This documentation is not intended to be a guide on TLS/mTLS certificate deployment or management. Manual certificate management might be acceptable for smaller deployments; however, there are solutions for automating certificate management. If you are exposing your Docker daemon to external network connections, then both proper TLS setup and management is a highly recommended.

Here are just a few available solutions:

Step-CA: Smallstep's private, self-hostable certificate authority. [Tutorial]
Vault: HashiCorp's secret management tool with PKI secrets engine for certificate generation
CFSSL: Cloudflare's PKI toolkit for certificate management

Docker Socket Proxies¶

You are highly encouraged to perform your own due diligence before using any software that interacts directly with the Docker socket.

Docker socket proxies provide a security layer between applications and the Docker daemon by filtering API calls and preventing unrestricted access to the Docker socket.

While this documentation focuses on TLS-based connections, socket proxies represent another approach for securing Docker daemon access in environments where full socket exposure is undesirable.

The following projects are examples of Docker socket proxies: