Using Private Plugins in Traefik Proxy 2.5

Traefik Proxy is a modular router by design, allowing you to place middleware into your routes, and to modify requests before they reach their intended backend service destinations. Traefik has many such middlewares builtin, and also allows you to load your own, in the form of plugins.

The easiest way to find and install middleware plugins, is via Traefik Pilot. Traefik Pilot is a Software-as-a-Service (SaaS) platform that offers a global metrics and alerting system, for all of your Traefik Proxy instances, and has a free to use built-in plugin store. Inside the store, you can browse all of the open source plugins available, and install them with the click of a button.

With the release of Traefik Proxy v2.5, there is a new way to load plugins directly from local storage (and without needing to enable Traefik Pilot). Simply place your plugin source code into a new directory called /plugins-local. (You would create this directory relative to your current working directory [from where you invoke traefik], which if you are using the traefik docker image, the entrypoint is always the root directory /.) Traefik Proxy itself will take care of building (interpreting) your plugin, so all you have to do is write the source code, and provide it in the right directory for Traefik Proxy to load it. Plugins are loaded only once per startup (ie. you must restart traefik each time you wish to reload your plugin source code).

In the following scenarios, you will find examples for composing your own Docker container images with Traefik Proxy v2.5, and bundling your plugin source code into the /plugins-local directory of that image. After testing your plugin in a development environment with Docker, (and possibly after creating Continuous Integration builds for it), you can push this image to a container registry, and reference this image in your production Docker server and/or Kubernetes clusters. You can keep your image private, or you can publish it, and share your plugin everywhere.

Build a Traefik Proxy container image and bundle the demo plugin

Here is an example Dockerfile that remixes the standard traefik:v2.5 docker image, and adds a plugin automatically cloned from a configurable git repository.

Create a temporary directory someplace, and inside of it create a new file called Dockerfile.demo:

# Dockerfile.demo - Example for Traefik Proxy and a demo plugin from git:
FROM alpine:3
ARG PLUGIN_MODULE=github.com/traefik/plugindemo
ARG PLUGIN_GIT_REPO=https://github.com/traefik/plugindemo.git
ARG PLUGIN_GIT_BRANCH=master
RUN apk add --update git && \
    git clone ${PLUGIN_GIT_REPO} /plugins-local/src/${PLUGIN_MODULE} \
      --depth 1 --single-branch --branch ${PLUGIN_GIT_BRANCH}

FROM traefik:v2.5
COPY --from=0 /plugins-local /plugins-local

The default build arguments load the example plugin demo published by Traefik Labs, which is essentially a clone of the builtin headers.customRequestHeaders middleware, but as a plugin.

In the same directory as Dockerfile.demo, build the image:

docker build -f Dockerfile.demo --tag traefik-with-demo-plugin .

You have now just built a docker image, containing Traefik v2.5 and the demo plugin. You can now run the image to test it:

docker run --rm -it traefik-with-demo-plugin \
  --log.level=DEBUG \
  --experimental.localPlugins.demo.moduleName=github.com/traefik/plugindemo

The log will print the config showing that the plugin is loaded and Traefik Proxy will be running. You can test this positively, press Ctrl-C to stop the container, and rerun the command changing the moduleName= to github.com/something/different, and you'll get an error saying that it doesn't exist and will immediately exit.

Build a Traefik Proxy container image with your custom plugin

To create a new plugin of your own design, fork this demo repository. (To do this directly on GitHub, you can click the green button labeled Use this template, or you can clone the repository to another server). You can choose to make this new repository public or private, but the instructions are different depending on if it requires authentication to clone it, or not, so each case will be covered separately.

Clone your forked repository to your workstation, and read the development instructions in the readme.md file. Create your plugin code, update the import line in .traefik.yml to match your repository name, commit the changes to git, and push your changes back to your git server (GitHub). You don't need to commit any changes if you just want to test the example plugin code. Furthermore, Traefik does not require the plugin source code to be compiled: plugins are loaded via raw source code and are interpreted at runtime by Yaegi.

Build the image from a public repository

If you made your repository public, building the image is easy. Open your shell terminal, and create these temporary environment variables to use as build arguments:

## Create temporary variables for your plugin and git repository details:
## Optionally save these to build-env.sh and run "source build-env.sh" after.
export DOCKER_IMAGE=traefik-with-my-plugin
export PLUGIN_MODULE=github.com/YOUR_NAME/YOUR_REPOSITORY
export PLUGIN_GIT_REPO=https://github.com/YOUR_NAME/YOUR_REPOSITORY.git
export PLUGIN_GIT_BRANCH=master

Change these variables to fit your forked plugin repository:

DOCKER_IMAGE is the tag for your new Docker image, which will bundle both Traefik and your plugin code.
PLUGIN_MODULE is the name of your plugin's Go module (eg. github.com/traefik/plugindemo). Use your own server, organization, and forked repository name.
PLUGIN_GIT_REPO is the full git clone URL for your plugin repository hub. (This example assumes a public repository is used, and no authentication is needed, otherwise see the next section.)
PLUGIN_GIT_BRANCH is the git branch name you wish to clone and install.

In the root directory of the cloned repository, create a new file named Dockerfile.public:

## Dockerfile.public - Bundle a Traefik plugin from a public git repository
FROM alpine:3
ARG PLUGIN_MODULE=github.com/traefik/plugindemo
ARG PLUGIN_GIT_REPO=https://github.com/traefik/plugindemo.git
ARG PLUGIN_GIT_BRANCH=master
RUN apk update && \
    apk add git && \
    git clone ${PLUGIN_GIT_REPO} /plugins-local/src/${PLUGIN_MODULE} \
      --depth 1 --single-branch --branch ${PLUGIN_GIT_BRANCH}

FROM traefik:v2.5
COPY --from=0 /plugins-local /plugins-local

Build and tag the image, passing the arguments from the environment:

docker build -f Dockerfile.public \
  --tag ${DOCKER_IMAGE} \
  --build-arg PLUGIN_MODULE \
  --build-arg PLUGIN_GIT_REPO \
  --build-arg PLUGIN_GIT_BRANCH .

Build the image from a private git repository

Building the image from a private git repository is a bit more challenging, because you need to pass your SSH credentials into the Docker build process, in order to clone from your private git repository as scripted in the Dockerfile.

You will need to update your Docker installation to version >=18.09, this allows loading the experimental BuildKit enhancements necessary to talk to your ssh-agent and to temporarily use your workstation user account's SSH keys, during the docker image build process.

Set these environment variables in your shell:

## Optionally save these to build-env.sh and run "source build-env.sh" after.
## Docker BuildKit is required for ssh-agent forwarding:
export DOCKER_BUILDKIT=1
## Edit these variables for your plugin and git repository:
export DOCKER_IMAGE=traefik-with-my-plugin
export PLUGIN_MODULE=github.com/YOUR_NAME/YOUR_REPOSITORY
export [email protected]:YOUR_NAME/YOUR_REPOSITORY.git
export PLUGIN_GIT_BRANCH=master

The Dockerfile needs to be modified for the host ssh-agent pass-through. Create a new file with the name Dockerfile.private:

# syntax=docker/dockerfile:1.0.0-experimental
# The above line is required to turn on experimental BuildKit features.
# Dockerfile.private - Build Traefik and plugin from a private git repository.
# Loads SSH keys from the host `ssh-agent` to allow git clone.
FROM alpine:3

# Clone your plugin git repositories:
ARG PLUGIN_MODULE=github.com/traefik/plugindemo
ARG [email protected]:traefik/plugindemo.git
ARG PLUGIN_GIT_BRANCH=master
RUN apk add --update git openssh && \
    mkdir -m 700 /root/.ssh && \
    touch -m 600 /root/.ssh/known_hosts && \
    ssh-keyscan github.com > /root/.ssh/known_hosts
RUN --mount=type=ssh git clone \
    --depth 1 --single-branch --branch ${PLUGIN_GIT_BRANCH} \
    ${PLUGIN_GIT_REPO} /plugins-local/src/${PLUGIN_MODULE} 
    
FROM traefik:v2.5
COPY --from=0 /plugins-local /plugins-local

Build the image, with the extra --ssh default option. This will hook into the build process with a connection to your host running ssh-agent, so that you can use your SSH keys during the build process, and clone the private git repository:

docker build -f Dockerfile.private \
  --ssh default --tag ${DOCKER_IMAGE} \
  --build-arg PLUGIN_MODULE \
  --build-arg PLUGIN_GIT_REPO \
  --build-arg PLUGIN_GIT_BRANCH .

Note: due to an open issue in docker-compose, you cannot currently utilize the --ssh parameter in docker-compose (and the connection to ssh-agent would fail), so if you want to use this modified Dockerfile along with docker-compose, you must manually build your container image first with the docker build command listed above. If you build the image first this way, docker-compose can then rely upon the build cache, or an explicit image name, without needing to build it again.

Using docker-compose as a plugin development environment

You can use docker-compose as an easy plugin development environment.

Clone your plugin repository to your workstation, and then create these new files into the root of the repository:

Create Dockerfile:

FROM traefik:v2.5
## Default module name (put your setting in .env to override)
ARG PLUGIN_MODULE=github.com/traefik/plugindemo
ADD . /plugins-local/src/${PLUGIN_MODULE}

Create .env settings file:

## Traefik Proxy local plugin .env file
## Configure your plugin name:
PLUGIN_NAME=demo
## Configure your module namespace:
PLUGIN_MODULE=github.com/traefik/plugindemo
## Configure whoami domain name for route testing:
WHOAMI_TRAEFIK_HOST=whoami.example.com
## Configure Email address for Let's Encrypt:
## Uncomment and configure this for production only:
# [email protected]

Create docker-compose.yaml:

# docker-compose.yaml for Traefik Proxy local plugin development
version: "3.3" networks: traefik-proxy: volumes: traefik-proxy:
  
services:
  traefik-proxy:
    build:
      context: .
      args:
        PLUGIN_MODULE: ${PLUGIN_MODULE}
    restart: unless-stopped
    networks:
    - traefik-proxy
    security_opt:
    - no-new-privileges:true
    command:
    #- "--log.level=DEBUG"
    - "--providers.docker=true"
    - "--providers.docker.exposedbydefault=false"
    - "--providers.docker.network=traefik-proxy"
    ## Entrypoints:
    - "--entrypoints.web.address=:80"
    - "--entrypoints.websecure.address=:443"
    - "--entrypoints.traefik.address=:9000"
    ## Automatically redirect HTTP to HTTPS
    - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
    ## ACME TLS config:
    - "--certificatesresolvers.default.acme.storage=/data/acme.json"
    ## Uncomment for production TLS certificates (Let's Encrypt):
    # - "--certificatesresolvers.default.acme.tlschallenge=true"
    # - "--certificatesresolvers.default.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
    # - "--certificatesresolvers.default.acme.email=${ACME_CA_EMAIL}"
    ## Enable Dashboard available only from the docker localhost:9000
    - "--api.dashboard=true"
    - "--api.insecure=true"
    ## Enable local plugins:
    - "--experimental.localPlugins.${PLUGIN_NAME}.moduleName=${PLUGIN_MODULE}"
    ports:
    - "80:80"
    - "443:443"
    - "127.0.0.1:9000:9000"
    volumes:
    - "traefik-proxy:/data"
    - "/var/run/docker.sock:/var/run/docker.sock:ro"

  ## The whoami container will run the demo plugin for testing purposes:
  whoami:
    image: traefik/whoami
    networks:
      - traefik-proxy
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`${WHOAMI_TRAEFIK_HOST}`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      # Configure the plugin as a new middleware:
      - "traefik.http.routers.whoami.middlewares=whoami-demo"
      # Add a test header to all incoming requests:
      # (the presense of this header in whoami response shows if the plugin works:)
      - "traefik.http.middlewares.whoami-demo.plugin.${PLUGIN_NAME}.headers.DoesPluginWork=YES"
      - "traefik.http.routers.whoami.tls.certresolver=default"

Create .dockerignore to exclude the .git directory from the image build:

# .dockerignore file exludes files from the image:
.git

Build the image and bring up the test instance:

docker-compose up

Edit your /etc/hosts file (or your local DNS server) and add the whoami route domain:

# ... excerpt from /etc/hosts
# Domain names for Traefik:
# Point to the IP address of your docker server:
127.0.0.1 whoami.example.com app1.example.com app2.example.com

Use curl to test that your DNS works, and that the plugin has taken effect (use the same domain name as you configured for WHOAMI_TRAEFIK_HOST and in /etc/hosts):

curl -k https://whoami.example.com

You should get the whoami response back, with this test header displayed amongst the output:

Doespluginwork: YES

This is the same header and value that the plugin was configured to inject into the request, and echoed back from whoami. If you see it, you know that your plugin is successfully configured.

Configure local DNS service for regular development work

When you need to test lots of different subdomains and Traefik Proxy Host router rules, a better solution for DNS, rather than continuously editing your /etc/hosts file, is to run dnsmasq on your workstation as a local DNS server, and it will respond to wildcard DNS A record queries, for an entire root domain, or subdomain names.

Configuration of dnsmasq is optional, and is supplemental to your /etc/hosts file. Installation instructions for dnsmasq are dependent on your operating system, but is available from most package managers. dnsmasq will make your development work much smoother, and is a good way to clean up your /etc/hosts file. Here is a sample /etc/dnsmasq.conf configuration file to setup a local DNS service with a wildcard domain. You will also need to edit your /etc/resolv.conf as noted in the comments:

# /etc/dnsmasq.conf
# Use this if you are tired of editing your /etc/hosts file.
# This is a local DNS service bound only to the looback device on localhost.
# To use this requires an /etc/resolv.conf file 
# with a single line (no leading space): nameserver 127.0.0.1
# To prevent any changes to the host DNS config,
# run: sudo chattr +i /etc/resolv.conf
#      (now all of your DNS queries will go through dnsmasq)
interface=lo
listen-address=::1,127.0.0.1
bind-interfaces
cache-size=1000
# Use cloudflare upstream DNS servers:
server=1.1.1.1
server=1.0.0.1
# Example wildcard domain names
# All *.example.com names point to a single docker server IP address:
address=/example.com/127.0.0.1
# dnsmasq also loads your /etc/hosts file, so those host names still work.

Check your operating system instructions for enabling the dnsmasq service, but usually it is with Systemd:

sudo systemctl enable --now dnsmasq.service

Edit /etc/resolv.conf to use the dnsmasq server for all system DNS queries:

domain your.domain.example.com
search domain.example.com
nameserver 127.0.0.1

Sometimes other services (systemd-resolvd) will like to overwrite this file, you can prevent that by applying the immutable flag on the file:

# This prevents editing the file, use -i to re-enable editing:
chattr +i /etc/resolv.conf

You can test that the DNS server is active with the dig, drill, or nslookup utilities:

# dig or drill:
dig test.example.com | grep -A1 "ANSWER SECTION"
# or nslookup:
nslookup test.example.com

The output from any of these tools should report the correct IP address of your docker host, and now you can use any subdomain you want in your Traefik Proxy routes.

To learn more and see it in action, watch the recording of our recent online meetup, "What's New in Traefik Proxy 2.5".