How Traefik Plugins Protect Your Apps Against the Log4j Vulnerability

On December 10th, 2021, a vulnerability in Apache Log4j2 was published (CVE-2021-44228). Dubbed Log4Shell, it’s an issue in a logging library for Java applications that is widely used across famous open source projects and enterprise-grade backend applications. Log4Shell introduces a critical security risk. In this article, we’ll show you how Traefik helps mitigate the issue using our plugin system.

The vulnerability

The Log4j vulnerability makes it possible for attackers to execute arbitrary code or retrieve confidential information from the system being attacked. This vulnerability can be mitigated by two different steps: either, you patch the actual vulnerability in the systems and redeploy them, or you block malicious requests coming in at the reverse proxy level. This is possible because of an issue when interpolating strings that could query LDAP Servers.

Blocking the Log4j vulnerability at the reverse proxy level

As soon as we learned of the vulnerability, we looked for the best way to help our users and customers mitigate the risk. As you can see from the diagram below, the most logical place to block attacks is right at the entrance to your network, before malicious requests reach a vulnerable system. That's exactly where Traefik Proxy sits.

While the internal architecture of the HeadersRegexp matcher allows you to block simple requests, such as curl 127.0.0.1:8080 -H 'User-Agent: jndi:ldap://127.0.0.1/a}, there is a way to mask the user agent easily and thus prevent simple matching against those headers. To help mitigate those issues as well, we took leverage of the Traefik plugin system.

The plugins

Since Traefik 2.3, Traefik can load and use plugins (with the help of our Go interpreter, Yaegi) which act as either middleware or provider. Handling and matching request headers, or user agent, falls into the responsibility of a middleware. To mitigate the Log4j issue we needed to act fast, without having to do an actual release of the software that would consume a good amount of time. So, we decided to implement the required functionality in a plugin.

The code of the plugin can be found here: https://github.com/traefik/plugin-log4shell

By leveraging the plugin system, every Traefik Proxy user or Traefik Enterprise customer can mitigate that CVE effectively with only a couple of lines of configuration and a restart of Traefik. This approach is beneficial to both sides. We save time as we don’t need to go through an actual release process while users and customers have quicker access to software that helps them mitigate that issue.

To use a plugin, as usual, you first have to declare it in the static configuration and restart Traefik afterward.

That can either be done through the command line:

--pilot.token=xxx
--experimental.plugins.log4shell.modulename=github.com/traefik/plugin-log4shell
--experimental.plugins.log4shell.version=v0.1.1

or in the static configuration as usual:

(yaml)

pilot:
  token: xxx

experimental:
  plugins:
    log4shell:
      modulename: github.com/traefik/plugin-log4shell
      version: v0.1.0

(toml)

[pilot]
    token = "xxx"

[experimental.plugins.log4shell]
    modulename = "github.com/traefik/plugin-log4shell"
    version = "v0.1.0"

Once the plugin is installed and Traefik restarted, it acts as a normal middleware and thus, needs to be referenced on the routers.

http:
  middlewares:
    log4shell-foo:
      plugin:
        log4shell:
          errorCode: 200

  routers:
    my-router:
      rule: Host(`localhost`)
      middlewares:
        - log4shell-foo
      service: my-service

  services:
    my-service:
      loadBalancer:
        servers:
          - url: 'http://127.0.0.1'

It can also be referenced on K8s Ingress or K8s Ingress Route.

K8s Ingress:

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: log4shell-foo
spec:
  plugin:
    log4shell:
      errorCode: 200

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: default-log4shell-foo@kubernetescrd

spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name:  whoami
                port:
                  number: 80

K8s Ingress Route:

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: log4shell-foo
spec:
  plugin:
    log4shell:
      errorCode: 200

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`whoami.example.com`)
      middlewares:
        - name: log4shell-foo
      services:
        - kind: Service
          name: whoami-svc
          port: 80

With this plugin middleware in place and ready, even more tricky queries, such as curl 127.0.0.1:8080 -H 'User-Agent: ${${lower:j}ndi:ldap://127.0.0.1/a} are blocked effectively at the Ingress Controller level and thus, the backend servers are protected from incoming malicious requests.

Conclusion

On top of the CVE-2021-44228 vuln, there was second a CVE published (CVE-2021-45046) for a less critical vulnerability in Log4shell which will force administrators, operators, and developers to yet again update all their apps and do a fresh set of deployments. With Traefik protecting the backend servers, there is less stress to do so, since Traefik blocks external attacks. This does not make updating the actual apps obsolete, but it removes the immediate stress to do so.

This mitigation is not the only way Traefik can help you secure your backend apps. Middleware, such as Rate-Limiting or the enhanced functionality of Traefik Enterprise, helps protect you against a variety of attacks.

Additional resources

• Security Enterprise Webinar
• Log4Shell Mitigation