/ CoreOS

Making Docker work with Consul on CoreOS

On the road to the unified and containerized environment production system, for me, one of the most significant milestones is an encounter with CoreOS. This operating system is ready for usage just after the start and supports a cluster mode out of the box. With a defined cloud config it feels like an engine that just needs to be turned on and immediately ready to serve.

On my way to the new architecture for multiple similar applications that have a lot of in common, I choose to use CoreOS as the main operating system because of a couple of reasons:

  1. our applications are docker containers
  2. I don't want to bother with the managing of docker machine
  3. I want to replace one node with a new one at any time

The new architecture should have been solving a couple of problems, and one of the was service discovery. To build internal DNS system I decided to use Consul from Hashicorp. Consul gives more than you can expect from just DNS server out of the box you have to check their page for details.

The idea of the internal DNS system is that it is seamlessly integrated into the OS and into the docker images, that is important in our CoreOS case. Consul works on port 8600 by default and DNS system on port 53.

One of the challenges is to make hosts discoverable and identifiable by the containers that it runs. In other words, make it possible to find a host which runs this or that container from another container on another host.

Table of Contents

  1. Introduction
  2. Technical assumptions
  3. Solution
  4. The dummy interface
  5. Run dnsmasq service
  6. Run Consul so it uses dummy interface
  7. Run application so it can communicate to the consul
  8. Cloud config
  9. Let's try it out
  10. To complete setup
  11. Conclusion

Technical assumptions

So let's assume that you have the next setup:

  • you have a couple instance with Consul servers on board that manage all discovery processes
  • you have working nodes that are based on the CoreOS
  • the applications that you run are containerized with Docker and runs in production mode on different working nodes
  • every node has a dynamic IP which you don't know beforehand

The problem is:

How to discover the IP address and the port of the node that runs a desired service within the docker container?

Solution

TLTR:

  • set the dummy network interface
  • bind Consul to this dummy interface
  • install dnsmasq service as a container
  • bind dnsmasq to listen to the dummy interface
  • set the containers to listen to the dummy interface as their DNS server

This is a principal schema of how discovery works

Ghost--docker--consul-1

Final CloudConfig you can find here.

In this article, I'll describe how to make it possible using CoreOS, that means that by its nature this Linux system runs every service as a docker or rkt container, so Consul, dnsmasq and our application are also containers and the host system is static.

To provision the CoreOS node I have used CloudInit and complete examples of the unit files will be listed at the bottom of the article and in the body I'll use only the essentials commands.

The dummy interface

Dummy interface serves as a mediator interface between docker container and DNS container. It also creates independent from docker/network interfaces layer that has a single serve purpose.

ip link add dummy0 type dummy
ip link set dev dummy0 up
ip link show type dummy
... output ...
ip addr add 169.254.1.1/32 dev dummy0
ip link set dev dummy0 up
ip addr show dev dummy0
... output ...

169.254.x.x is usually used as a local provisioning and introspection and points to the local cluster.

Configure dummy interface

File: /etc/systemd/network/dummy0.netdev

[NetDev]
Name=dummy0
Kind=dummy

File: /etc/systemd/network/dummy0.network

[Match]
Name=dummy0

[Network]
Address=169.254.1.1/32

After restart network interface

sudo systemctl restart systemd-network

Run dnsmasq service

Using dnsmasq we will reroute traffic that belongs to our domain to Consul. In CoreOS the dnsmasq is also a docker container, so we have to start it on the host network and give network admin capabilities.

Let's configure it by creating config file "/etc/dnsmasq.d/consul.conf":

server=/corp/169.254.1.1#8600
listen-address=127.0.0.1
listen-address=169.254.1.1

And what it actually does is:

  1. it listens to the loopback interface and to our dummy interface
  2. it directs traffic to the dummy interface if it tries to reach domain "corp", for instance, application.service.corp

And then we can run service:

docker run --name dnsmasq-service \
            -v /etc/dnsmasq.d/consul.conf:/etc/dnsmasq.d/consul.conf \
            --net=host \
            --cap-add=NET_ADMIN \
            quay.io/coreos/dnsmasq -d -q

Run Consul so it uses dummy interface

As a next step, let's utilize dummy interface and start Consul properly

docker run --name consul-agent -v /etc/consul.d:/etc/consul.d --net=host \
          consul agent -data-dir=/data -config-dir=/etc/consul.d \
          -bind=192.168.55.231 \
          -domain=corp \
          -client=169.254.1.1 \
          -recursor="8.8.8.8" \
          -recursor="8.8.4.4" \
          -advertise=192.168.55.231 \
          -enable-script-checks=true -retry-join=192.168.55.100

Run application so it can communicate to the consul

In the final step, let's run our application in the way that it can resolve our internal "corp" domains:

docker run --name corp-application --dns 169.254.1.1 -p 3000:3000 foo:latest

As you can see it is not a lot of options to start the application, we just have to indicate the DNS option, which is part of the default docker options to run a container.

Cloud config

When we put everything together:

#cloud-config

hostname: "HOSTNAME"

coreos:
  units:
    - name: "setup-network-environment.service"
      command: start
      content: |
        [Unit]
        Description=Setup Network Environment
        Documentation=https://github.com/kelseyhightower/setup-network-environment
        Requires=systemd-networkd-wait-online.service
        After=systemd-networkd-wait-online.service

        [Service]
        ExecStartPre=-/usr/bin/mkdir -p /opt/bin
        ExecStartPre=/usr/bin/wget -N -P /opt/bin https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment

        ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment
        ExecStart=/opt/bin/setup-network-environment
        RemainAfterExit=yes
        Type=oneshot
    - name: "application.service"
      command: start
      runtime: true
      content: |
        [Unit]
        Description=My Corp Application
        Requires=docker.service

        [Service]
        Restart=always
        Environment=IMAGE_NAME=nginx:latest
        Environment=CONTAINER_APPLICATION_NAME=application
        ExecStartPre=/usr/bin/docker pull $IMAGE_NAME
        ExecStartPre=-/usr/bin/docker kill $CONTAINER_APPLICATION_NAME
        ExecStartPre=-/usr/bin/docker rm $CONTAINER_APPLICATION_NAME
        ExecStart=/usr/bin/docker run --name $CONTAINER_APPLICATION_NAME --dns 169.254.1.1 -p 80:80 $IMAGE_NAME
        ExecStop=/usr/bin/docker stop $CONTAINER_APPLICATION_NAME
    - name: "dnsmasq.service"
      content: |
        [Unit]
        Description=Dnsmasq service
        Requires=docker.service
        Requires=setup-network-environment.service
        After=docker.service
        After=setup-network-environment.service

        [Service]
        Restart=always
        EnvironmentFile=/etc/network-environment
        ExecStartPre=-/usr/bin/env ip link add dummy0 type dummy
        ExecStartPre=-/usr/bin/env ip link set dev dummy0 up
        ExecStartPre=-/usr/bin/env ip addr add 169.254.1.1/32 dev dummy0
        ExecStartPre=-/usr/bin/env ip link set dev dummy0 up
        ExecStartPre=/usr/bin/env systemctl restart systemd-networkd
        ExecStartPre=-/usr/bin/docker kill dnsmasq-service
        ExecStartPre=-/usr/bin/docker rm dnsmasq-service
        ExecStartPre=/usr/bin/docker pull quay.io/coreos/dnsmasq
        ExecStart=/usr/bin/docker run --name dnsmasq-service \
            -v /etc/dnsmasq.d/consul.conf:/etc/dnsmasq.d/consul.conf \
            --net=host \
            --cap-add=NET_ADMIN \
            quay.io/coreos/dnsmasq -d -q
        ExecStop=/usr/bin/docker stop dnsmasq-service
    - name: "consul.service"
      command: start
      content: |
        [Unit]
        Description=Consul agent
        Requires=docker.service
        Requires=dnsmasq.service
        Requires=setup-network-environment.service
        After=docker.service
        After=setup-network-environment.service
        After=dnsmasq.service

        [Service]
        Restart=always 
        EnvironmentFile=/etc/network-environment
        Environment=CONSULE_SERVER_HOST=<HERE_HAVE_TO_BE_CONSUL_SERVER_IP>
        Environment=COMPANY_DOMAIN=corp
        ExecStartPre=-/usr/bin/docker kill consul-agent
        ExecStartPre=-/usr/bin/docker rm consul-agent
        ExecStartPre=/usr/bin/docker pull consul:1.0.2
        ExecStart=/usr/bin/docker run --name consul-agent -v /etc/consul.d:/etc/consul.d --net=host \
          consul agent -data-dir=/tmp/consul -config-dir=/etc/consul.d \
          -bind=${DEFAULT_IPV4} \
          -domain=${COMPANY_DOMAIN}\
          -client=169.254.1.1 \
          -recursor="85.158.4.230" \
          -recursor="85.158.7.30" \
          -recursor="8.8.8.8" \
          -recursor="8.8.4.4" \
          -advertise=${DEFAULT_IPV4} \
          -enable-script-checks=true -retry-join=${CONSULE_SERVER_HOST}
        ExecStop=/usr/bin/docker stop consul-agent
write_files:
  - path: /etc/systemd/network/dummy0.netdev
    content: |
      [NetDev]
      Name=dummy0
      Kind=dummy
  - path: /etc/systemd/network/dummy0.network
    content: |
      [Match]
      Name=dummy0

      [Network]
      Address=169.254.1.1/32
  - path: /etc/dnsmasq.d/consul.conf
    content: |
      server=/corp/169.254.1.1#8600
      listen-address=127.0.0.1
      listen-address=169.254.1.1

Let's try it out

You can easily try it on DigitalOcean

  • Create new droplet choosing CoreOS under "Container distributions"
  • Check "User Data" under "Additional Options" and paste cloud config there
  • Start, give it some time to bootstrap and open, and you see a welcome page from Nginx

To complete setup

To complete setup you need to install your Consul server, or maybe you have already one and replace <HERE_HAVE_TO_BE_CONSUL_SERVER_IP> with server IP.
Usually, Consul server is up and running in a cluster, so you can indicate either LoadBalancer IP or the IP of one of the server. I would not recommend you to use IP of one of the servers, it may be changed and your setup will be broken.

Cloud config from the example is ready to be used in production, just add whenever you are missing.

Conclusion

Using Consul with CoreOS you can easily achieve dynamic setup for any docker based application and use a small trick with dummy interface to put all service together.

Please feel free to write me if you found some inconsistencies or the ways how to improve these setup.