This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module.

Part 1 – Setting up SkyDNS for service discovery of dockerized microservices
Part 2 – Automating SkyDNS service discovery with simple Node.js service
Part 3 – Setting up nginx as a reverse-proxy for dockerized microservices

 

What is DNS and how it can help with service discovery

In first approximation, DNS is a service that maps alpha-numeric, human readable and memorisable host names to corresponding IP addresses, required for establishing TCP/IP connection. Process of finding IP address for a given host name is called DNS resolution. It takes place every time when we type in a url in a browser or send an email. DNS resolvers are implemented and configured on OS level. They utilise caching to speed up the process and minimise load for DNS servers. All network client applications, whether it’s curl, sendmail or Google Chrome use these resolvers when establishing connection to destination services. Here are the most important benefits of using DNS (and host / domain names)

  • domain name is meaningful and easy to remember for people
  • domain name is permanent, while IP can change (for example when you move from one hosting provider to another)
  • one domain name can be backed by multiple IP addresses / hosts – for load balancing
  • one IP address (and server) can be used to host multiple domains – to save on hosting costs and simplify management
  • domain name is required for setting up secure SSL / HTTPS communication

Most of these possibilities can improve and enrich our hypothetical microservices environment. The most important, however, is that it gives any service a persistent and meaningful name by which it can be accessed. It enables easy service-to-service and consumer-to-service communication, load balancing and zero-downtime upgrades.

Dynamic nature of microservices (they get deployed, upgraded, and undeployed quite often) poses extra requirements for our DNS service – it has to be dynamic as well. SkyDNS is an API-driven DNS server backed by etcd – distributed key-value configuration store used by many modern services. It is a perfect solution for a problem we are trying to solve.

Setting up etcd and SkyDNS

I assume you have some flavour of Unix for OS and it has fairly recent version of Docker installed (I am using CoreOS 1068.8.0 with docker 1.10.3). We will install all services in docker containers just to simplify and unify instructions. Please note, that you can use etcd installed on your host machine – for production it would be more appropriate. Following instructions are not intended for any sort of production use – only for demo / dev purposes. If you want me to prepare instructions more suitable for production, please leave a comment below.

First we need to find IP address of your host.  Don’t use loopback IP or virtual adapter IP. Adapter name you’re after normally starts with e*

As you can see, our IP address is  147.10.7.61 I will use it in the following instructions, you, please replace it with IP address for your host machine.

Now, we install etcd in docker container. I am going to setup it to use non-standard port – 4002 (as I already have etcd running on my host) and non-persistent disk storage inside container (so don’t expect it to keep the data if you recreate container). We are starting docker container in bridged network mode (default) and mapping port 4002 to the same port on our host. Also, we are changing default etcd configuration by supplying command line parameters

After image is pulled from docker registry and container is started, we can verify etcd health status by execuring etcd client – etcdctl cluster-health command inside running container

Before proceeding with SkyDNS installation, we need to pre-configure it via etcd. Using the same etcdctl tool we will add /skydns/config entry to override default SkyDNS settings:

  • dns_addr = “0.0.0.0:53” – listen on all available network interfaces, port 53 (standard port for DNS servers)
  • ttl = 60 – default TTL (time to live) for DNS entries in seconds. It defines maximum time interval before DNS entry has to be revalidated by client (or for how long it can be cached). We are making it 60 seconds, which is a very short interval, because microservices are extremely dynamic by nature and we want all updates to DNS records to be propagated as soon as possible. This will, however, result in more queries coming to DNS server, having some performance impact.
  • domain = “cluster.local.” – “home” domain for our DNS – we will use it later to configure DNS resolution.
  • nameservers = [“8.8.8.8:53″,”8.8.4.4:53”] – “upstream” DNS for SkyDNS. When SkyDNS cannot resolve entry itself, it will forward request to one of those servers. This enables delegation of responsibilities – SkyDNS only resolves local resources (our microservices) and the rest is resolved by external DNS servers. 8.8.8.8:53 and 8.8.4.4:53 are IP addresses of Google publish DNS servers that should be accessible everywhere.

Now we are ready to start SkyDNS

 

Setting up DNS resolution

In order to make our DNS server used by DNS resolvers, we need to update their configuration. There are 3 options available depending on the desired scope

Host-wide

In this case, our SkyDNS will be used by all applications running both on host and in docker containers.

Update file /etc/resolv.conf on host machine

and add following lines at the top of the file

For all docker containers

SkyDNS will be used only inside docker containers (container must run in bridged network mode)

Update file /etc/sysconfig/docker on host machine

Add following to OPTIONS variable

Then restart docker

For individual docker containers

In case, if we only need individual containers to use SkyDNS, we can add these options to docker run  command while starting a container

We will use the latter option, as it doesn’t require any permanent configuration changes.

 

Testing service discovery

As our target service, we will use simple dockerized microservice developed in one of the pervious tutorials.

Let’s find IP address of service1 running in docker container

Now we can test if service can be resolved by DNS. We will use curl inside docker container

As we see, name service1 could not be resolved, which is expected. To make host name “service1” resolvable by DNS we need to add an entry for it:

When we run “curl” test again

we get our service name successfully resolved.

Another usage scenario for DNS is load balancing combined with service discovery. To achieve that, we will start 2 more instances of our service and configure DNS entry for “service” domain name

Now let’s verify that it works as intended

As we can see, when we are target host “service” we hit all 3 running services one by one – this behaviour called load balancing.

Obviously enough, there is a gap in our software stack – we have to create DNS entires manually. First getting IP address of a container, then use etcdctl tool for updating SkyDNS configuration in etcd. In the next article of this series we will develop a simple Node.js service to automate this task.

If you have any questions or want further details regarding any of the aspects of this tutorial – please leave a comment below.

Share This

Share This

Share this post with your friends!