Streaming Docker Logs to the Elastic Stack (ELK Stack) using Fluentd

Kibana

Introduction

Fluentd and Docker’s native logging driver for Fluentd makes it easy to stream Docker logs from multiple running containers to the Elastic Stack. In this post, we will use Fluentd to stream Docker logs from multiple instances of a Dockerized Spring Boot RESTful service and MongoDB, to the Elastic Stack (ELK).

log_message_flow_notype

In a recent post, Distributed Service Configuration with Consul, Spring Cloud, and Docker, we built a Consul cluster using Docker swarm mode, to host distributed configurations for a Spring Boot service. We will use the resulting swarm cluster from the previous post as a foundation for this post.

Fluentd

According to the Fluentd website, Fluentd is described as an open source data collector, which unifies data collection and consumption for a better use and understanding of data. Fluentd combines all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. Fluentd structures data as JSON as much as possible.

Logging Drivers

Docker includes multiple logging mechanisms to get logs from running containers and services. These mechanisms are called logging drivers. Fluentd is one of the ten current Docker logging drivers. According to Docker, The fluentd logging driver sends container logs to the Fluentd collector as structured log data. Then, users can utilize any of the various output plugins, from Fluentd, to write these logs to various destinations.

Elastic Stack

The ELK Stack, now known as the Elastic Stack, is the combination of Elastic’s very popular products: Elasticsearch, Logstash, and Kibana. According to Elastic, the Elastic Stack provides real-time insights from almost any type of structured and unstructured data source.

Setup

All code for this post has been tested on both MacOS and Linux. For this post, I am provisioning and deploying to a Linux workstation, running the most recent release of Fedora and Oracle VirtualBox. If you want to use AWS or another infrastructure provider instead of VirtualBox to build your swarm, it is fairly easy to switch the Docker Machine driver and change a few configuration items in the vms_create.sh script (see Provisioning, below).

Required Software

If you want to follow along with this post, you will need the latest versions of git, Docker, Docker Machine, Docker Compose, and VirtualBox installed.

cat /etc/*release | sed -n '1p'
# Fedora release 25 (Twenty Five)
git –version \
docker –version \
docker-compose –version \
docker-machine –version \
virtualbox –help | sed -n '1p'
# git version 2.9.3
# Docker version 17.04.0-ce, build 4845c56
# docker-compose version 1.11.2, build dfed245
# docker-machine version 0.10.0, build 76ed2a6
# Oracle VM VirtualBox Manager 5.1.18

view raw
check_versions.txt
hosted with ❤ by GitHub

Source Code

All source code for this post is located in two GitHub repositories. The first repository contains scripts to provision the VMs, create an overlay network and persistent host-mounted volumes, build the Docker swarm, and deploy Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack. The second repository contains scripts to deploy two instances of the Widget Spring Boot RESTful service and a single instance of MongoDB. You can execute all scripts manually, from the command-line, or from a CI/CD pipeline, using tools such as Jenkins.

Provisioning the Swarm

To start, clone the first repository, and execute the single run_all.sh script, or execute the seven individual scripts necessary to provision the VMs, create the overlay network and host volumes, build the swarm, and deploy Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack. Follow the steps below to complete this part.

git clone –depth 1 –branch fluentd \
https://github.com/garystafford/microservice-docker-demo-consul.git
cd microservice-docker-demo-consul/scripts/
sh ./run_all.sh # single uber-script
# alternately, run the individual scripts
sh ./vms_create.sh # creates vms using docker machine
sh ./swarm_create.sh # creates the swarm
sh ./ntwk_vols_create.sh # creates overlay network and volumes
sh ./consul_deploy.sh # deploys consul to all nodes
sh ./registrator_deploy.sh # deploys registrator
sh ./stack_deploy.sh # deploys fluentd, visualizer, elastic stack
sh ./stack_validate.sh # waits/tests for all containers to start

view raw
install_swarm.sh
hosted with ❤ by GitHub

When the scripts have completed, the resulting swarm should be configured similarly to the diagram below. Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack containers should be distributed across the three swarm manager nodes and the three swarm worker nodes (VirtualBox VMs).

swarm_fluentd_diagram

Deploying the Application

Next, clone the second repository, and execute the single run_all.sh script, or execute the four scripts necessary to deploy the Widget Spring Boot RESTful service and a single instance of MongoDB. Follow the steps below to complete this part.

git clone –depth 1 –branch fluentd \
https://github.com/garystafford/microservice-docker-demo-widget.git
cd ../../microservice-docker-demo-widget/scripts/
sh ./run_all.sh # single uber-script
# alternately, run the individual scripts
sh ./profiles_to_consul.sh # pushes widget spring profiles to consul
sh ./stack_deploy.sh # deploys widget and mongodb containers
sh ./stack_validate.sh # waits/tests for all containers to start
sh ./seed_widgets.sh # creates a series of sample widget entries

view raw
install_widgets.sh
hosted with ❤ by GitHub

When the scripts have completed, the Widget service and MongoDB containers should be distributed across two of the three swarm worker nodes (VirtualBox VMs).

swarm_fluentd_diagram_b

To confirm the final state of the swarm and the running container stacks, use the following Docker commands.

$ docker-machine env manager1 \
&& eval $(docker-machine env manager1)
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
manager1 * virtualbox Running tcp://192.168.99.100:2376 v17.04.0-ce
manager2 – virtualbox Running tcp://192.168.99.101:2376 v17.04.0-ce
manager3 – virtualbox Running tcp://192.168.99.102:2376 v17.04.0-ce
worker1 – virtualbox Running tcp://192.168.99.103:2376 v17.04.0-ce
worker2 – virtualbox Running tcp://192.168.99.104:2376 v17.04.0-ce
worker3 – virtualbox Running tcp://192.168.99.105:2376 v17.04.0-ce
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
1c7lyvega5qqwqrz0nsudfmpb * manager1 Ready Active Leader
86md2td2a1zxh4x82tp8zwnim worker2 Ready Active
iw82eojks2q82nvmdu07kz7ap manager3 Ready Active Reachable
valelhqb6p6rtvi1jw970cwfu worker3 Ready Active
vx34axhfwil16iudyba09pb38 manager2 Ready Active Reachable
xhnpumc22fht87z49y5ze3zkh worker1 Ready Active
$ docker stack ls
NAME SERVICES
monitoring_stack 3
widget_stack 2
$ docker service ls
ID NAME MODE REPLICAS IMAGE
6jf2bkbu3k2w widget_stack_mongodb replicated 1/1 mongo:latest
iyu9yvs3xnly widget_stack_widget global 2/2 garystafford/microservice-docker-demo-widget:latest
rzarts0oi0se monitoring_stack_visualizer global 3/3 manomarks/visualizer:latest
yjnpg9tmcss8 monitoring_stack_fluentd global 2/2 garystafford/custom-fluentd:latest
zf0oovhzf0ow monitoring_stack_elk replicated 1/1 sebp/elk:latest

view raw
swarm_output.txt
hosted with ❤ by GitHub

Open the Swarm Visualizer web UI, using any of the swarm manager node IPs, on port 5001, to confirm the swarm health, as well as the running container’s locations.

Visualizer

Lastly, open the Consul Web UI, using any of the swarm manager node IPs, on port 5601, to confirm the running container’s health, as well as their placement on the swarm nodes.

Consul_1

Streaming Logs

Elastic Stack

If you read the previous post, Distributed Service Configuration with Consul, Spring Cloud, and Docker, you will notice we deployed a few additional components this time. First, the Elastic Stack (aka ELK), is deployed to the worker3 swarm worker node, within a single container. I have increased the CPU count and RAM assigned to this VM, to minimally run the Elastic Stack. If you review the docker-compose.yml file, you will note I am using Sébastien Pujadas’ sebp/elk:latest Docker base image from Docker Hub to provision the Elastic Stack. At the time of the post, this was based on the 5.3.0 version of ELK.

Docker Logging Driver

The Widget stack’s docker-compose.yml file has been modified since the last post. The compose file now incorporates a Fluentd logging configuration section for each service. The logging configuration includes the address of the Fluentd instance, on the same swarm worker node. The logging configuration also includes a tag for each log message.

version: '3.0'
services:
widget:
image: garystafford/microservice-docker-demo-widget:fluentd
hostname: widget
depends_on:
mongodb
ports:
8030:8030/tcp
networks:
demo_overlay_net
logging:
driver: fluentd
options:
tag: docker.{{.Name}}
fluentd-address: localhost:24224
labels: com.widget.environment
env: SERVICE_NAME, SERVICE_TAGS
labels:
"com.widget.environment: ${ENVIRONMENT}"
deploy:
mode: global
placement:
constraints:
node.role == worker
node.hostname != worker3
environment:
"CONSUL_SERVER_URL=${CONSUL_SERVER}"
"SERVICE_NAME=widget"
"SERVICE_TAGS=service"
command: "java -Dspring.profiles.active=${ACTIVE_PROFILE} \
-Djava.security.egd=file:/dev/./urandom \
-jar widget/widget-service.jar"

view raw
docker-compose.yml
hosted with ❤ by GitHub

Fluentd

In addition to the Elastic Stack, we have deployed Fluentd to the worker1 and worker2 swarm nodes. This is also where the Widget and MongoDB containers are deployed. Again, looking at the docker-compose.yml file, you will note we are using a custom Fluentd Docker image, garystafford/custom-fluentd:latest, which I created. The custom image is available on Docker Hub.

version: '3.0'
services:
fluentd:
image: garystafford/custom-fluentd:latest
hostname: fluentd
ports:
"24224:24224/tcp"
"24224:24224/udp"
networks:
demo_overlay_net
deploy:
mode: global
placement:
constraints:
node.role == worker
node.hostname != worker3
environment:
SERVICE_NAME: fluentd
SERVICE_TAGS: monitoring

view raw
docker-compose.yml
hosted with ❤ by GitHub

The custom Fluentd Docker image is based on Fluentd’s official onbuild Docker image, fluent/fluentd:onbuild. Fluentd provides instructions for building your own custom images, from their onbuild base images.

FROM fluent/fluentd:onbuild
LABEL maintainer "Gary A. Stafford <garystafford@rochester.rr.com>"
ENV REFRESHED_AT 2017-04-02
USER root
RUN apk add –update –virtual .build-deps \
sudo build-base ruby-dev \
# cutomize following instruction as you wish
&& sudo -u fluent gem install \
fluent-plugin-secure-forward \
fluent-plugin-elasticsearch \
fluent-plugin-concat \
&& sudo -u fluent gem sources –clear-all \
&& apk del .build-deps \
&& rm -rf /var/cache/apk/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gem
USER fluent

view raw
Dockerfile
hosted with ❤ by GitHub

There were two reasons I chose to create a custom Fluentd Docker image. First, I added the Uken Games’ Fluentd Elasticsearch Plugin, to the Docker Image. This highly configurable Fluentd Output Plugin allows us to push Docker logs, processed by Fluentd to the Elasticsearch. Adding additional plugins is a common reason for creating a custom Fluentd Docker image.

The second reason to create a custom Fluentd Docker image was configuration. Instead of bind-mounting host directories or volumes to the multiple Fluentd containers, to provide Fluentd’s configuration, I baked the configuration file into the immutable Docker image. The bare-bones, basicFluentd configuration file defines three processes, which are Input, Filter, and Output. These processes are accomplished using Fluentd plugins. Fluentd has 6 types of plugins: Input, Parser, Filter, Output, Formatter and Buffer. Fluentd is primarily written in Ruby, and its plugins are Ruby gems.

<source>
@type forward
@id input1
@label @mainstream
port 24224
</source>
<filter **>
@type stdout
</filter>
<label @mainstream>
<match **>
@type copy
<store>
@type file
@id output_docker1
path /fluentd/log/docker.*.log
symlink_path /fluentd/log/docker.log
append true
time_slice_format %Y%m%d
time_slice_wait 1m
time_format %Y%m%dT%H%M%S%z
buffer_path /fluentd/log/docker.*.log
</store>
<store>
@type elasticsearch
logstash_format true
flush_interval 5s
host elk
port 9200
index_name fluentd
type_name fluentd
</store>
</match>
</label>

view raw
fluent.conf
hosted with ❤ by GitHub

Fluentd listens for input on tcp port 24224, using the forward Input Plugin. Docker logs are streamed locally on each swarm node, from the Widget and MongoDB containers to the local Fluentd container, over tcp port 24224, using Docker’s fluentd logging driver, introduced earlier. Fluentd

Fluentd then filters all input using the stdout Filter Plugin. This plugin prints events to stdout, or logs if launched with daemon mode. This is the most basic method of filtering.

Lastly, Fluentd outputs the filtered input to two destinations, a local log file and Elasticsearch. First, the Docker logs are sent to a local Fluentd log file. This is only for demonstration purposes and debugging. Outputting log files is not recommended for production, nor does it meet the 12-factor application recommendations for logging. Second, Fluentd outputs the Docker logs to Elasticsearch, over tcp port 9200, using the Fluentd Elasticsearch Plugin, introduced above.

log_message_flow

Additional Metadata

In addition to the log message itself, in JSON format, the fluentd log driver sends the following metadata in the structured log message: container_id, container_name, and source. This is helpful in identifying and categorizing log messages from multiple sources. Below is a sample of log messages from the raw Fluentd log file, with the metadata tags highlighted in yellow. At the bottom of the output is a log message parsed with jq, for better readability.

fluentd_logs

Using Elastic Stack

Now that our two Docker stacks are up and running on our swarm, we should be streaming logs to Elasticsearch. To confirm this, open the Kibana web console, which should be available at the IP address of the worker3 swarm worker node, on port 5601.

Kibana

For the sake of this demonstration, I increased the verbosity of the Spring Boot Widget service’s log level, from INFO to DEBUG, in Consul. At this level of logging, the two Widget services and the single MongoDB instance were generating an average of 250-400 log messages every 30 seconds, according to Kibana.

If that seems like a lot, keep in mind, these are Docker logs, which are single-line log entries. We have not aggregated multi-line messages, such as Java exceptions and stack traces messages, into single entries. That is for another post. Also, the volume of debug-level log messages generated by the communications between the individual services and Consul is fairly verbose.

Kibana_3

Inspecting log entries in Kibana, we find the metadata tags contained in the raw Fluentd log output are now searchable fields: container_id, container_name, and source, as well as log. Also, note the _type field, with a value of ‘fluentd’. We injected this field in the output section of our Fluentd configuration, using the Fluentd Elasticsearch Plugin. The _type fiel allows us to differentiate these log entries from other potential data sources.

Kibana_2.png

References

All opinions in this post are my own and not necessarily the views of my current employer or their clients.

, , , , , , ,

  1. #1 by amir on July 12, 2017 - 7:01 pm

    Hey — your repo links are invalid; they’re missing your username.
    Thanks for the article and repos — very well organized.
    Cheers

  2. #2 by max on August 24, 2017 - 10:16 am

    This is a great article. Got me started with Elastic stack in docker swarm, with my own applications pretty smoothly.

  3. #3 by Michael Hobbs on January 18, 2019 - 9:10 am

    Gary, you saved me so much time! Awesome work!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: