Posts Tagged ELK
Docker Log Aggregation and Visualization Options with the Elastic Stack
Posted by Gary A. Stafford in AWS, Cloud, DevOps, Software Development on September 28, 2017
As a Developer and DevOps Engineer, it wasn’t that long ago, I spent a lot of time requesting logs from Operations teams for applications running in Production. Many organizations I’ve worked with have created elaborate systems for requesting, granting, and revoking access to application logs. Requesting and obtaining access to logs typically took hours or days, or simply never got approved. Since most enterprise applications are composed of individual components running on multiple application and web servers, it was necessary to request multiple logs. What was often a simple problem to diagnose and fix, became an unnecessarily time-consuming ordeal.
Hopefully, you are still not in this situation. Given the average complexity of today’s modern, distributed, containerized application platforms, accessing individual logs is simply unrealistic and ineffective. The solution is log aggregation and visualization.
Log Aggregation and Visualization
In the context of this post, log aggregation and visualization is defined as the collection, centralized storage, and the ability to simultaneously display application logs from multiple, dissimilar sources. Take a typical modern web application. The frontend UI might be built with Angular, React, or Node. The UI is likely backed by multiple RESTful services, possibly built in Java Spring Boot or Python Flask, and a database or databases, such as MongoDB or MySQL. To support the application, there are auxiliary components, such as API gateways, load-balancers, and messaging brokers. These components are likely deployed as multiple instances, for performance and availability. All instances generate application logs in varying formats.
When troubleshooting an application, such as the one described above, you must often trace a user’s transaction from UI through firewalls and gateways, to the web server, back through the API gateway, to multiple backend services via load-balancers, through message queues, to databases, possibly to external third-party APIs, and back to the client. This is why log aggregation and visualization is essential.
Logging Options
Log aggregation and visualization solutions typically come in three varieties: cloud-hosted by a SaaS provider, a service provided by your Cloud provider, and self-hosted, either on-premises or in the cloud. Cloud-hosted SaaS solutions include Loggly, Splunk, Logentries, and Sumo Logic. Some of these solutions, such as Splunk, are also available as a self-hosted service. Cloud-provider solutions include AWS CloudWatch and Azure Application Insights. Most hosted solutions have reoccurring pricing models based on the volume of logs or the number of server nodes being monitored.
Self-hosted solutions include Graylog 2, Nagios Log Server, Splunk Free, and Elastic’s Elastic Stack. The ELK Stack (Elasticsearch, Logstash, and Kibana), as it was previously known, has been re-branded the Elastic Stack, which now includes Beats. Beats is Elastic’s lightweight shipper that send data from edge machines to Logstash and Elasticsearch.
Often, you will see other components mentioned in the self-hosted space, such as Fluentd, syslog, and Kafka. These are examples of log aggregators or datastores for logs. They lack the combined abilities to collect, store, and display multiple logs. These components are generally part of a larger log aggregation and visualization solution.
This post will explore self-hosted log aggregation and visualization of a Dockerized application on AWS, using the Elastic Stack. The post details three common variations of log collection and routing to Elasticsearch, using various Docker logging drivers, along with Logspout, Fluentd, and GELF (Graylog Extended Log Format).
Docker Swarm Cluster
The post’s example application is deployed to a Docker Swarm, built on AWS, using Docker CE for AWS. Docker has automated the creation of a Swarm on AWS using Docker Cloud, right from your desktop. Creating a Swarm is as easy as inputting a few options and clicking build. Docker uses an AWS CloudFormation script to provision all the necessary AWS resources for the Docker Swarm.
For this post’s logging example, I built a minimally configured Docker Swarm cluster, consisting of a single Manager Node and three Worker Nodes. The four Swarm nodes, all EC2 instances, are behind an AWS ELB, inside a new AWS VPC.
As seen with the docker node ls
command, the Docker Swarm will look similar to the following.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ docker node ls | |
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS | |
tc2mwa29jl21bp4x3buwbhlhx ip-172-31-5-65.ec2.internal Ready Active | |
ueonffqdgocfo6dwb7eckjedz ip-172-31-29-135.ec2.internal Ready Active | |
n9zportqziqyzqsav3cbfakng ip-172-31-39-165.ec2.internal Ready Active | |
ao55bv77cs0542x9arm6droyo * ip-172-31-47-42.ec2.internal Ready Active Leader |
Sample Application Components
Multiple containerized copies of a simple Java Spring Boot RESTful Hello-World service, available on GitHub, along with the associated logging aggregators, are deployed to Worker Node 1 and Worker Node 2. We will explore each of these application components later in the post. The containerized components consist of the following:
- Fluentd (garystafford/custom-fluentd)
- Logspout (garystafford/custom-logspout)
- NGINX (garystafford/custom-nginx)
- Hello-World Service using Docker’s default JSON file logging driver
- Hello-World Service using Docker’s GELF logging driver
- Hello-World Service using Docker’s Fluentd logging driver
NGINX is used as a simple frontend API gateway, which to routes HTTP requests to each of the three logging variations of the Hello-World service (garystafford/hello-world).
A single container, running the entire Elastic Stack (garystafford/custom-elk) is deployed to Worker Node 3. This is to isolate the Elastic Stack from the application. Typically, in a real environment, the Elastic Stack would be running on separate infrastructure for performance and security, not alongside your application. Running a docker service ls
, the deployed services appear as follows.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ docker service ls | |
ID NAME MODE REPLICAS IMAGE PORTS | |
6va602lzfl4y dockercloud-server-proxy global 1/1 dockercloud/server-proxy *:2376->2376/tcp | |
jjjddhuczx35 elk-demo_elk replicated 1/1 garystafford/custom-elk:latest *:5000->5000/udp,*:5044->5044/tcp,*:5601->5601/tcp,*:9200->9200/tcp,*:12201->12201/udp | |
mwm1rbo3dp3t elk-demo_fluentd global 2/2 garystafford/custom-fluentd:latest *:24224->24224/tcp,*:24224->24224/udp | |
ofo02g2kbhg7 elk-demo_hello-fluentd replicated 2/2 garystafford/hello-world:latest | |
05txkpmizwxq elk-demo_hello-gelf replicated 2/2 garystafford/hello-world:latest | |
pjs614raq37y elk-demo_hello-logspout replicated 2/2 garystafford/hello-world:latest | |
9h0l0w2ej1yw elk-demo_logspout global 2/2 garystafford/custom-logspout:latest | |
wpxjji5wwd4j elk-demo_nginx replicated 2/2 garystafford/custom-nginx:latest *:80->80/tcp | |
w0y5inpryaql elk-demo_portainer global 1/1 portainer/portainer:latest *:9000->9000/tcp |
Portainer
A single instance of Portainer (Docker Hub: portainer/portainer) is deployed on the single Manager Node. Portainer, amongst other things, provides a detailed view of Docker Swarm, showing each Swarm Node and the service containers deployed to them.
In my opinion, Portainer provides a much better user experience than Docker Enterprise Edition’s most recent Universal Control Plane (UCP). In the past, I have also used Visualizer (dockersamples/visualizer), one of the first open source solutions in this space. However, since the Visualizer project moved to Docker, it seems like the development of new features has completely stalled out. A good list of container tools can be found on StackShare.
Deployment
All the Docker service containers are deployed to the AWS-based Docker Swarm using a single Docker Compose file. The order of service startup is critical. Elasticsearch should fully startup first, followed by Fluentd and Logspout, then the three sets of Hello-World instances, and finally NGINX.
To deploy and start all the Docker services correctly, there are two scripts in the GitHub repository. First, execute the following command, sh ./stack_deploy.sh
. This will deploy the Docker service stack and create an overlay network, containing all the services as configured in the docker-compose.yml
file. Then, to ensure the services start in the correct sequence, execute sh ./service_update.sh
. This will restart each service in the correct order, with pauses between services to allow time for startup; a bit of a hack, but effective.
Collection and Routing Examples
Below is a diagram showing all the components comprising this post’s examples, and includes the protocols and ports on which they communicate. Following, we will look at three variations of self-hosted log collection and routing options for the Elastic Stack.
Example 1: Fluentd
The first example of log aggregation and visualization uses Fluentd, a Cloud Native Computing Foundation (CNCF) hosted project. Fluentd is described as ‘an open source data collector for unified logging layer.’ A container running Fluentd with a custom configuration runs globally on each Worker Node where the applications are deployed, in this case, the hello-fluentd Docker service. Here is the custom Fluentd configuration file (fluent.conf
):
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# fluentd config for logging demo | |
<source> | |
@type forward | |
port 24224 | |
</source> | |
<filter **> | |
@type concat | |
key log | |
stream_identity_key container_id | |
multiline_start_regexp /^\S+/ | |
multiline_end_regexp /\s+.*more$/ | |
flush_interval 120s | |
timeout_label @processdata | |
</filter> | |
<label @ERROR> | |
<match **> | |
@type stdout | |
</match> | |
</label> | |
<label @processdata> | |
<match **> | |
@type stdout | |
</match> | |
</label> | |
<match **> | |
@type elasticsearch | |
logstash_format true | |
host elk | |
port 9200 | |
index_name fluentd | |
type_name fluentd | |
</match> |
The Hello-World service is configured through the Docker Compose file to use the Fluentd Docker logging driver. The log entries from the Hello-World containers on the Worker Nodes are diverted from being output to JSON files, using the default JSON file logging driver, to the Fluentd container instance on the same host as the Hello-World container. The Fluentd container is listening for TCP traffic on port 24224.
Fluentd then sends the individual log entries to Elasticsearch directly, bypassing Logstash. Fluentd log entries are sent via HTTP to port 9200, Elasticsearch’s JSON interface.
Using Fluentd as a transport method, log entries appear as JSON documents in Elasticsearch, as shown below. This Elasticsearch JSON document is an example of a single line log entry. Note the primary field container identifier, when using Fluentd, is container_id
. This field will vary depending on the Docker driver and log collector, as seen in the next two logging examples.
The next example shows a Fluentd multiline log entry. Using the Fluentd Concat filter plugin (fluent-plugin-concat
), the individual lines of a stack trace from a Java runtime exception, thrown by the hello-fluentd Docker service, have been recombined into a single Elasticsearch JSON document.
In the above log entries, note the DEPLOY_ENV
and SERVICE_NAME
fields. These values were injected into the Docker Compose file, as environment variables, during deployment of the Hello-World service. The Fluentd Docker logging driver applies these as env
options, as shown in the example Docker Compose snippet, below, lines 5-9.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hello-fluentd: | |
image: garystafford/hello-world:latest | |
networks: | |
– elk-demo | |
logging: | |
driver: fluentd | |
options: | |
tag: docker.{{.Name}} | |
env: SERVICE_NAME,DEPLOY_ENV | |
deploy: | |
placement: | |
constraints: | |
– node.role == worker | |
– node.hostname != ${WORKER_NODE_3} | |
replicas: 2 | |
update_config: | |
parallelism: 1 | |
delay: 10s | |
restart_policy: | |
condition: on-failure | |
max_attempts: 3 | |
delay: 5s | |
environment: | |
SERVICE_NAME: hello-fluentd | |
DEPLOY_ENV: ${DEPLOY_ENV} | |
LOGSPOUT: ignore | |
command: "java \ | |
-Dspring.profiles.active=${DEPLOY_ENV} \ | |
-Djava.security.egd=file:/dev/./urandom \ | |
-jar hello-world.jar" |
Example 2: Logspout
The second example of log aggregation and visualization uses GliderLabs’ Logspout. Logspout is described by GliderLabs as ‘a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system.’ In the post’s example, a container running Logspout with a custom configuration runs globally on each Worker Node where the applications are deployed, identical to Fluentd.
The hello-logspout Docker service is configured through the Docker Compose file to use the default JSON file logging driver. According to Docker, ‘by default, Docker captures the standard output (and standard error) of all your containers and writes them in files using the JSON format. The JSON format annotates each line with its origin (stdout or stderr) and its timestamp. Each log file contains information about only one container.’
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hello-logspout: | |
image: garystafford/hello-world:latest | |
networks: | |
– elk-demo | |
logging: | |
driver: json-file | |
options: | |
env: SERVICE_NAME,DEPLOY_ENV | |
max-size: 10m | |
deploy: | |
placement: | |
constraints: | |
– node.role == worker | |
– node.hostname != ${WORKER_NODE_3} | |
replicas: 2 | |
update_config: | |
parallelism: 1 | |
delay: 10s | |
restart_policy: | |
condition: on-failure | |
max_attempts: 3 | |
delay: 5s | |
environment: | |
SERVICE_NAME: hello-logspout | |
DEPLOY_ENV: ${DEPLOY_ENV} | |
command: "java \ | |
-Dspring.profiles.active=${DEPLOY_ENV} \ | |
-Djava.security.egd=file:/dev/./urandom \ | |
-jar hello-world.jar" |
Normally, it is not necessary to explicitly set the default Docker logging driver to JSON files. However, in this case, Docker CE for AWS automatically configured each Swarm Node’s Docker daemon default logging driver to Amazon CloudWatch Logs logging driver. The default drive may be seen by running the docker info
command while attached to the Docker daemon. Note line 12 in the snippet below.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
docker info | |
Containers: 6 | |
Running: 6 | |
Paused: 0 | |
Stopped: 0 | |
Images: 10 | |
Server Version: 17.07.0-ce | |
Storage Driver: overlay2 | |
Backing Filesystem: extfs | |
Supports d_type: true | |
Native Overlay Diff: true | |
Logging Driver: awslogs | |
Cgroup Driver: cgroupfs | |
Plugins: | |
Volume: local | |
Network: bridge host ipvlan macvlan null overlay | |
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog | |
Swarm: active |
The hello-fluentd Docker service containers on the Worker Nodes send log entries to individual JSON files. The Fluentd container on each host then retrieves and routes those JSON log entries to Logstash, within the Elastic Stack container running on Worker Node 3, over UDP to port 5000. Logstash, which is explicitly listening for JSON via UDP on port 5000, then outputs those log entries to Elasticsearch, via HTTP to port 9200, Elasticsearch’s JSON interface.
Using Logspout as a transport method, log entries appear as JSON documents in Elasticsearch, as shown below. Note the field differences between the Fluentd log entry above and this entry. There are a number of significant variations, making it difficult to use both methods, across the same distributed application. For example, the main body of the log entry is contained in the message
field using Logspout, but in the log
field using Fluentd. The name of the Docker container, which serves as the primary means of identifying the container instance, is the docker.name
field with Logspout, but container.name
for Fluentd.
Another helpful field, provided by Logspout, is the docker.image
field. This is beneficial when associating code issues to a particular code release. In this example, the Hello-World service uses the latest
Docker image tag, which is not considered best practice. However, in a real production environment, the Docker tags often represents the incremental build number from the CI/CD system, which is tied to a specific build of the code.
The other challenge I have had with Logspout is passing the
env
and tag
options, such as DEPLOY_ENV
and SERVICE_NAME
, as seen previously with the Fluentd example. Note they are blank in the above sample. It is possible, but not as straightforward as with Fluentd, and requires interacting directly with the Docker daemon on each Worker node.
Example 3: Graylog Extended Format (GELF)
The third and final example of log aggregation and visualization uses the Docker Graylog Extended Format (GELF) logging driver. According to the GELF website, ‘the Graylog Extended Log Format (GELF) is a log format that avoids the shortcomings of classic plain syslog.’ These syslog shortcomings include a maximum length of 1024 bytes, no data types, multiple dialects making parsing difficult, and no compression.
The GELF format, designed to work with the Graylog Open Source Log Management Server, work equally as well with the Elastic Stack. With the GELF logging driver, there is no intermediary logging collector and router, as with Fluentd and Logspout. The hello-gelf Docker service is configured through its Docker Compose file to use the GELF logging driver. The two hello-gelf Docker service containers on the Worker Nodes send log entries directly to Logstash, running within the Elastic Stack container, running on Worker Node 3, via UDP to port 12201.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hello-gelf: | |
image: garystafford/hello-world:latest | |
networks: | |
– elk-demo | |
logging: | |
driver: gelf | |
options: | |
gelf-address: "udp://${ELK_IP}:12201" | |
tag: docker.{{.Name}} | |
env: SERVICE_NAME,DEPLOY_ENV | |
deploy: | |
placement: | |
constraints: | |
– node.role == worker | |
– node.hostname != ${WORKER_NODE_3} | |
replicas: 2 | |
update_config: | |
parallelism: 1 | |
delay: 10s | |
restart_policy: | |
condition: on-failure | |
max_attempts: 3 | |
delay: 5s | |
environment: | |
LOGSPOUT: ignore | |
SERVICE_NAME: hello-gelf | |
DEPLOY_ENV: ${DEPLOY_ENV} | |
command: "java \ | |
-Dspring.profiles.active=${DEPLOY_ENV} \ | |
-Djava.security.egd=file:/dev/./urandom \ | |
-jar hello-world.jar" |
Logstash, which is explicitly listening for UDP traffic on port 12201, then outputs those log entries to Elasticsearch, via HTTP to port 9200, Elasticsearch’s JSON interface.
Using the Docker Graylog Extended Format (GELF) logging driver as a transport method, log entries appear as JSON documents in Elasticsearch, as shown below. They are the most verbose of the three formats.
Again, note the field differences between the Fluentd and Logspout log entries above, and this GELF entry. Both the field names of the main body of the log entry and the name of the Docker container are different from both previous examples.
Another bonus with GELF, each entry contains the command field, which stores the command used to start the container’s process. This can be helpful when troubleshooting application startup issues. Often, the exact container startup command might have been injected into the Docker Compose file at deploy time by the CI Server and contained variables, as is the case with the Hello-World service. Reviewing the log entry in Kibana for the command is much easier and safer than logging into the container and executing commands to check the running process for the startup command.
Unlike Logspout, and similar to Fluentd, note the DEPLOY_ENV
and SERVICE_NAME
fields are present in the GELF entry. These were injected into the Docker Compose file as environment variables during deployment of the Hello-World service. The GELF Docker logging driver applies these as env
options. With GELF the entry also gets the optional tag, which was passed in the Docker Compose file’s service definition, tag: docker.{{.Name}}
.
Unlike Fluentd, GELF and Logspout do not easily handle multiline logs. Below is an example of a multiline Java runtime exception thrown by the hello-gelf Docker service. The stack trace is not recombined into a single JSON document in Elasticsearch, like in the Fluentd example. The stack trace exists as multiple JSON documents, making troubleshooting much more difficult. Logspout entries will look similar to GELF.
Pros and Cons
In my opinion, and based on my level of experience with each of the self-hosted logging collection and routing options, the following some of their pros and cons.
Fluentd
- Pros
- Part of CNCF, Fluentd is becoming the defacto logging standard for cloud-native applications
- Easily extensible via a large number of plugins
- Easily containerized
- Ability to easily handle multiline log entries (ie. Java stack trace)
- Ability to use the Fluentd container’s service name as the Fluentd address, not an IP address or DNS resolvable hostname
- Cons
- Using Docker’s Fluentd logging driver, if the Fluentd container is not available on the container’s host, the container logging to Fluentd will fail (major con!)
Logspout
- Pros
- Doesn’t require a change to the default Docker JSON file logging driver, logs are still viewable via docker logs command (big plus!)
- Easily to add and remove functionality via Golang modules
- Easily containerized
- Cons
- Inability to easily handle multiline log entries (ie. Java stack trace)
- Logspout containers must be restarted if the Elastic Stack is restarted to restart logging
- To reach Logstash, Logspout must use a DNS resolvable hostname or IP address, not the name of the Elastic Stack container on the same overlay network (big con!)
GELF
- Pros
- Application containers, using Docker GELF logging driver will not fail if the downstream Logspout container is unavailable
- Docker GELF logging driver allows compression of logs for shipment to Logspout
- Cons
- Inability to easily handle multiline log entries (ie. Java stack trace)
Conclusion
Of course, there are other self-hosted logging collection and routing options, including Elastic’s Beats, journald, and various syslog servers. Each has their pros and cons, depending on your project’s needs. After building and maintaining several self-hosted mission-critical log aggregation and visualization solutions, it is easy to see the appeal of an off-the-shelf cloud-hosted SaaS solution such as Splunk or Cloud provider solutions such as Application Insights.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Streaming Docker Logs to the Elastic Stack (ELK Stack) using Fluentd
Posted by Gary A. Stafford in Bash Scripting, DevOps, Enterprise Software Development, Software Development on April 10, 2017
Introduction
Fluentd and Docker’s native logging driver for Fluentd makes it easy to stream Docker logs from multiple running containers to the Elastic Stack. In this post, we will use Fluentd to stream Docker logs from multiple instances of a Dockerized Spring Boot RESTful service and MongoDB, to the Elastic Stack (ELK).
In a recent post, Distributed Service Configuration with Consul, Spring Cloud, and Docker, we built a Consul cluster using Docker swarm mode, to host distributed configurations for a Spring Boot service. We will use the resulting swarm cluster from the previous post as a foundation for this post.
Fluentd
According to the Fluentd website, Fluentd is described as an open source data collector, which unifies data collection and consumption for a better use and understanding of data. Fluentd combines all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. Fluentd structures data as JSON as much as possible.
Logging Drivers
Docker includes multiple logging mechanisms to get logs from running containers and services. These mechanisms are called logging drivers. Fluentd is one of the ten current Docker logging drivers. According to Docker, The fluentd logging driver sends container logs to the Fluentd collector as structured log data. Then, users can utilize any of the various output plugins, from Fluentd, to write these logs to various destinations.
Elastic Stack
The ELK Stack, now known as the Elastic Stack, is the combination of Elastic’s very popular products: Elasticsearch, Logstash, and Kibana. According to Elastic, the Elastic Stack provides real-time insights from almost any type of structured and unstructured data source.
Setup
All code for this post has been tested on both MacOS and Linux. For this post, I am provisioning and deploying to a Linux workstation, running the most recent release of Fedora and Oracle VirtualBox. If you want to use AWS or another infrastructure provider instead of VirtualBox to build your swarm, it is fairly easy to switch the Docker Machine driver and change a few configuration items in the vms_create.sh
script (see Provisioning, below).
Required Software
If you want to follow along with this post, you will need the latest versions of git, Docker, Docker Machine, Docker Compose, and VirtualBox installed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cat /etc/*release | sed -n '1p' | |
# Fedora release 25 (Twenty Five) | |
git –version \ | |
docker –version \ | |
docker-compose –version \ | |
docker-machine –version \ | |
virtualbox –help | sed -n '1p' | |
# git version 2.9.3 | |
# Docker version 17.04.0-ce, build 4845c56 | |
# docker-compose version 1.11.2, build dfed245 | |
# docker-machine version 0.10.0, build 76ed2a6 | |
# Oracle VM VirtualBox Manager 5.1.18 |
Source Code
All source code for this post is located in two GitHub repositories. The first repository contains scripts to provision the VMs, create an overlay network and persistent host-mounted volumes, build the Docker swarm, and deploy Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack. The second repository contains scripts to deploy two instances of the Widget Spring Boot RESTful service and a single instance of MongoDB. You can execute all scripts manually, from the command-line, or from a CI/CD pipeline, using tools such as Jenkins.
Provisioning the Swarm
To start, clone the first repository, and execute the single run_all.sh
script, or execute the seven individual scripts necessary to provision the VMs, create the overlay network and host volumes, build the swarm, and deploy Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack. Follow the steps below to complete this part.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git clone –depth 1 –branch fluentd \ | |
https://github.com/garystafford/microservice-docker-demo-consul.git | |
cd microservice-docker-demo-consul/scripts/ | |
sh ./run_all.sh # single uber-script | |
# alternately, run the individual scripts | |
sh ./vms_create.sh # creates vms using docker machine | |
sh ./swarm_create.sh # creates the swarm | |
sh ./ntwk_vols_create.sh # creates overlay network and volumes | |
sh ./consul_deploy.sh # deploys consul to all nodes | |
sh ./registrator_deploy.sh # deploys registrator | |
sh ./stack_deploy.sh # deploys fluentd, visualizer, elastic stack | |
sh ./stack_validate.sh # waits/tests for all containers to start |
When the scripts have completed, the resulting swarm should be configured similarly to the diagram below. Consul, Registrator, Swarm Visualizer, Fluentd, and the Elastic Stack containers should be distributed across the three swarm manager nodes and the three swarm worker nodes (VirtualBox VMs).
Deploying the Application
Next, clone the second repository, and execute the single run_all.sh
script, or execute the four scripts necessary to deploy the Widget Spring Boot RESTful service and a single instance of MongoDB. Follow the steps below to complete this part.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git clone –depth 1 –branch fluentd \ | |
https://github.com/garystafford/microservice-docker-demo-widget.git | |
cd ../../microservice-docker-demo-widget/scripts/ | |
sh ./run_all.sh # single uber-script | |
# alternately, run the individual scripts | |
sh ./profiles_to_consul.sh # pushes widget spring profiles to consul | |
sh ./stack_deploy.sh # deploys widget and mongodb containers | |
sh ./stack_validate.sh # waits/tests for all containers to start | |
sh ./seed_widgets.sh # creates a series of sample widget entries |
When the scripts have completed, the Widget service and MongoDB containers should be distributed across two of the three swarm worker nodes (VirtualBox VMs).
To confirm the final state of the swarm and the running container stacks, use the following Docker commands.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
$ docker-machine env manager1 \ | |
&& eval $(docker-machine env manager1) | |
$ docker-machine ls | |
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS | |
manager1 * virtualbox Running tcp://192.168.99.100:2376 v17.04.0-ce | |
manager2 – virtualbox Running tcp://192.168.99.101:2376 v17.04.0-ce | |
manager3 – virtualbox Running tcp://192.168.99.102:2376 v17.04.0-ce | |
worker1 – virtualbox Running tcp://192.168.99.103:2376 v17.04.0-ce | |
worker2 – virtualbox Running tcp://192.168.99.104:2376 v17.04.0-ce | |
worker3 – virtualbox Running tcp://192.168.99.105:2376 v17.04.0-ce | |
$ docker node ls | |
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS | |
1c7lyvega5qqwqrz0nsudfmpb * manager1 Ready Active Leader | |
86md2td2a1zxh4x82tp8zwnim worker2 Ready Active | |
iw82eojks2q82nvmdu07kz7ap manager3 Ready Active Reachable | |
valelhqb6p6rtvi1jw970cwfu worker3 Ready Active | |
vx34axhfwil16iudyba09pb38 manager2 Ready Active Reachable | |
xhnpumc22fht87z49y5ze3zkh worker1 Ready Active | |
$ docker stack ls | |
NAME SERVICES | |
monitoring_stack 3 | |
widget_stack 2 | |
$ docker service ls | |
ID NAME MODE REPLICAS IMAGE | |
6jf2bkbu3k2w widget_stack_mongodb replicated 1/1 mongo:latest | |
iyu9yvs3xnly widget_stack_widget global 2/2 garystafford/microservice-docker-demo-widget:latest | |
rzarts0oi0se monitoring_stack_visualizer global 3/3 manomarks/visualizer:latest | |
yjnpg9tmcss8 monitoring_stack_fluentd global 2/2 garystafford/custom-fluentd:latest | |
zf0oovhzf0ow monitoring_stack_elk replicated 1/1 sebp/elk:latest |
Open the Swarm Visualizer web UI, using any of the swarm manager node IPs, on port 5001, to confirm the swarm health, as well as the running container’s locations.
Lastly, open the Consul Web UI, using any of the swarm manager node IPs, on port 5601, to confirm the running container’s health, as well as their placement on the swarm nodes.
Streaming Logs
Elastic Stack
If you read the previous post, Distributed Service Configuration with Consul, Spring Cloud, and Docker, you will notice we deployed a few additional components this time. First, the Elastic Stack (aka ELK), is deployed to the worker3
swarm worker node, within a single container. I have increased the CPU count and RAM assigned to this VM, to minimally run the Elastic Stack. If you review the docker-compose.yml
file, you will note I am using Sébastien Pujadas’ sebp/elk:latest
Docker base image from Docker Hub to provision the Elastic Stack. At the time of the post, this was based on the 5.3.0 version of ELK.
Docker Logging Driver
The Widget stack’s docker-compose.yml
file has been modified since the last post. The compose file now incorporates a Fluentd logging configuration section for each service. The logging configuration includes the address of the Fluentd instance, on the same swarm worker node. The logging configuration also includes a tag for each log message.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: '3.0' | |
services: | |
widget: | |
image: garystafford/microservice-docker-demo-widget:fluentd | |
hostname: widget | |
depends_on: | |
– mongodb | |
ports: | |
– 8030:8030/tcp | |
networks: | |
– demo_overlay_net | |
logging: | |
driver: fluentd | |
options: | |
tag: docker.{{.Name}} | |
fluentd-address: localhost:24224 | |
labels: com.widget.environment | |
env: SERVICE_NAME, SERVICE_TAGS | |
labels: | |
– "com.widget.environment: ${ENVIRONMENT}" | |
deploy: | |
mode: global | |
placement: | |
constraints: | |
– node.role == worker | |
– node.hostname != worker3 | |
environment: | |
– "CONSUL_SERVER_URL=${CONSUL_SERVER}" | |
– "SERVICE_NAME=widget" | |
– "SERVICE_TAGS=service" | |
command: "java -Dspring.profiles.active=${ACTIVE_PROFILE} \ | |
-Djava.security.egd=file:/dev/./urandom \ | |
-jar widget/widget-service.jar" |
Fluentd
In addition to the Elastic Stack, we have deployed Fluentd to the worker1
and worker2
swarm nodes. This is also where the Widget and MongoDB containers are deployed. Again, looking at the docker-compose.yml
file, you will note we are using a custom Fluentd Docker image, garystafford/custom-fluentd:latest
, which I created. The custom image is available on Docker Hub.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: '3.0' | |
services: | |
fluentd: | |
image: garystafford/custom-fluentd:latest | |
hostname: fluentd | |
ports: | |
– "24224:24224/tcp" | |
– "24224:24224/udp" | |
networks: | |
– demo_overlay_net | |
deploy: | |
mode: global | |
placement: | |
constraints: | |
– node.role == worker | |
– node.hostname != worker3 | |
environment: | |
SERVICE_NAME: fluentd | |
SERVICE_TAGS: monitoring |
The custom Fluentd Docker image is based on Fluentd’s official onbuild Docker image, fluent/fluentd:onbuild
. Fluentd provides instructions for building your own custom images, from their onbuild
base images.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM fluent/fluentd:onbuild | |
LABEL maintainer "Gary A. Stafford <garystafford@rochester.rr.com>" | |
ENV REFRESHED_AT 2017-04-02 | |
USER root | |
RUN apk add –update –virtual .build-deps \ | |
sudo build-base ruby-dev \ | |
# cutomize following instruction as you wish | |
&& sudo -u fluent gem install \ | |
fluent-plugin-secure-forward \ | |
fluent-plugin-elasticsearch \ | |
fluent-plugin-concat \ | |
&& sudo -u fluent gem sources –clear-all \ | |
&& apk del .build-deps \ | |
&& rm -rf /var/cache/apk/* \ | |
/home/fluent/.gem/ruby/2.3.0/cache/*.gem | |
USER fluent |
There were two reasons I chose to create a custom Fluentd Docker image. First, I added the Uken Games’ Fluentd Elasticsearch Plugin, to the Docker Image. This highly configurable Fluentd Output Plugin allows us to push Docker logs, processed by Fluentd to the Elasticsearch. Adding additional plugins is a common reason for creating a custom Fluentd Docker image.
The second reason to create a custom Fluentd Docker image was configuration. Instead of bind-mounting host directories or volumes to the multiple Fluentd containers, to provide Fluentd’s configuration, I baked the configuration file into the immutable Docker image. The bare-bones, basicFluentd configuration file defines three processes, which are Input, Filter, and Output. These processes are accomplished using Fluentd plugins. Fluentd has 6 types of plugins: Input, Parser, Filter, Output, Formatter and Buffer. Fluentd is primarily written in Ruby, and its plugins are Ruby gems.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<source> | |
@type forward | |
@id input1 | |
@label @mainstream | |
port 24224 | |
</source> | |
<filter **> | |
@type stdout | |
</filter> | |
<label @mainstream> | |
<match **> | |
@type copy | |
<store> | |
@type file | |
@id output_docker1 | |
path /fluentd/log/docker.*.log | |
symlink_path /fluentd/log/docker.log | |
append true | |
time_slice_format %Y%m%d | |
time_slice_wait 1m | |
time_format %Y%m%dT%H%M%S%z | |
buffer_path /fluentd/log/docker.*.log | |
</store> | |
<store> | |
@type elasticsearch | |
logstash_format true | |
flush_interval 5s | |
host elk | |
port 9200 | |
index_name fluentd | |
type_name fluentd | |
</store> | |
</match> | |
</label> |
Fluentd listens for input on tcp port 24224, using the forward Input Plugin. Docker logs are streamed locally on each swarm node, from the Widget and MongoDB containers to the local Fluentd container, over tcp port 24224, using Docker’s fluentd logging driver, introduced earlier. Fluentd
Fluentd then filters all input using the stdout Filter Plugin. This plugin prints events to stdout, or logs if launched with daemon mode. This is the most basic method of filtering.
Lastly, Fluentd outputs the filtered input to two destinations, a local log file and Elasticsearch. First, the Docker logs are sent to a local Fluentd log file. This is only for demonstration purposes and debugging. Outputting log files is not recommended for production, nor does it meet the 12-factor application recommendations for logging. Second, Fluentd outputs the Docker logs to Elasticsearch, over tcp port 9200, using the Fluentd Elasticsearch Plugin, introduced above.
Additional Metadata
In addition to the log
message itself, in JSON format, the fluentd log driver sends the following metadata in the structured log message: container_id
, container_name
, and source
. This is helpful in identifying and categorizing log messages from multiple sources. Below is a sample of log messages from the raw Fluentd log file, with the metadata tags highlighted in yellow. At the bottom of the output is a log message parsed with jq, for better readability.
Using Elastic Stack
Now that our two Docker stacks are up and running on our swarm, we should be streaming logs to Elasticsearch. To confirm this, open the Kibana web console, which should be available at the IP address of the worker3
swarm worker node, on port 5601.
For the sake of this demonstration, I increased the verbosity of the Spring Boot Widget service’s log level, from INFO to DEBUG, in Consul. At this level of logging, the two Widget services and the single MongoDB instance were generating an average of 250-400 log messages every 30 seconds, according to Kibana.
If that seems like a lot, keep in mind, these are Docker logs, which are single-line log entries. We have not aggregated multi-line messages, such as Java exceptions and stack traces messages, into single entries. That is for another post. Also, the volume of debug-level log messages generated by the communications between the individual services and Consul is fairly verbose.
Inspecting log entries in Kibana, we find the metadata tags contained in the raw Fluentd log output are now searchable fields: container_id
, container_name
, and source
, as well as log
. Also, note the _type
field, with a value of ‘fluentd’. We injected this field in the output section of our Fluentd configuration, using the Fluentd Elasticsearch Plugin. The _type
fiel allows us to differentiate these log entries from other potential data sources.
References
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Spring Music Revisited: Java-Spring-MongoDB Web App with Docker 1.12
Posted by Gary A. Stafford in Build Automation, Continuous Delivery, DevOps, Enterprise Software Development, Java Development, Software Development on August 7, 2016
Build, test, deploy, and monitor a multi-container, MongoDB-backed, Java Spring web application, using the new Docker 1.12.
Introduction
This post and the post’s example project represent an update to a previous post, Build and Deploy a Java-Spring-MongoDB Application using Docker. This new post incorporates many improvements made in Docker 1.12, including the use of the new Docker Compose v2 YAML format. The post’s project was also updated to use Filebeat with ELK, as opposed to Logspout, which was used previously.
In this post, we will demonstrate how to build, test, deploy, and manage a Java Spring web application, hosted on Apache Tomcat, load-balanced by NGINX, monitored by ELK with Filebeat, and all containerized with Docker.
We will use a sample Java Spring application, Spring Music, available on GitHub from Cloud Foundry. The Spring Music sample record album collection application was originally designed to demonstrate the use of database services on Cloud Foundry, using the Spring Framework. Instead of Cloud Foundry, we will host the Spring Music application locally, using Docker on VirtualBox, and optionally on AWS.
All files necessary to build this project are stored on the docker_v2
branch of the garystafford/spring-music-docker repository on GitHub. The Spring Music source code is stored on the springmusic_v2
branch of the garystafford/spring-music repository, also on GitHub.
Application Architecture
The Java Spring Music application stack contains the following technologies: Java, Spring Framework, AngularJS, Bootstrap, jQuery, NGINX, Apache Tomcat, MongoDB, the ELK Stack, and Filebeat. Testing frameworks include the Spring MVC Test Framework, Mockito, Hamcrest, and JUnit.
A few changes were made to the original Spring Music application to make it work for this demonstration, including:
- Move from Java 1.7 to 1.8 (including newer Tomcat version)
- Add unit tests for Continuous Integration demonstration purposes
- Modify MongoDB configuration class to work with non-local, containerized MongoDB instances
- Add Gradle
warNoStatic
task to build WAR without static assets - Add Gradle
zipStatic
task to ZIP up the application’s static assets for deployment to NGINX - Add Gradle
zipGetVersion
task with a versioning scheme for build artifacts - Add
context.xml
file andMANIFEST.MF
file to the WAR file - Add Log4j
RollingFileAppender
appender to send log entries to Filebeat - Update versions of several dependencies, including Gradle, Spring, and Tomcat
We will use the following technologies to build, publish, deploy, and host the Java Spring Music application: Gradle, git, GitHub, Travis CI, Oracle VirtualBox, Docker, Docker Compose, Docker Machine, Docker Hub, and optionally, Amazon Web Services (AWS).
NGINX
To increase performance, the Spring Music web application’s static content will be hosted by NGINX. The application’s WAR file will be hosted by Apache Tomcat 8.5.4. Requests for non-static content will be proxied through NGINX on the front-end, to a set of three load-balanced Tomcat instances on the back-end. To further increase application performance, NGINX will also be configured for browser caching of the static content. In many enterprise environments, the use of a Java EE application server, like Tomcat, is still not uncommon.
Reverse proxying and caching are configured thought NGINX’s default.conf
file, in the server
configuration section:
server { | |
listen 80; | |
server_name proxy; | |
location ~* \/assets\/(css|images|js|template)\/* { | |
root /usr/share/nginx/; | |
expires max; | |
add_header Pragma public; | |
add_header Cache-Control "public, must-revalidate, proxy-revalidate"; | |
add_header Vary Accept-Encoding; | |
access_log off; | |
} |
The three Tomcat instances will be manually configured for load-balancing using NGINX’s default round-robin load-balancing algorithm. This is configured through the default.conf
file, in the upstream
configuration section:
upstream backend { | |
server music_app_1:8080; | |
server music_app_2:8080; | |
server music_app_3:8080; | |
} |
Client requests are received through port 80
on the NGINX server. NGINX redirects requests, which are not for non-static assets, to one of the three Tomcat instances on port 8080
.
MongoDB
The Spring Music application was designed to work with a number of data stores, including MySQL, Postgres, Oracle, MongoDB, Redis, and H2, an in-memory Java SQL database. Given the choice of both SQL and NoSQL databases, we will select MongoDB.
The Spring Music application, hosted by Tomcat, will store and modify record album data in a single instance of MongoDB. MongoDB will be populated with a collection of album data from a JSON file, when the Spring Music application first creates the MongoDB database instance.
ELK
Lastly, the ELK Stack with Filebeat, will aggregate NGINX, Tomcat, and Java Log4j log entries, providing debugging and analytics to our demonstration. A similar method for aggregating logs, using Logspout instead of Filebeat, can be found in this previous post.
Continuous Integration
In this post’s example, two build artifacts, a WAR file for the application and ZIP file for the static web content, are built automatically by Travis CI, whenever source code changes are pushed to the springmusic_v2
branch of the garystafford/spring-music repository on GitHub.
Following a successful build and a small number of unit tests, Travis CI pushes the build artifacts to the build-artifacts
branch on the same GitHub project. The build-artifacts
branch acts as a pseudo binary repository for the project, much like JFrog’s Artifactory. These artifacts are used later by Docker to build the project’s immutable Docker images and containers.
Build Notifications
Travis CI pushes build notifications to a Slack channel, which eliminates the need to actively monitor Travis CI.
Automation Scripting
The .travis.yaml
file, custom gradle.build
Gradle tasks, and the deploy_travisci.sh
script handles the Travis CI automation described, above.
Travis CI .travis.yaml
file:
language: java | |
jdk: oraclejdk8 | |
before_install: | |
- chmod +x gradlew | |
before_deploy: | |
- chmod ugo+x deploy_travisci.sh | |
script: | |
- "./gradlew clean build" | |
- "./gradlew warNoStatic warCopy zipGetVersion zipStatic" | |
- sh ./deploy_travisci.sh | |
env: | |
global: | |
- GH_REF: github.com/garystafford/spring-music.git | |
- secure: <GH_TOKEN_secure_hash_here> | |
- secure: <COMMIT_AUTHOR_EMAIL_secure_hash_here> | |
notifications: | |
slack: | |
- secure: <SLACK_secure_hash_here> |
Custom gradle.build
tasks:
// new Gradle build tasks | |
task warNoStatic(type: War) { | |
// omit the version from the war file name | |
version = '' | |
exclude '**/assets/**' | |
manifest { | |
attributes | |
'Manifest-Version': '1.0', | |
'Created-By': currentJvm, | |
'Gradle-Version': GradleVersion.current().getVersion(), | |
'Implementation-Title': archivesBaseName + '.war', | |
'Implementation-Version': artifact_version, | |
'Implementation-Vendor': 'Gary A. Stafford' | |
} | |
} | |
task warCopy(type: Copy) { | |
from 'build/libs' | |
into 'build/distributions' | |
include '**/*.war' | |
} | |
task zipGetVersion (type: Task) { | |
ext.versionfile = | |
new File("${projectDir}/src/main/webapp/assets/buildinfo.properties") | |
versionfile.text = 'build.version=' + artifact_version | |
} | |
task zipStatic(type: Zip) { | |
from 'src/main/webapp/assets' | |
appendix = 'static' | |
version = '' | |
} |
The deploy.sh
file:
#!/bin/bash | |
set -e | |
cd build/distributions | |
git init | |
git config user.name "travis-ci" | |
git config user.email "${COMMIT_AUTHOR_EMAIL}" | |
git add . | |
git commit -m "Deploy Travis CI Build #${TRAVIS_BUILD_NUMBER} artifacts to GitHub" | |
git push --force --quiet "https://${GH_TOKEN}@${GH_REF}" master:build-artifacts > /dev/null 2>&1 |
You can easily replicate the project’s continuous integration automation using your choice of toolchains. GitHub or BitBucket are good choices for distributed version control. For continuous integration and deployment, I recommend Travis CI, Semaphore, Codeship, or Jenkins. Couple those with a good persistent chat application, such as Glider Labs’ Slack or Atlassian’s HipChat.
Building the Docker Environment
Make sure VirtualBox, Docker, Docker Compose, and Docker Machine, are installed and running. At the time of this post, I have the following versions of software installed on my Mac:
- Mac OS X 10.11.6
- VirtualBox 5.0.26
- Docker 1.12.1
- Docker Compose 1.8.0
- Docker Machine 0.8.1
To build the project’s VirtualBox VM, Docker images, and Docker containers, execute the build script, using the following command: sh ./build_project.sh
. A build script is useful when working with CI/CD automation tools, such as Jenkins CI or ThoughtWorks go. However, to understand the build process, I suggest first running the individual commands, locally.
#!/bin/sh | |
set -ex | |
# clone project | |
git clone -b docker_v2 --single-branch \ | |
https://github.com/garystafford/spring-music-docker.git music \ | |
&& cd "$_" | |
# provision VirtualBox VM | |
docker-machine create --driver virtualbox springmusic | |
# set new environment | |
docker-machine env springmusic \ | |
&& eval "$(docker-machine env springmusic)" | |
# mount a named volume on host to store mongo and elk data | |
# ** assumes your project folder is 'music' ** | |
docker volume create --name music_data | |
docker volume create --name music_elk | |
# create bridge network for project | |
# ** assumes your project folder is 'music' ** | |
docker network create -d bridge music_net | |
# build images and orchestrate start-up of containers (in this order) | |
docker-compose -p music up -d elk && sleep 15 \ | |
&& docker-compose -p music up -d mongodb && sleep 15 \ | |
&& docker-compose -p music up -d app \ | |
&& docker-compose scale app=3 && sleep 15 \ | |
&& docker-compose -p music up -d proxy && sleep 15 | |
# optional: configure local DNS resolution for application URL | |
#echo "$(docker-machine ip springmusic) springmusic.com" | sudo tee --append /etc/hosts | |
# run a simple connectivity test of application | |
for i in {1..9}; do curl -I $(docker-machine ip springmusic); done |
Deploying to AWS
By simply changing the Docker Machine driver to AWS EC2 from VirtualBox, and providing your AWS credentials, the springmusic
environment may also be built on AWS.
Build Process
Docker Machine provisions a single VirtualBox springmusic
VM on which host the project’s containers. VirtualBox provides a quick and easy solution that can be run locally for initial development and testing of the application.
Next, the script creates a Docker data volume and project-specific Docker bridge network.
Next, using the project’s individual Dockerfiles, Docker Compose pulls base Docker images from Docker Hub for NGINX, Tomcat, ELK, and MongoDB. Project-specific immutable Docker images are then built for NGINX, Tomcat, and MongoDB. While constructing the project-specific Docker images for NGINX and Tomcat, the latest Spring Music build artifacts are pulled and installed into the corresponding Docker images.
Docker Compose builds and deploys (6) containers onto the VirtualBox VM: (1) NGINX, (3) Tomcat, (1) MongoDB, and (1) ELK.
The NGINX Dockerfile
:
# NGINX image with build artifact | |
FROM nginx:latest | |
MAINTAINER Gary A. Stafford <garystafford@rochester.rr.com> | |
ENV REFRESHED_AT 2016-09-17 | |
ENV GITHUB_REPO https://github.com/garystafford/spring-music/raw/build-artifacts | |
ENV STATIC_FILE spring-music-static.zip | |
RUN apt-get update -qq \ | |
&& apt-get install -qqy curl wget unzip nano \ | |
&& apt-get clean \ | |
\ | |
&& wget -O /tmp/${STATIC_FILE} ${GITHUB_REPO}/${STATIC_FILE} \ | |
&& unzip /tmp/${STATIC_FILE} -d /usr/share/nginx/assets/ | |
COPY default.conf /etc/nginx/conf.d/default.conf | |
# tweak nginx image set-up, remove log symlinks | |
RUN rm /var/log/nginx/access.log /var/log/nginx/error.log | |
# install Filebeat | |
ENV FILEBEAT_VERSION=filebeat_1.2.3_amd64.deb | |
RUN curl -L -O https://download.elastic.co/beats/filebeat/${FILEBEAT_VERSION} \ | |
&& dpkg -i ${FILEBEAT_VERSION} \ | |
&& rm ${FILEBEAT_VERSION} | |
# configure Filebeat | |
ADD filebeat.yml /etc/filebeat/filebeat.yml | |
# CA cert | |
RUN mkdir -p /etc/pki/tls/certs | |
ADD logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt | |
# start Filebeat | |
ADD ./start.sh /usr/local/bin/start.sh | |
RUN chmod +x /usr/local/bin/start.sh | |
CMD [ "/usr/local/bin/start.sh" ] |
The Tomcat Dockerfile
:
# Apache Tomcat image with build artifact | |
FROM tomcat:8.5.4-jre8 | |
MAINTAINER Gary A. Stafford <garystafford@rochester.rr.com> | |
ENV REFRESHED_AT 2016-09-17 | |
ENV GITHUB_REPO https://github.com/garystafford/spring-music/raw/build-artifacts | |
ENV APP_FILE spring-music.war | |
ENV TERM xterm | |
ENV JAVA_OPTS -Djava.security.egd=file:/dev/./urandom | |
RUN apt-get update -qq \ | |
&& apt-get install -qqy curl wget \ | |
&& apt-get clean \ | |
\ | |
&& touch /var/log/spring-music.log \ | |
&& chmod 666 /var/log/spring-music.log \ | |
\ | |
&& wget -q -O /usr/local/tomcat/webapps/ROOT.war ${GITHUB_REPO}/${APP_FILE} \ | |
&& mv /usr/local/tomcat/webapps/ROOT /usr/local/tomcat/webapps/_ROOT | |
COPY tomcat-users.xml /usr/local/tomcat/conf/tomcat-users.xml | |
# install Filebeat | |
ENV FILEBEAT_VERSION=filebeat_1.2.3_amd64.deb | |
RUN curl -L -O https://download.elastic.co/beats/filebeat/${FILEBEAT_VERSION} \ | |
&& dpkg -i ${FILEBEAT_VERSION} \ | |
&& rm ${FILEBEAT_VERSION} | |
# configure Filebeat | |
ADD filebeat.yml /etc/filebeat/filebeat.yml | |
# CA cert | |
RUN mkdir -p /etc/pki/tls/certs | |
ADD logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt | |
# start Filebeat | |
ADD ./start.sh /usr/local/bin/start.sh | |
RUN chmod +x /usr/local/bin/start.sh | |
CMD [ "/usr/local/bin/start.sh" ] |
Docker Compose v2 YAML
This post was recently updated for Docker 1.12, and to use Docker Compose v2 YAML file format. The post’s docker-compose.yml
takes advantage of improvements in Docker 1.12 and Docker Compose v2 YAML. Improvements to the YAML file include eliminating the need to link containers and expose ports, and the addition of named networks and volumes.
version: '2' | |
services: | |
proxy: | |
build: nginx/ | |
ports: | |
- 80:80 | |
networks: | |
- net | |
depends_on: | |
- app | |
hostname: proxy | |
container_name: proxy | |
app: | |
build: tomcat/ | |
ports: | |
- 8080 | |
networks: | |
- net | |
depends_on: | |
- mongodb | |
hostname: app | |
mongodb: | |
build: mongodb/ | |
ports: | |
- 27017:27017 | |
networks: | |
- net | |
depends_on: | |
- elk | |
hostname: mongodb | |
container_name: mongodb | |
volumes: | |
- music_data:/data/db | |
- music_data:/data/configdb | |
elk: | |
image: sebp/elk:latest | |
ports: | |
- 5601:5601 | |
- 9200:9200 | |
- 5044:5044 | |
- 5000:5000 | |
networks: | |
- net | |
volumes: | |
- music_elk:/var/lib/elasticsearch | |
hostname: elk | |
container_name: elk | |
volumes: | |
music_data: | |
external: true | |
music_elk: | |
external: true | |
networks: | |
net: | |
driver: bridge |
The Results
Below are the results of building the project.
# Resulting Docker Machine VirtualBox VM: | |
$ docker-machine ls | |
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS | |
springmusic * virtualbox Running tcp://192.168.99.100:2376 v1.12.1 | |
# Resulting external volume: | |
$ docker volume ls | |
DRIVER VOLUME NAME | |
local music_data | |
local music_elk | |
# Resulting bridge network: | |
$ docker network ls | |
NETWORK ID NAME DRIVER SCOPE | |
f564dfa1b440 music_net bridge local | |
# Resulting Docker images - (4) base images and (3) project images: | |
$ docker images | |
REPOSITORY TAG IMAGE ID CREATED SIZE | |
music_proxy latest 7a8dd90bcf32 About an hour ago 250.2 MB | |
music_app latest c93c713d03b8 About an hour ago 393 MB | |
music_mongodb latest fbcbbe9d4485 25 hours ago 366.4 MB | |
tomcat 8.5.4-jre8 98cc750770ba 2 days ago 334.5 MB | |
mongo latest 48b8b08dca4d 2 days ago 366.4 MB | |
nginx latest 4efb2fcdb1ab 10 days ago 183.4 MB | |
sebp/elk latest 07a3e78b01f5 13 days ago 884.5 MB | |
# Resulting (6) Docker containers | |
$ docker ps | |
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | |
b33922767517 music_proxy "/usr/local/bin/start" 3 hours ago Up 13 minutes 0.0.0.0:80->80/tcp, 443/tcp proxy | |
e16d2372f2df music_app "/usr/local/bin/start" 3 hours ago Up About an hour 0.0.0.0:32770->8080/tcp music_app_3 | |
6b7accea7156 music_app "/usr/local/bin/start" 3 hours ago Up About an hour 0.0.0.0:32769->8080/tcp music_app_2 | |
2e94f766df1b music_app "/usr/local/bin/start" 3 hours ago Up About an hour 0.0.0.0:32768->8080/tcp music_app_1 | |
71f8dc574148 sebp/elk:latest "/usr/local/bin/start" 3 hours ago Up About an hour 0.0.0.0:5000->5000/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5601->5601/tcp, 0.0.0.0:9200->9200/tcp, 9300/tcp elk | |
f7e7d1af7cca music_mongodb "/entrypoint.sh mongo" 20 hours ago Up About an hour 0.0.0.0:27017->27017/tcp mongodb |
Testing the Application
Below are partial results of the curl test, hitting the NGINX endpoint. Note the different IP addresses in the Upstream-Address
field between requests. This test proves NGINX’s round-robin load-balancing is working across the three Tomcat application instances: music_app_1
, music_app_2
, and music_app_3
.
Also, note the sharp decrease in the Request-Time
between the first three requests and subsequent three requests. The Upstream-Response-Time
to the Tomcat instances doesn’t change, yet the total Request-Time
is much shorter, due to caching of the application’s static assets by NGINX.
for i in {1..6}; do curl -I $(docker-machine ip springmusic);done | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:50 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.575 | |
Upstream-Address: 172.18.0.4:8080 | |
Upstream-Response-Time: 1474137230.048 | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:51 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.711 | |
Upstream-Address: 172.18.0.5:8080 | |
Upstream-Response-Time: 1474137230.865 | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:52 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.326 | |
Upstream-Address: 172.18.0.6:8080 | |
Upstream-Response-Time: 1474137231.812 | |
# assets now cached... | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:53 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.012 | |
Upstream-Address: 172.18.0.4:8080 | |
Upstream-Response-Time: 1474137233.111 | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:53 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.017 | |
Upstream-Address: 172.18.0.5:8080 | |
Upstream-Response-Time: 1474137233.350 | |
HTTP/1.1 200 | |
Server: nginx/1.11.4 | |
Date: Sat, 17 Sep 2016 18:33:53 GMT | |
Content-Type: text/html;charset=ISO-8859-1 | |
Content-Length: 2094 | |
Connection: keep-alive | |
Accept-Ranges: bytes | |
ETag: W/"2094-1473924940000" | |
Last-Modified: Thu, 15 Sep 2016 07:35:40 GMT | |
Content-Language: en | |
Request-Time: 0.013 | |
Upstream-Address: 172.18.0.6:8080 | |
Upstream-Response-Time: 1474137233.594 |
Spring Music Application Links
Assuming the springmusic
VM is running at 192.168.99.100
, the following links can be used to access various project endpoints. Note the (3) Tomcat instances each map to randomly exposed ports. These ports are not required by NGINX, which maps to port 8080 for each instance. The port is only required if you want access to the Tomcat Web Console. The port, shown below, 32771, is merely used as an example.
- Spring Music Application: 192.168.99.100
- NGINX Status: 192.168.99.100/nginx_status
- Tomcat Web Console – music_app_1*: 192.168.99.100:32771/manager
- Environment Variables – music_app_1: 192.168.99.100:32771/env
- Album List (RESTful endpoint) – music_app_1: 192.168.99.100:32771/albums
- Elasticsearch Info: 192.168.99.100:9200
- Elasticsearch Status: 192.168.99.100:9200/_status?pretty
- Kibana Web Console: 192.168.99.100:5601
* The Tomcat user name is admin
and the password is t0mcat53rv3r
.
Helpful Links
- Cloud Foundry’s Spring Music Example
- Getting Started with Gradle for Java
- Introduction to Gradle
- Spring Framework
- Understanding Nginx HTTP Proxying, Load Balancing, Buffering, and Caching
- Common conversion patterns for log4j’s PatternLayout
- Spring @PropertySource example
- Java log4j logging
TODOs
- Automate the Docker image build and publish processes
- Automate the Docker container build and deploy processes
- Automate post-deployment verification testing of project infrastructure
- Add Docker Swarm multi-host capabilities with overlay networking
- Update Spring Music with latest CF project revisions
- Include scripting example to stand-up project on AWS
- Add Consul and Consul Template for NGINX configuration
Containerized Microservice Log Aggregation and Visualization using ELK Stack and Logspout
Posted by Gary A. Stafford in Build Automation, Continuous Delivery, DevOps, Enterprise Software Development on August 2, 2015
Log aggregation, visualization, analysis, and monitoring of Dockerized microservices using the ELK Stack (Elasticsearch, Logstash, and Kibana) and Logspout
Introduction
In the last series of posts, we learned how to use Jenkins CI, Maven, Docker, Docker Compose, and Docker Machine to take a set of Java-based microservices from source control on GitHub, to a fully tested set of integrated Docker containers running within an Oracle VirtualBox VM. We performed integration tests, using a scripted set of synthetic transactions, to make sure the microservices were functioning as expected, within their containers.
In this post, we will round out our Virtual-Vehicles microservices REST API project by adding log aggregation, visualization, analysis, and monitoring, using the ELK Stack (Elasticsearch, Logstash, and Kibana) and Logspout.
All code for this post is available on GitHub, release version v3.1.0 on the ‘master’ branch (after running ‘git clone …’, run a ‘git checkout tags/v3.1.0’ command).
Logging
If you’re using Docker, then you’re familiar with the command, ‘docker logs container-name command
‘. This command streams the log output of running services within a container, commonly used to debugging and troubleshooting. It sure beats ‘docker exec -it container-name cat /var/logs/foo/foo.log
‘ and so on, for each log we need to inspect within a container.
With Docker Compose, we gain the command, ‘docker-compose logs
‘. This command stream the log output of running services, of all containers defined in our ‘docker-compose.yml
‘ file. Although moderately more useful for debugging, I’ve also found it fairly buggy when used with Docker Machine and Docker Swarm.
As helpful as these type of Docker commands are, when you start scaling from one container, to ten containers, to hundreds of containers, individually inspecting container logs from the command line is time-consuming and of little value. Correlating log events between containers is impossible. That’s where solutions such as the ELK Stack and Logspout really shine for containerized environments.
ELK Stack
Although not specifically designed for the purpose, the ELK Stack (Elasticsearch, Logstash, and Kibana) is an ideal tool-chain for log aggregation, visualization, analysis, and monitoring. Individually setting up Elasticsearch, Logstash, and Kibana, and configuring them to communicate with each other is not a small task. Luckily, there are several ready-made Docker images on Docker Hub, whose authors have already done much of the hard work for us. After trying several ELK containers on Docker Hub, I chose on the willdurand/elk image. This image is easy to get started with, and is easily used to build containers using Docker Compose.
Logspout
Using the ELK Stack, we have a way to collect (Logstash), store and search (Elasticsearch), and visualize and analyze (Kibana) our container’s log events. Although Logstash is capable of collecting our log events, to integrate more easily with Docker, we will add another component, Glider Lab’s Logspout, to our tool-chain. Logspout advertises itself as “a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system.”
Since Logspout is extensible through third-party modules, we will use one last component, Loop Lab’s Logspout/Logstash Adapter. Written in the go programming language, the adapter is described as “a minimalistic adapter for Glider Lab’s Logspout to write to Logstash UDP”. This adapter will allow us to collect Docker’s log events with Logspout and send them to Logstash using User Datagram Protocol (UDP).
In order to use the Logspout/Logstash adapter, we need to build a Logspout container from the /logspout Docker image, which contains a customized version of Logspout’s modules.go
configuration file. This is explained in the Custom Logspout Builds section of Logspout’s README.md. Below is the modified configuration module with the addition of the adapter (see last import statement).
package main import ( _ "github.com/gliderlabs/logspout/adapters/raw" _ "github.com/gliderlabs/logspout/adapters/syslog" _ "github.com/gliderlabs/logspout/httpstream" _ "github.com/gliderlabs/logspout/routesapi" _ "github.com/gliderlabs/logspout/httpstream" _ "github.com/gliderlabs/logspout/transports/udp" _ "github.com/looplab/logspout-logstash" )
One note with Logspout, according to their website, for now it Logspout only captures stdout and stderr, but a module to collect container syslog is planned. Although syslog is common centralized log collection method, the Docker logs we will collect are sent to stdout and stderr, the lack of syslog support is not a limitation for us, in this demonstration.
We will configure Logstash to accept log events from Logspout, using UDP on port 5000. Below is an abridged version of the logstash-logspout-log4j2.conf
configuration file. The except from the configuration file, below, instructs Logstash to listen for Logspout’s messages over UDP on port 5000, and passes them to Elasticsearch.
input { udp { port => 5000 codec => json type => "dockerlogs" } # filtering section not shown... output { elasticsearch { protocol => "http" } stdout { codec => rubydebug } }
We could spend several posts on the configuration of Logstash. There are an infinite number of input, filter, and output combinations, to collect, transform, and push log events to various programs, including Logstash. The filtering section alone takes some time to learn exactly how to filter and transform log events, based upon the requirements for visualization and analysis.
Apache Log4j Logs
What about our Virtual-Vehicle microservice’s Log4j 2 logs? In the previous posts, you’ll recall we were sending our log events to physical log files within each container, using Log4j’s Rolling File appender.
<Appenders> <RollingFile name="RollingFile" fileName="${log-path}/virtual-authentication.log" filePattern="${log-path}/virtual-authentication-%d{yyyy-MM-dd}-%i.log" > <PatternLayout> <pattern>%d{dd/MMM/yyyy HH:mm:ss,SSS}- %c{1}: %m%n</pattern> </PatternLayout> <Policies> <SizeBasedTriggeringPolicy size="1024 KB" /> </Policies> <DefaultRolloverStrategy max="4"/> </RollingFile> </Appenders>
Given the variety of appenders available with Log4j 2, we have a few options to leverage the ELK Stack with these logs events. The least disruptive change would be to send the Log4j log events to Logspout by redirecting Log4j output from the physical log file to stdout. We could do this by running a Linux link command in each microservice’s Dockerfile, as in the following example with Authentication microservice.
RUN touch /var/log/virtual-authentication.log && \ ln -sf /dev/stdout /var/log/virtual-authentication.log
This method would not require us to change the log4j2.xml
configuration files, and rebuild the services. However, the alternative we will use in this post is switching to Log4j’s Syslog appender. According to Log4j documentation, the Syslog appender is a Socket appender that writes its output to a remote destination specified by a host and port in a format that conforms with either the BSD Syslog format or the RFC 5424 format. The data can be sent over either TCP or UDP.
To use the Syslog appender option, we do need to change each log4j2.xml
configuration file, and then rebuild each of the microservices. Instead of using UDP over port 5000, which is the port Logspout is currently using to communicate with Logstash, we will use UDP over port 5001. Below is a sample of the log4j2.xml
configuration files for the Authentication microservice.
<Appenders> <Syslog name="RFC5424" format="RFC5424" host="elk" port="5001" protocol="UDP" appName="virtual-authentication" includeMDC="true" facility="SYSLOG" enterpriseNumber="18060" newLine="true" messageId="log4j2" mdcId="mdc" id="App" connectTimeoutMillis="1000" reconnectionDelayMillis="5000"> <LoggerFields> <KeyValuePair key="thread" value="%t"/> <KeyValuePair key="priority" value="%p"/> <KeyValuePair key="category" value="%c"/> <KeyValuePair key="exception" value="%ex"/> <KeyValuePair key="message" value="%m"/> </LoggerFields> </Syslog> </Appenders>
To communicate with Logstash over port 5001 with the Syslog appender, we also need to modify the logstash-logspout-log4j2.conf
configuration file, again. Below is the unabridged version of the configuration file, with both the Logspout (UDP port 5000) and Log4j (UDP port 5001) configurations.
input { udp { port => 5000 codec => json type => "dockerlogs" } udp { type => "log4j2" port => 5001 } } filter { if [type] == "log4j2" { mutate { gsub => ['message', "\n", " "] gsub => ['message', "\t", " "] } } if [type] == "dockerlogs" { if ([message] =~ "^\tat ") { drop {} } grok { break_on_match => false match => [ "message", " responded with %{NUMBER:status_code:int}" ] tag_on_failure => [] } grok { break_on_match => false match => [ "message", " in %{NUMBER:response_time:int}ms" ] tag_on_failure => [] } } } output { elasticsearch { protocol => "http" } stdout { codec => rubydebug } }
You will note some basic filtering in the configuration. I will touch upon this in the next section. Below is a diagram showing the complete flow of log events from both Log4j and from the Docker containers to Logspout and the ELK Stack.
Troubleshooting and Debugging
Trying to troubleshoot why log events may not be showing up in Kibana can be frustrating, without methods to debug the flow of log events along with way. Were the stdout Docker log events successfully received by Logspout? Did Logspout successfully forward the log events to Logstash? Did Log4j successfully push the microservice’s log events to Logstash? Probably the most frustrating of all issues, did you properly configure the Logstash configuration file(s) to receive, filter, transform, and push the log events to Elasticsearch. I spent countless hours debugging filtering, alone. Luckily, there are several ways to ensure log events are flowing. The below diagram shows some of the debug points along the way.
First, we can check that the log events are making to Logspout from Docker by cURLing or browsing port 8000. Executing ‘curl -X GET --url http://api.virtual-vehicles.com:8000/logs
‘ will tail incoming log events received to Logspout. You should see log events flowing into Logspout as you call the microservices through NGINX, by running the project’s integration tests, as shown in the example, below.
Second, we can cURL or browse port 9200. This port will display information about Elasticsearch. There are several useful endpoints exposed by Elasticsearch’s REST API interface. Executing ‘curl -X GET --url http://api.virtual-vehicles.com:9200/_status?pretty
‘ will display statistics about Elasticsearch, including the number of log events, referred to as ‘documents’ to Elasticsearch’s structured JSON document-based NoSQL datastore. Note the line, ‘"num_docs": 469
‘, indicating 469 log events were captured by Elasticsearch as documents.
{ "_shards": { "total": 32, "successful": 16, "failed": 0 }, "indices": { "logstash-2015.08.01": { "index": { "primary_size_in_bytes": 525997, "size_in_bytes": 525997 }, "translog": { "operations": 492 }, "docs": { "num_docs": 469, "max_doc": 469, "deleted_docs": 0 } } } }
If you find log events are not flowing into Logstash, a quick way to start debugging issues is to check Logstash’s log:
docker exec -it jenkins_elk_1 cat /var/log/logstash/stdout.log
If you find log events are flowing into Logstash, but not being captured by Elasticsearch, it’s probably your Logstash configuration file. Either the input, filter, and/or output sections are wrong. A quick way to debug these types of issues is to check Elasticsearch’s log. I’ve found this log often contains useful and specific error messages, which can help fix Logstash configuration issues.
docker exec -it jenkins_elk_1 cat /var/log/elasticsearch/logstash.log
Without log event documents in Elasticsearch, there is no sense moving on to Kibana. Kibana will have no data available to display.
Kibana
If you recall from our last post, the project already has Graphite and StatsD configured and running, as shown below. On its own, Graphite provides important monitoring and performance information about our microservices. In fact, we could choose to also send all our Docker log events, through Logstash, to Graphite. This would require some additional filtering and output configuration.
However, our main interest in this post is the ELK Stack. The way we visualize and analyze the log events we have captured is through Kibana. Kibana resembles other popular log aggregators and log search and analysis products, like Splunk, Graylog, and Sumo Logic. I suggest you familiarize yourself with Kibana before diving into the this part of the demonstration. Kibana can be confusing at first, if you are not familiar with it’s indexing, discovery, and search features.
We can access Kibana from our browser, at port 8200, ‘http://api.virtual-vehicles.com:8200
‘. The first interactions with Kibana will be through the Discover view, as seen in the screen grab shown below. Kibana displays the typical vertical bar chart event timeline, based on log event timestamps. The details of each log event are displayed below the timeline. You can filter and search within this view. Searches can be saved and used later.
Heck, just the ability to view and search all our log events in one place is a huge improvement over the command line. If you look a little closer at the actual log events, as shown below, you will notice two types, ‘dockerlogs
‘ and ‘log4j2
‘. Looking at the Logstash configuration file again, shown previously, you see we applied the ‘type
‘ tag to the log events as they were being processed by Logstash.
In the Logstash configuration file, shown previously, you will also note the use of a few basic filters. I created a ‘status_code
‘ and ‘response_time
‘ filter, specifically for the Docker log events. Each Docker log event is passed through the filters. The two fields, ‘status_code
‘ and ‘response_time
‘, are extracted from the main log event text and added as separate, indexable, and searchable fields. Below is an example of one such Docker log event, an HTTP DELETE call to the Valet microservice, shown as JSON. Note the two fields, showing a response time of 13ms and a http status code of 204.
{ "_index": "logstash-2015.08.01", "_type": "dockerlogs", "_id": "AU7rcyxTA4OY8JukKyIv", "_score": null, "_source": { "message": "DELETE http://api.virtual-vehicles.com/valets/55bd30c2e4b0818a113883a6 responded with 204 No Content in 13ms", "docker.name": "/jenkins_valet_1", "docker.id": "7ef368f9fdca2d338786ecd8fe612011aebbfc9ad9b677c21578332f7c46cf2b", "docker.image": "jenkins_valet", "docker.hostname": "7ef368f9fdca", "@version": "1", "@timestamp": "2015-08-01T22:47:49.649Z", "type": "dockerlogs", "host": "172.17.0.7", "status_code": 204, "response_time": 13 }, "fields": { "@timestamp": [ 1438469269649 ] }, "sort": [ 1438469269649 ] }
For comparison, here is a sample Log4j 2 log event, generated by a JsonParseException
. Note the different field structure. With more time spent modifying the Log4j event format, and configuring Logstash’s filtering and transforms, we could certainly improve the usability of Log4j log events.
{ "_index": "logstash-2015.08.02", "_type": "log4j2", "_id": "AU7wJt8zA4OY8JukKyrt", "_score": null, "_source": { "message": "<43>1 2015-08-02T20:42:35.067Z bc45ce804859 virtual-authentication - log4j2 [mdc@18060 category=\"com.example.authentication.objectid.JwtController\" exception=\"\" message=\"validateJwt() failed: JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value at [Source: java.io.StringReader@12a24457; line: 1, column: 27\\]\" priority=\"ERROR\" thread=\"nioEventLoopGroup-3-9\"] validateJwt() failed: JsonParseException: Unexpected end-of-input: was expecting closing quote for a string value at [Source: java.io.StringReader@12a24457; line: 1, column: 27] ", "@version": "1", "@timestamp": "2015-08-02T20:42:35.188Z", "type": "log4j2", "host": "172.17.0.9" }, "fields": { "@timestamp": [ 1438548155188 ] }, "sort": [ 1438548155188 ] }
Kibana Dashboard
To demonstrate the visualization capabilities of Kibana, we will create a Dashboard. Our Dashboard will be composed of a series of Kibana Visualizations. Visualizations are charts, graphs, tables, and metrics, based on the log events we see in the Discovery view. Below, I have created a rather basic Dashboard, containing some simple data visualization, based on our Docker and Log4j log events, collected over a 1-hour period. This one small screen-grab does not begin to do justice to the real power of Kibana.
In the dashboard above, you see a few basic metrics, such as request response times, response http status code, a chart of which containers are logging events, a graph that shows log events captured per minute, and so forth. Along with Searches, Visualizations and Dashboards can also be saved in Kibana. Note this demonstration’s Docker Compose YAML file does not configure volume mapping between the containers and host. If you destroy the containers, you destroy anything you saved in Kibana.
A key feature of Kibana’s Dashboards is their interactive capabilities. Rolling over any piece of a Visualization brings up an informative pop-up with additional details. For example, as shown below, rolling over the http status code ‘500’ pie chart slice, pops up the number of status code 500 responses. In this case, 15 log events, or 1.92% of the total 2,994 log events captured, had a ‘status_code’ field of ‘500’, within the 24-hour period the Dashboard analyzed.
Conveniently, Kibana also allows you to switch from a visual mode to a data table mode, for any Visualization on the Dashboard, as shown below, for a 24-hour period.
Conclusion
The ELK Stack is just one of a number of enterprise-class tools available to monitor and analyze the overall health of your applications running within a Dockerized environment. Having well planned logging, monitoring, and analytics strategies is key to this type of project. They should be implemented from the beginning of the project, to increase development and testing velocity, as well as provide quick troubleshooting, key business metrics, and proactive monitoring, once the application is in production.
Notes on Running the GitHub Project
If you download and run this project from GitHub, there is two key steps you should note. First, you need add an entry to your local /etc/hosts
file. The IP address will be that of the Docker Machine VM, ‘test’. The hostname is ‘api.virtual-vehicles.com’. which matches the one I used throughout the demo. You should run the following bash command before building your containers from the docker-compose.yml
file, but after you have built your VM using Docker Machine. The ‘test’ VM must already exist.
echo "$(docker-machine ip test) api.virtual-vehicles.com" \ | sudo tee --append /etc/hosts
If you want to override this domain name with your own, you will need to modify and re-build the microservices project, first. Then, copy those build artifacts into this project, replacing the ones you pulled from GitHub.
Second, in order to achieve HATEOAS in my REST API responses, I have included some variables in my docker-compose.yml
file. Wait, docker-compose.yml
doesn’t support variables? Well, it can if you use a template file (docker-compose-template.yml
) and run a script (compose_replace.sh
) to provide variable expansion. My gist explains the technique a little better.
You should also run this command before building your containers from the docker-compose.yml
file, but after you have built your VM using Docker Machine. Again, the ‘test’ VM must already exist.
sh compose_replace.sh
Lastly, remember, we can run our integration tests to generate log events, using the following command.
sh tests_color.sh api.virtual-vehicles.com