BLACK BOX APPLICATION MONITORING WITH NAGIOS AND DOCKER - AT HOME !

This guide assumes you have a working installation of Docker!

This blog post will show how to monitor a web service running in a Docker container using black box monitoring techniques with Nagios.

Monitoring is:

Examining your software in the act of running in production;

Looking for error conditions and recording data for trending purposes;

Being able to correlate events for troubleshooting;

Alerting and possibly recovering when things go wrong.

The very final goal of monitoring application service is to lower your MTTDetect, MTTDiagnose, MTTTRepair where MTT == Mean Time To.

There are two kind of monitoring techniques: black box and white box.

Black box techniques take in consideration the global state os the service as a whole. 

Black-box is:

Simulated queries.

Verifies end-to-end operation.

You really don’t have anything except the input and output. 

Important to monitor all sets of dependencies. e.g. DNS. "Single points of failure." 

White-box is:

Monitoring where you know about the inside of the app. 

You may have access to the source or other instrumentation 

You can evaluate metrics derived from instrumentation.

In this post we'll recreate a production environment using three docker virtual machines.

The first virtual machine, we are going to call www, will contain a car service web application. Technically speaking this is a docker container running Jboss with a Car Service web application deployed in it.

The first step is to pull this virtual machine from docker hub:

docker pull jwasilewski/lab4-www

and run it with:

docker run -i -t --name www -p 8080:8080 jwasilewski/lab4-www /bin/bash

And starting Jboss:

/opt/wildfly/bin/standalone.sh -c standalone-full.xml -b 0.0.0.0

We can get the ip address of boot2docker (is you are running Docker on a Mac like me) with the following command:

boot2docker ip

Which returns on my machine:

192.168.59.103

If we open the browser at 192.168.59.103:8080 we should now see the web application up and running: 

The second step will be to pull a Nagios docker container from docker hub with:

docker pull cpuguy83/nags

Nagios is an open source computer system monitoring, network monitoring and infrastructure monitoring software application. Nagios offers monitoring and alerting services for servers, switches, applications, and services. It alerts the users when things go wrong and alerts them a second time when the problem has been resolved.

In starting this docker container, we want to link it with our web-application because they will need to continuously communicate with each other:

sudo docker run -i -t  --name nagios -p 80:80 --link www:www cpuguy83/nagios /bin/bash

The username and password for the Nagios Docker images are:

username = nagiosadmin

password = nagios

Let’s make sure that the Nagios container can communicate with the web application docker container (www), trying to ping the latter from the Nagios container shell: 

Let’s also make sure that the Nagios docker container can directly contact the Jboss web application with a simple wget;

The index.html page only represent the WildFly landing page:

It is now time to create a couple of checks in the Nagios container.

In the folder objects, let’s create a new file called www.cfg. This file will contain our nagios check:

cd /opt/nagios/etc/objects

vi www.cfg

So now we have to tell Nagios that there is a new configuration file to read; We have to edit the file /opt/nagios/etc/nagios.cfg: 

We can finally start Nagios:

/usr/local/bin/start_nagios

We should now be able to see Nagios user interface at http://192.168.59.103:80, where 192.168.59.103 is the Boot2Docker host Ip address: 

The image below is a snapshot of the “Hosts” Nagios page, that shows that localhost and www (which is our web Docker Container) are both up at moment. This is a “ping” check ! 

The next images is a snapshot of the “Services” page that shows that we are monitoring the Jboss service inside the “www” docker container.
 This is a Http Check on the port 8080 which is the port where Jboss is currently running: 

Then we go back to our “www” virtual machine and we shutdown Jboss. Nagios, in the next check iteration, will show Jboss down as shown in the image below:

Many other checks can be performed using Nagios.

Some of them are listed here.