Monitoring your Docker stack with Grafana and Prometheus

Have you ever dreamt of having smooth looking dashboards showing you the status of your servers in real time? Look no further! Grafana and Prometheus are really awesome at exactly that! Plus, you get alerting whenever any parameter you define stray outside acceptable limits. And the really great news is; it isn't that difficult!

So, what is Grafana and Prometheus? In general terms, Grafana is a visualizing tool, whereas Prometheus is a database for time series. Prometheus also happens to be a very well supported back-end for Grafana, but Grafana can use back-ends like New Relic or Elasticsearch. If you have lots of, lots of services you'd like to monitor, there are other solutionts that scale better. However, Prometheus has sort of been scaled up to 1M machines.

If you do go with Grafana+Prometheus, which I recommend for the kind of setup I have, you will need a little more than just those to components. Prometheus requires that you publish monitoring data on a supported format somewhere. When you've taken care of that, Prometheus can poll that data on intervals you define. Fortunately, there are tons of pre-made data exporters out there. Two of the most useful exporters for a Docker stack are the built-in exporter in cAdvisor, plus the Prometheus node exporter.

The cAdvisor exporter primarily gives you metrics and live status for your docker containers, while the Prometheus node exporter gives you metrics on your host system, or whatever server or container the node exporter is installed on. Keep in mind that it's NOT recommended to install the node exporter in a Docker container, especially if the main goal is to monitor the hosts resource consumption.

To explore the data, Prometheus has its own query language, PromQL. That's also how you get data for your visualizations in Grafana, and that's how you define conditions for when alerts should be triggered. If you'd like being notified of said alerts, you should also set up the Prometheus Alertmanager. The Alertmanager supports a wide variety of notification methods, recipients and useful functions like grouping, aggregation and muting of certain alarms.

So - how do you get this ensemble to play together? Luckily, TheAwesomeGarage has it all documeted with complete configuration examples.

The highlights follow:

docker-compose.yml

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    networks:
      - frontend
      - backend
    expose:
      - "9090"
    environment:
      - TZ=${DOCKER_TZ}
      - VIRTUAL_HOST=${PROMETHEUS_DOMAIN}
      - VIRTUAL_PORT=9090
      - VIRTUAL_PROTO=http
      - LETSENCRYPT_HOST=${PROMETHEUS_DOMAIN}
      - LETSENCRYPT_EMAIL=${NOTIFICATION_EMAIL}
    volumes:
      - ${MY_DOCKER_DATA_DIR}/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ${MY_DOCKER_DATA_DIR}/prometheus/alert.rules:/etc/prometheus/alert.rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  prometheus-alertmanager:
    container_name: prometheus-alertmanager
    image: prom/alertmanager:latest
    restart: always
    environment:
      - TZ=${DOCKER_TZ}
    networks:
      - backend
      - email
    volumes:
      - ${MY_DOCKER_DATA_DIR}/prometheus/alertmanager.yml:/alertmanager.yml
    command:
      - '--config.file=/alertmanager.yml'

  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    restart: always
    depends_on:
      - prometheus
    networks:
      - frontend
      - email
    expose:
      - "3000"
    environment:
      - TZ=${DOCKER_TZ}
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=smtp-relay:25
      - GF_SMTP_FROM_ADDRESS=${NOTIFICATION_EMAIL}
      - GF_SERVER_DOMAIN=${GRAFANA_DOMAIN}
      - GF_SERVER_ROOT_URL=${GRAFANA_ROOT_URL}
      - VIRTUAL_HOST=${GRAFANA_DOMAIN}
      - VIRTUAL_PORT=3000
      - VIRTUAL_PROTO=http
      - LETSENCRYPT_HOST=${GRAFANA_DOMAIN}
      - LETSENCRYPT_EMAIL=${NOTIFICATION_EMAIL}
    volumes:
      - grafana_data:/var/lib/grafana

  smtp-relay:
    container_name: smtp-relay
    #build: ./smtp-relay/.
    image: docker.remim.com/smtp-relay:latest
    restart: always
    networks:
      - email
    environment:
      - TZ=${DOCKER_TZ}
      - RELAY_HOST=smtp.gmail.com
      - RELAY_PORT=587
      - RELAY_USERNAME=${GMAIL_USERNAME}@gmail.com
      - RELAY_PASSWORD=${GMAIL_PASSWORD}

What you have here is a docker-compose installation and configuration of Prometheus, Alertmanager, Grafana and an smtp-relay to send alerts via e-mail. The central configurations to be done are located in the following files:

• prometheus.yml: how often and from where is monitoring data fetched/scraped
• alert.rules: no further comment needed
• alertmanager.yml: notification settings

In the docker-compose.yml extract, you'll see a lot of variables. They origin from the .env and need to be defined correctly in order for the stack to work. The whole setup of course depends on my complete Docker stack with Nginx, SSL-certificate generation, +++. Clone the entire repo, read the read.me and play with the setup - or just pick the parts you need and try and make it work on your rig.

When it comes to defining the actual dashboards - I'll see if I have time to do a write up on that later!