Some fun with monitoring

Some fun with monitoring

I recently switched my home lab from ProxMox VMs to HashiCorp Nomad, Consul, Vault and containers running on bare metal servers, and I needed a way to monitor this stack, my applications as well as external systems that I care about.

The Monitoring Stack

I looked at a bunch of options and finally settled on Prometheus, Grafana and some other smaller exporters that could scrape the data I needed and present it in a simple way.

Exporters

If you’re not familiar with how Prometheus works, the basic essential you need to know is that Prometheus servers will scrape targets periodically to collect metrics. This is a problem if you need to scrape data from a network device that only supports SNMP, or you need connection metrics for a website where you can’t install more detailed web server metrics exporters. I can’t install a Prometheus exporter on my Juniper firewall for example, well I probably could since it’s a BSD OS underneath but that would most likely mean a lot of pain during software upgrades. However, there are two exporters that can help out here:

The configuration

I’m running Prometheus, Alertmanager, snmp-exporter and blackbox-exporter in Nomad with Consul service discovery. My Prometheus config for the blackbox-exporter and snmp-exporter are below:

Blackbox Exporter Config
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - https://sigterm.sh
        - https://lg.gxa.dn42.io
        - https://mx.gxa.io/ping/
        - https://nyc-1.us.gxa.dn42.io/api/bgp
        - https://sfo-1.us.gxa.dn42.io/api/bgp
        - https://fra-1.de.gxa.dn42.io/api/bgp
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.service.consul:9115
        # ^ The blackbox exporter's real hostname:port.

  - job_name: 'blackbox-imap'
    metrics_path: /probe
    params:
      module: [imap_starttls]
    static_configs:
      - targets:
        - mx.gxa.io:143
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.service.consul:9115
        # ^ The blackbox exporter's real hostname:port.

  - job_name: 'blackbox-smtp'
    metrics_path: /probe
    params:
      module: [smtp_starttls]
    static_configs:
      - targets:
        - mx.gxa.io:587
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.service.consul:9115
        # ^ The blackbox exporter's real hostname:port.
SNMP Exporter Config
scrape_configs:
  - job_name: 'snmp'
    scrape_interval: 60s
    scrape_timeout: 55s
    static_configs:
      - targets:
        - 172.29.255.6
        - 172.29.255.7
        - 172.29.60.254
        - 172.29.58.1
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: snmp-exporter.service.consul:9116
        # ^ The SNMP exporter's real hostname:port.

The Final Result

Here are a few graphs I threw together demonstrating some SNMP data from my core firewalls and switches, along with some temperature and relative humidity graphs from my pine64 project, a monitor for SMTP and IMAP connection latency and finally some SSL certificate expiration monitoring.

Gregory Armer avatar
About Gregory Armer
Sometimes I just want to give it all up and become a handsome billionaire.
comments powered by Disqus