I've used a lot of monitoring systems over the years. Nagios, Zabbix, Munin, Cacti, and several others that have thankfully been forgotten. Prometheus with Grafana is the first combination that I actually enjoy using. Not because it's flashy — it's not. Because it works the way a sysadmin thinks: pull metrics, store them, query them, alert on them. No magic, no hidden state, no "it should be working but isn't."
What Monitoring Actually Needs to Do
Before installing anything, be clear about what monitoring is for. It's not for making pretty dashboards (though those are nice). It's for three things:
- Alert you when something breaks. Before a user notices, ideally. Before a customer notices, absolutely.
- Help you diagnose problems. When something breaks at 3 AM, you need data — what changed, when did it start, what's correlated.
- Show trends over time. Is disk usage growing? Are connections increasing? Is latency creeping up? Trends tell you what's going to break before it breaks.
A monitoring system that does these three things well is worth more than one that produces beautiful graphs of irrelevant metrics.
Architecture Overview
[Your Servers] [Monitoring Stack]
┌──────────────┐ ┌──────────────┐
│ node_exporter ├──(scrape)──────┤ Prometheus │
│ :9100 │ │ :9090 │
└──────────────┘ └──────┬───────┘
┌──────────────┐ │
│ custom_export├──(scrape)──────────────┘
│ :9101 │ ┌──────┴───────┐
└──────────────┘ │ Grafana │
│ :3000 │
└──────┬───────┘
│
┌──────┴───────┐
│ Alertmanager │
│ :9093 │
└──────────────┘Prometheus pulls (scrapes) metrics from exporters on your servers. Grafana connects to Prometheus as a data source and visualizes the metrics. Alertmanager receives alert notifications from Prometheus and routes them to email, Slack, PagerDuty, etc.
Prometheus Setup
# Download and install:
wget https://github.com/prometheus/prometheus/releases/download/v2.51.0/prometheus-2.51.0.linux-amd64.tar.gz
tar xzf prometheus-2.51.0.linux-amd64.tar.gz
sudo mv prometheus-2.51.0.linux-amd64 /opt/prometheus
# Create user:
sudo useradd --no-create-home --shell /bin/false prometheus
# Configuration:
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheusMinimal /etc/prometheus/prometheus.yml:
global:
scrape_interval: 15s # How often to scrape targets
evaluation_interval: 15s # How often to evaluate rules
rule_files:
- "alerts.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "nodes"
static_configs:
- targets:
- "server1.example.com:9100"
- "server2.example.com:9100"
- "irc.example.com:9100"Create a systemd service:
# /etc/systemd/system/prometheus.service:
[Unit]
Description=Prometheus
After=network.target
[Service]
User=prometheus
ExecStart=/opt/prometheus/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--storage.tsdb.retention.time=90d \
--web.listen-address=:9090
Restart=always
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now prometheusOn the web-facing side, if your IRC network has a website or documentation portal, tools like Koeki can help you understand how people discover your network through search engines. Monitoring isn't just about uptime — it's about visibility too.
Node Exporter: System Metrics
Install node_exporter on every server you want to monitor:
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/Node exporter provides hundreds of metrics out of the box: CPU, memory, disk, network, filesystem, and more. The metrics you'll use most often:
# CPU usage (as a percentage):
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory usage:
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
# Disk usage (percentage):
100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)
# Network traffic (bytes/sec):
rate(node_network_receive_bytes_total[5m])
rate(node_network_transmit_bytes_total[5m])
# System load:
node_load1, node_load5, node_load15Grafana Setup and Dashboards
sudo apt install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_10.4.0_amd64.deb
sudo dpkg -i grafana_10.4.0_amd64.deb
sudo systemctl enable --now grafana-serverAccess Grafana at http://server:3000 (default login: admin/admin). Add Prometheus as a data source (URL: http://localhost:9090).
Dashboard design tips from someone who's stared at them during outages:
- Top row: status indicators. Green/red panels showing whether critical services are up. No numbers, no graphs — just "OK" or "PROBLEM." This is what you look at first.
- Second row: key metrics. CPU, memory, disk, connections. Time series graphs with reasonable time ranges (last 6 hours default, with easy zoom to 24h and 7d).
- Third row: detail panels. Network throughput, per-service metrics, error rates. Only look at these when diagnosing.
- Don't overload. A dashboard with 30 panels is useless during an incident. Your "overview" dashboard should have 8-12 panels maximum.
Alerting Rules
Create /etc/prometheus/alerts.yml:
groups:
- name: critical
rules:
- alert: InstanceDown
expr: up == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} is down"
- alert: DiskSpaceLow
expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
for: 5m
labels:
severity: warning
annotations:
summary: "Disk space below 10% on {{ $labels.instance }}"
- alert: HighMemoryUsage
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "Memory usage above 90% on {{ $labels.instance }}"
- alert: HighCPU
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
for: 10m
labels:
severity: warning
annotations:
summary: "CPU usage above 90% for 10 min on {{ $labels.instance }}"The for clause is crucial — it prevents alerts from firing on brief spikes. A CPU spike that lasts 30 seconds isn't an incident. One that lasts 10 minutes is.
Custom Exporters
For IRC-specific metrics, you'll need a custom exporter. A simple Python exporter using the prometheus_client library:
from prometheus_client import start_http_server, Gauge
import subprocess, time
irc_users = Gauge('irc_users_total', 'Total connected IRC users')
irc_channels = Gauge('irc_channels_total', 'Total active IRC channels')
def collect_metrics():
# Parse ircd stats — adapt to your server's output
# This is a simplified example
try:
result = subprocess.run(['grep', '-c', 'Client connecting',
'/home/ircd/unrealircd/logs/ircd.log'],
capture_output=True, text=True)
irc_users.set(float(result.stdout.strip()))
except Exception:
pass
if __name__ == '__main__':
start_http_server(9101)
while True:
collect_metrics()
time.sleep(15)A proper exporter would query the IRC server's statistics directly (via the server's stats interface or by connecting as an oper and parsing /LUSERS output). The approach depends on your ircd and what it exposes.
Monitoring isn't something you set up once and forget. It's something you tune over time — adding alerts when you discover blind spots, removing alerts that cry wolf, adjusting thresholds as your understanding of normal behavior improves. The best monitoring setup is one that's been refined through actual incidents. The second best is one that exists at all.
Frequently Asked Questions
How much disk space does Prometheus need?
It depends on the number of time series and the retention period. A rough estimate: 1-2 bytes per sample, with samples every 15 seconds. For a small setup (5 servers, node_exporter only), expect 1-2 GB per month. For larger setups, use the Prometheus storage calculator or monitor the actual usage after initial deployment.
Should I monitor the monitoring system?
Yes. At minimum, have an external check (a simple cron job that curls the Prometheus health endpoint and alerts via a different channel if it fails). The monitoring system failing silently is worse than no monitoring at all — it gives false confidence.
Prometheus or Zabbix?
Different tools for different philosophies. Zabbix is a traditional monitoring system with agent-based collection, built-in dashboards, and a database backend. Prometheus is a metrics-focused system with a pull model and PromQL. For cloud-native and container environments, Prometheus is the standard. For traditional infrastructure with a preference for all-in-one solutions, Zabbix is solid. Both work; neither is wrong.
How do I monitor servers behind NAT?
Options: push metrics via Pushgateway (Prometheus component for pushing instead of pulling), VPN or SSH tunnel from the monitoring server to the target, or run a local Prometheus instance that federates to a central server. Pushgateway is the simplest but least Prometheus-native approach.
FAQ
Can I monitor IRC with Prometheus?
Yes. You can export IRC metrics using custom scripts that query the IRCd's stats interface or use dedicated exporters. Common metrics include connection count, channel count, message throughput, and server uptime.
What should I monitor on an IRC server?
Key metrics: concurrent connections, connection rate, bandwidth usage, CPU and memory, message throughput, netsplit events, and service availability. Set alerts for sudden drops in user count (possible netsplit) and connection spikes (possible flood).
Is Grafana useful for IRC monitoring?
Very much so. Grafana dashboards give you visual insight into traffic patterns, peak usage times, and long-term trends. Combined with Prometheus alerting, you get a complete monitoring stack for your IRC infrastructure.