← Wiki index

Monitoring service resource usage using systemd

Topics: systemd

This note provides systemd settings and a watchdog script for monitoring and restarting services based on CPU use.

systemd settings to limit CPU quota:

[Service]
CPUAccounting=yes
CPUQuota=10%
Restart=on-failure
RestartSec=10s
KillSignal=SIGTERM
TimeoutStopSec=30s

Script that restarts the service if the service’s CPU usage is exceeding its quota by 5% or continuously reaching 95% of quota:

#!/bin/bash
set -euo pipefail

UNIT="$1"
INTERVAL=30

# Policy thresholds
PEGGED_NS=$((INTERVAL * 1000000000 * 9 / 10))   # ~90% of quota window
SUSTAINED_NS=$((INTERVAL * 1000000000 * 5 / 100)) # 5% CPU

STATE="/run/cpu-watch-${UNIT}.state"

current=$(systemctl show "$UNIT" -p CPUUsageNSec --value)
previous=0
[[ -f "$STATE" ]] && previous=$(cat "$STATE")
echo "$current" > "$STATE"

delta=$((current - previous))

# Restart if pegged (hitting CPUQuota)
if (( delta >= PEGGED_NS )); then
  logger -t cpu-watch "CPU pegged for $UNIT (${delta}ns), restarting"
  systemctl restart "$UNIT"
  exit 0
fi

# Restart if consistently above 5%
if (( delta >= SUSTAINED_NS )); then
  logger -t cpu-watch "Sustained CPU abuse for $UNIT (${delta}ns), restarting"
  systemctl restart "$UNIT"
fi

systemd service and timer templates that run the monitoring script regularly:

# /etc/systemd/system/cpu-watch@.service
[Unit]
Description=CPU watchdog for %i
After=%i.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-watch.sh %i.service

# /etc/systemd/system/cpu-watch@.timer
[Unit]
Description=Periodic CPU watchdog for %i

[Timer]
OnBootSec=2min
OnUnitActiveSec=30s
AccuracySec=5s

[Install]
WantedBy=timers.target

Enable monitoring of a specific service:

sudo systemctl enable --now cpu-watch@avahi-daemon.timer

Bookmarks

Questions