When disk space runs out, it rarely announces itself politely

Disk pressure is one of those problems that grows quietly. A server starts clean: logs are small, package caches are tidy, and application data fits comfortably. Then time passes. A new service writes more logs than expected. A backup job keeps one extra copy “just in case.” A container runtime accumulates layers. A database grows steadily, and one day a routine deploy fails because /var is full. The outage is not dramatic at first—just a few errors. Then it cascades: services can’t write PID files, journald can’t persist logs, databases can’t checkpoint, and suddenly we are troubleshooting symptoms instead of the cause.

We are going to prevent that kind of slow-motion failure with a controlled, production-grade approach on Linux: reliable disk usage monitoring and automated CLI alerts. No dashboards, no dependency on a browser, and no “someone has to remember to check.” We will build a small, intentional alerting path that survives reboots, runs with least privilege, and is easy to verify.

Prerequisites and assumptions

Before we touch commands, we need to be explicit about the environment we are designing for. Disk monitoring is simple in concept, but production safety comes from the assumptions we make upfront.

• Platform: Linux. The steps below assume a systemd-based distribution (common in Ubuntu Server 20.04/22.04/24.04, Debian 11/12, RHEL 8/9, Rocky/Alma 8/9). If systemd is not present, the scheduling step must be adapted.
• Access: We need a shell with sudo privileges. We will create a dedicated service account and restrict what it can do.
• Network: Outbound connectivity is required only if we choose email delivery via SMTP. If outbound is restricted, we can still alert to local syslog/journal and rely on existing log shipping.
• Mail path: For CLI alerts, we will use local mail submission via msmtp (lightweight SMTP client). This avoids running a full mail server. If an organization already has an approved mail relay, we will use it.
• Security posture: We will avoid storing secrets in world-readable files, keep permissions tight, and ensure the monitoring job cannot modify system state.
• Scope: We will monitor mounted filesystems that matter for outages (typically /, /var, /home, application mounts, and database volumes). We will ignore pseudo filesystems like tmpfs and devtmpfs.

Design overview

We will implement three components:

• A disk check script that evaluates filesystem usage and emits a clear, single-line summary per filesystem that crosses thresholds.
• A CLI alert path that can send an email via an SMTP relay, and always logs to the system journal for auditability.
• A systemd timer that runs the check on a schedule, persists across reboots, and provides a clean operational surface (systemctl status, logs, and exit codes).

This keeps the solution small, observable, and easy to reason about under pressure.

Step 1: Confirm what we are monitoring and why

Before we automate anything, we need to see what the system considers “disk usage” and which mounts are real risk. We will list filesystems with types and usage so we can decide what to include and what to ignore.

df -hT

This shows each mounted filesystem, its type, and its current usage. In production, the mounts that usually matter are persistent storage volumes (ext4, xfs, btrfs, zfs, LVM-backed mounts, cloud block volumes). Pseudo filesystems like tmpfs can fill too, but they are memory-backed and typically handled differently; we will exclude them by default to avoid noisy alerts.

Step 2: Install minimal dependencies for CLI alerts

We need a reliable way to send an alert from the command line. We will use msmtp as a lightweight SMTP client and mailutils to provide the mail command. This keeps the footprint small and avoids running a full MTA.

First, we will detect which package manager is available so the commands remain copy/paste safe across common Linux families.

if command -v apt-get >/dev/null 2>&1; then
  echo "Detected apt-based system"
elif command -v dnf >/dev/null 2>&1; then
  echo "Detected dnf-based system"
elif command -v yum >/dev/null 2>&1; then
  echo "Detected yum-based system"
else
  echo "No supported package manager detected (apt/dnf/yum)."
fi

This prints which family we are on. Now we will install the packages using the appropriate manager.

Install on apt-based systems

We will update package metadata and install msmtp and mailutils. This gives us a standard CLI mail interface and a simple SMTP sender.

sudo apt-get update
sudo apt-get install -y msmtp msmtp-mta mailutils ca-certificates

This installs the mail tooling and ensures TLS certificates are present for secure SMTP connections.

Install on dnf/yum-based systems

We will install msmtp and a mail client. On some RHEL-like systems, the mail client package name differs; mailx is commonly available.

if command -v dnf >/dev/null 2>&1; then
  sudo dnf install -y msmtp mailx ca-certificates
else
  sudo yum install -y msmtp mailx ca-certificates
fi

This provides the SMTP sender and a CLI mail command. If the distribution uses a different mail client package, we will address it in troubleshooting.

Step 3: Create a dedicated monitoring identity

We want the monitoring job to read filesystem usage and send alerts, not to modify system state. We will create a dedicated system user with no login shell. This is a small step that pays off later when we audit permissions.

sudo useradd --system --home /var/lib/diskmon --create-home --shell /usr/sbin/nologin diskmon

This creates a system account named diskmon with a home directory for its configuration and no interactive login.

Step 4: Configure SMTP for CLI alerts (securely)

Now we will configure msmtp for the diskmon user. We will store credentials in a file readable only by that user. In enterprise environments, the SMTP relay is usually internal and authenticated; if authentication is not required, we can omit the password section.

First, we will create the configuration directory and file with strict permissions.

sudo -u diskmon mkdir -p /var/lib/diskmon
sudo -u diskmon touch /var/lib/diskmon/.msmtprc
sudo chown -R diskmon:diskmon /var/lib/diskmon
sudo chmod 700 /var/lib/diskmon
sudo chmod 600 /var/lib/diskmon/.msmtprc

This ensures only diskmon can read the SMTP configuration, which is critical if we store credentials.

Next, we will write a complete msmtp configuration. We will keep it explicit and TLS-first. We will also set a log file for SMTP send attempts, which helps during incident response.

Because SMTP details vary by environment, we will set them as shell variables first so the commands remain copy/paste safe and we avoid half-edited config files.

SMTP_HOST="smtp.example.com"
SMTP_PORT="587"
SMTP_USER="[email protected]"
ALERT_FROM="[email protected]"
ALERT_TO="[email protected]"
SMTP_TLS="on"
SMTP_STARTTLS="on"

These variables define the relay and recipients. We will now prompt for the SMTP password without echoing it to the terminal, then write the config file.

read -r -s -p "Enter SMTP password for ${SMTP_USER}: " SMTP_PASS
echo
sudo -u diskmon bash -c "cat > /var/lib/diskmon/.msmtprc" <<EOF
# msmtp configuration for disk space alerts
# File must be chmod 600 and owned by diskmon

defaults
auth           on
tls            ${SMTP_TLS}
tls_starttls   ${SMTP_STARTTLS}
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/lib/diskmon/msmtp.log
timeout        10

account        alerts
host           ${SMTP_HOST}
port           ${SMTP_PORT}
user           ${SMTP_USER}
password       ${SMTP_PASS}
from           ${ALERT_FROM}

account default : alerts
EOF
unset SMTP_PASS

This writes a complete .msmtprc for the diskmon user and immediately unsets the password variable. The password is now stored only in a file protected by permissions. The log file path is also under /var/lib/diskmon, keeping everything contained.

Now we will verify that the mail path works before we build automation around it. We will send a single test email using msmtp directly so we can see clear errors if the relay rejects us.

HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)"
sudo -u diskmon bash -c "printf 'Subject: Disk monitor test on %snnThis is a test message from disk monitoring.n' '${HOSTNAME_FQDN}' | msmtp -t '${ALERT_TO}'"

This attempts to deliver a test message to the on-call address. If it succeeds, we have proven outbound SMTP connectivity, credentials, and TLS trust. If it fails, we will use the troubleshooting section to interpret the error and fix it before proceeding.

Firewall considerations for CLI alerts

This solution does not require inbound firewall changes. If a host firewall is enabled, we only need to ensure outbound TCP to the SMTP relay is allowed (commonly port 587 for STARTTLS or 465 for implicit TLS). In tightly controlled environments, outbound is often default-deny, so we should confirm policy with the network team.

Step 5: Implement the disk usage check script

Now we will create the script that does the actual monitoring. The goal is simple: check real filesystems, compare usage against thresholds, and alert only when action is needed. We will also make the output predictable so it is easy to parse in logs and easy to read in an email.

We will place the script in /usr/local/sbin because it is a standard location for locally managed system executables. We will keep it root-owned and not writable by regular users to prevent tampering.

sudo tee /usr/local/sbin/disk-usage-alert.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Disk usage monitoring with CLI alerts
# - Logs to journald via logger
# - Sends email via msmtp when thresholds are exceeded
#
# Exit codes:
# 0 = OK (no thresholds exceeded)
# 1 = Warning threshold exceeded
# 2 = Critical threshold exceeded
# 3 = Script/config error

WARN_PCT="${WARN_PCT:-80}"
CRIT_PCT="${CRIT_PCT:-90}"
ALERT_TO="${ALERT_TO:[email protected]}"
MSMTP_BIN="${MSMTP_BIN:-/usr/bin/msmtp}"
MSMTP_CONFIG="${MSMTP_CONFIG:-/var/lib/diskmon/.msmtprc}"

HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)"
NOW_UTC="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"

log() {
  # Tag logs for easy filtering: journalctl -t diskmon
  logger -t diskmon -- "$*"
}

send_mail() {
  local subject="$1"
  local body="$2"

  if [[ ! -x "${MSMTP_BIN}" ]]; then
    log "ERROR: msmtp not found at ${MSMTP_BIN}"
    return 1
  fi
  if [[ ! -r "${MSMTP_CONFIG}" ]]; then
    log "ERROR: msmtp config not readable at ${MSMTP_CONFIG}"
    return 1
  fi

  # Use msmtp directly to avoid dependency differences across distros
  printf 'Subject: %snTo: %snn%sn' "${subject}" "${ALERT_TO}" "${body}" | "${MSMTP_BIN}" -C "${MSMTP_CONFIG}" -t
}

# Build a list of real filesystems:
# -P: POSIX output (stable columns)
# -T: include filesystem type
# Exclude pseudo filesystems and read-only special mounts by type.
mapfile -t LINES < <(df -P -T | tail -n +2 | awk '
  $2 !~ /^(tmpfs|devtmpfs|overlay|squashfs|proc|sysfs|cgroup2?|pstore|securityfs|debugfs|tracefs|mqueue|hugetlbfs|rpc_pipefs|autofs|fusectl)$/ { print }
')

if [[ "${#LINES[@]}" -eq 0 ]]; then
  log "ERROR: No filesystems found to monitor"
  exit 3
fi

warn_hits=()
crit_hits=()

for line in "${LINES[@]}"; do
  # Fields: Filesystem Type 1024-blocks Used Available Capacity Mounted_on
  # Capacity is like "42%"
  fs="$(awk '{print $1}' <<< "${line}")"
  fstype="$(awk '{print $2}' <<< "${line}")"
  usepct_raw="$(awk '{print $6}' <<< "${line}")"
  mnt="$(awk '{print $7}' <<< "${line}")"

  usepct="${usepct_raw%%}"

  # Skip if parsing failed
  if [[ -z "${usepct}" ]] || ! [[ "${usepct}" =~ ^[0-9]+$ ]]; then
    log "ERROR: Unable to parse usage for line: ${line}"
    continue
  fi

  if (( usepct >= CRIT_PCT )); then
    crit_hits+=("${usepct}% ${mnt} (${fs}, ${fstype})")
  elif (( usepct >= WARN_PCT )); then
    warn_hits+=("${usepct}% ${mnt} (${fs}, ${fstype})")
  fi
done

if [[ "${#crit_hits[@]}" -eq 0 && "${#warn_hits[@]}" -eq 0 ]]; then
  log "OK: Disk usage within thresholds (warn=${WARN_PCT}%, crit=${CRIT_PCT}%)"
  exit 0
fi

body="Time (UTC): ${NOW_UTC}
Host: ${HOSTNAME_FQDN}
Thresholds: warn=${WARN_PCT}%, crit=${CRIT_PCT}%

"

exit_code=0
subject_prefix="WARNING"

if [[ "${#crit_hits[@]}" -gt 0 ]]; then
  subject_prefix="CRITICAL"
  exit_code=2
  body+="Critical filesystems:n"
  for item in "${crit_hits[@]}"; do
    body+="- ${item}n"
  done
  body+="n"
fi

if [[ "${#warn_hits[@]}" -gt 0 ]]; then
  if [[ "${#crit_hits[@]}" -eq 0 ]]; then
    exit_code=1
  fi
  body+="Warning filesystems:n"
  for item in "${warn_hits[@]}"; do
    body+="- ${item}n"
  done
  body+="n"
fi

body+="Top space consumers (quick view):n"
body+="(This is a hint, not a full investigation.)nn"
body+="Largest directories under /var (if accessible):n"
if command -v du >/dev/null 2>&1; then
  # Avoid expensive scans: limit depth and ignore errors
  body+="$(du -x -h -d 2 /var 2>/dev/null | sort -h | tail -n 15)n"
else
  body+="du not availablen"
fi

subject="${subject_prefix}: Disk usage on ${HOSTNAME_FQDN}"
log "${subject} - warn=${WARN_PCT}% crit=${CRIT_PCT}% - crit_hits=${#crit_hits[@]} warn_hits=${#warn_hits[@]}"

if ! send_mail "${subject}" "$(printf "%b" "${body}")"; then
  log "ERROR: Failed to send alert email"
  # Still return a non-zero code to surface the condition to systemd
  exit 3
fi

exit "${exit_code}"
EOF
sudo chown root:root /usr/local/sbin/disk-usage-alert.sh
sudo chmod 0755 /usr/local/sbin/disk-usage-alert.sh

This creates a hardened script with strict shell settings, stable parsing, journald logging, and an email alert path. It exits with meaningful codes so systemd can reflect health. It also includes a small “top consumers” hint to speed up triage without turning the check into a heavy scan.

Verify the script runs and logs correctly

Before scheduling, we will run the script once with explicit environment variables so we can confirm behavior without editing the file. We will also run it as the diskmon user to match production execution.

HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)"
sudo -u diskmon env 
  WARN_PCT=80 
  CRIT_PCT=90 
  ALERT_TO="[email protected]" 
  MSMTP_BIN="/usr/bin/msmtp" 
  MSMTP_CONFIG="/var/lib/diskmon/.msmtprc" 
  /usr/local/sbin/disk-usage-alert.sh || true

sudo journalctl -t diskmon -n 50 --no-pager

This runs the check and then prints the last 50 log lines tagged diskmon. If disk usage is below thresholds, we should see an “OK” log and a zero exit. If thresholds are exceeded, we should see a warning/critical log and receive an email. If email fails, the journal will show why.

Step 6: Make it persistent with a systemd service and timer

Running a script manually is not monitoring. We need a scheduler that is reliable, observable, and survives reboots. systemd timers give us all of that, and they integrate cleanly with journald.

Create the systemd service unit

We will create a oneshot service that runs the script as diskmon. We will also harden the service with systemd sandboxing options to reduce blast radius.

sudo tee /etc/systemd/system/disk-usage-alert.service >/dev/null <<'EOF'
[Unit]
Description=Disk usage monitoring and CLI alerts
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=diskmon
Group=diskmon

# Environment for thresholds and alert destination
Environment=WARN_PCT=80
Environment=CRIT_PCT=90
[email protected]
Environment=MSMTP_BIN=/usr/bin/msmtp
Environment=MSMTP_CONFIG=/var/lib/diskmon/.msmtprc

ExecStart=/usr/local/sbin/disk-usage-alert.sh

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictRealtime=true
RestrictSUIDSGID=true
RestrictNamespaces=true

# Allow reading filesystem stats and msmtp config/log location
ReadOnlyPaths=/
ReadWritePaths=/var/lib/diskmon

# Networking is required for SMTP
PrivateNetwork=false

# Resource safety
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

[Install]
WantedBy=multi-user.target
EOF

This service runs as a non-privileged user, uses explicit environment variables for thresholds and recipients, and applies a strict filesystem protection model. We allow write access only to /var/lib/diskmon for the SMTP log and config access. The service remains network-capable because SMTP requires outbound connectivity.

Create the systemd timer unit

Now we will schedule the service. We will run it every 5 minutes, and we will enable persistence so missed runs during downtime execute shortly after boot. This is important because disk pressure often appears after maintenance windows and reboots.

sudo tee /etc/systemd/system/disk-usage-alert.timer >/dev/null <<'EOF'
[Unit]
Description=Run disk usage monitoring on a schedule

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Persistent=true
Unit=disk-usage-alert.service

[Install]
WantedBy=timers.target
EOF

This timer triggers the service two minutes after boot and then every five minutes. With Persistent=true, systemd will catch up after downtime, which reduces blind spots.

Enable and start the timer, then verify

We will reload systemd to pick up the new units, enable the timer so it persists across reboots, and start it immediately. Then we will verify status and recent runs.

sudo systemctl daemon-reload
sudo systemctl enable --now disk-usage-alert.timer

sudo systemctl status disk-usage-alert.timer --no-pager
sudo systemctl list-timers --all | grep -F "disk-usage-alert" || true

This activates the schedule and confirms systemd recognizes it. The timer list should show the next and last run times.

Now we will trigger the service once on demand to validate end-to-end execution under systemd.

sudo systemctl start disk-usage-alert.service
sudo systemctl status disk-usage-alert.service --no-pager
sudo journalctl -u disk-usage-alert.service -n 100 --no-pager

This forces a run and then shows the service result and logs. If the service fails, the journal output will contain the reason, including SMTP errors or permission issues.

Operational checks we should keep in our pocket

When disk alerts fire, we want fast confirmation and fast triage. These commands are safe and give immediate clarity.

• Confirm current usage with filesystem types

  df -hT

  This confirms whether the alert matches reality and which mount is under pressure.

• Find large directories on the affected mount

  MOUNTPOINT="/var"
  sudo du -x -h -d 2 "${MOUNTPOINT}" 2>/dev/null | sort -h | tail -n 30

  This stays on the same filesystem (-x) and gives a quick shortlist of where space is going.

• Check recent logs for the monitoring job

  sudo journalctl -t diskmon -n 200 --no-pager
  sudo journalctl -u disk-usage-alert.service -n 200 --no-pager

  This shows both the script’s tagged logs and the systemd unit logs.

Troubleshooting

When this setup fails, it usually fails in predictable ways. We want symptoms, likely causes, and fixes that are safe under pressure.

Symptom: No emails arrive, but the service shows “success”

• Likely cause: Disk usage is below thresholds, so no alert is sent.
• Fix: Confirm by checking logs and optionally lowering thresholds temporarily for a controlled test.

sudo journalctl -t diskmon -n 50 --no-pager

sudo systemctl stop disk-usage-alert.timer
sudo systemctl start disk-usage-alert.service

# Controlled test: lower thresholds for one run
sudo systemctl set-environment WARN_PCT=1 CRIT_PCT=2
sudo systemctl start disk-usage-alert.service
sudo systemctl unset-environment WARN_PCT CRIT_PCT

sudo systemctl start disk-usage-alert.timer

This checks logs and forces a one-time run with very low thresholds to validate the alert path. We then remove the temporary environment overrides and restore the timer.

Symptom: Service fails with “msmtp config not readable”

• Likely cause: Wrong path in MSMTP_CONFIG or permissions are too restrictive/incorrect ownership.
• Fix: Confirm ownership and permissions, then re-run the service.

sudo ls -la /var/lib/diskmon/.msmtprc
sudo chown diskmon:diskmon /var/lib/diskmon/.msmtprc
sudo chmod 600 /var/lib/diskmon/.msmtprc

sudo systemctl start disk-usage-alert.service
sudo journalctl -u disk-usage-alert.service -n 50 --no-pager

This ensures the config is readable by diskmon and not exposed to other users.

Symptom: Email send fails with authentication or TLS errors

• Likely cause: Wrong credentials, relay requires different port, STARTTLS mismatch, or missing CA bundle path on non-Debian systems.
• Fix: Inspect msmtp logs and adjust the config accordingly.

sudo -u diskmon tail -n 200 /var/lib/diskmon/msmtp.log 2>/dev/null || true

# Validate CA bundle location (varies by distro)
ls -l /etc/ssl/certs/ca-certificates.crt 2>/dev/null || true
ls -l /etc/pki/tls/certs/ca-bundle.crt 2>/dev/null || true

If the CA bundle path differs (common on RHEL-like systems), we should update tls_trust_file in /var/lib/diskmon/.msmtprc to the correct path and re-test the direct msmtp send command.

Symptom: Timer is enabled, but the service never runs

• Likely cause: Timer not started, unit name mismatch, or system time issues.
• Fix: Check timer status, list timers, and confirm system time.

sudo systemctl status disk-usage-alert.timer --no-pager
sudo systemctl list-timers --all | grep -F "disk-usage-alert" || true
timedatectl status

This confirms the timer is active and scheduled, and that system time is sane. If time is wrong, timers can behave unexpectedly.

Symptom: Service fails with “Read-only file system” or permission denied

• Likely cause: systemd hardening is blocking a path the script needs to write to, or the script is trying to write outside /var/lib/diskmon.
• Fix: Confirm the script only writes to allowed paths, and ensure ReadWritePaths=/var/lib/diskmon is present.

sudo systemctl cat disk-usage-alert.service
sudo systemctl restart disk-usage-alert.service
sudo journalctl -u disk-usage-alert.service -n 100 --no-pager

This shows the effective unit file and the exact failure message. If we need additional write paths (rare), we should add them explicitly rather than weakening ProtectSystem.

Common mistakes

Mistake: Editing the script to hardcode recipients and thresholds

Symptom: Changes work once, then drift across servers and become inconsistent.

Fix: Keep the script generic and set values in the systemd unit via Environment=. That keeps configuration in one predictable place.

Mistake: Running the check as root “because it’s easier”

Symptom: The monitoring job has unnecessary privileges, and a script bug becomes a security incident.

Fix: Run as diskmon and keep systemd hardening enabled. If a specific mount requires elevated access, we should address that mount’s permissions rather than escalating the whole job.

Mistake: Using a mail relay without confirming outbound firewall policy

Symptom: msmtp times out, logs show connection failures, and alerts silently never leave the host.

Fix: Confirm outbound TCP to the relay and port is allowed. Then re-run the direct msmtp test send and check /var/lib/diskmon/msmtp.log.

Mistake: Monitoring everything, including pseudo filesystems

Symptom: Noisy alerts for tmpfs or container overlays that do not represent persistent disk exhaustion.

Fix: Keep the filesystem type filter in place. If a specific pseudo filesystem matters in our environment, we should add it intentionally with a clear reason.

How do we at NIILAA look at this

This setup is not impressive because it is complex. It is impressive because it is controlled. Every component is intentional. Every configuration has a reason. This is how infrastructure should scale — quietly, predictably, and without drama.

At NIILAA, we help organizations design, deploy, secure, and maintain monitoring and alerting patterns like this in real production environments—aligned with least privilege, operational clarity, and the realities of on-call response. When disk pressure is handled early and consistently, outages stop being surprises and start being routine maintenance.

• Website: https://www.niilaa.com
• Email: [email protected]
• LinkedIn: https://www.linkedin.com/company/niilaa
• Facebook: https://www.facebook.com/niilaa.llc