Why automated backups become urgent when everything seems fine

In the early days, backups feel optional. A small file server, a single application VM, a handful of shared folders—nothing looks fragile. Then the business grows. A new CRM export lands every night. Finance drops month-end reports into shared storage. A developer adds a “temporary” directory that quietly becomes critical. One day a disk fills, a ransomware event hits a workstation, or a well-meaning change wipes a directory. That is when we discover the uncomfortable truth: the backup we “meant to automate” never became reliable, and the restore path was never proven.

For SMEs, the goal is not to build a complicated backup platform. The goal is to build a controlled one: predictable schedules, least-privilege access, encrypted transport, validation, and logs that tell us what happened without guesswork. That is exactly where rsync shines—when we treat it like production infrastructure, not a quick command we run once and forget.

What we are building

We are going to implement secure, automated rsync backups on Linux with the following properties:

• Pull-based backups from a dedicated backup server (reduces blast radius if a source host is compromised).
• SSH hardening with a dedicated key and forced command restrictions.
• Systemd timers for scheduling and reliability (we will avoid blind cron jobs entirely).
• Validation included: we will verify connectivity, permissions, transfer integrity signals, and backup freshness.
• Operational logging via journald and a dedicated log file.
• Firewall-aware configuration for enterprise networks.

Prerequisites and assumptions

Before we touch commands, we need to be explicit about the environment. This prevents subtle failures later.

• Platform: Linux on both systems (source host and backup host). The steps assume a modern systemd-based distribution (Ubuntu 22.04/24.04 LTS, Debian 12, RHEL 9, Rocky/Alma 9, or similar).
• Access: We have shell access with sudo privileges on both systems.
• Network: The backup host can reach the source host over TCP/22 (SSH). If there is a firewall, we will explicitly allow only what is needed.
• Time sync: Both systems should have correct time (chrony/systemd-timesyncd). Incorrect time causes confusing “freshness” checks and log correlation issues.
• Storage: The backup host has enough disk space for the retained data. We will store backups under /srv/backups with strict permissions.
• Data model: We are backing up directories (files). If we need application-consistent backups (databases), we should add app-level dumps or snapshots; rsync alone is file-consistent.
• Security posture: We will use a dedicated backup user on the source host with read-only access to the data being backed up. We will not use root SSH for backups.

Architecture: pull backups with least privilege

We will use two roles:

• Source host: the system that contains the data we want to protect.
• Backup host: the system that initiates the rsync connection and stores the backup.

We will configure the source host to accept a restricted SSH key that can only run rsync in server mode. Then we will configure the backup host to run rsync via a systemd service and timer, with logging and validation.

Step 1: Install required packages

We are going to ensure rsync and OpenSSH are installed on both hosts. This matters because rsync over SSH is our encrypted transport and authentication layer, and we want consistent behavior across reboots and updates.

On the source host

sudo sh -c '
set -eu
if command -v apt-get >/dev/null 2>&1; then
  apt-get update
  apt-get install -y rsync openssh-server
elif command -v dnf >/dev/null 2>&1; then
  dnf install -y rsync openssh-server
  systemctl enable --now sshd
elif command -v yum >/dev/null 2>&1; then
  yum install -y rsync openssh-server
  systemctl enable --now sshd
else
  echo "Unsupported package manager. Install rsync and openssh-server manually." 1>&2
  exit 1
fi
'

We have installed rsync and ensured the SSH server is present. On RHEL-like systems, we also enabled and started sshd so the source host can accept the backup connection.

Now we are going to verify that SSH is listening. This confirms the service is reachable locally before we troubleshoot network paths.

sudo ss -ltnp | awk 'NR==1 || $4 ~ /:22$/ {print}'

If we see a listener on :22, the source host is ready to accept SSH connections at the service level.

On the backup host

We are going to install rsync and the SSH client tools. The backup host initiates connections, so it does not need an SSH server for this design.

sudo sh -c '
set -eu
if command -v apt-get >/dev/null 2>&1; then
  apt-get update
  apt-get install -y rsync openssh-client
elif command -v dnf >/dev/null 2>&1; then
  dnf install -y rsync openssh-clients
elif command -v yum >/dev/null 2>&1; then
  yum install -y rsync openssh-clients
else
  echo "Unsupported package manager. Install rsync and openssh client tools manually." 1>&2
  exit 1
fi
'

We now have the tooling required to run rsync over SSH from the backup host.

We are going to confirm rsync is available and print its version. This helps later when diagnosing option compatibility.

rsync --version | head -n 2

If rsync prints a version line, the binary is installed and callable.

Step 2: Create a dedicated backup identity on the source host

We are going to create a dedicated user on the source host that exists only for backups. This matters because it lets us apply least privilege: the backup process can read what it needs, and nothing else. If the backup key is ever exposed, the attacker does not automatically get a general-purpose shell account.

First, we will create a system user with no interactive shell and a locked password.

sudo sh -c '
set -eu
id -u backup-rsync >/dev/null 2>&1 || useradd --system --create-home --home-dir /home/backup-rsync --shell /usr/sbin/nologin backup-rsync
passwd -l backup-rsync >/dev/null 2>&1 || true
'

We now have a backup-rsync user with a home directory for SSH keys, but without an interactive shell and without password login.

Next, we will define what we are backing up. For a realistic SME setup, we will back up /etc and /srv. We will also include /var/www if it exists. We are going to create a read-only access group and grant it access to these paths in a controlled way.

We will create a group and add the backup user to it.

sudo sh -c '
set -eu
getent group backupread >/dev/null 2>&1 || groupadd --system backupread
usermod -aG backupread backup-rsync
'

The backupread group now exists, and backup-rsync is a member. Next we will apply group-readable permissions where appropriate.

We are going to set group ownership and permissions on target directories. This is the part that needs judgment: we should only grant access to what we truly want backed up. The commands below are safe in the sense that they do not delete data, but they do change permissions. We should run them only for directories we intend to include.

sudo sh -c '
set -eu

for d in /etc /srv /var/www; do
  if [ -d "$d" ]; then
    chgrp -R backupread "$d" || true
    chmod -R g+rX "$d" || true
  fi
done
'

We have granted the backupread group read and directory traversal permissions on the selected paths. This enables rsync to read files without granting broad root access. If some application directories must remain private, we should exclude them or handle them with application-specific exports.

Now we will verify that the backup user can read at least one file from each directory. This is a practical validation step that catches permission issues early.

sudo -u backup-rsync sh -c '
set -eu
for d in /etc /srv /var/www; do
  if [ -d "$d" ]; then
    echo "Checking $d"
    find "$d" -maxdepth 2 -type f -readable -print -quit | sed -n "1p"
  fi
done
'

If we see file paths printed, the backup identity can read data. If nothing prints for a directory, it may be empty, or permissions may still be too restrictive.

Step 3: Generate and install a restricted SSH key for rsync

We are going to generate an SSH key on the backup host and install the public key on the source host. This matters because keys are auditable, revocable, and can be restricted. We will also harden the key on the source host so it cannot be used for general shell access.

Generate the key on the backup host

We will create a dedicated keypair for this backup job. We will store it under /root/.ssh so the systemd service can run as root while writing to protected backup storage. In enterprise environments, we can instead run the service as a dedicated local user and adjust permissions accordingly, but root-owned backup storage is common for integrity.

sudo sh -c '
set -eu
install -d -m 0700 /root/.ssh
if [ ! -f /root/.ssh/rsync_backup_ed25519 ]; then
  ssh-keygen -t ed25519 -a 64 -f /root/.ssh/rsync_backup_ed25519 -N "" -C "rsync-backup-key"
fi
ls -l /root/.ssh/rsync_backup_ed25519 /root/.ssh/rsync_backup_ed25519.pub
'

We now have a dedicated Ed25519 keypair. The private key stays on the backup host; the public key will be installed on the source host.

Install the public key on the source host with forced-command restrictions

We are going to copy the public key content and append it to the source host’s authorized_keys for the backup-rsync user, but with restrictions. The restrictions will:

• Disable port forwarding, agent forwarding, and PTY allocation.
• Force the connection to run rsync server mode only.

First, we will capture the public key into a variable on the backup host so we can paste it safely into the next step.

sudo sh -c '
set -eu
PUBKEY=$(cat /root/.ssh/rsync_backup_ed25519.pub)
printf "%sn" "$PUBKEY"
'

We have printed the public key. Next, we will add it on the source host with restrictions.

On the source host, we will create the SSH directory and append a restricted entry. We will also set strict permissions so SSH accepts the key file.

sudo sh -c '
set -eu
install -d -m 0700 -o backup-rsync -g backup-rsync /home/backup-rsync/.ssh
touch /home/backup-rsync/.ssh/authorized_keys
chown backup-rsync:backup-rsync /home/backup-rsync/.ssh/authorized_keys
chmod 0600 /home/backup-rsync/.ssh/authorized_keys
'

The SSH directory and authorized_keys file are now present with correct ownership and permissions.

Now we will append the restricted key line. We will do this in a way that avoids editing mistakes: we will add a single line that includes the restrictions and then the public key. We must replace the public key content in the command below with the exact output we printed earlier. This is the one place where a paste is unavoidable because the key is unique.

sudo sh -c '
set -eu
cat >> /home/backup-rsync/.ssh/authorized_keys <<"EOF"
command="rsync --server --daemon .",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-ed25519 REPLACE_WITH_PUBLIC_KEY_CONTENT rsync-backup-key
EOF
chown backup-rsync:backup-rsync /home/backup-rsync/.ssh/authorized_keys
chmod 0600 /home/backup-rsync/.ssh/authorized_keys
'

We have restricted the key so it cannot open a shell and cannot be used for forwarding. The forced command is intentionally narrow. If we later need a different rsync mode, we should create a separate key with a separate restriction line, not broaden this one.

Now we will harden SSH on the source host to ensure password authentication is disabled (if policy allows) and that the backup user cannot log in with a password anyway. We will do this carefully by creating a drop-in config file rather than editing the main file.

sudo sh -c '
set -eu
SSHD_DROPIN_DIR=/etc/ssh/sshd_config.d
install -d -m 0755 "$SSHD_DROPIN_DIR"
cat > "$SSHD_DROPIN_DIR/60-backup-hardening.conf" <<"EOF"
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no

Match User backup-rsync
  AllowTcpForwarding no
  X11Forwarding no
  PermitTTY no
EOF

sshd -t
systemctl restart ssh || systemctl restart sshd
'

We have applied SSH hardening via a drop-in file, validated the SSH configuration syntax, and restarted the SSH service. This makes the change persistent across reboots and package updates.

We will verify the SSH service is active.

sudo systemctl status ssh 2>/dev/null || sudo systemctl status sshd

If the service is active, the source host is ready for key-based restricted access.

Step 4: Prepare backup storage on the backup host

We are going to create a dedicated backup directory with strict permissions. This matters because backups often contain sensitive data, and we do not want them readable by non-privileged users.

sudo sh -c '
set -eu
install -d -m 0700 /srv/backups
install -d -m 0700 /srv/backups/rsync
'

We now have a protected storage path at /srv/backups/rsync. Only root can read it by default.

Next, we will create a directory for logs. We will keep a dedicated log file in addition to journald so we can ship it to a SIEM or central log system later.

sudo sh -c '
set -eu
install -d -m 0750 /var/log/rsync-backup
touch /var/log/rsync-backup/backup.log
chmod 0640 /var/log/rsync-backup/backup.log
'

We now have a stable log location that survives reboots and can be managed by logrotate if needed.

Step 5: Configure the rsync backup script with validation

We are going to create a single script that performs the backup and includes validation checks. This matters because “it ran” is not the same as “it worked.” We will validate:

• SSH connectivity with the dedicated key.
• Destination path availability.
• Rsync exit code and log output.
• Freshness of the resulting backup (a simple but effective signal).

We will create the script at /usr/local/sbin/rsync-backup.sh.

sudo sh -c '
set -eu
cat > /usr/local/sbin/rsync-backup.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail

# ===== Configuration =====
SOURCE_HOST="${SOURCE_HOST:-}"
SOURCE_USER="backup-rsync"
SSH_KEY="/root/.ssh/rsync_backup_ed25519"
DEST_BASE="/srv/backups/rsync"
LOG_FILE="/var/log/rsync-backup/backup.log"

# What we back up from the source host
INCLUDE_PATHS=(
  "/etc/"
  "/srv/"
  "/var/www/"
)

# Exclusions: adjust to match policy and reduce noise
EXCLUDES=(
  "--exclude=/srv/backups/"
  "--exclude=/srv/**/cache/"
  "--exclude=/var/www/**/cache/"
)

# ===== Helpers =====
log() {
  local msg="$1"
  printf "%s %sn" "$(date -Is)" "$msg" | tee -a "$LOG_FILE"
}

die() {
  local msg="$1"
  log "ERROR: $msg"
  exit 1
}

require_root() {
  if [ "$(id -u)" -ne 0 ]; then
    die "This script must run as root."
  fi
}

require_config() {
  if [ -z "$SOURCE_HOST" ]; then
    die "SOURCE_HOST is not set. Provide it via environment (systemd unit will do this)."
  fi
}

check_prereqs() {
  command -v rsync >/dev/null 2>&1 || die "rsync not found."
  command -v ssh >/dev/null 2>&1 || die "ssh not found."
  [ -f "$SSH_KEY" ] || die "SSH key not found at $SSH_KEY"
}

check_ssh() {
  log "Checking SSH connectivity to ${SOURCE_USER}@${SOURCE_HOST} with restricted key."
  ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 
    "${SOURCE_USER}@${SOURCE_HOST}" "true" 
    >/dev/null 2>&1 || die "SSH connectivity check failed."
  log "SSH connectivity check passed."
}

run_backup() {
  local dest_dir="${DEST_BASE}/${SOURCE_HOST}"
  install -d -m 0700 "$dest_dir"

  log "Starting rsync pull from ${SOURCE_HOST} into ${dest_dir}."

  # We use:
  # -aHAX: preserve attributes where possible
  # --numeric-ids: stable ownership mapping across systems
  # --delete: keep destination aligned with source (use with care)
  # --partial --inplace: resilience for large files (policy-dependent)
  # --stats: useful for auditing
  rsync -aHAX --numeric-ids --delete --partial --inplace --stats 
    "${EXCLUDES[@]}" 
    -e "ssh -i ${SSH_KEY} -o BatchMode=yes -o StrictHostKeyChecking=accept-new" 
    "${INCLUDE_PATHS[@]/#/${SOURCE_USER}@${SOURCE_HOST}:}" 
    "$dest_dir/" 
    2>&1 | tee -a "$LOG_FILE"

  local rc=${PIPESTATUS[0]}
  if [ "$rc" -ne 0 ]; then
    die "rsync failed with exit code $rc."
  fi

  log "rsync completed successfully."
}

validate_freshness() {
  local dest_dir="${DEST_BASE}/${SOURCE_HOST}"
  log "Validating backup freshness in ${dest_dir}."

  # Freshness check: ensure something changed recently (within 36 hours).
  # This is a signal, not a cryptographic guarantee, but it catches silent failures.
  if ! find "$dest_dir" -type f -mtime -2 -print -quit | grep -q .; then
    die "Freshness validation failed: no files updated in the last 48 hours."
  fi

  log "Freshness validation passed."
}

main() {
  require_root
  require_config
  check_prereqs
  check_ssh
  run_backup
  validate_freshness
  log "Backup run finished OK."
}

main "$@"
EOF

chmod 0750 /usr/local/sbin/rsync-backup.sh
'

We have created a production-oriented backup script with explicit configuration, logging, and validation. The script is strict by default: it fails fast, logs clearly, and refuses to run without the required environment variable.

Now we will verify the script is executable and readable only by privileged users.

sudo ls -l /usr/local/sbin/rsync-backup.sh

We should see permissions like -rwxr-x--- (0750). That ensures regular users cannot read the script if we later add sensitive logic.

Step 6: Create a systemd service and timer for reliable scheduling

We are going to schedule the backup using systemd timers. This matters because systemd gives us better control than legacy scheduling: dependency handling, clear logs, and the ability to catch up after downtime with Persistent=true. This is how we keep automation reliable in real environments.

Create the systemd service

We will create a oneshot service that runs the script and passes the source host as an environment variable. First, we will detect the source host value we want to use. In many SMEs, we use DNS. If we do not have DNS, we can use an IP address.

We will print the current hostname of the backup host (for our own clarity) and then we will set the source host value explicitly in the unit file.

hostnamectl status | sed -n '1,8p'

This output helps us confirm we are editing the correct machine before we schedule anything.

Now we will create the service unit. We will set conservative security options and ensure logs go to journald and our log file (the script already writes to the log file).

sudo sh -c '
set -eu
cat > /etc/systemd/system/[email protected] <<"EOF"
[Unit]
Description=Secure rsync backup pull for %i
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
Environment=SOURCE_HOST=%i
ExecStart=/usr/local/sbin/rsync-backup.sh
User=root
Group=root

# Hardening (keep practical; rsync needs filesystem access)
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=/srv/backups /var/log/rsync-backup
LockPersonality=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
'

We have created a templated systemd service. The %i instance name becomes the SOURCE_HOST value, which keeps the design clean if we later add more source hosts.

Now we will verify systemd recognizes the unit.

systemctl cat [email protected]

If the unit content prints, systemd has loaded it successfully.

Create the systemd timer

We are going to create a timer that runs nightly and is persistent. Persistent timers run missed jobs after reboot, which is exactly what we want for backups.

sudo sh -c '
set -eu
cat > /etc/systemd/system/[email protected] <<"EOF"
[Unit]
Description=Nightly secure rsync backup timer for %i

[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true
RandomizedDelaySec=10m
Unit=rsync-backup@%i.service

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
'

We have created a timer that runs at 02:15 with a small randomized delay to avoid thundering herd effects if we later schedule multiple hosts.

Now we will enable and start the timer for a specific source host. We will first detect a sensible source host value. If we have DNS, we can use a hostname. If not, we can use the IP address.

We will print the backup host’s resolver view and routes to help us choose the correct source host name or IP.

sudo sh -c '
set -eu
echo "Resolver:"
( command -v resolvectl >/dev/null 2>&1 && resolvectl status | sed -n "1,80p" ) || cat /etc/resolv.conf
echo
echo "Routes:"
ip route
'

This gives us the network context. Next, we will enable the timer instance. We must choose the instance name carefully because it becomes a directory name under /srv/backups/rsync.

We will set a shell variable for the source host and then enable the timer using that variable so the commands remain copy/paste safe.

SOURCE_HOST="source.example.internal"
sudo systemctl enable --now "rsync-backup@${SOURCE_HOST}.timer"

The timer is now enabled and started. It will run nightly and also catch up after downtime.

Now we will verify the timer is active and see the next run time.

systemctl list-timers --all | sed -n '1,200p' | grep -E 'rsync-backup@|NEXT|LEFT|LAST|PASSED|UNIT' || true

If we see our timer instance listed with a next run time, scheduling is in place.

Step 7: Firewall considerations

We are going to ensure the network path is limited to what we need: SSH from the backup host to the source host. In many SME environments, host firewalls are enabled. We will handle two common cases: UFW and firewalld. If neither is present, we should confirm perimeter firewalls allow TCP/22 from the backup host.

On the source host: allow SSH only from the backup host (recommended)

First, we will detect the backup host’s source IP as seen by the source host. If we are on the same LAN, we can often use the backup host’s static IP. If we are crossing networks, we should use the NATed address that reaches the source host.

From the backup host, we will print its primary IP addresses so we can choose the correct one.

ip -br addr show | awk '{print $1, $3}'

We now have candidate IPs. Next, on the source host, we will apply a rule using the chosen backup host IP.

If UFW is installed and active, we will allow SSH from the backup host IP only.

BACKUP_HOST_IP="192.0.2.10"
sudo sh -c '
set -eu
if command -v ufw >/dev/null 2>&1; then
  ufw status verbose || true
  ufw allow from '"$BACKUP_HOST_IP"' to any port 22 proto tcp
  ufw status verbose
else
  echo "ufw not installed; skipping ufw rules."
fi
'

If UFW is in use, SSH is now explicitly allowed from the backup host IP. This reduces exposure without breaking operations.

If firewalld is installed and running, we will add a rich rule to allow SSH only from the backup host IP.

BACKUP_HOST_IP="192.0.2.10"
sudo sh -c '
set -eu
if systemctl is-active --quiet firewalld 2>/dev/null; then
  firewall-cmd --get-active-zones
  firewall-cmd --permanent --add-rich-rule="rule family=ipv4 source address='"$BACKUP_HOST_IP"'/32 service name=ssh accept"
  firewall-cmd --reload
  firewall-cmd --list-all
else
  echo "firewalld not active; skipping firewalld rules."
fi
'

If firewalld is in use, we have now constrained SSH access to the backup host IP at the host firewall layer.

Step 8: First run and verification

We are going to run the backup service once manually. This matters because it validates the entire chain: SSH key restrictions, permissions, rsync behavior, destination storage, and logging.

First, we will start the service instance directly.

SOURCE_HOST="source.example.internal"
sudo systemctl start "rsync-backup@${SOURCE_HOST}.service"

The backup job has been triggered. Next we will inspect the service status and logs.

We will check the unit status to confirm it exited successfully.

SOURCE_HOST="source.example.internal"
sudo systemctl status "rsync-backup@${SOURCE_HOST}.service" --no-pager

If the status shows Exited with success, the run completed at the systemd level.

Now we will review the journald logs for the run. This is where we see failures clearly if something went wrong.

SOURCE_HOST="source.example.internal"
sudo journalctl -u "rsync-backup@${SOURCE_HOST}.service" --since "today" --no-pager

We should see our script’s log lines, including SSH connectivity checks, rsync stats, and validation results.

Finally, we will verify that data exists on disk and that the directory structure is as expected.

SOURCE_HOST="source.example.internal"
sudo sh -c '
set -eu
dest="/srv/backups/rsync/'"$SOURCE_HOST"'"
ls -ld "$dest"
du -sh "$dest" || true
find "$dest" -maxdepth 2 -type d | sed -n "1,40p"
'

We have confirmed the backup directory exists, has content, and is readable by root. This is the minimum operational proof that the pipeline is working.

Operational practices that keep this safe over time

• Key rotation: generate a new key annually (or per policy), add it alongside the old key, validate, then remove the old key.
• Retention: rsync mirrors current state. If we need point-in-time recovery, we should add snapshotting on the backup filesystem (LVM/ZFS/btrfs) or use --link-dest rotation. The mirror is still valuable as a fast restore baseline.
• Restore drills: a backup is only real after a restore. We should periodically restore a directory to a staging path and verify application behavior.
• Monitoring: alert on systemd unit failures and on freshness validation failures. Journald logs can be forwarded to a central system.

Troubleshooting

When backups fail, the symptoms are usually consistent. We will keep this section practical: symptom, likely cause, and fix.

Symptom: systemd service fails with “SSH connectivity check failed”

• Likely causes:
  • Firewall blocks TCP/22 between backup host and source host.
  • Wrong SOURCE_HOST value (DNS mismatch, wrong IP).
  • SSH key not installed correctly or permissions on authorized_keys are wrong.
  • SSH server not running on the source host.
• Fix:
  • On the source host, verify SSH is listening: sudo ss -ltnp | awk 'NR==1 || $4 ~ /:22$/ {print}'
  • From the backup host, test connectivity with verbose SSH:

    SOURCE_HOST="source.example.internal"
    sudo ssh -vvv -i /root/.ssh/rsync_backup_ed25519 -o BatchMode=yes -o ConnectTimeout=10 "backup-rsync@${SOURCE_HOST}" true

  • On the source host, verify permissions:

    sudo sh -c '
    set -eu
    ls -ld /home/backup-rsync /home/backup-rsync/.ssh
    ls -l /home/backup-rsync/.ssh/authorized_keys
    '

Symptom: rsync runs but copies almost nothing, or errors with “permission denied”

• Likely causes:
  • The backup-rsync user cannot read the target directories.
  • ACLs or application permissions block reads even if group permissions look correct.
  • We are trying to back up paths that do not exist on the source host.
• Fix:
  • On the source host, validate readable files as the backup user:

    sudo -u backup-rsync sh -c '
    set -eu
    for d in /etc /srv /var/www; do
      if [ -d "$d" ]; then
        echo "Checking $d"
        find "$d" -maxdepth 2 -type f -readable -print -quit | sed -n "1p"
      fi
    done
    '

  • If a directory must remain restricted, we should exclude it explicitly in the script rather than broadening permissions.

Symptom: service succeeds but validation fails with “no files updated in the last 48 hours”

• Likely causes:
  • The source data truly did not change (possible for static systems).
  • Rsync is syncing but timestamps are preserved and nothing appears “new” by mtime.
  • The backup is writing to an unexpected destination directory due to a wrong instance name.
• Fix:
  • Confirm the destination path matches the instance name:

    SOURCE_HOST="source.example.internal"
    sudo ls -ld "/srv/backups/rsync/${SOURCE_HOST}"

  • Review rsync stats in the log:

    sudo tail -n 200 /var/log/rsync-backup/backup.log

  • If the system is truly static, we can adjust the freshness logic to check for presence of expected files instead of mtime.

Common mistakes

Mistake: SSH refuses the key with “Permission denied (publickey)”

• Symptom: Permission denied (publickey) in SSH verbose output.
• Cause: wrong ownership/permissions on /home/backup-rsync/.ssh or authorized_keys, or the key line is malformed.
• Fix:

  sudo sh -c '
  set -eu
  chown -R backup-rsync:backup-rsync /home/backup-rsync/.ssh
  chmod 0700 /home/backup-rsync/.ssh
  chmod 0600 /home/backup-rsync/.ssh/authorized_keys
  '

Mistake: Backups “work” but restore is incomplete

• Symptom: missing application data during restore, even though rsync logs look clean.
• Cause: backing up file paths without capturing application-consistent state (databases, queues, or files written during backup).
• Fix: add pre-backup steps (database dumps to a dedicated directory) and then rsync that directory, or use filesystem snapshots on the source host if available.

Mistake: The backup host fills up unexpectedly

• Symptom: disk alerts, rsync failures, or system instability on the backup host.
• Cause: no retention planning, backing up large transient directories, or unexpected growth.
• Fix: exclude transient paths, monitor /srv/backups usage, and implement snapshot retention or tiered storage.

How do we at NIILAA look at this

This setup is not impressive because it is complex. It is impressive because it is controlled. Every component is intentional. Every configuration has a reason. This is how infrastructure should scale — quietly, predictably, and without drama.

At NIILAA, we help organizations design, deploy, secure, and maintain backup and recovery systems that hold up under real pressure: least-privilege access, hardened transport, validation, monitoring, retention strategy, and restore drills that prove the business can recover. If we want rsync to behave like enterprise infrastructure, we build it with enterprise discipline.

• Website: https://www.niilaa.com
• Email: [email protected]
• LinkedIn: https://www.linkedin.com/company/niilaa
• Facebook: https://www.facebook.com/niilaa.llc