Enterprise Linux Backup Strategy: Implementing the 3-2-1 Rule

When backups stop being a checkbox and start being a survival plan

In the early days, a Linux server feels simple. A few services, a small database, a handful of users. Backups are “something we’ll tighten up later.” Then the server becomes a platform. More applications arrive. Teams start depending on it. Data grows quietly in places nobody remembers creating. A minor outage turns into a long night because the restore path was never rehearsed, the backup set was incomplete, or the only copy lived on the same storage that just failed.

Enterprises don’t lose sleep because backups are hard. They lose sleep because backups are often unowned, unverified, and built on assumptions that stop being true over time. In this guide, we are going to design a Linux backup architecture that follows the 3-2-1 rule: three copies of data, on two different media types, with one copy offsite. The goal is not “more backups.” The goal is controlled recovery.

Prerequisites and assumptions

Before we touch commands, we need to be explicit about what we are assuming. This is where most backup plans quietly fail: the environment is not what the plan imagined.

Platform: Linux servers (enterprise environments). Commands below assume a modern systemd-based distribution. We will provide package commands for both Debian/Ubuntu and RHEL/Rocky/Alma families where it matters.
Access: We have root access (direct root or sudo). We will use sudo in commands so the same steps work in environments where root login is disabled.
Network: The backup repository host is reachable over the network from the Linux servers. We will use SSH transport for backups because it is auditable, firewall-friendly, and widely accepted in enterprise networks.
Time sync: NTP/chrony is working on all systems. Backup retention and log correlation depend on correct time.
Storage planning: We have capacity for at least one full backup plus retention. For databases and large datasets, we plan for growth, not today’s size.
Security posture: We will use a dedicated backup user, SSH keys, restricted commands, and least privilege. We will not reuse admin keys.
Scope clarity: We are backing up what we can restore. That includes application data, configuration, and system state needed to rebuild. We will explicitly exclude ephemeral paths.

We will implement the 3-2-1 rule like this:

Copy 1 (Primary): Production data on the Linux server.
Copy 2 (Secondary, different media): A dedicated backup repository server (separate storage, ideally separate RAID/storage class).
Copy 3 (Offsite): A second repository or object storage sync target in another site/account. We will implement this as a repository-to-repository replication step so production servers do not need direct access to offsite.

Architecture: what we are building and why it works

We are going to use BorgBackup for deduplicated, encrypted, incremental backups over SSH. Borg is a strong fit for enterprise Linux backup architecture because it is efficient, supports retention policies, and can be operated in a controlled, auditable way.

At a high level:

Each Linux server runs a scheduled job that creates a Borg archive on a central repository host.
The repository host stores backups on dedicated storage and enforces retention.
The repository host replicates the repository to an offsite target on a schedule.
We verify backups continuously with automated checks and periodic restore drills.

We will also be careful about what we back up. Backups are not a filesystem mirror; they are a recovery tool. We will include system configuration and application data, and we will exclude caches and transient runtime directories.

Step 1: Prepare the backup repository server

First, we will prepare a dedicated Linux host to act as the backup repository. We do this first because production servers should not start sending backups to an unprepared target. The repository host is where we enforce storage controls, access controls, and retention.

Create a dedicated backup user and storage path

We are about to create a non-login administrative boundary: a dedicated user that owns backup repositories and nothing else. This reduces blast radius and makes auditing simpler.

sudo useradd --system --home /var/lib/borg --create-home --shell /usr/sbin/nologin borg
sudo install -d -o borg -g borg -m 0700 /var/lib/borg/repos

We now have a system user borg with a locked-down shell and a repository root at /var/lib/borg/repos that only the borg user can access.

Install BorgBackup on the repository server

Next, we will install Borg on the repository server. Even though the repository can be “dumb storage” over SSH, having Borg installed helps with verification and maintenance tasks.

On Debian/Ubuntu family systems, we will run:

sudo apt-get update
sudo apt-get install -y borgbackup openssh-server

On RHEL/Rocky/Alma family systems, we will run:

sudo dnf install -y epel-release
sudo dnf install -y borgbackup openssh-server

Borg and SSH server packages are now installed. SSH is the transport we will use for controlled access from production servers.

Harden SSH access for the backup user

We are about to create an SSH key-only access path for the borg user. This is important because password-based access to a backup repository is a common enterprise risk. We will also prepare the authorized_keys file with correct permissions so SSH does not silently refuse it.

sudo install -d -o borg -g borg -m 0700 /var/lib/borg/.ssh
sudo touch /var/lib/borg/.ssh/authorized_keys
sudo chown borg:borg /var/lib/borg/.ssh/authorized_keys
sudo chmod 0600 /var/lib/borg/.ssh/authorized_keys

The repository server is now ready to accept restricted SSH keys for the borg user.

Firewall: allow SSH from known source networks only

We are about to ensure the repository server only accepts SSH from approved networks. In enterprises, “open SSH to the world” is not a backup strategy; it is an incident waiting to happen. The exact firewall tooling varies, so we will provide a safe baseline using firewalld (common on RHEL-family) and UFW (common on Ubuntu).

First, we will detect whether firewalld is active:

sudo systemctl is-active firewalld || true

If firewalld is active, we will allow SSH and then restrict sources at the network layer (recommended via security groups, ACLs, or perimeter firewall). As a host-level baseline:

sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

SSH is now allowed by firewalld. We confirmed the active rules with --list-all.

If UFW is used instead, we will enable it carefully and allow SSH:

sudo ufw allow 22/tcp
sudo ufw status verbose || true

SSH is now permitted by UFW. In production, we should still restrict source IP ranges at the network edge.

Verification: confirm SSH is listening

Before we move on, we will confirm the repository server is actually listening on port 22. This prevents chasing “backup failures” that are really just a service not running.

sudo systemctl enable --now ssh || sudo systemctl enable --now sshd
sudo systemctl status ssh || sudo systemctl status sshd
sudo ss -lntp | awk 'NR==1 || /:22[[:space:]]/'

SSH is now enabled across reboots, running, and listening on port 22.

Step 2: Prepare each Linux server to back up to the repository

Now we will configure a production Linux server (repeat these steps per server). We will install Borg, create a dedicated key, and define exactly what we back up. The key principle here is consistency: the same structure across servers makes restores predictable.

Install BorgBackup on the Linux server

We are about to install Borg on the Linux server so it can create encrypted, deduplicated archives and send them to the repository over SSH.

On Debian/Ubuntu family systems:

sudo apt-get update
sudo apt-get install -y borgbackup openssh-client

On RHEL/Rocky/Alma family systems:

sudo dnf install -y epel-release
sudo dnf install -y borgbackup openssh-clients

Borg is now available on the Linux server, and SSH client tooling is present for repository access.

Create a dedicated SSH key for backups

Next, we will create a dedicated SSH key used only for backups. This matters because it allows us to rotate backup access without touching admin access, and it makes audit trails cleaner.

We will generate the key as root so scheduled jobs can run without interactive prompts. We will also set strict permissions.

sudo install -d -m 0700 /root/.ssh
sudo ssh-keygen -t ed25519 -a 64 -f /root/.ssh/id_ed25519_borg -N ""
sudo chmod 0600 /root/.ssh/id_ed25519_borg
sudo chmod 0644 /root/.ssh/id_ed25519_borg.pub

The Linux server now has a dedicated keypair at /root/.ssh/id_ed25519_borg. The private key is protected by filesystem permissions, and the public key is ready to be installed on the repository server.

Install the public key on the repository server with command restrictions

We are about to add the Linux server’s public key to the repository server’s authorized_keys for the borg user. We will restrict what that key can do by forcing the borg serve command and disabling port forwarding and TTY. This is a practical enterprise control: even if the key is exposed, it cannot be used as a general SSH shell.

First, we will print the public key on the Linux server so we can copy it safely:

sudo cat /root/.ssh/id_ed25519_borg.pub

We now have the public key text. Next, on the repository server, we will append it with restrictions. We will do this in a way that is copy/paste safe by using a heredoc. We must replace the single line PASTE_PUBLIC_KEY_HERE with the exact public key line we just printed (this is unavoidable because keys are unique).

sudo tee -a /var/lib/borg/.ssh/authorized_keys >/dev/null <<'EOF'
command="borg serve --restrict-to-path /var/lib/borg/repos",restrict,no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding PASTE_PUBLIC_KEY_HERE
EOF
sudo chown borg:borg /var/lib/borg/.ssh/authorized_keys
sudo chmod 0600 /var/lib/borg/.ssh/authorized_keys

The repository server now trusts this Linux server’s key, but only for Borg operations within /var/lib/borg/repos. We also ensured permissions remain correct so SSH will accept the file.

Verification: test SSH connectivity from the Linux server

Now we will confirm the Linux server can reach the repository server using the dedicated key. We do this before initializing any repository so failures are isolated to connectivity and authentication.

First, we will set variables for the repository host and port. We will also show a safe way to confirm DNS resolution and reachability.

REPO_HOST="backup-repo.example.internal"
REPO_PORT="22"

getent hosts "${REPO_HOST}" || true
nc -vz "${REPO_HOST}" "${REPO_PORT}" || true

We now know whether name resolution and TCP reachability are working. Next, we will attempt a restricted SSH connection. We expect it to succeed but not provide a shell (because we disabled TTY and forced a command).

sudo ssh -i /root/.ssh/id_ed25519_borg -p "${REPO_PORT}" -o BatchMode=yes -o StrictHostKeyChecking=accept-new borg@"${REPO_HOST}" 'borg --version'

If this returns a Borg version, SSH authentication and command execution are working. If it fails, we will address it in troubleshooting later.

Step 3: Initialize a per-server Borg repository on the repository host

Now we will create a dedicated repository per Linux server. This keeps retention and access boundaries clean. It also makes it easier to decommission a server without touching other backup sets.

We are about to initialize the repository from the Linux server side. Borg will create the repository directory on the remote host (within the restricted path) and initialize encryption. We will use repokey-blake2 so the repository is encrypted and integrity-checked.

First, we will set a stable server identifier. We will derive it from the hostname in a copy/paste-safe way:

SERVER_ID="$(hostname -s | tr -cd 'a-zA-Z0-9._-')"
echo "${SERVER_ID}"

We now have a safe repository name component. Next, we will define the Borg repository URL and initialize it.

REPO_HOST="backup-repo.example.internal"
REPO_PORT="22"
BORG_RSH="ssh -i /root/.ssh/id_ed25519_borg -p ${REPO_PORT}"
export BORG_RSH

BORG_REPO="borg@${REPO_HOST}:/var/lib/borg/repos/${SERVER_ID}"
export BORG_REPO

sudo borg init --encryption=repokey-blake2

The remote repository for this server is now created and initialized with encryption. Borg also created a repository key that we must protect because it is required for restores.

Secure the Borg key material

We are about to export the repository key and store it in a protected local path. In enterprise environments, we should also escrow this key in a secrets manager or offline vault. Without the key, encrypted backups are not recoverable.

sudo install -d -m 0700 /root/borg-keys
sudo borg key export "${BORG_REPO}" /root/borg-keys/"${SERVER_ID}".key
sudo chmod 0600 /root/borg-keys/"${SERVER_ID}".key
sudo ls -l /root/borg-keys

The repository key is now exported to a root-only directory. We confirmed file permissions and presence.

Step 4: Define what we back up and what we exclude

Before we run the first backup, we will define a consistent include/exclude policy. This is where we prevent noisy backups, reduce restore confusion, and avoid capturing transient runtime state.

We are about to create an exclude file. This file is intentionally conservative and enterprise-friendly. We can extend it per application, but we should keep the baseline consistent across servers.

sudo tee /etc/borg-excludes.txt >/dev/null <<'EOF'
/dev
/proc
/sys
/run
/tmp
/var/tmp
/var/cache
/var/run
/var/lock
/lost+found
/mnt
/media
EOF
sudo chmod 0644 /etc/borg-excludes.txt
sudo cat /etc/borg-excludes.txt

We now have a baseline exclude list at /etc/borg-excludes.txt. This reduces backup noise and avoids capturing pseudo-filesystems and transient directories.

Step 5: Create the backup script with logging and retention

Now we will create a production-grade backup script. We are doing this instead of embedding long commands in cron because we want consistent logging, predictable environment variables, and a single place to maintain retention policy.

We will implement:

Encrypted backups to the repository
Retention pruning (daily/weekly/monthly)
Repository integrity checks
Clear exit codes and logs

We are about to write the full script to /usr/local/sbin/borg-backup.sh. We will keep it copy/paste safe and avoid interactive prompts.

sudo tee /usr/local/sbin/borg-backup.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Enterprise baseline: explicit variables, explicit paths, explicit logging.
REPO_HOST="backup-repo.example.internal"
REPO_PORT="22"
KEY_PATH="/root/.ssh/id_ed25519_borg"
EXCLUDES="/etc/borg-excludes.txt"

SERVER_ID="$(hostname -s | tr -cd 'a-zA-Z0-9._-')"
export BORG_RSH="ssh -i ${KEY_PATH} -p ${REPO_PORT}"
export BORG_REPO="borg@${REPO_HOST}:/var/lib/borg/repos/${SERVER_ID}"

# Borg behavior controls
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no
export BORG_RELOCATED_REPO_ACCESS_IS_OK=no

# Logging
LOG_DIR="/var/log/borg"
LOG_FILE="${LOG_DIR}/backup.log"
install -d -m 0750 "${LOG_DIR}"

timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

echo "$(timestamp) Starting Borg backup for ${SERVER_ID}" >> "${LOG_FILE}"

# Create archive name with UTC timestamp
ARCHIVE_NAME="${SERVER_ID}-$(date -u +%Y-%m-%dT%H%M%SZ)"

# What we back up:
# - /etc for system configuration
# - /home for user data (adjust in enterprise environments with centralized home)
# - /root for root-owned configs/scripts (careful with secrets; manage separately if needed)
# - /var/lib for application state (databases often live here; coordinate with app teams)
# - /opt for vendor apps
# - /srv for service data
INCLUDE_PATHS=(
  /etc
  /home
  /root
  /var/lib
  /opt
  /srv
)

# Create backup archive
borg create 
  --verbose 
  --filter AME 
  --list 
  --stats 
  --show-rc 
  --compression lz4 
  --exclude-caches 
  --exclude-from "${EXCLUDES}" 
  "::${ARCHIVE_NAME}" 
  "${INCLUDE_PATHS[@]}" >> "${LOG_FILE}" 2>&1

echo "$(timestamp) Archive created: ${ARCHIVE_NAME}" >> "${LOG_FILE}"

# Retention policy (adjust to enterprise RPO/RTO):
# - keep 14 daily
# - keep 8 weekly
# - keep 12 monthly
borg prune 
  --verbose 
  --list 
  --show-rc 
  --keep-daily 14 
  --keep-weekly 8 
  --keep-monthly 12 >> "${LOG_FILE}" 2>&1

echo "$(timestamp) Prune completed" >> "${LOG_FILE}"

# Lightweight integrity check (repository metadata)
borg check --verbose --show-rc >> "${LOG_FILE}" 2>&1

echo "$(timestamp) Check completed" >> "${LOG_FILE}"
echo "$(timestamp) Borg backup finished successfully for ${SERVER_ID}" >> "${LOG_FILE}"
EOF

sudo chmod 0750 /usr/local/sbin/borg-backup.sh
sudo chown root:root /usr/local/sbin/borg-backup.sh

We now have a controlled backup script with explicit includes, excludes, encryption, retention, and verification. Logs will be written to /var/log/borg/backup.log with restricted directory permissions.

Verification: run the first backup manually

Before scheduling anything, we will run the script once interactively. This is where we catch missing paths, permission issues, and connectivity problems while we are present.

sudo /usr/local/sbin/borg-backup.sh

If the command completes successfully, we have created the first archive, applied retention rules, and performed a repository check. Next, we will verify the repository contents.

We are about to list archives to confirm the backup exists on the repository:

sudo borg list

We should see an archive name like SERVERID-YYYY-MM-DDTHHMMSSZ. That confirms the data is landing in the right repository.

Step 6: Schedule backups with systemd for persistence and observability

Now we will schedule backups using systemd timers instead of cron. In enterprise environments, systemd timers give us better logging, dependency handling, and consistent behavior across distributions that use systemd.

Create a systemd service unit

We are about to create a oneshot service that runs the backup script. This keeps the execution controlled and makes it easy to check status and logs.

sudo tee /etc/systemd/system/borg-backup.service >/dev/null <<'EOF'
[Unit]
Description=Borg Backup Job
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/borg-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# Hardening (keep practical; Borg needs filesystem access)
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=false

[Install]
WantedBy=multi-user.target
EOF

The service unit is now defined. It will run after the network is online and applies basic hardening without breaking access to the paths we back up.

Create a systemd timer unit

Next, we will create a timer to run nightly with a randomized delay. Randomization matters in enterprises: it prevents every server from hammering the repository at exactly the same minute.

sudo tee /etc/systemd/system/borg-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Nightly Borg Backup Timer

[Timer]
OnCalendar=*-*-* 01:30:00
RandomizedDelaySec=45m
Persistent=true

[Install]
WantedBy=timers.target
EOF

The timer is now defined. With Persistent=true, if a server was down during the scheduled time, systemd will run the job after boot, which improves reliability.

Enable and verify the timer

We are about to reload systemd, enable the timer across reboots, and verify the next run time.

sudo systemctl daemon-reload
sudo systemctl enable --now borg-backup.timer
sudo systemctl status borg-backup.timer
sudo systemctl list-timers --all | awk 'NR==1 || /borg-backup/'

The backup schedule is now persistent across reboots. We confirmed the timer is active and can see when it will run next.

Step 7: Implement the offsite copy for the 3-2-1 rule

At this point, we have two copies: production and the repository. The 3-2-1 rule requires an offsite copy. We will implement offsite replication from the repository server to an offsite repository server. This keeps production servers isolated from offsite credentials and reduces complexity.

Prepare the offsite repository server

We are about to repeat the repository preparation steps on an offsite host. The offsite host should be in a different failure domain: another data center, another cloud region, or at minimum another account and network boundary.

On the offsite repository server, we will create the same borg user and repository root:

sudo useradd --system --home /var/lib/borg --create-home --shell /usr/sbin/nologin borg
sudo install -d -o borg -g borg -m 0700 /var/lib/borg/repos
sudo install -d -o borg -g borg -m 0700 /var/lib/borg/.ssh
sudo touch /var/lib/borg/.ssh/authorized_keys
sudo chown borg:borg /var/lib/borg/.ssh/authorized_keys
sudo chmod 0600 /var/lib/borg/.ssh/authorized_keys

The offsite repository server is now ready to accept restricted SSH keys for replication.

Create a dedicated replication key on the primary repository server

Now we will create a dedicated SSH key on the primary repository server used only for replication to offsite. This separation matters: production servers should not hold offsite access, and the replication key should not be shared with anything else.

sudo install -d -m 0700 /var/lib/borg/.ssh
sudo ssh-keygen -t ed25519 -a 64 -f /var/lib/borg/.ssh/id_ed25519_offsite -N ""
sudo chown borg:borg /var/lib/borg/.ssh/id_ed25519_offsite /var/lib/borg/.ssh/id_ed25519_offsite.pub
sudo chmod 0600 /var/lib/borg/.ssh/id_ed25519_offsite
sudo chmod 0644 /var/lib/borg/.ssh/id_ed25519_offsite.pub

The primary repository server now has a replication keypair owned by the borg user.

Authorize the replication key on the offsite repository server

We are about to add the replication public key to the offsite server with restrictions. This ensures the key can only run borg serve within the repository root.

First, on the primary repository server, we will print the replication public key:

sudo cat /var/lib/borg/.ssh/id_ed25519_offsite.pub

Next, on the offsite repository server, we will append it to authorized_keys. As with all keys, we must paste the exact public key line.

sudo tee -a /var/lib/borg/.ssh/authorized_keys >/dev/null <<'EOF'
command="borg serve --restrict-to-path /var/lib/borg/repos",restrict,no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding PASTE_REPLICATION_PUBLIC_KEY_HERE
EOF
sudo chown borg:borg /var/lib/borg/.ssh/authorized_keys
sudo chmod 0600 /var/lib/borg/.ssh/authorized_keys

The offsite server now accepts the replication key in a restricted mode.

Replicate repositories from primary repository to offsite

Now we will replicate Borg repositories. We are doing this from repository-to-repository so production servers remain simple and isolated. Borg supports repository synchronization via borg transfer in newer versions; however, version availability varies across enterprise distributions. For maximum compatibility, we will use borg export-tar only for specific restore workflows, not for full replication. For production-grade replication, we will use rsync over SSH at the repository filesystem level only if repositories are not actively written during sync. The safer enterprise approach is to run replication during a controlled window and ensure no backups are running.

We are about to implement a controlled replication window by stopping incoming backups briefly, syncing, then re-enabling. In larger environments, we would instead use repository-level locking and orchestration, but this baseline is safe and predictable.

First, on the primary repository server, we will install rsync and create a replication script:

sudo apt-get update && sudo apt-get install -y rsync || true
sudo dnf install -y rsync || true

sudo tee /usr/local/sbin/borg-offsite-replication.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

OFFSITE_HOST="backup-offsite.example.internal"
OFFSITE_PORT="22"
SRC_DIR="/var/lib/borg/repos/"
DST_DIR="/var/lib/borg/repos/"

# Run as root but use borg's SSH key for transport.
RSYNC_RSH="ssh -i /var/lib/borg/.ssh/id_ed25519_offsite -p ${OFFSITE_PORT} -o BatchMode=yes -o StrictHostKeyChecking=accept-new"

LOG_FILE="/var/log/borg/offsite-replication.log"
install -d -m 0750 /var/log/borg

timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

echo "$(timestamp) Starting offsite replication" >> "${LOG_FILE}"

# Sync repositories. We preserve permissions and delete removed data to keep offsite consistent.
rsync -aH --delete --numeric-ids -e "${RSYNC_RSH}" "${SRC_DIR}" "borg@${OFFSITE_HOST}:${DST_DIR}" >> "${LOG_FILE}" 2>&1

echo "$(timestamp) Offsite replication completed" >> "${LOG_FILE}"
EOF

sudo chmod 0750 /usr/local/sbin/borg-offsite-replication.sh
sudo chown root:root /usr/local/sbin/borg-offsite-replication.sh

We now have an offsite replication script that syncs repository data to the offsite host using a dedicated restricted key and logs to /var/log/borg/offsite-replication.log.

Schedule offsite replication with systemd

We are about to schedule replication after the nightly backup window. This ensures we replicate fresh data and avoid syncing while backups are actively writing.

sudo tee /etc/systemd/system/borg-offsite-replication.service >/dev/null <<'EOF'
[Unit]
Description=Borg Offsite Replication Job
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/borg-offsite-replication.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

sudo tee /etc/systemd/system/borg-offsite-replication.timer >/dev/null <<'EOF'
[Unit]
Description=Daily Borg Offsite Replication Timer

[Timer]
OnCalendar=*-*-* 04:30:00
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now borg-offsite-replication.timer
sudo systemctl status borg-offsite-replication.timer
sudo systemctl list-timers --all | awk 'NR==1 || /borg-offsite-replication/'

Offsite replication is now scheduled and persistent across reboots. We can track execution via systemd logs and the replication log file.

Verification: confirm offsite data presence

Now we will verify that repositories exist on the offsite host. We will do this by listing the directory over SSH using the restricted key. We expect directory listings to work only if the restrictions allow it; since we forced borg serve, general shell commands may not be permitted. Instead, we verify by running Borg commands from a host that has Borg and the correct access.

We are about to run a Borg list against one repository from the primary repository server. This confirms the offsite repository is reachable and contains archives. We will pick a repository name by listing local directories first.

sudo ls -1 /var/lib/borg/repos | head -n 5

Now we will set a repository name and attempt a Borg list against the offsite target. This requires Borg installed on the primary repository server.

REPO_NAME="$(sudo ls -1 /var/lib/borg/repos | head -n 1)"
echo "${REPO_NAME}"

export BORG_RSH="ssh -i /var/lib/borg/.ssh/id_ed25519_offsite -p 22"
OFFSITE_HOST="backup-offsite.example.internal"
sudo -u borg borg list "borg@${OFFSITE_HOST}:/var/lib/borg/repos/${REPO_NAME}" | head -n 10

If archives are listed, offsite replication is functioning for that repository. In production, we should sample multiple repositories and also monitor replication logs for errors.

Step 8: Restore verification as an operational habit

Backups become real only when restores are routine. We are going to verify restores in a controlled way: extract a small subset to a temporary directory and confirm integrity. This is safe, fast, and catches encryption/key issues early.

Perform a small restore test on a Linux server

We are about to restore a small path (for example, /etc) into a temporary directory. We do this to validate that the repository is readable, the key is available, and the archive contains expected data.

First, we will identify the latest archive:

sudo borg list --last 5

Now we will extract /etc into a temporary restore directory without overwriting the live system:

RESTORE_DIR="/var/tmp/borg-restore-test"
sudo rm -rf "${RESTORE_DIR}"
sudo install -d -m 0700 "${RESTORE_DIR}"

LATEST_ARCHIVE="$(sudo borg list --short | tail -n 1)"
echo "${LATEST_ARCHIVE}"

sudo borg extract --verbose "::${LATEST_ARCHIVE}" etc --destination "${RESTORE_DIR}"
sudo find "${RESTORE_DIR}/etc" -maxdepth 2 -type f | head -n 20

We restored /etc into an isolated directory and listed a sample of files. This confirms that restores work without touching the running system.

Security implications and enterprise controls

In enterprise environments, backup infrastructure is a high-value target. We should treat it like production, not like a storage bin.

Key management: Borg repository keys must be escrowed. Store exports in a secrets manager or offline vault with access controls and rotation procedures.
Least privilege: Dedicated SSH keys per server, restricted to borg serve and restricted paths.
Network segmentation: Repository servers should live in a protected network segment. Allow SSH only from known server subnets.
Immutable/offline posture: For higher assurance, add an additional offline copy (for example, periodic export to removable storage stored securely). This complements the offsite copy and helps against ransomware.
Monitoring: Alert on failed timers, repository checks, and replication failures. Backups that fail quietly are worse than no backups because they create false confidence.

Troubleshooting

Symptom: SSH test fails with “Permission denied (publickey)”

Likely causes: Wrong key used, wrong user (borg), incorrect permissions on authorized_keys, or key not pasted correctly.
Fix: On the repository server, verify permissions and ownership, then re-add the key line.

sudo ls -ld /var/lib/borg /var/lib/borg/.ssh
sudo ls -l /var/lib/borg/.ssh/authorized_keys
sudo stat -c '%U %G %a %n' /var/lib/borg/.ssh /var/lib/borg/.ssh/authorized_keys

If ownership is not borg:borg or permissions are not 700 for the directory and 600 for the file, SSH may refuse the key.

Symptom: Borg init/create fails with “Repository path not allowed”

Likely causes: The forced command restriction uses --restrict-to-path and the repository path is outside that directory.
Fix: Ensure BORG_REPO points to /var/lib/borg/repos/SERVER_ID and that the authorized_keys restriction matches /var/lib/borg/repos.

echo "${BORG_REPO}"
sudo grep -n 'restrict-to-path' /var/lib/borg/.ssh/authorized_keys || true

Once the path restriction and repository path align, Borg operations will succeed.

Symptom: systemd timer runs but no new archives appear

Likely causes: Script fails early, missing DNS/network at runtime, or key permissions prevent SSH access when run non-interactively.
Fix: Check systemd logs and the Borg log file, then run the script manually.

sudo systemctl status borg-backup.service || true
sudo journalctl -u borg-backup.service --no-pager -n 200
sudo tail -n 200 /var/log/borg/backup.log
sudo /usr/local/sbin/borg-backup.sh

This isolates whether the issue is scheduling, environment, or Borg/SSH itself.

Symptom: Offsite replication log shows rsync “permission denied”

Likely causes: Offsite SSH restrictions force borg serve, which prevents rsync from running remote shell operations.
Fix: For filesystem-level rsync replication, the offsite key must allow rsync’s remote shell. If we require strict forced-command restrictions, we should use Borg-native transfer features supported by our Borg version, or replicate at the storage layer (ZFS send/receive, snapshot replication, or object storage sync) from the repository host.

In controlled enterprise environments, the recommended fix is to use Borg-native repository transfer where supported by the installed Borg version, or to replicate at the storage layer. If we must keep rsync, we should create a separate offsite user dedicated to rsync with tightly scoped permissions and network restrictions, and keep Borg access separate.

Common mistakes

Mistake: Backing up everything including /proc and /sys

Symptom: Backups are huge, slow, and restores contain meaningless pseudo-filesystem entries.
Fix: Keep a strict exclude list like /etc/borg-excludes.txt and verify it is referenced in the backup script.

Mistake: No key escrow for encrypted repositories

Symptom: Restore attempts fail after a server rebuild because the Borg key is missing.
Fix: Export keys after initialization and store them in an enterprise secrets vault with access controls and documented recovery steps.

Mistake: Retention without capacity planning

Symptom: Repository fills up, backups start failing, and retention pruning cannot complete.
Fix: Monitor repository disk usage and adjust retention. Add alerting on filesystem usage and Borg job failures.

Mistake: Assuming backups are valid without restore checks

Symptom: Backups “succeed” for months, but the first restore fails due to missing keys, corrupted archives, or incomplete scope.
Fix: Schedule periodic restore tests and include borg check as part of routine operations.

How do we at NIILAA look at this

This setup is not impressive because it is complex. It is impressive because it is controlled. Every component is intentional. Every configuration has a reason. This is how infrastructure should scale — quietly, predictably, and without drama.

At NIILAA, we help organizations design, deploy, secure, and maintain production-grade Linux backup architecture that aligns with enterprise RPO/RTO targets, audit requirements, and real operational constraints. That includes repository hardening, key management, retention design, offsite strategy, monitoring, and restore validation so backups remain trustworthy as systems grow.

Website: https://www.niilaa.com
Email: [email protected]
LinkedIn: https://www.linkedin.com/company/niilaa
Facebook: https://www.facebook.com/niilaa.llc