How secure access quietly becomes the hardest problem in the room

In the beginning, access is simple. A small team, a few servers, and a handful of trusted networks. Then the organization grows. New environments appear. Vendors need temporary access. Engineers rotate on-call. A new region comes online. Suddenly, “who can reach what” becomes a moving target, and the old habit of letting administrators connect straight to internal systems starts to feel less like convenience and more like risk.

This is where bastion hosts (jump servers) earn their keep. Not as a shiny security feature, but as a calm, controlled choke point: one place to authenticate, one place to log, one place to enforce policy. The goal is controlled access that scales without turning operations into a daily firefight.

Architecture we are implementing

We are going to implement a production-grade bastion host on Ubuntu that provides controlled administrative access to private systems without exposing those systems to the internet. The bastion becomes the only inbound entry point, and everything else stays private.

• Bastion host: Ubuntu server with a public IP, hardened, tightly firewalled, and heavily logged.
• Private targets: application servers, databases, and internal services reachable only from the bastion (or internal networks).
• Controlled access: identity-based access, least privilege, auditable sessions, and explicit network paths.

We will also explicitly avoid the common anti-pattern of exposing internal servers directly to the internet. The bastion is the boundary.

Prerequisites and assumptions

Before we touch configuration, we need to be clear about the environment assumptions. These are the conditions that make the steps below safe and predictable in enterprise environments.

• OS: Ubuntu Server 22.04 LTS or 24.04 LTS on the bastion host. The commands below are compatible with both.
• Access method for initial provisioning: We assume we already have console access (cloud console, IPMI/iDRAC, or equivalent) or an existing secure management path to perform initial setup. This avoids relying on risky ad-hoc exposure during bootstrap.
• Privileges: We assume we have a user with sudo privileges on the bastion.
• Networking:
  • The bastion has a public IP and at least one network interface connected to the internet.
  • The bastion can reach private targets over an internal network (VPC/VNet/subnet, VPN, or peering).
  • Private targets do not have public inbound exposure.
• Identity: We will use local Linux users and SSH public keys for access control. In enterprise deployments, this typically maps to centralized identity (SSO/IdP) via PAM/SSSD, but we will keep the implementation self-contained and production-safe.
• Logging: We will enable strong audit trails (system logs + session recording via a hardened shell wrapper). For deeper compliance, this can be extended to a SIEM.
• Change control: We assume we can schedule a maintenance window for firewall and access policy changes.

Step 1: Baseline the bastion host and lock down the OS

We will start by updating the system, installing a minimal set of security tools, and enabling time synchronization. This reduces drift, closes known vulnerabilities, and ensures logs have reliable timestamps.

sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install ufw fail2ban auditd unattended-upgrades chrony jq
sudo systemctl enable --now chrony
sudo systemctl enable --now auditd

We have now patched the host, installed a host firewall (UFW), brute-force protection (Fail2ban), auditing (auditd), automatic security updates, and time sync (chrony). This is the baseline that makes later controls meaningful.

We will verify that time sync and auditing are active, because broken time or missing audit trails undermines incident response.

systemctl status chrony --no-pager
chronyc tracking
systemctl status auditd --no-pager

We have confirmed that chrony is tracking a time source and auditd is running. If either is inactive, we should fix that before proceeding.

Step 2: Create controlled admin identities and remove casual privilege paths

We will create a dedicated group for bastion access and a dedicated admin user. In enterprises, this maps cleanly to role-based access: membership in a group becomes the gate.

We will also ensure that only explicitly authorized users can access the bastion, and that privilege escalation is auditable.

sudo groupadd --force bastion-access
sudo adduser --disabled-password --gecos "" bastionadmin
sudo usermod -aG bastion-access,sudo bastionadmin

We have created a bastion-access group and a bastionadmin user, then granted controlled administrative capability via sudo. Next, we will enforce key-based authentication for this user.

We will create the SSH directory with correct permissions. Permissions matter here: loose permissions can cause key authentication to fail or create security gaps.

sudo install -d -m 0700 -o bastionadmin -g bastionadmin /home/bastionadmin/.ssh
sudo install -m 0600 -o bastionadmin -g bastionadmin /dev/null /home/bastionadmin/.ssh/authorized_keys

We have created a secure home for SSH keys. Now we will add a public key. Because keys differ per organization, we will place the key via an editor in a controlled way rather than embedding placeholders into commands.

We will open the authorized keys file and paste the approved public key on a single line.

sudoedit /home/bastionadmin/.ssh/authorized_keys

We have now established a key-based identity for the bastion admin. Next, we will harden the SSH daemon to enforce controlled access patterns.

Step 3: Harden SSH on the bastion for controlled access

We will harden the SSH service to reduce attack surface and enforce policy. The key principles are:

• Disable password authentication to reduce credential stuffing risk.
• Disable root login to force accountability through named users.
• Restrict access to a specific group (bastion-access).
• Reduce forwarding features to only what we explicitly need for controlled administration.
• Increase logging signal for investigations.

We will first back up the existing SSH configuration so we can roll back safely.

sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.$(date +%F_%H%M%S)

We have created a timestamped backup of the SSH configuration. Now we will replace the configuration with a hardened, explicit version to avoid “mystery defaults” across OS releases.

sudo tee /etc/ssh/sshd_config >/dev/null <<'EOF'
# Hardened SSH configuration for a bastion host (Ubuntu)
# Principle: controlled access, least privilege, strong auditability.

Port 22
Protocol 2

# Identity and authentication
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
PubkeyAuthentication yes
AuthenticationMethods publickey

# Restrict who can access the bastion
AllowGroups bastion-access

# Reduce attack surface
X11Forwarding no
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
GatewayPorts no
PermitUserEnvironment no

# Session behavior
ClientAliveInterval 300
ClientAliveCountMax 2
LoginGraceTime 30
MaxAuthTries 3
MaxSessions 5

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Host keys and crypto defaults are managed by Ubuntu/OpenSSH packages.
EOF

We have enforced key-only authentication, blocked root login, restricted access to the bastion-access group, and disabled forwarding features by default. This is a controlled baseline. If we later need specific forwarding for operational reasons, we will enable it narrowly and intentionally.

We will validate the SSH configuration before restarting the service. This prevents lockouts caused by syntax errors.

sudo sshd -t
sudo systemctl restart ssh
sudo systemctl status ssh --no-pager

We have validated the configuration and restarted SSH successfully. The service status confirms it is running with the new policy.

Step 4: Enforce host firewall policy with UFW

We will now enforce a firewall policy that matches the bastion’s purpose: minimal inbound exposure and explicit outbound control. In most enterprise environments, inbound should be limited to known corporate egress IP ranges or VPN ranges.

Because IP ranges vary, we will implement a safe baseline: deny inbound by default, allow outbound by default, and allow inbound SSH only. Then we will show how to restrict inbound to known ranges in a copy/paste-safe way by first printing the current rules and interface details.

We will enable UFW with a deny-by-default inbound stance.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp comment 'Bastion access (restrict by source IP in enterprise)'
sudo ufw --force enable
sudo ufw status verbose

We have activated the firewall, blocked all unsolicited inbound traffic, and allowed inbound TCP/22. The status output confirms the active policy. In enterprise deployments, we should now restrict that inbound rule to known source ranges.

We will restrict inbound access by source IP ranges. First, we will list current rules so we can remove the broad allow rule safely.

sudo ufw status numbered

We now have numbered rules. Next, we will delete the broad allow rule and replace it with source-restricted rules. Because source ranges differ, we will add them as variables in a safe way by prompting for values at runtime.

We will set shell variables for approved source ranges and apply them. We will keep the commands copy/paste-safe by using read to capture values interactively.

read -r -p "Enter approved source CIDR #1 (e.g., corporate egress): " SRC1
read -r -p "Enter approved source CIDR #2 (optional, press Enter to skip): " SRC2

# Remove the broad allow rule if it exists (we will delete by matching rule text manually if needed).
# If the broad rule is present, delete it by number shown in 'ufw status numbered'.
echo "Now run: sudo ufw status numbered  (and delete the broad 22/tcp allow rule by number if present)"

sudo ufw allow from "$SRC1" to any port 22 proto tcp comment 'Bastion access - approved source #1'
if [ -n "$SRC2" ]; then
  sudo ufw allow from "$SRC2" to any port 22 proto tcp comment 'Bastion access - approved source #2'
fi

sudo ufw status verbose

We have now constrained bastion access to approved source networks. This is one of the most important “controlled access” moves: even strong authentication benefits from reduced exposure.

Step 5: Add brute-force protection with Fail2ban

We will configure Fail2ban to watch authentication logs and temporarily ban IPs that generate repeated failures. This is not a substitute for strong authentication, but it reduces noise and slows down opportunistic scanning.

We will create a dedicated jail configuration so package updates do not overwrite our settings.

sudo tee /etc/fail2ban/jail.d/sshd-bastion.local >/dev/null <<'EOF'
[sshd]
enabled = true
port = 22
backend = systemd
maxretry = 5
findtime = 10m
bantime = 1h
EOF

sudo systemctl enable --now fail2ban
sudo systemctl restart fail2ban

We have enabled Fail2ban for SSH and set a reasonable retry window and ban duration. This will persist across reboots because the service is enabled.

We will verify Fail2ban is running and that the SSH jail is active.

sudo systemctl status fail2ban --no-pager
sudo fail2ban-client status
sudo fail2ban-client status sshd

We have confirmed Fail2ban is active and monitoring SSH authentication events.

Step 6: Implement session accountability and command logging

Enterprises rarely fail because they cannot connect. They fail because they cannot explain what happened after someone connected. We will add two layers:

• Auditd for system-level auditing.
• Session logging for interactive shell sessions on the bastion.

We will configure a controlled session logging mechanism using script so every interactive shell session is recorded to a protected directory. This is not perfect “screen recording,” but it is a strong, practical baseline that works well in production.

We will create a protected log directory and a profile script that starts session recording for interactive shells.

sudo install -d -m 0730 -o root -g bastion-access /var/log/bastion-sessions

sudo tee /etc/profile.d/bastion-session-logging.sh >/dev/null <<'EOF'
# Start session logging for interactive shells on the bastion.
# Logs are stored under /var/log/bastion-sessions and are readable only by root and bastion-access group.

if [ -n "$PS1" ] && [ -z "$BASTION_SESSION_LOGGED" ]; then
  export BASTION_SESSION_LOGGED=1
  TS="$(date -u +%Y%m%dT%H%M%SZ)"
  LOG_DIR="/var/log/bastion-sessions"
  LOG_FILE="${LOG_DIR}/${USER}_${TS}_$$.log"

  umask 077
  exec /usr/bin/script -q -f "$LOG_FILE"
fi
EOF

We have created a protected directory for session logs and a profile script that automatically records interactive sessions. The logs are protected by filesystem permissions and will persist across reboots.

We will verify permissions and confirm the profile script is present.

ls -ld /var/log/bastion-sessions
ls -l /etc/profile.d/bastion-session-logging.sh

We have confirmed the directory permissions and the logging hook. On the next interactive login, a session log file will be created automatically.

Step 7: Controlled access to private targets through the bastion

Now we connect the bastion to its real job: reaching private targets without exposing them publicly. The enterprise pattern is simple:

• Private targets accept administrative access only from the bastion’s private IP (or bastion subnet).
• Administrators authenticate to the bastion with strong identity controls.
• From the bastion, administrators reach targets over private networking.

We will first identify the bastion’s private IP address so we can use it in target-side firewall rules and security groups.

ip -br addr show
BAS_PRIVATE_IP=$(ip -4 route get 1.1.1.1 | awk '{for(i=1;i<=NF;i++) if ($i=="src") {print $(i+1); exit}}')
echo "Bastion private IP detected as: $BAS_PRIVATE_IP"

We have printed interface addressing and captured the bastion’s primary source IP used for outbound routing. This is typically the private IP in cloud networks, but we should confirm it matches our internal routing design.

Next, we will define the target-side rule conceptually: allow administrative access only from $BAS_PRIVATE_IP (or the bastion subnet). The exact implementation depends on whether targets use cloud security groups, on-host firewalls, or both. The key is that targets should not accept inbound administrative access from the internet.

Target-side firewall example (Ubuntu targets using UFW)

We will show a safe, explicit example for Ubuntu targets using UFW. We will deny inbound by default and allow administrative access only from the bastion’s private IP. We will also keep outbound allowed so the target can reach patch repositories and internal services.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow administrative access only from the bastion private IP
# Replace the value below with the bastion private IP printed earlier.
read -r -p "Enter bastion private IP (from bastion output): " BAS_PRIVATE_IP
sudo ufw allow from "$BAS_PRIVATE_IP" to any port 22 proto tcp comment 'Admin access only from bastion'

sudo ufw --force enable
sudo ufw status verbose

We have now enforced controlled access on the target: only the bastion can initiate administrative connections. This is the core security win of the bastion pattern.

Step 8: Operational guardrails (patching, reboots, and persistence)

We will ensure security updates are applied automatically for critical packages. This reduces the window of exposure without relying on perfect human scheduling.

sudo dpkg-reconfigure -plow unattended-upgrades
sudo systemctl enable --now unattended-upgrades
systemctl status unattended-upgrades --no-pager

We have enabled unattended upgrades and confirmed the service is active. In enterprise environments, we typically pair this with maintenance windows and staged rollouts, but the bastion should never be allowed to drift unpatched.

We will also verify that our key services are enabled to persist across reboots.

systemctl is-enabled ssh
systemctl is-enabled ufw
systemctl is-enabled fail2ban
systemctl is-enabled auditd
systemctl is-enabled chrony

We have confirmed that the bastion’s critical controls will survive reboots and maintenance cycles.

Verification checklist (what “good” looks like)

We will now verify the bastion is behaving like a controlled access point. These checks are designed to be quick, repeatable, and meaningful during audits and incident response.

• SSH policy is enforced: password authentication is disabled, root login is disabled, and group restriction is active.
• Firewall is active: inbound is denied by default, and access is restricted to approved sources.
• Brute-force protection is active: Fail2ban is running and monitoring SSH.
• Audit and session logs exist: auditd is running and session logs are being written.

We will run a consolidated set of verification commands.

sudo sshd -T | egrep -i 'passwordauthentication|permitrootlogin|allowgroups|loglevel|allowtcpforwarding|allowagentforwarding|x11forwarding'
sudo ufw status verbose
sudo fail2ban-client status sshd
sudo systemctl status auditd --no-pager
sudo ls -ld /var/log/bastion-sessions
sudo journalctl -u ssh --no-pager -n 50

We have validated the effective SSH settings, confirmed firewall posture, confirmed Fail2ban jail status, confirmed auditd health, confirmed session log directory permissions, and reviewed recent SSH service logs.

Troubleshooting

Symptom: We cannot log in after hardening SSH

• Likely cause: The user is not in the bastion-access group, or the public key is not installed correctly.
• Fix: From console access, confirm group membership and key permissions.

id bastionadmin
sudo getent group bastion-access
sudo ls -ld /home/bastionadmin /home/bastionadmin/.ssh
sudo ls -l /home/bastionadmin/.ssh/authorized_keys
sudo stat -c "%a %U %G %n" /home/bastionadmin/.ssh /home/bastionadmin/.ssh/authorized_keys

We have confirmed whether the user is in the correct group and whether file permissions match secure expectations (0700 for .ssh, 0600 for authorized_keys).

Symptom: Authentication fails even with the correct key

• Likely cause: Wrong ownership/permissions, or the key line is malformed (wrapped lines, extra characters).
• Fix: Reapply strict permissions and re-paste the key as a single line.

sudo chown -R bastionadmin:bastionadmin /home/bastionadmin/.ssh
sudo chmod 0700 /home/bastionadmin/.ssh
sudo chmod 0600 /home/bastionadmin/.ssh/authorized_keys
sudoedit /home/bastionadmin/.ssh/authorized_keys

We have restored correct ownership and permissions and reopened the key file for a clean, single-line key entry.

Symptom: We can reach the bastion, but we cannot reach private targets from it

• Likely cause: Missing route to private subnets, target firewall/security group does not allow bastion private IP, or DNS resolution is not configured.
• Fix: Validate routing, test connectivity, and confirm target-side rules.

# On the bastion: confirm route and test connectivity to a target IP
ip route
read -r -p "Enter a private target IP to test: " TGT_IP
ping -c 3 "$TGT_IP" || true
nc -vz -w 3 "$TGT_IP" 22 || true

# Check DNS if targets are referenced by name
resolvectl status
read -r -p "Enter a private target hostname to resolve: " TGT_HOST
getent ahosts "$TGT_HOST" || true

We have checked routing, basic reachability, port connectivity, and name resolution. The results point directly to whether the issue is network path, firewall policy, or DNS.

Symptom: Fail2ban is running but no bans occur

• Likely cause: Wrong backend/log source, or SSH logs are not being parsed as expected.
• Fix: Confirm the jail is enabled and check Fail2ban logs for parsing errors.

sudo fail2ban-client status sshd
sudo journalctl -u fail2ban --no-pager -n 200
sudo journalctl -u ssh --no-pager -n 200

We have confirmed the jail status and inspected recent Fail2ban and SSH logs to identify parsing or backend mismatches.

Symptom: Session logs are not being created in /var/log/bastion-sessions

• Likely cause: The shell is non-interactive, the profile script is not executed, or permissions prevent writing.
• Fix: Confirm interactive shell behavior and directory permissions.

sudo ls -l /etc/profile.d/bastion-session-logging.sh
sudo ls -ld /var/log/bastion-sessions
echo "$SHELL"
echo "$PS1"

We have confirmed the logging hook exists, the directory is writable for the intended sessions, and the shell context is interactive (session logging triggers only for interactive shells by design).

Common mistakes

Mistake: Leaving inbound access open to the world

• Symptom: UFW shows 22/tcp ALLOW Anywhere and logs show constant authentication noise.
• Fix: Replace broad allow rules with source-restricted rules for corporate egress/VPN ranges.

sudo ufw status numbered

We should delete the broad rule by number and add source-restricted rules as shown earlier. After that, the noise drops and exposure shrinks dramatically.

Mistake: Not restricting bastion access to a dedicated group

• Symptom: Any local user can authenticate if they have a key, and access reviews become messy.
• Fix: Enforce AllowGroups bastion-access and manage membership as the access control plane.

sudo sshd -T | egrep -i 'allowgroups'
getent group bastion-access

We have confirmed the SSH daemon is enforcing group restriction and we can review membership centrally on the host.

Mistake: Treating the bastion like a general-purpose server

• Symptom: Extra services are installed, more ports are opened, and the bastion becomes a “pet server” with unclear purpose.
• Fix: Keep the bastion minimal, patch aggressively, and open only what is required for controlled access.

sudo ss -tulpen
sudo ufw status verbose
dpkg -l | wc -l

We have listed listening services, confirmed firewall posture, and taken a quick inventory signal. The bastion should remain intentionally small and boring.

How do we at NIILAA look at this

This setup is not impressive because it is complex. It is impressive because it is controlled. Every component is intentional. Every configuration has a reason. This is how infrastructure should scale — quietly, predictably, and without drama.

At NIILAA, we help organizations design, deploy, secure, and maintain bastion host patterns that hold up under real operational pressure: access reviews, incident response, audits, and growth. We align identity, network boundaries, logging, and change control so controlled access stays controlled as teams and environments expand.

Website: https://www.niilaa.com
Email: [email protected]
LinkedIn: https://www.linkedin.com/company/niilaa
Facebook: https://www.facebook.com/niilaa.llc