NIST-800.53-VM/README.md

# NIST 800-53 Hardened Ubuntu 24.04 - Proxmox VM Template

A hardened Ubuntu 24.04 LTS VM template for Proxmox VE, aligned with NIST 800-53 security controls and validated against CIS Ubuntu 24.04 benchmarks via Wazuh SCA.

The template is built through three scripts run in sequence on a fresh Ubuntu 24.04 Server (minimal) installation. Once complete, the VM is converted to a Proxmox template for rapid, secure clone deployment.

---

## Table of Contents

- [Architecture Overview](#architecture-overview)
- [Prerequisites](#prerequisites)
- [VM Creation in Proxmox](#vm-creation-in-proxmox)
- [SSH Key Setup (Windows)](#ssh-key-setup-windows)
- [Script 1 - partition-layout.sh](#script-1--partition-layoutsh)
- [Script 2 - nist-800-53-harden-v2.sh](#script-2--nist-800-53-harden-v2sh)
- [Script 3 - remediate.sh](#script-3--remediatesh)
- [Deployment Workflow](#deployment-workflow)
- [Post-Template: Cloning](#post-template-cloning)
- [Firewall Management](#firewall-management)
- [NIST 800-53 Control Mapping](#nist-800-53-control-mapping)
- [Wazuh SCA Check Coverage](#wazuh-sca-check-coverage)
- [Known Exceptions](#known-exceptions)
- [Maintenance](#maintenance)
- [Troubleshooting](#troubleshooting)

---

## Architecture Overview

```
┌──────────────────────────────────────────────────────────┐
│                    Proxmox VM                            │
├──────────────┬───────────────────────────────────────────┤
│  Disk 1      │  Disk 2 (LVM - vg_nist)                   │
│  /dev/sda    │  /dev/sdb                                 │
│  64 GB       │  50 GB                                    │
│              │                                           │
│  sda1: EFI   │  lv_var ............ /var        (10 GB)  │
│  sda2: /boot │  lv_var_log ........ /var/log     (8 GB)  │
│  sda3: LVM   │  lv_var_log_audit .. /var/log/audit (4 GB)│
│    └─ /      │  lv_home ........... /home        (8 GB)  │
│              │  lv_tmp ............ /tmp         (2 GB)  │
│              │  lv_var_tmp ........ /var/tmp     (2 GB)  │
│              │  [~16 GB free for future growth]          │
├──────────────┴───────────────────────────────────────────┤
│  Cloud-Init Drive (auto-injected ISO on /dev/sr0)        │
└──────────────────────────────────────────────────────────┘
```

Each mount point carries restrictive options:

| Mount Point       | Options                          |
|-------------------|----------------------------------|
| `/home`           | `nosuid,nodev`                   |
| `/tmp`            | `nosuid,nodev,noexec`            |
| `/var`            | `nosuid,nodev`                   |
| `/var/tmp`        | `nosuid,nodev,noexec`            |
| `/var/log`        | `nosuid,nodev,noexec`            |
| `/var/log/audit`  | `nosuid,nodev,noexec`            |
| `/dev/shm`        | `nosuid,nodev,noexec`            |

---

## Prerequisites

- Proxmox VE 8.x or later
- Ubuntu 24.04 LTS Server ISO (minimal install)
- An SSH key pair on your admin workstation (see [SSH Key Setup](#ssh-key-setup-windows))
- Wazuh agent (optional, for compliance scanning - the scripts pre-configure outbound firewall rules for Wazuh on ports 1514/1515)

---

## VM Creation in Proxmox

Create the VM with these settings before installing Ubuntu:

**System tab:**

- Machine: `q35`
- BIOS: `OVMF (UEFI)` with EFI Disk, pre-enroll keys checked (Secure Boot)
- TPM: `v2.0`
- SCSI Controller: `VirtIO SCSI single`
- Qemu Agent: checked

**Disks:**

- Disk 1 (OS): SCSI, 20–64 GB, Discard checked, IO Thread checked
- Disk 2 (LVM): Added after OS install - SCSI, 50 GB, Discard checked, IO Thread checked

**CPU:** Type `host` (exposes AES-NI), 2+ cores

**Memory:** 2048 MB minimum, Ballooning checked

**Network:** VirtIO (paravirtualized), Firewall checked

**After OS installation, add:**

- The second 50 GB disk (Hardware → Add → Hard Disk)
- A Cloud-Init drive (Hardware → Add → CloudInit Drive)

**Cloud-Init tab:**

- User: your admin username (e.g., `chris`)
- SSH public key: paste your `id_ed25519.pub`
- IP Config: DHCP or static
- DNS: your internal servers

---

## SSH Key Setup (Windows)

Generate an Ed25519 key pair from PowerShell on your admin workstation:

```powershell
ssh-keygen -t ed25519 -C "admin@proxmox"
```

Accept the default path (`C:\Users\YourName\.ssh\id_ed25519`) and set a strong passphrase.

Copy the public key:

```powershell
cat ~/.ssh/id_ed25519.pub
```

This one public key is used for all VMs cloned from the template. Paste it into both the Cloud-Init tab in Proxmox and (during initial setup) the template VM's `~/.ssh/authorized_keys`:

```bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "ssh-ed25519 AAAA...your-key..." >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```

**Optional - enable the SSH agent** so you don't retype the passphrase every session:

```powershell
# One-time (run as Admin)
Get-Service ssh-agent | Set-Service -StartupType Automatic -PassThru | Start-Service

# Then from regular PowerShell
ssh-add ~/.ssh/id_ed25519
```

---

## Script 1 - partition-layout.sh

**Purpose:** Migrates a default single-partition Ubuntu install into a NIST/CIS-compliant LVM layout with six separate mount points on the second disk.

**When to run:** After Ubuntu install, after attaching the second disk, before the hardening script.

```bash
chmod +x partition-layout.sh
sudo ./partition-layout.sh /dev/sdb
```

**What it does:**

1. Installs LVM2 if missing
2. Creates a GPT partition and LVM volume group (`vg_nist`) on the second disk
3. Creates six logical volumes with ext4 filesystems
4. Migrates existing data from `/var`, `/home`, `/tmp`, etc. using rsync
5. Writes fstab entries with NIST-compliant mount options
6. Mounts everything and sets correct permissions (sticky bits, directory modes)
7. Leaves ~16 GB free in the volume group for future growth

**Post-run:** Reboot and verify all six volumes mount automatically:

```bash
sudo reboot
# After reboot:
df -h | grep vg_nist
findmnt -t ext4 | grep vg_nist
```

**Growing a volume later:**

```bash
sudo lvextend -L +5G /dev/vg_nist/lv_var_log
sudo resize2fs /dev/vg_nist/lv_var_log
```

**Known issue:** The script may fail to mount `/var/log/audit` because the mount point doesn't exist on the freshly mounted `/var/log` volume. If this happens:

```bash
sudo mkdir -p /var/log/audit
sudo mount /var/log/audit
sudo chmod 700 /var/log/audit
```

---

## Script 2 - nist-800-53-harden-v2.sh

**Purpose:** Comprehensive OS hardening covering NIST 800-53 controls and CIS Ubuntu 24.04 benchmarks. Uses iptables (not UFW) for Docker compatibility and clean compliance scans.

**When to run:** After partition-layout.sh and a successful reboot confirming all mounts.

```bash
# Optional: set admin username (defaults to chris)
export ADMIN_USER="chris"

chmod +x nist-800-53-harden-v2.sh
sudo ./nist-800-53-harden-v2.sh
```

**What it does (by NIST control family):**

**Access Control (AC):**

- Locks root account, removes unnecessary users
- Configures pam_faillock (5 attempts / 15-min lockout)
- Sets login warning banners (`/etc/issue`, `/etc/issue.net`, `/etc/motd`)
- 15-minute idle session timeout (TMOUT with readonly guard)
- SSH: key-only auth, AllowUsers, PermitRootLogin no, DisableForwarding

**Audit and Accountability (AU):**

- Comprehensive auditd rules covering CIS checks 35731–35748 (sudoers, user emulation, time changes, network changes, file access, DAC modifications, mounts, deletions, MAC policy, chcon/setfacl/chacl/usermod, kernel modules)
- Audit log retention: `keep_logs`, space warnings, admin halt on full
- Chrony NTP (systemd-timesyncd disabled and masked)
- Journald: persistent, compressed, rotated, size-limited
- Rsyslog: file creation mode 0640, remote forwarding placeholder

**Identification and Authentication (IA):**

- Password quality: minlen=15, minclass=4, maxsequence=3, maxrepeat=3
- Password aging: 60-day max, 1-day min, 14-day warning, 30-day inactive lock
- PAM: faillock, pwhistory (remember=24, enforce_for_root), nullok removed
- opasswd file created with strict permissions

**System and Communications Protection (SC):**

- iptables firewall with default deny on all chains
- ICMP type-filtered with rate-limited ping (1/sec, burst 4)
- SSH rate-limited (4 new/min per source IP)
- Outbound allowed: DNS (53), HTTP (80), HTTPS (443), NTP (123), Wazuh (1514/1515)
- DOCKER-USER chain pre-created for Docker compatibility
- Rules persisted via iptables-persistent
- SSH cryptography: Ed25519/RSA-SHA2 host keys, Curve25519/AES-GCM ciphers, SHA2-ETM MACs
- /dev/shm hardened with noexec,nosuid,nodev

**System and Information Integrity (SI):**

- Kernel hardening: ASLR, ptrace restriction, BPF hardening, core dumps disabled, dmesg restricted
- AIDE file integrity monitoring with audit tool coverage
- Unattended security upgrades enabled
- AppArmor enforced and added to GRUB command line
- `audit=1 audit_backlog_limit=8192` added to GRUB

**Configuration Management (CM):**

- Unnecessary services disabled and masked
- Filesystem modules blacklisted (afs, ceph, cifs, exfat, fat, fscache, fuse, gfs2, nfs, nfsd, smbfs, cramfs, squashfs, etc.)
- Network protocol modules blacklisted (dccp, sctp, rds, tipc)
- USB storage disabled
- Telnet, FTP, rsync, nftables, UFW, apport, snapd removed
- File permissions hardened, cron/at restricted, su restricted to sudo group
- Sudo I/O logging enabled

**Template Preparation:**

- QEMU guest agent enabled
- Cloud-init configured for Proxmox (NoCloud/ConfigDrive)
- Machine-id truncated (regenerated on clone)
- SSH host keys removed (regenerated on first boot via firstboot-harden.service)
- initramfs rebuilt, apt cache cleaned, logs truncated

**Post-run:** AIDE database initialization takes 10–15 minutes. After the script completes, reboot:

```bash
sudo reboot
```

---

## Script 3 - remediate.sh

**Purpose:** Post-scan remediation script that fixes remaining Wazuh SCA failures after the initial hardening. Addresses specific check IDs with targeted fixes.

**When to run:** After the hardening script and a Wazuh SCA scan reveals remaining failures. Can also be run standalone on an already-hardened system.

> **Note:** This script was written during the initial hardening iteration and uses UFW for firewall rules (sections 35619–35639). If you ran `nist-800-53-harden-v2.sh` (which uses iptables and removes UFW), the UFW sections will be skipped harmlessly since UFW won't be installed. The iptables rules from the v2 hardening script satisfy the same controls.

```bash
chmod +x remediate.sh
sudo ./remediate.sh
```

**What it fixes (by Wazuh check ID):**

| Check ID    | Description                                        |
|-------------|----------------------------------------------------|
| 35509       | Filesystem kernel modules blacklisted              |
| 35522       | `nodev` added to `/var` mount                      |
| 35537       | AppArmor enabled in GRUB command line              |
| 35545       | Apport disabled, masked, and purged                |
| 35573       | rsync removed                                      |
| 35585       | telnet removed                                     |
| 35587       | ftp removed                                        |
| 35589       | systemd-timesyncd masked (chrony is NTP source)    |
| 35600       | `/etc/cron.allow` created                          |
| 35601       | `/etc/at.allow` created                            |
| 35604–35607 | dccp, sctp, rds, tipc modules blacklisted          |
| 35619–35639 | Firewall fully configured (UFW in remediate.sh, iptables in v2) |
| 35641–35642 | SSH host key permissions set                       |
| 35643–35661 | Complete SSH hardening rewrite                     |
| 35664       | Sudo logfile configured                            |
| 35668       | `su` restricted to sudo group via pam_wheel        |
| 35672–35690 | Full PAM stack rewrite (common-auth, common-account, common-password, common-session) |
| 35681/35683 | Password quality: minclass, maxsequence added      |
| 35694–35698 | Password aging and inactive lock on all users      |
| 35703       | Root umask set in `.bashrc` and `.bash_profile`    |
| 35708       | Journald rotation configured                       |
| 35719       | Rsyslog file creation mode 0640                    |
| 35722       | Log file permissions tightened                     |
| 35725–35726 | `audit=1 audit_backlog_limit=8192` in GRUB         |
| 35728–35730 | auditd log retention: keep_logs, space alerts      |
| 35731–35748 | Complete CIS-compliant audit ruleset               |
| 35752       | Audit config file permissions (640)                |
| 35755       | Audit tool permissions                             |
| 35760       | AIDE integrity rules for audit tools               |
| 35770       | opasswd/opasswd.old permissions                    |

---

## Deployment Workflow

The full sequence from start to template:

```
1.  Create VM in Proxmox (settings above)
2.  Install Ubuntu 24.04 Server (minimal)
3.  Attach second 50 GB disk in Proxmox Hardware tab
4.  Add Cloud-Init drive in Proxmox Hardware tab
5.  Boot VM, log in via Proxmox console (noVNC)
6.  Add your SSH public key to ~/.ssh/authorized_keys
7.  Copy scripts to the VM
8.  Run:  sudo ./partition-layout.sh /dev/sdb
9.  Reboot, verify mounts with df -h
10. Run:  sudo ./nist-800-53-harden-v2.sh
11. Reboot, verify SSH access with key auth
12. (Optional) Install Wazuh agent, run SCA scan
13. (Optional) Run:  sudo ./remediate.sh
14. Reboot, re-scan, review results
15. Set GRUB password (manual - see below)
16. Remove Ubuntu install ISO from CD drive
17. Shut down VM
18. Right-click → Convert to Template (or: qm template <VMID>)
```

### Setting the GRUB Password (Manual Step)

This is the one control that cannot be automated because it requires interactive hash generation:

```bash
# Generate the hash
sudo grub-mkpasswd-pbkdf2
# Enter and confirm your password, copy the output hash

# Create the GRUB config
sudo tee /etc/grub.d/40_custom > /dev/null <<EOF
#!/bin/sh
exec tail -n +3 \$0
set superusers="grubadmin"
password_pbkdf2 grubadmin <PASTE_YOUR_HASH_HERE>
EOF

sudo chmod 755 /etc/grub.d/40_custom
sudo update-grub
```

---

## Post-Template: Cloning

To deploy a new VM from the template:

1. Right-click the template → **Clone** → **Full Clone**
2. Set VM ID and name
3. Go to the clone's **Cloud-Init tab** and configure:
   - User (or keep the default)
   - SSH public key (or keep the template's key)
   - IP configuration (static or DHCP)
   - DNS servers
4. Start the VM

On first boot, the `firstboot-harden.service` automatically:

- Regenerates SSH host keys
- Sets correct host key permissions
- Rebuilds the AIDE integrity database
- Creates `/var/lib/firstboot-done` to prevent re-running

The Cloud-Init subsystem handles hostname, user creation, SSH key injection, and network configuration.

---

## Firewall Management

The v2 hardening script uses iptables with persistent rules. Common management tasks:

**View current rules:**

```bash
sudo iptables -L -v -n --line-numbers
sudo ip6tables -L -v -n --line-numbers
```

**Add a new inbound service (e.g., HTTPS):**

```bash
# Insert before the LOG rule (check line numbers first)
sudo iptables -I INPUT <line> -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4
```

**Add a new outbound service:**

```bash
sudo iptables -I OUTPUT <line> -p tcp --dport 8080 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4
```

**Docker compatibility:** The script pre-creates a `DOCKER-USER` chain. When Docker is installed, it inserts its own rules but respects the `DOCKER-USER` chain for custom filtering. To restrict Docker container traffic:

```bash
sudo iptables -I DOCKER-USER -s 10.0.0.0/8 -j DROP
sudo iptables-save | sudo tee /etc/iptables/rules.v4
```

**Persistent rules** are stored in `/etc/iptables/rules.v4` and `/etc/iptables/rules.v6`, loaded on boot by `netfilter-persistent`.

---

## NIST 800-53 Control Mapping

| Control    | Title                                  | Implementation                                             |
|------------|----------------------------------------|------------------------------------------------------------|
| AC-2       | Account Management                     | Root locked, unnecessary users removed, umask 027          |
| AC-3       | Access Enforcement                     | AppArmor, su restricted to sudo group                      |
| AC-7       | Unsuccessful Logon Attempts            | pam_faillock: 5 attempts, 15-min lockout                   |
| AC-8       | System Use Notification                | /etc/issue, /etc/issue.net, /etc/motd banners              |
| AC-11      | Session Lock                           | TMOUT=900 (15-min idle timeout)                            |
| AC-17      | Remote Access                          | SSH: key-only, AllowUsers, modern crypto                   |
| AU-2/3/12  | Audit Events, Content, Generation      | Comprehensive auditd rules (CIS 4.1.3.1–18)                |
| AU-4       | Audit Log Storage                      | keep_logs, space alerts, admin halt                        |
| AU-8       | Time Stamps                            | Chrony NTP, systemd-timesyncd disabled                     |
| CM-2       | Baseline Configuration                 | Full system update before hardening                        |
| CM-6       | Configuration Settings                 | File permissions, sudo logging, cron/at restrictions       |
| CM-7       | Least Functionality                    | Services disabled, modules blacklisted, packages purged    |
| IA-5       | Authenticator Management               | Password quality/aging, PAM stack, opasswd                 |
| MP-2       | Media Access                           | Mount options (noexec, nosuid, nodev)                      |
| MP-7       | Media Use                              | USB storage module disabled                                |
| SC-5       | Denial-of-Service Protection           | SYN cookies, iptables rate limiting                        |
| SC-7       | Boundary Protection                    | iptables default-deny, connection tracking                 |
| SC-8/13    | Transmission Confidentiality/Crypto    | SSH: Curve25519, AES-GCM, SHA2-ETM                         |
| SC-28      | Protection of Information at Rest      | Filesystem mount hardening                                 |
| SI-2       | Flaw Remediation                       | Unattended security upgrades                               |
| SI-7       | Software Integrity                     | AIDE file integrity monitoring                             |
| SI-16      | Memory Protection                      | ASLR, ptrace, BPF, core dumps disabled                     |

---

## Wazuh SCA Check Coverage

Total checks addressed across all three scripts: **70+**

Checks are tracked by their Wazuh SCA IDs (35xxx series) corresponding to the CIS Ubuntu Linux 24.04 LTS Benchmark. See the summary tables at the end of each script for a complete list of check IDs mapped to their fixes.

---

## Known Exceptions

These checks will continue to fail or require environment-specific action. Document them as compensating controls for auditors.

| Check ID | Description                        | Reason / Compensating Control                                                  |
|----------|------------------------------------|--------------------------------------------------------------------------------|
| 35509    | `fat` module not disabled          | Required for UEFI boot partition (`/boot/efi`). Built into kernel on 24.04 - blacklisting the loadable module is safe but the check may still flag it. |
| 35540    | GRUB bootloader password           | Requires interactive hash generation. Set manually (see Deployment Workflow).  |
| 35589    | systemd-timesyncd not enabled      | Chrony is the NTP source (AU-8). timesyncd is intentionally disabled. Both satisfy the control; the scanner may not detect chrony. |
| 35710    | systemd-journal-upload auth        | Only required if using journal-upload to a remote journal gateway. Most setups use Wazuh or rsyslog instead. |
| 35720    | rsyslog remote logging             | Environment-specific. Placeholder config at `/etc/rsyslog.d/99-remote.conf`. Wazuh agent serves as a compensating control for centralized log collection. |

---

## Maintenance

**Quarterly review checklist:**

- Run Wazuh SCA scan and review any new failures
- Review `/var/log/sudo.log` and audit logs
- Verify AIDE integrity: `sudo aide --check`
- Check that unattended-upgrades is running: `sudo systemctl status unattended-upgrades`
- Review and rotate known-exception documentation
- Verify NTP sync: `chronyc tracking`
- Check LVM free space: `sudo vgdisplay vg_nist | grep Free`

**Updating the template:**

1. Clone the template to a temporary VM
2. Boot, apply updates (`sudo apt update && sudo apt upgrade`)
3. Re-run the hardening script (it's idempotent)
4. Rebuild AIDE database: `sudo aideinit -y -f`
5. Clean up and shut down
6. Convert to new template, retire the old one

---

## Troubleshooting

**Can't SSH in after hardening:**

The hardening script sets `PasswordAuthentication no` and `AllowUsers chris` (or your configured admin user). Verify your SSH key is in `~/.ssh/authorized_keys` and your username matches `AllowUsers`. Use the Proxmox noVNC console as a fallback - it's a virtual console, not SSH.

**TMOUT readonly variable error on login:**

The v2 script includes a guard for this. If you see the error on an older install, replace the timeout script:

```bash
sudo tee /etc/profile.d/nist-timeout.sh > /dev/null <<'EOF'
if [ -z "${TMOUT:-}" ]; then
    readonly TMOUT=900
    export TMOUT
fi
EOF
```

**AIDE takes a long time:**

Initial database creation scans all of `/boot`, `/bin`, `/sbin`, `/lib`, `/usr`, and `/etc`. This takes 10–20 minutes on a VM. Subsequent checks are faster. Verify it's running with `top -c | grep aide`.

**Docker bypasses firewall rules:**

The v2 script pre-creates the `DOCKER-USER` chain. Docker respects rules inserted there. To block specific traffic to containers, add rules to `DOCKER-USER`, not `INPUT`.

**iptables rules lost after reboot:**

Verify `netfilter-persistent` is enabled: `sudo systemctl is-enabled netfilter-persistent`. Rules should be in `/etc/iptables/rules.v4` and `rules.v6`. To re-save current rules: `sudo iptables-save | sudo tee /etc/iptables/rules.v4`.

**Wazuh still flags firewall checks (35626–35639):**

If you ran `remediate.sh` (which uses UFW) and then `nist-800-53-harden-v2.sh` (which removes UFW), the iptables rules satisfy the same CIS controls. Some Wazuh SCA checks are written for a specific firewall tool. Add Wazuh rule-level exceptions for the ones that don't match your chosen tool (see the Wazuh exception guidance in the conversation that produced these scripts).

---

## File Reference

```
.
├── README.md                       # This file
├── partition-layout.sh             # Step 1: LVM partition layout
├── nist-800-53-harden-v2.sh        # Step 2: OS hardening (iptables, CIS/Wazuh aligned)
└── remediate.sh                    # Step 3: Post-scan targeted fixes
```
---

## Post-Clone Identity Reset

When cloning from the template, each new VM must have a unique machine ID and SSH host keys. The `firstboot-harden.service` handles SSH key regeneration automatically, and cloud-init regenerates the machine ID — but if cloud-init is not configured or the firstboot service didn't run, perform these steps manually after the first boot of each clone.

**Regenerate the machine ID:**

```bash
sudo rm -f /etc/machine-id
sudo systemd-machine-id-setup
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
```

**Regenerate SSH host keys:**

```bash
sudo rm -f /etc/ssh/ssh_host_*
sudo ssh-keygen -A
sudo systemctl restart ssh
```

Without these steps, cloned VMs will share the same machine ID (which breaks DHCP leases, logging, and systemd journal identification) and the same SSH host keys (which triggers host key mismatch warnings on your admin workstation and would allow impersonation between clones).

---

## License

These scripts are provided as-is for educational and operational use. Review and test thoroughly in a non-production environment before deployment. Compliance is a shared responsibility - these scripts address technical controls but do not replace organizational policies, procedures, or risk assessments.