# Wazuh SIEM

Self-hosted Wazuh all-in-one deployment (Manager + Indexer + Dashboard) running on a Proxmox VM converted from the official Wazuh Amazon Machine Image (AMI) OVA.

---

## Repository Contents

```
wazuh/
├── configs/
│   ├── ossec.conf                          # Manager main config
│   ├── local_rules.xml                     # Custom detection rules + suppression overrides
│   ├── local_decoder.xml                   # Custom decoders (pfSense, Gitea, Mailcow)
│   ├── internal_options.conf               # Internal tuning options
│   └── filebeat.yml                        # Filebeat config (manager → indexer pipeline)
└── README.md
```

---

## Infrastructure

| Component | Role |
|-----------|------|
| Wazuh Manager | Event analysis, rule matching, alert generation |
| Wazuh Indexer (OpenSearch) | Alert storage and search - `https://127.0.0.1:9200` |
| Wazuh Dashboard | Web UI |
| Filebeat | Ships manager alerts → indexer |

**VM origin:** Official Wazuh AMI OVA exported from AWS and imported into Proxmox. Standard ext4 on `/dev/sda1` - no LVM. Expand disk via Proxmox resize + `growpart /dev/sda 1` + `resize2fs /dev/sda1`.

---

## `ossec.conf` - Key Settings

### Global / Email Alerts

| Setting | Value |
|---------|-------|
| `email_notification` | `yes` |
| `email_to` | `chris@wittenberger.us` |
| `smtp_server` | `localhost` (Postfix relay - see email setup below) |
| `email_from` | `<redacted>` |
| `email_maxperhour` | `12` |
| `email_alert_level` | **12** |
| `log_alert_level` | `3` |
| `logall` | **`yes`** (raw archives enabled - monitor `/var/ossec/logs/archives/` disk usage) |
| `logall_json` | `no` |

> **Email setup:** Wazuh sends to `localhost`. Postfix is installed on the Wazuh VM and configured to relay through Mailcow (`mail.wittenberger.us:587`) using SASL auth. The relay credential is stored in `/etc/postfix/sasl_passwd` and hashed with `postmap`. Sender address is `wazuh-alerts@wittenberger.us`.

> **Disk note:** `logall` is enabled, meaning raw events are written to `/var/ossec/logs/archives/`. Combined with the indexer this is the main source of disk growth. Monitor with `du -sh /var/ossec/logs/archives/` and consider disabling if disk becomes an issue.

### Agent Communication

| Port | Protocol | Purpose |
|------|----------|---------|
| `1514` | TCP | Agent event forwarding |
| `1515` | TCP | Agent enrollment |
| `514` | UDP | Syslog ingestion (pfSense, `10.0.0.0/8` allowed) |
| `1516` | TCP | Cluster (disabled) |

### Active Integrations

| Integration | Status | Notes |
|-------------|--------|-------|
| VirusTotal | **Enabled** | Monitors syscheck rule IDs 550, 553, 554 - API key in `ossec.conf` |
| Vulnerability Detection | Enabled | Feed update every 60 minutes |
| SCA | Enabled | Scans every 12 hours |
| Syscheck (FIM) | Enabled | Monitors `/etc`, `/usr/bin`, `/usr/sbin`, `/bin`, `/sbin`, `/boot` |
| Rootcheck | Enabled | Every 12 hours; ignores `/var/lib/docker/overlay2` and `/var/lib/containerd` |
| Syscollector | Enabled | Hardware, OS, packages, ports, processes, users, services, browser extensions |
| CIS-CAT | Disabled | - |
| Osquery | Disabled | - |

### Whitelisted IPs

Active response whitelist is in the second `<global>` block - replace `WHITELISTED_IPs` placeholder with your LAN ranges.

### Log Sources (Manager-local)

| Source | Format |
|--------|--------|
| journald | `journald` |
| `/var/log/audit/audit.log` | `audit` |
| `/var/ossec/logs/active-responses.log` | `syslog` |
| `df -P` (command) | `command` - every 360s |
| `netstat -tulpn` (command) | `full_command` - every 360s |
| `last -n 20` (command) | `full_command` - every 360s |

---

## `local_decoder.xml` - Custom Decoders

### pfSense (syslog over UDP 514)

Decodes `filterlog` lines forwarded from pfSense. Extracts: `id`, `action`, `protocol`, `srcip`, `dstip`, `srcport`, `dstport`. Uses `offset="after_regex"` chaining across multiple child decoders to walk the CSV filterlog format.

### Mailcow journald unwrap

Catches double-wrapped postfix log lines that arrive via journald with the format:
```
postfix(PID): HOSTNAME TIMESTAMP HOSTNAME message
```
Unwraps and passes to standard Postfix decoders.

### Gitea

Simple `<program_name>gitea</program_name>` parent match. Child decoder `gitea-auth-fail` extracts `user` and `srcip` from failed authentication lines:
```
2026/05/29 14:19:59 routers/web/... [W] Failed authentication attempt for <user> from <ip>
```
> **Note:** Earlier decoder iterations using `<prematch>` with regex failed because OS_Regex does not support PCRE alternation (`|`). The current working approach uses `<program_name>` for the parent match and a targeted `<prematch>` only on the specific child decoder that needs it.

---

## `local_rules.xml` - Custom Rules

### pfSense Rules

| Rule ID | Level | Description |
|---------|-------|-------------|
| `87699` | 0 | pfSense wrapped syslog parent (bridge rule) |
| `87761` | 5 | pfSense firewall drop event |
| `87762` | 10 | Multiple pfSense blocks from same source (18 hits / 45s, ignore 240s) - MITRE T1110 |

### AlienVault Reputation

| Rule ID | Level | Description |
|---------|-------|-------------|
| `100100` | 10 | Source IP matched in AlienVault blacklist - fires on any web/attack group alert |

### Dovecot / Mailcow Suppression

| Rule ID | Suppresses | Reason |
|---------|-----------|--------|
| `100200` | SID 9705 | Mailcow watchdog health check user |
| `100201` | SID 9707 | Mailcow watchdog IMAP probe disconnect (internal `172.22.1.x`) |
| `100202` | SID 9706 | Own-mailbox routine session disconnect |
| `100300` | SID 9701 | All Dovecot successful logins (routine IMAP polling noise) |
| `100301` | SID 9706 | Mailcow watchdog managesieve healthcheck disconnect |

### Gitea Rules

| Rule ID | Level | Description |
|---------|-------|-------------|
| `100400` | 0 | Gitea parent (all gitea-decoded events) |
| `100401` | 0 | Router polling - suppressed |
| `100402` | 0 | Router completed request - suppressed |
| `100410` | 5 | Failed authentication attempt - MITRE T1110 |
| `100411` | 10 | Brute force: 5+ failures in 120s - MITRE T1110 |
| `100420` | 5 | New user account created |
| `100421` | 7 | User account deleted |
| `100430` | 5 | SSH key added to account |
| `100431` | 3 | SSH key removed from account |
| `100440` | 5 | Access token activity |
| `100450` | 5 | Password reset / account recovery |
| `100460` | 7 | Repository deleted |
| `100470` | 5 | 2FA event |

---

## `filebeat.yml`

Ships manager alerts to the local indexer at `127.0.0.1:9200` over HTTPS. Archives shipping is **disabled** (`archives.enabled: false`). Certificates are at `/etc/filebeat/certs/`. Credentials are loaded from environment variables (`${username}` / `${password}`) - not hardcoded.

Log retention: 7 files at `/var/log/filebeat/`.

---

## `internal_options.conf`

Largely stock AMI defaults. Notable values:

| Option | Value | Notes |
|--------|-------|-------|
| `monitord.keep_log_days` | `31` | Manager log rotation - 31 days |
| `monitord.size_rotate` | `512` MB | Rotate logs at 512 MB |
| `logcollector.input_threads` | `4` | Parallel log ingestion threads |
| `analysisd.rlimit_nofile` | `458752` | High FD limit for busy manager |
| `remoted.worker_pool` | `4` | Agent communication workers |

All debug levels are `0` - production setting.

---

## Post-Install: Changing Default Passwords

The Wazuh AMI ships with default credentials for the Dashboard and indexer API. These **must** be changed before putting the instance on any network. The official tool is `wazuh-passwords-tool.sh`, included with the installation.

### Change the Dashboard (web UI) admin password

```bash
# Stop dependent services first
systemctl stop wazuh-dashboard filebeat

# Run the password tool
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh \
  -u admin -p '<new_password>'

# Restart services
systemctl start wazuh-dashboard filebeat
systemctl restart wazuh-manager
```

### Change all passwords at once (recommended on fresh install)

```bash
systemctl stop wazuh-dashboard filebeat

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -a

systemctl start wazuh-dashboard filebeat
systemctl restart wazuh-manager
```

The `-a` flag rotates all internal users (admin, kibanaserver, logstash, etc.) and prints the new passwords to stdout - **save these immediately**, they are not stored anywhere recoverable.

> **Filebeat credentials:** After any password change, update the Filebeat keystore to match. `filebeat.yml` loads credentials via `${username}` / `${password}` from the keystore, not from the file directly. If Filebeat stops shipping alerts after a password rotation, the keystore is the first place to check:
> ```bash
> filebeat keystore list
> echo '<new_password>' | filebeat keystore add password --force --stdin
> systemctl restart filebeat
> ```

---

## Disk Management

`logall` is **enabled** - raw archives accumulate at `/var/ossec/logs/archives/`. Monitor:

```bash
df -h /
du -sh /var/ossec/logs/archives/
du -sh /var/lib/wazuh-indexer/
```

Indexer retention is managed via **ISM policies** in the Dashboard:
- **Indexer Management → Index State Management → Policies**
- Recommended homelab retention: **30 days**

To emergency-delete old indexer indices:
```bash
curl -u admin:<password> -k -X DELETE \
  "https://localhost:9200/wazuh-alerts-4.x-YYYY.MM.DD"
```

---

## Useful Commands

**Validate config/decoders before any restart:**
```bash
/var/ossec/bin/wazuh-analysisd -t && echo "CONFIG OK"
```

**Test a log line through the full pipeline:**
```bash
echo '<log line>' | /var/ossec/bin/wazuh-logtest
```

**Service control:**
```bash
systemctl restart wazuh-manager
systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
systemctl restart filebeat
/var/ossec/bin/wazuh-control status
```

**Live alert stream:**
```bash
tail -f /var/ossec/logs/alerts/alerts.log
```

**Agent list:**
```bash
/var/ossec/bin/agent_control -l
```

---

## Known Gaps / Future Work

- **XFF / real client IP** - Web traffic through NPM means web attack alerts attribute NPM's IP as the source. Custom XFF decoder not yet written.
- **Mailcow Postfix/rspamd rules** - Suppression rules are in place; detection rules (spam relay attempts, auth failures) not yet written.
- **Active response** - `firewall-drop` and `host-deny` commands are defined but no active response blocks are configured. Not enabled.
- **Whitelist placeholder** - `WHITELISTED_IPs` in the second `<global>` block needs replacing with actual LAN ranges.
- **`WAZUH_SERVER_IP` placeholder** - syslog remote listener has a placeholder local IP that needs setting.
- **`email_from` placeholder** - `YOUR_EMAIL` in `ossec.conf` needs a real address.

---

## Troubleshooting

**Decoder/rule syntax error on restart**
Always run `-t` first. OS_Regex (used in decoder `<prematch>`) does not support `|` alternation, `?`, or `{n,m}`. Use `\w`, `\d`, or character classes `[...]` instead.

**Agents disconnected**
Verify TCP 1514 and 1515 are open inbound from agent IPs.

**pfSense logs not arriving**
Verify UDP 514 is open, pfSense is sending to the Wazuh IP, and `allowed-ips` covers the pfSense source.

**VirusTotal alerts not firing**
Check API key is valid and not rate-limited. VirusTotal free tier is 4 requests/minute.

**Email not arriving**
Wazuh sends to `localhost` → Postfix relays to Mailcow via SASL. Check each step:

```bash
# Is Postfix running?
systemctl status postfix

# Test the full relay path manually
echo "Test from Wazuh" | mail -s "Postfix Test" -r "wazuh-alerts@wittenberger.us" chris@wittenberger.us

# Check Postfix relay logs
tail -50 /var/log/maillog
```

SASL credentials are in `/etc/postfix/sasl_passwd` (hashed via `postmap`):
```
[mail.wittenberger.us]:587  wazuh-alerts@wittenberger.us:<password>
```

If credentials changed in Mailcow, update the file and rehash:
```bash
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
```

---

## pfSense Syslog Integration

pfSense does **not** run a Wazuh agent - it ships firewall logs over the network via **syslog-ng** to the Wazuh manager's syslog listener on **UDP 514**.

### pfSense Side (syslog-ng)

In pfSense: **Status → System Logs → Settings → Remote Logging**

| Setting | Value |
|---------|-------|
| Remote log server | `<wazuh-ip>:514` |
| Protocol | UDP |
| Log contents | Firewall events (filterlog) |

syslog-ng wraps the filterlog lines in a standard syslog envelope before forwarding. The lines arrive at Wazuh looking like:
```
<134>May 29 12:00:01 pfsense.local filterlog[1234]: 5,,,,...
```

### Wazuh Side

The syslog remote listener in `ossec.conf` accepts these:
```xml
<remote>
  <connection>syslog</connection>
  <port>514</port>
  <protocol>udp</protocol>
  <allowed-ips>10.0.0.0/8</allowed-ips>
  <local_ip>WAZUH_SERVER_IP</local_ip>
</remote>
```

> **Placeholder:** `WAZUH_SERVER_IP` in `ossec.conf` needs to be set to the actual Wazuh VM IP.

The `pfsense-wrapped` decoder in `local_decoder.xml` matches on `filterlog` and extracts `id`, `action`, `protocol`, `srcip`, `dstip`, `srcport`, `dstport` via chained `offset="after_regex"` decoders walking the CSV format. Rules 87761/87762 in `local_rules.xml` handle single block events and brute-force correlation respectively.

---

## Agent Setup

### Installation (Linux - Debian/Ubuntu/Amazon Linux)

```bash
# Add Wazuh repo
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
  > /etc/apt/sources.list.d/wazuh.list
apt update && apt install -y wazuh-agent

# For RHEL/Amazon Linux:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# dnf install -y wazuh-agent
```

### Enrollment

```bash
WAZUH_MANAGER="<wazuh-ip>" WAZUH_AGENT_NAME="<hostname>" systemctl enable --now wazuh-agent
```

Or manually enroll then start:
```bash
/var/ossec/bin/agent-auth -m <wazuh-ip> -A <hostname>
systemctl enable --now wazuh-agent
```

Verify from the **manager**:
```bash
/var/ossec/bin/agent_control -l
```

> **Cloned VMs:** After cloning a Proxmox template, delete `/var/ossec/etc/client.keys` before starting the agent so it re-enrolls with a fresh identity. Stale/duplicate agent entries on the manager should be cleared periodically via `manage_agents` → Remove.

---

## Agent `ossec.conf` - Log Sources

Each agent's `/var/ossec/etc/ossec.conf` controls what that host ships to the manager. The `<localfile>` blocks must be on the **agent**, not the manager.

### Standard blocks on all Linux agents

```xml
<!-- System journal -->
<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
</localfile>

<!-- Audit log -->
<localfile>
  <log_format>audit</log_format>
  <location>/var/log/audit/audit.log</location>
</localfile>
```

### Docker hosts - journald filter method (preferred)

For hosts running Docker containers that log to journald (via `logging: driver: journald` in compose), use a filtered journald block. This is cleaner than tailing JSON log files and feeds Wazuh's built-in decoders directly.

**Mailcow host** (`/opt/mailcow-dockerized/docker-compose.override.yml` sets `tag:` per container):
```xml
<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
  <filter field="CONTAINER_TAG">^(postfix|dovecot|rspamd)$</filter>
</localfile>
```

**Gitea host:**
```xml
<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
  <filter field="CONTAINER_TAG">^gitea$</filter>
</localfile>
```

The `CONTAINER_TAG` field is set by the `tag:` option in the container's `logging:` block in `docker-compose.override.yml`. Without this, all container output lands in the journal under the container ID, which is unusable for filtering.

**Prerequisite - journald must be persistent:**
```bash
# Check
ls /var/log/journal/ && echo "PERSISTENT" || echo "VOLATILE"

# Fix if volatile
sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald
```

### Docker hosts - bind-mount file method (web app logs)

For containers writing nginx/apache-format logs to a bind-mounted host path, use `apache` format to feed Wazuh's built-in web attack ruleset (5500/5700 range):

```xml
<localfile>
  <log_format>apache</log_format>
  <location>/path/to/app/logs/nginx/access.log</location>
</localfile>
<localfile>
  <log_format>apache</log_format>
  <location>/path/to/app/logs/nginx/error.log</location>
</localfile>
```

Add to compose to enable the bind mount:
```yaml
volumes:
  - ./logs/nginx:/var/log/nginx
```

> **XFF note:** Web apps behind NPM log NPM's IP as the source, not the real client. The `X-Forwarded-For` field contains the real IP but the standard apache decoder ignores it. A custom XFF decoder is needed for accurate source attribution - not yet implemented.

### Non-Docker hosts - app-specific log files

```xml
<!-- Vaultwarden (LXC) -->
<localfile>
  <log_format>syslog</log_format>
  <location>/opt/vaultwarden/data/logs/access.log</location>
</localfile>
```

> **File permissions:** The agent runs as the `wazuh` user. Log files must be world-readable (`chmod o+r`) or the agent user must be added to the owning group. Verify with `sudo -u wazuh cat <logfile>`.

---

## Agent Troubleshooting

**Agent not appearing on manager**
```bash
# On agent
systemctl status wazuh-agent
tail -50 /var/ossec/logs/ossec.log   # look for "Connected to the server"
```

**Log file not being monitored**
```bash
grep -i "Analyzing file\|localfile" /var/ossec/logs/ossec.log
```
If the file path doesn't appear, the `<localfile>` block is either missing, on the wrong machine (manager instead of agent), or the agent wasn't restarted after the edit.

**journald filter not working**
```bash
# Confirm the CONTAINER_TAG field exists
journalctl CONTAINER_TAG=<tag> -n 20
```
If nothing returns, the container isn't logging to journald. Check the compose `logging:` block and recreate the container.

**Duplicate agent IDs after cloning**
Delete `/var/ossec/etc/client.keys` on the clone before starting the agent. On the manager, remove the stale entry:
```bash
/var/ossec/bin/manage_agents
# choose R (remove), enter the old agent ID
```