531 lines
18 KiB
Markdown
531 lines
18 KiB
Markdown
# Wazuh SIEM
|
|
|
|
Self-hosted Wazuh all-in-one deployment (Manager + Indexer + Dashboard) running on a Proxmox VM converted from the official Wazuh Amazon Machine Image (AMI) OVA.
|
|
|
|
---
|
|
|
|
## Repository Contents
|
|
|
|
```
|
|
wazuh/
|
|
├── configs/
|
|
│ ├── ossec.conf # Manager main config
|
|
│ ├── local_rules.xml # Custom detection rules + suppression overrides
|
|
│ ├── local_decoder.xml # Custom decoders (pfSense, Gitea, Mailcow)
|
|
│ ├── internal_options.conf # Internal tuning options
|
|
│ └── filebeat.yml # Filebeat config (manager → indexer pipeline)
|
|
└── README.md
|
|
```
|
|
|
|
---
|
|
|
|
## Infrastructure
|
|
|
|
| Component | Role |
|
|
|-----------|------|
|
|
| Wazuh Manager | Event analysis, rule matching, alert generation |
|
|
| Wazuh Indexer (OpenSearch) | Alert storage and search - `https://127.0.0.1:9200` |
|
|
| Wazuh Dashboard | Web UI |
|
|
| Filebeat | Ships manager alerts → indexer |
|
|
|
|
**VM origin:** Official Wazuh AMI OVA exported from AWS and imported into Proxmox. Standard ext4 on `/dev/sda1` - no LVM. Expand disk via Proxmox resize + `growpart /dev/sda 1` + `resize2fs /dev/sda1`.
|
|
|
|
---
|
|
|
|
## `ossec.conf` - Key Settings
|
|
|
|
### Global / Email Alerts
|
|
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| `email_notification` | `yes` |
|
|
| `email_to` | `chris@wittenberger.us` |
|
|
| `smtp_server` | `localhost` (Postfix relay - see email setup below) |
|
|
| `email_from` | `<redacted>` |
|
|
| `email_maxperhour` | `12` |
|
|
| `email_alert_level` | **12** |
|
|
| `log_alert_level` | `3` |
|
|
| `logall` | **`yes`** (raw archives enabled - monitor `/var/ossec/logs/archives/` disk usage) |
|
|
| `logall_json` | `no` |
|
|
|
|
> **Email setup:** Wazuh sends to `localhost`. Postfix is installed on the Wazuh VM and configured to relay through Mailcow (`mail.wittenberger.us:587`) using SASL auth. The relay credential is stored in `/etc/postfix/sasl_passwd` and hashed with `postmap`. Sender address is `wazuh-alerts@wittenberger.us`.
|
|
|
|
> **Disk note:** `logall` is enabled, meaning raw events are written to `/var/ossec/logs/archives/`. Combined with the indexer this is the main source of disk growth. Monitor with `du -sh /var/ossec/logs/archives/` and consider disabling if disk becomes an issue.
|
|
|
|
### Agent Communication
|
|
|
|
| Port | Protocol | Purpose |
|
|
|------|----------|---------|
|
|
| `1514` | TCP | Agent event forwarding |
|
|
| `1515` | TCP | Agent enrollment |
|
|
| `514` | UDP | Syslog ingestion (pfSense, `10.0.0.0/8` allowed) |
|
|
| `1516` | TCP | Cluster (disabled) |
|
|
|
|
### Active Integrations
|
|
|
|
| Integration | Status | Notes |
|
|
|-------------|--------|-------|
|
|
| VirusTotal | **Enabled** | Monitors syscheck rule IDs 550, 553, 554 - API key in `ossec.conf` |
|
|
| Vulnerability Detection | Enabled | Feed update every 60 minutes |
|
|
| SCA | Enabled | Scans every 12 hours |
|
|
| Syscheck (FIM) | Enabled | Monitors `/etc`, `/usr/bin`, `/usr/sbin`, `/bin`, `/sbin`, `/boot` |
|
|
| Rootcheck | Enabled | Every 12 hours; ignores `/var/lib/docker/overlay2` and `/var/lib/containerd` |
|
|
| Syscollector | Enabled | Hardware, OS, packages, ports, processes, users, services, browser extensions |
|
|
| CIS-CAT | Disabled | - |
|
|
| Osquery | Disabled | - |
|
|
|
|
### Whitelisted IPs
|
|
|
|
Active response whitelist is in the second `<global>` block - replace `WHITELISTED_IPs` placeholder with your LAN ranges.
|
|
|
|
### Log Sources (Manager-local)
|
|
|
|
| Source | Format |
|
|
|--------|--------|
|
|
| journald | `journald` |
|
|
| `/var/log/audit/audit.log` | `audit` |
|
|
| `/var/ossec/logs/active-responses.log` | `syslog` |
|
|
| `df -P` (command) | `command` - every 360s |
|
|
| `netstat -tulpn` (command) | `full_command` - every 360s |
|
|
| `last -n 20` (command) | `full_command` - every 360s |
|
|
|
|
---
|
|
|
|
## `local_decoder.xml` - Custom Decoders
|
|
|
|
### pfSense (syslog over UDP 514)
|
|
|
|
Decodes `filterlog` lines forwarded from pfSense. Extracts: `id`, `action`, `protocol`, `srcip`, `dstip`, `srcport`, `dstport`. Uses `offset="after_regex"` chaining across multiple child decoders to walk the CSV filterlog format.
|
|
|
|
### Mailcow journald unwrap
|
|
|
|
Catches double-wrapped postfix log lines that arrive via journald with the format:
|
|
```
|
|
postfix(PID): HOSTNAME TIMESTAMP HOSTNAME message
|
|
```
|
|
Unwraps and passes to standard Postfix decoders.
|
|
|
|
### Gitea
|
|
|
|
Simple `<program_name>gitea</program_name>` parent match. Child decoder `gitea-auth-fail` extracts `user` and `srcip` from failed authentication lines:
|
|
```
|
|
2026/05/29 14:19:59 routers/web/... [W] Failed authentication attempt for <user> from <ip>
|
|
```
|
|
> **Note:** Earlier decoder iterations using `<prematch>` with regex failed because OS_Regex does not support PCRE alternation (`|`). The current working approach uses `<program_name>` for the parent match and a targeted `<prematch>` only on the specific child decoder that needs it.
|
|
|
|
---
|
|
|
|
## `local_rules.xml` - Custom Rules
|
|
|
|
### pfSense Rules
|
|
|
|
| Rule ID | Level | Description |
|
|
|---------|-------|-------------|
|
|
| `87699` | 0 | pfSense wrapped syslog parent (bridge rule) |
|
|
| `87761` | 5 | pfSense firewall drop event |
|
|
| `87762` | 10 | Multiple pfSense blocks from same source (18 hits / 45s, ignore 240s) - MITRE T1110 |
|
|
|
|
### AlienVault Reputation
|
|
|
|
| Rule ID | Level | Description |
|
|
|---------|-------|-------------|
|
|
| `100100` | 10 | Source IP matched in AlienVault blacklist - fires on any web/attack group alert |
|
|
|
|
### Dovecot / Mailcow Suppression
|
|
|
|
| Rule ID | Suppresses | Reason |
|
|
|---------|-----------|--------|
|
|
| `100200` | SID 9705 | Mailcow watchdog health check user |
|
|
| `100201` | SID 9707 | Mailcow watchdog IMAP probe disconnect (internal `172.22.1.x`) |
|
|
| `100202` | SID 9706 | Own-mailbox routine session disconnect |
|
|
| `100300` | SID 9701 | All Dovecot successful logins (routine IMAP polling noise) |
|
|
| `100301` | SID 9706 | Mailcow watchdog managesieve healthcheck disconnect |
|
|
|
|
### Gitea Rules
|
|
|
|
| Rule ID | Level | Description |
|
|
|---------|-------|-------------|
|
|
| `100400` | 0 | Gitea parent (all gitea-decoded events) |
|
|
| `100401` | 0 | Router polling - suppressed |
|
|
| `100402` | 0 | Router completed request - suppressed |
|
|
| `100410` | 5 | Failed authentication attempt - MITRE T1110 |
|
|
| `100411` | 10 | Brute force: 5+ failures in 120s - MITRE T1110 |
|
|
| `100420` | 5 | New user account created |
|
|
| `100421` | 7 | User account deleted |
|
|
| `100430` | 5 | SSH key added to account |
|
|
| `100431` | 3 | SSH key removed from account |
|
|
| `100440` | 5 | Access token activity |
|
|
| `100450` | 5 | Password reset / account recovery |
|
|
| `100460` | 7 | Repository deleted |
|
|
| `100470` | 5 | 2FA event |
|
|
|
|
---
|
|
|
|
## `filebeat.yml`
|
|
|
|
Ships manager alerts to the local indexer at `127.0.0.1:9200` over HTTPS. Archives shipping is **disabled** (`archives.enabled: false`). Certificates are at `/etc/filebeat/certs/`. Credentials are loaded from environment variables (`${username}` / `${password}`) - not hardcoded.
|
|
|
|
Log retention: 7 files at `/var/log/filebeat/`.
|
|
|
|
---
|
|
|
|
## `internal_options.conf`
|
|
|
|
Largely stock AMI defaults. Notable values:
|
|
|
|
| Option | Value | Notes |
|
|
|--------|-------|-------|
|
|
| `monitord.keep_log_days` | `31` | Manager log rotation - 31 days |
|
|
| `monitord.size_rotate` | `512` MB | Rotate logs at 512 MB |
|
|
| `logcollector.input_threads` | `4` | Parallel log ingestion threads |
|
|
| `analysisd.rlimit_nofile` | `458752` | High FD limit for busy manager |
|
|
| `remoted.worker_pool` | `4` | Agent communication workers |
|
|
|
|
All debug levels are `0` - production setting.
|
|
|
|
---
|
|
|
|
## Post-Install: Changing Default Passwords
|
|
|
|
The Wazuh AMI ships with default credentials for the Dashboard and indexer API. These **must** be changed before putting the instance on any network. The official tool is `wazuh-passwords-tool.sh`, included with the installation.
|
|
|
|
### Change the Dashboard (web UI) admin password
|
|
|
|
```bash
|
|
# Stop dependent services first
|
|
systemctl stop wazuh-dashboard filebeat
|
|
|
|
# Run the password tool
|
|
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh \
|
|
-u admin -p '<new_password>'
|
|
|
|
# Restart services
|
|
systemctl start wazuh-dashboard filebeat
|
|
systemctl restart wazuh-manager
|
|
```
|
|
|
|
### Change all passwords at once (recommended on fresh install)
|
|
|
|
```bash
|
|
systemctl stop wazuh-dashboard filebeat
|
|
|
|
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -a
|
|
|
|
systemctl start wazuh-dashboard filebeat
|
|
systemctl restart wazuh-manager
|
|
```
|
|
|
|
The `-a` flag rotates all internal users (admin, kibanaserver, logstash, etc.) and prints the new passwords to stdout - **save these immediately**, they are not stored anywhere recoverable.
|
|
|
|
> **Filebeat credentials:** After any password change, update the Filebeat keystore to match. `filebeat.yml` loads credentials via `${username}` / `${password}` from the keystore, not from the file directly. If Filebeat stops shipping alerts after a password rotation, the keystore is the first place to check:
|
|
> ```bash
|
|
> filebeat keystore list
|
|
> echo '<new_password>' | filebeat keystore add password --force --stdin
|
|
> systemctl restart filebeat
|
|
> ```
|
|
|
|
---
|
|
|
|
## Disk Management
|
|
|
|
`logall` is **enabled** - raw archives accumulate at `/var/ossec/logs/archives/`. Monitor:
|
|
|
|
```bash
|
|
df -h /
|
|
du -sh /var/ossec/logs/archives/
|
|
du -sh /var/lib/wazuh-indexer/
|
|
```
|
|
|
|
Indexer retention is managed via **ISM policies** in the Dashboard:
|
|
- **Indexer Management → Index State Management → Policies**
|
|
- Recommended homelab retention: **30 days**
|
|
|
|
To emergency-delete old indexer indices:
|
|
```bash
|
|
curl -u admin:<password> -k -X DELETE \
|
|
"https://localhost:9200/wazuh-alerts-4.x-YYYY.MM.DD"
|
|
```
|
|
|
|
---
|
|
|
|
## Useful Commands
|
|
|
|
**Validate config/decoders before any restart:**
|
|
```bash
|
|
/var/ossec/bin/wazuh-analysisd -t && echo "CONFIG OK"
|
|
```
|
|
|
|
**Test a log line through the full pipeline:**
|
|
```bash
|
|
echo '<log line>' | /var/ossec/bin/wazuh-logtest
|
|
```
|
|
|
|
**Service control:**
|
|
```bash
|
|
systemctl restart wazuh-manager
|
|
systemctl restart wazuh-indexer
|
|
systemctl restart wazuh-dashboard
|
|
systemctl restart filebeat
|
|
/var/ossec/bin/wazuh-control status
|
|
```
|
|
|
|
**Live alert stream:**
|
|
```bash
|
|
tail -f /var/ossec/logs/alerts/alerts.log
|
|
```
|
|
|
|
**Agent list:**
|
|
```bash
|
|
/var/ossec/bin/agent_control -l
|
|
```
|
|
|
|
---
|
|
|
|
## Known Gaps / Future Work
|
|
|
|
- **XFF / real client IP** - Web traffic through NPM means web attack alerts attribute NPM's IP as the source. Custom XFF decoder not yet written.
|
|
- **Mailcow Postfix/rspamd rules** - Suppression rules are in place; detection rules (spam relay attempts, auth failures) not yet written.
|
|
- **Active response** - `firewall-drop` and `host-deny` commands are defined but no active response blocks are configured. Not enabled.
|
|
- **Whitelist placeholder** - `WHITELISTED_IPs` in the second `<global>` block needs replacing with actual LAN ranges.
|
|
- **`WAZUH_SERVER_IP` placeholder** - syslog remote listener has a placeholder local IP that needs setting.
|
|
- **`email_from` placeholder** - `YOUR_EMAIL` in `ossec.conf` needs a real address.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**Decoder/rule syntax error on restart**
|
|
Always run `-t` first. OS_Regex (used in decoder `<prematch>`) does not support `|` alternation, `?`, or `{n,m}`. Use `\w`, `\d`, or character classes `[...]` instead.
|
|
|
|
**Agents disconnected**
|
|
Verify TCP 1514 and 1515 are open inbound from agent IPs.
|
|
|
|
**pfSense logs not arriving**
|
|
Verify UDP 514 is open, pfSense is sending to the Wazuh IP, and `allowed-ips` covers the pfSense source.
|
|
|
|
**VirusTotal alerts not firing**
|
|
Check API key is valid and not rate-limited. VirusTotal free tier is 4 requests/minute.
|
|
|
|
**Email not arriving**
|
|
Wazuh sends to `localhost` → Postfix relays to Mailcow via SASL. Check each step:
|
|
|
|
```bash
|
|
# Is Postfix running?
|
|
systemctl status postfix
|
|
|
|
# Test the full relay path manually
|
|
echo "Test from Wazuh" | mail -s "Postfix Test" -r "wazuh-alerts@wittenberger.us" chris@wittenberger.us
|
|
|
|
# Check Postfix relay logs
|
|
tail -50 /var/log/maillog
|
|
```
|
|
|
|
SASL credentials are in `/etc/postfix/sasl_passwd` (hashed via `postmap`):
|
|
```
|
|
[mail.wittenberger.us]:587 wazuh-alerts@wittenberger.us:<password>
|
|
```
|
|
|
|
If credentials changed in Mailcow, update the file and rehash:
|
|
```bash
|
|
postmap /etc/postfix/sasl_passwd
|
|
systemctl restart postfix
|
|
```
|
|
|
|
---
|
|
|
|
## pfSense Syslog Integration
|
|
|
|
pfSense does **not** run a Wazuh agent - it ships firewall logs over the network via **syslog-ng** to the Wazuh manager's syslog listener on **UDP 514**.
|
|
|
|
### pfSense Side (syslog-ng)
|
|
|
|
In pfSense: **Status → System Logs → Settings → Remote Logging**
|
|
|
|
| Setting | Value |
|
|
|---------|-------|
|
|
| Remote log server | `<wazuh-ip>:514` |
|
|
| Protocol | UDP |
|
|
| Log contents | Firewall events (filterlog) |
|
|
|
|
syslog-ng wraps the filterlog lines in a standard syslog envelope before forwarding. The lines arrive at Wazuh looking like:
|
|
```
|
|
<134>May 29 12:00:01 pfsense.local filterlog[1234]: 5,,,,...
|
|
```
|
|
|
|
### Wazuh Side
|
|
|
|
The syslog remote listener in `ossec.conf` accepts these:
|
|
```xml
|
|
<remote>
|
|
<connection>syslog</connection>
|
|
<port>514</port>
|
|
<protocol>udp</protocol>
|
|
<allowed-ips>10.0.0.0/8</allowed-ips>
|
|
<local_ip>WAZUH_SERVER_IP</local_ip>
|
|
</remote>
|
|
```
|
|
|
|
> **Placeholder:** `WAZUH_SERVER_IP` in `ossec.conf` needs to be set to the actual Wazuh VM IP.
|
|
|
|
The `pfsense-wrapped` decoder in `local_decoder.xml` matches on `filterlog` and extracts `id`, `action`, `protocol`, `srcip`, `dstip`, `srcport`, `dstport` via chained `offset="after_regex"` decoders walking the CSV format. Rules 87761/87762 in `local_rules.xml` handle single block events and brute-force correlation respectively.
|
|
|
|
---
|
|
|
|
## Agent Setup
|
|
|
|
### Installation (Linux - Debian/Ubuntu/Amazon Linux)
|
|
|
|
```bash
|
|
# Add Wazuh repo
|
|
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
|
|
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
|
|
> /etc/apt/sources.list.d/wazuh.list
|
|
apt update && apt install -y wazuh-agent
|
|
|
|
# For RHEL/Amazon Linux:
|
|
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
|
|
# dnf install -y wazuh-agent
|
|
```
|
|
|
|
### Enrollment
|
|
|
|
```bash
|
|
WAZUH_MANAGER="<wazuh-ip>" WAZUH_AGENT_NAME="<hostname>" systemctl enable --now wazuh-agent
|
|
```
|
|
|
|
Or manually enroll then start:
|
|
```bash
|
|
/var/ossec/bin/agent-auth -m <wazuh-ip> -A <hostname>
|
|
systemctl enable --now wazuh-agent
|
|
```
|
|
|
|
Verify from the **manager**:
|
|
```bash
|
|
/var/ossec/bin/agent_control -l
|
|
```
|
|
|
|
> **Cloned VMs:** After cloning a Proxmox template, delete `/var/ossec/etc/client.keys` before starting the agent so it re-enrolls with a fresh identity. Stale/duplicate agent entries on the manager should be cleared periodically via `manage_agents` → Remove.
|
|
|
|
---
|
|
|
|
## Agent `ossec.conf` - Log Sources
|
|
|
|
Each agent's `/var/ossec/etc/ossec.conf` controls what that host ships to the manager. The `<localfile>` blocks must be on the **agent**, not the manager.
|
|
|
|
### Standard blocks on all Linux agents
|
|
|
|
```xml
|
|
<!-- System journal -->
|
|
<localfile>
|
|
<log_format>journald</log_format>
|
|
<location>journald</location>
|
|
</localfile>
|
|
|
|
<!-- Audit log -->
|
|
<localfile>
|
|
<log_format>audit</log_format>
|
|
<location>/var/log/audit/audit.log</location>
|
|
</localfile>
|
|
```
|
|
|
|
### Docker hosts - journald filter method (preferred)
|
|
|
|
For hosts running Docker containers that log to journald (via `logging: driver: journald` in compose), use a filtered journald block. This is cleaner than tailing JSON log files and feeds Wazuh's built-in decoders directly.
|
|
|
|
**Mailcow host** (`/opt/mailcow-dockerized/docker-compose.override.yml` sets `tag:` per container):
|
|
```xml
|
|
<localfile>
|
|
<log_format>journald</log_format>
|
|
<location>journald</location>
|
|
<filter field="CONTAINER_TAG">^(postfix|dovecot|rspamd)$</filter>
|
|
</localfile>
|
|
```
|
|
|
|
**Gitea host:**
|
|
```xml
|
|
<localfile>
|
|
<log_format>journald</log_format>
|
|
<location>journald</location>
|
|
<filter field="CONTAINER_TAG">^gitea$</filter>
|
|
</localfile>
|
|
```
|
|
|
|
The `CONTAINER_TAG` field is set by the `tag:` option in the container's `logging:` block in `docker-compose.override.yml`. Without this, all container output lands in the journal under the container ID, which is unusable for filtering.
|
|
|
|
**Prerequisite - journald must be persistent:**
|
|
```bash
|
|
# Check
|
|
ls /var/log/journal/ && echo "PERSISTENT" || echo "VOLATILE"
|
|
|
|
# Fix if volatile
|
|
sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
|
|
mkdir -p /var/log/journal
|
|
systemd-tmpfiles --create --prefix /var/log/journal
|
|
systemctl restart systemd-journald
|
|
```
|
|
|
|
### Docker hosts - bind-mount file method (web app logs)
|
|
|
|
For containers writing nginx/apache-format logs to a bind-mounted host path, use `apache` format to feed Wazuh's built-in web attack ruleset (5500/5700 range):
|
|
|
|
```xml
|
|
<localfile>
|
|
<log_format>apache</log_format>
|
|
<location>/path/to/app/logs/nginx/access.log</location>
|
|
</localfile>
|
|
<localfile>
|
|
<log_format>apache</log_format>
|
|
<location>/path/to/app/logs/nginx/error.log</location>
|
|
</localfile>
|
|
```
|
|
|
|
Add to compose to enable the bind mount:
|
|
```yaml
|
|
volumes:
|
|
- ./logs/nginx:/var/log/nginx
|
|
```
|
|
|
|
> **XFF note:** Web apps behind NPM log NPM's IP as the source, not the real client. The `X-Forwarded-For` field contains the real IP but the standard apache decoder ignores it. A custom XFF decoder is needed for accurate source attribution - not yet implemented.
|
|
|
|
### Non-Docker hosts - app-specific log files
|
|
|
|
```xml
|
|
<!-- Vaultwarden (LXC) -->
|
|
<localfile>
|
|
<log_format>syslog</log_format>
|
|
<location>/opt/vaultwarden/data/logs/access.log</location>
|
|
</localfile>
|
|
```
|
|
|
|
> **File permissions:** The agent runs as the `wazuh` user. Log files must be world-readable (`chmod o+r`) or the agent user must be added to the owning group. Verify with `sudo -u wazuh cat <logfile>`.
|
|
|
|
---
|
|
|
|
## Agent Troubleshooting
|
|
|
|
**Agent not appearing on manager**
|
|
```bash
|
|
# On agent
|
|
systemctl status wazuh-agent
|
|
tail -50 /var/ossec/logs/ossec.log # look for "Connected to the server"
|
|
```
|
|
|
|
**Log file not being monitored**
|
|
```bash
|
|
grep -i "Analyzing file\|localfile" /var/ossec/logs/ossec.log
|
|
```
|
|
If the file path doesn't appear, the `<localfile>` block is either missing, on the wrong machine (manager instead of agent), or the agent wasn't restarted after the edit.
|
|
|
|
**journald filter not working**
|
|
```bash
|
|
# Confirm the CONTAINER_TAG field exists
|
|
journalctl CONTAINER_TAG=<tag> -n 20
|
|
```
|
|
If nothing returns, the container isn't logging to journald. Check the compose `logging:` block and recreate the container.
|
|
|
|
**Duplicate agent IDs after cloning**
|
|
Delete `/var/ossec/etc/client.keys` on the clone before starting the agent. On the manager, remove the stale entry:
|
|
```bash
|
|
/var/ossec/bin/manage_agents
|
|
# choose R (remove), enter the old agent ID
|
|
``` |