wazuh/README.md

Wazuh SIEM

Self-hosted Wazuh all-in-one deployment (Manager + Indexer + Dashboard) running on a Proxmox VM converted from the official Wazuh Amazon Machine Image (AMI) OVA.

---

Repository Contents

wazuh/
├── configs/
│   ├── ossec.conf                          # Manager main config
│   ├── local_rules.xml                     # Custom detection rules + suppression overrides
│   ├── local_decoder.xml                   # Custom decoders (pfSense, Gitea, Mailcow)
│   ├── internal_options.conf               # Internal tuning options
│   └── filebeat.yml                        # Filebeat config (manager → indexer pipeline)
└── README.md

---

Infrastructure

Component	Role
Wazuh Manager	Event analysis, rule matching, alert generation
Wazuh Indexer (OpenSearch)	Alert storage and search - https://127.0.0.1:9200
Wazuh Dashboard	Web UI
Filebeat	Ships manager alerts → indexer

VM origin: Official Wazuh AMI OVA exported from AWS and imported into Proxmox. Standard ext4 on /dev/sda1 - no LVM. Expand disk via Proxmox resize + growpart /dev/sda 1 + resize2fs /dev/sda1.

---

ossec.conf - Key Settings

Global / Email Alerts

Setting	Value
email_notification	yes
email_to	chris@wittenberger.us
smtp_server	localhost (Postfix relay - see email setup below)
email_from	<redacted>
email_maxperhour	12
email_alert_level	12
log_alert_level	3
logall	yes (raw archives enabled - monitor /var/ossec/logs/archives/ disk usage)
logall_json	no

Email setup: Wazuh sends to localhost. Postfix is installed on the Wazuh VM and configured to relay through Mailcow (mail.wittenberger.us:587) using SASL auth. The relay credential is stored in /etc/postfix/sasl_passwd and hashed with postmap. Sender address is wazuh-alerts@wittenberger.us.

Disk note: logall is enabled, meaning raw events are written to /var/ossec/logs/archives/. Combined with the indexer this is the main source of disk growth. Monitor with du -sh /var/ossec/logs/archives/ and consider disabling if disk becomes an issue.

Agent Communication

Port	Protocol	Purpose
1514	TCP	Agent event forwarding
1515	TCP	Agent enrollment
514	UDP	Syslog ingestion (pfSense, 10.0.0.0/8 allowed)
1516	TCP	Cluster (disabled)

Active Integrations

Integration	Status	Notes
VirusTotal	Enabled	Monitors syscheck rule IDs 550, 553, 554 - API key in ossec.conf
Vulnerability Detection	Enabled	Feed update every 60 minutes
SCA	Enabled	Scans every 12 hours
Syscheck (FIM)	Enabled	Monitors /etc, /usr/bin, /usr/sbin, /bin, /sbin, /boot
Rootcheck	Enabled	Every 12 hours; ignores /var/lib/docker/overlay2 and /var/lib/containerd
Syscollector	Enabled	Hardware, OS, packages, ports, processes, users, services, browser extensions
CIS-CAT	Disabled	-
Osquery	Disabled	-

Whitelisted IPs

Active response whitelist is in the second <global> block - replace WHITELISTED_IPs placeholder with your LAN ranges.

Log Sources (Manager-local)

Source	Format
journald	journald
/var/log/audit/audit.log	audit
/var/ossec/logs/active-responses.log	syslog
df -P (command)	command - every 360s
netstat -tulpn (command)	full_command - every 360s
last -n 20 (command)	full_command - every 360s

---

local_decoder.xml - Custom Decoders

pfSense (syslog over UDP 514)

Decodes filterlog lines forwarded from pfSense. Extracts: id, action, protocol, srcip, dstip, srcport, dstport. Uses offset="after_regex" chaining across multiple child decoders to walk the CSV filterlog format.

Mailcow journald unwrap

Catches double-wrapped postfix log lines that arrive via journald with the format:

postfix(PID): HOSTNAME TIMESTAMP HOSTNAME message

Unwraps and passes to standard Postfix decoders.

Gitea

Simple <program_name>gitea</program_name> parent match. Child decoder gitea-auth-fail extracts user and srcip from failed authentication lines:

2026/05/29 14:19:59 routers/web/... [W] Failed authentication attempt for <user> from <ip>

Note: Earlier decoder iterations using <prematch> with regex failed because OS_Regex does not support PCRE alternation (|). The current working approach uses <program_name> for the parent match and a targeted <prematch> only on the specific child decoder that needs it.

---

local_rules.xml - Custom Rules

pfSense Rules

Rule ID	Level	Description
87699	0	pfSense wrapped syslog parent (bridge rule)
87761	5	pfSense firewall drop event
87762	10	Multiple pfSense blocks from same source (18 hits / 45s, ignore 240s) - MITRE T1110

AlienVault Reputation

Rule ID	Level	Description
100100	10	Source IP matched in AlienVault blacklist - fires on any web/attack group alert

Dovecot / Mailcow Suppression

Rule ID	Suppresses	Reason
100200	SID 9705	Mailcow watchdog health check user
100201	SID 9707	Mailcow watchdog IMAP probe disconnect (internal 172.22.1.x)
100202	SID 9706	Own-mailbox routine session disconnect
100300	SID 9701	All Dovecot successful logins (routine IMAP polling noise)
100301	SID 9706	Mailcow watchdog managesieve healthcheck disconnect

Gitea Rules

Rule ID	Level	Description
100400	0	Gitea parent (all gitea-decoded events)
100401	0	Router polling - suppressed
100402	0	Router completed request - suppressed
100410	5	Failed authentication attempt - MITRE T1110
100411	10	Brute force: 5+ failures in 120s - MITRE T1110
100420	5	New user account created
100421	7	User account deleted
100430	5	SSH key added to account
100431	3	SSH key removed from account
100440	5	Access token activity
100450	5	Password reset / account recovery
100460	7	Repository deleted
100470	5	2FA event

---

filebeat.yml

Ships manager alerts to the local indexer at 127.0.0.1:9200 over HTTPS. Archives shipping is disabled (archives.enabled: false). Certificates are at /etc/filebeat/certs/. Credentials are loaded from environment variables (${username} / ${password}) - not hardcoded.

Log retention: 7 files at /var/log/filebeat/.

---

internal_options.conf

Largely stock AMI defaults. Notable values:

Option	Value	Notes
monitord.keep_log_days	31	Manager log rotation - 31 days
monitord.size_rotate	512 MB	Rotate logs at 512 MB
logcollector.input_threads	4	Parallel log ingestion threads
analysisd.rlimit_nofile	458752	High FD limit for busy manager
remoted.worker_pool	4	Agent communication workers

All debug levels are 0 - production setting.

---

Post-Install: Changing Default Passwords

The Wazuh AMI ships with default credentials for the Dashboard and indexer API. These must be changed before putting the instance on any network. The official tool is wazuh-passwords-tool.sh, included with the installation.

Change the Dashboard (web UI) admin password

# Stop dependent services first
systemctl stop wazuh-dashboard filebeat

# Run the password tool
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh \
  -u admin -p '<new_password>'

# Restart services
systemctl start wazuh-dashboard filebeat
systemctl restart wazuh-manager

Change all passwords at once (recommended on fresh install)

systemctl stop wazuh-dashboard filebeat

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -a

systemctl start wazuh-dashboard filebeat
systemctl restart wazuh-manager

The -a flag rotates all internal users (admin, kibanaserver, logstash, etc.) and prints the new passwords to stdout - save these immediately, they are not stored anywhere recoverable.

Filebeat credentials: After any password change, update the Filebeat keystore to match. filebeat.yml loads credentials via ${username} / ${password} from the keystore, not from the file directly. If Filebeat stops shipping alerts after a password rotation, the keystore is the first place to check:

filebeat keystore list
echo '<new_password>' | filebeat keystore add password --force --stdin
systemctl restart filebeat

---

Disk Management

logall is enabled - raw archives accumulate at /var/ossec/logs/archives/. Monitor:

df -h /
du -sh /var/ossec/logs/archives/
du -sh /var/lib/wazuh-indexer/

Indexer retention is managed via ISM policies in the Dashboard:

• Indexer Management → Index State Management → Policies
• Recommended homelab retention: 30 days

To emergency-delete old indexer indices:

curl -u admin:<password> -k -X DELETE \
  "https://localhost:9200/wazuh-alerts-4.x-YYYY.MM.DD"

---

Useful Commands

Validate config/decoders before any restart:

/var/ossec/bin/wazuh-analysisd -t && echo "CONFIG OK"

Test a log line through the full pipeline:

echo '<log line>' | /var/ossec/bin/wazuh-logtest

Service control:

systemctl restart wazuh-manager
systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
systemctl restart filebeat
/var/ossec/bin/wazuh-control status

Live alert stream:

tail -f /var/ossec/logs/alerts/alerts.log

Agent list:

/var/ossec/bin/agent_control -l

---

Known Gaps / Future Work

• XFF / real client IP - Web traffic through NPM means web attack alerts attribute NPM's IP as the source. Custom XFF decoder not yet written.
• Mailcow Postfix/rspamd rules - Suppression rules are in place; detection rules (spam relay attempts, auth failures) not yet written.
• Active response - firewall-drop and host-deny commands are defined but no active response blocks are configured. Not enabled.
• Whitelist placeholder - WHITELISTED_IPs in the second <global> block needs replacing with actual LAN ranges.
• WAZUH_SERVER_IP placeholder - syslog remote listener has a placeholder local IP that needs setting.
• email_from placeholder - YOUR_EMAIL in ossec.conf needs a real address.

---

Troubleshooting

Decoder/rule syntax error on restart Always run -t first. OS_Regex (used in decoder <prematch>) does not support | alternation, ?, or {n,m}. Use \w, \d, or character classes [...] instead.

Agents disconnected Verify TCP 1514 and 1515 are open inbound from agent IPs.

pfSense logs not arriving Verify UDP 514 is open, pfSense is sending to the Wazuh IP, and allowed-ips covers the pfSense source.

VirusTotal alerts not firing Check API key is valid and not rate-limited. VirusTotal free tier is 4 requests/minute.

Email not arriving Wazuh sends to localhost → Postfix relays to Mailcow via SASL. Check each step:

# Is Postfix running?
systemctl status postfix

# Test the full relay path manually
echo "Test from Wazuh" | mail -s "Postfix Test" -r "wazuh-alerts@wittenberger.us" chris@wittenberger.us

# Check Postfix relay logs
tail -50 /var/log/maillog

SASL credentials are in /etc/postfix/sasl_passwd (hashed via postmap):

[mail.wittenberger.us]:587  wazuh-alerts@wittenberger.us:<password>

If credentials changed in Mailcow, update the file and rehash:

postmap /etc/postfix/sasl_passwd
systemctl restart postfix

---

pfSense Syslog Integration

pfSense does not run a Wazuh agent - it ships firewall logs over the network via syslog-ng to the Wazuh manager's syslog listener on UDP 514.

pfSense Side (syslog-ng)

In pfSense: Status → System Logs → Settings → Remote Logging

Setting	Value
Remote log server	<wazuh-ip>:514
Protocol	UDP
Log contents	Firewall events (filterlog)

syslog-ng wraps the filterlog lines in a standard syslog envelope before forwarding. The lines arrive at Wazuh looking like:

<134>May 29 12:00:01 pfsense.local filterlog[1234]: 5,,,,...

Wazuh Side

The syslog remote listener in ossec.conf accepts these:

<remote>
  <connection>syslog</connection>
  <port>514</port>
  <protocol>udp</protocol>
  <allowed-ips>10.0.0.0/8</allowed-ips>
  <local_ip>WAZUH_SERVER_IP</local_ip>
</remote>

Placeholder: WAZUH_SERVER_IP in ossec.conf needs to be set to the actual Wazuh VM IP.

The pfsense-wrapped decoder in local_decoder.xml matches on filterlog and extracts id, action, protocol, srcip, dstip, srcport, dstport via chained offset="after_regex" decoders walking the CSV format. Rules 87761/87762 in local_rules.xml handle single block events and brute-force correlation respectively.

---

Agent Setup

Installation (Linux - Debian/Ubuntu/Amazon Linux)

# Add Wazuh repo
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" \
  > /etc/apt/sources.list.d/wazuh.list
apt update && apt install -y wazuh-agent

# For RHEL/Amazon Linux:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# dnf install -y wazuh-agent

Enrollment

WAZUH_MANAGER="<wazuh-ip>" WAZUH_AGENT_NAME="<hostname>" systemctl enable --now wazuh-agent

Or manually enroll then start:

/var/ossec/bin/agent-auth -m <wazuh-ip> -A <hostname>
systemctl enable --now wazuh-agent

Verify from the manager:

/var/ossec/bin/agent_control -l

Cloned VMs: After cloning a Proxmox template, delete /var/ossec/etc/client.keys before starting the agent so it re-enrolls with a fresh identity. Stale/duplicate agent entries on the manager should be cleared periodically via manage_agents → Remove.

---

Agent ossec.conf - Log Sources

Each agent's /var/ossec/etc/ossec.conf controls what that host ships to the manager. The <localfile> blocks must be on the agent, not the manager.

Standard blocks on all Linux agents

<!-- System journal -->
<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
</localfile>

<!-- Audit log -->
<localfile>
  <log_format>audit</log_format>
  <location>/var/log/audit/audit.log</location>
</localfile>

Docker hosts - journald filter method (preferred)

For hosts running Docker containers that log to journald (via logging: driver: journald in compose), use a filtered journald block. This is cleaner than tailing JSON log files and feeds Wazuh's built-in decoders directly.

Mailcow host (/opt/mailcow-dockerized/docker-compose.override.yml sets tag: per container):

<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
  <filter field="CONTAINER_TAG">^(postfix|dovecot|rspamd)$</filter>
</localfile>

Gitea host:

<localfile>
  <log_format>journald</log_format>
  <location>journald</location>
  <filter field="CONTAINER_TAG">^gitea$</filter>
</localfile>

The CONTAINER_TAG field is set by the tag: option in the container's logging: block in docker-compose.override.yml. Without this, all container output lands in the journal under the container ID, which is unusable for filtering.

Prerequisite - journald must be persistent:

# Check
ls /var/log/journal/ && echo "PERSISTENT" || echo "VOLATILE"

# Fix if volatile
sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Docker hosts - bind-mount file method (web app logs)

For containers writing nginx/apache-format logs to a bind-mounted host path, use apache format to feed Wazuh's built-in web attack ruleset (5500/5700 range):

<localfile>
  <log_format>apache</log_format>
  <location>/path/to/app/logs/nginx/access.log</location>
</localfile>
<localfile>
  <log_format>apache</log_format>
  <location>/path/to/app/logs/nginx/error.log</location>
</localfile>

Add to compose to enable the bind mount:

volumes:
  - ./logs/nginx:/var/log/nginx

XFF note: Web apps behind NPM log NPM's IP as the source, not the real client. The X-Forwarded-For field contains the real IP but the standard apache decoder ignores it. A custom XFF decoder is needed for accurate source attribution - not yet implemented.

Non-Docker hosts - app-specific log files

<!-- Vaultwarden (LXC) -->
<localfile>
  <log_format>syslog</log_format>
  <location>/opt/vaultwarden/data/logs/access.log</location>
</localfile>

File permissions: The agent runs as the wazuh user. Log files must be world-readable (chmod o+r) or the agent user must be added to the owning group. Verify with sudo -u wazuh cat <logfile>.

---

Agent Troubleshooting

Agent not appearing on manager

# On agent
systemctl status wazuh-agent
tail -50 /var/ossec/logs/ossec.log   # look for "Connected to the server"

Log file not being monitored

grep -i "Analyzing file\|localfile" /var/ossec/logs/ossec.log

If the file path doesn't appear, the <localfile> block is either missing, on the wrong machine (manager instead of agent), or the agent wasn't restarted after the edit.

journald filter not working

# Confirm the CONTAINER_TAG field exists
journalctl CONTAINER_TAG=<tag> -n 20

If nothing returns, the container isn't logging to journald. Check the compose logging: block and recreate the container.

Duplicate agent IDs after cloning Delete /var/ossec/etc/client.keys on the clone before starting the agent. On the manager, remove the stale entry:

/var/ossec/bin/manage_agents
# choose R (remove), enter the old agent ID