Watchtower Stack

Project Overview

This guide walks you through deploying a comprehensive monitoring stack on a Proxmox LXC container. The "Watchtower" combines three powerful tools to give you complete visibility into your infrastructure's security, performance, and health.

The Stack

Infrastructure Specifications

For this deployment, we're using a single LXC container with the following specs:

What You'll Get

• Security Monitoring: Wazuh provides real-time threat detection, vulnerability assessment, and compliance monitoring
• Performance Metrics: Prometheus collects and stores time-series metrics from your entire infrastructure
• Visual Dashboards: Grafana transforms raw metrics into beautiful, actionable visualizations
• Proxmox Integration: Monitor your hypervisor, VMs, and containers from a single pane of glass

Step 1: Deploy Wazuh SIEM

Wazuh is our security information and event management (SIEM) platform. It provides threat detection, integrity monitoring, incident response, and compliance capabilities.

Installation

The Wazuh installation is straightforward with their all-in-one installer:

curl -sO https://packages.wazuh.com/4.14/wazuh-install.sh
sudo bash ./wazuh-install.sh -a

Change Default Password

For security, immediately change the default admin password:

# Download the password tool
curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/4.7/wazuh-passwords-tool.sh

# Set your new password
bash wazuh-passwords-tool.sh -u admin -p YourSecurePassword

# Restart the dashboard
sudo service wazuh-dashboard restart

Secure Your Command History

Since you just typed a password in plaintext, clean your command history:

# Clear current session history
history -c

# Clear persistent history
cat /dev/null > ~/.bash_history

Access the Dashboard

Once installation is complete, access the Wazuh dashboard at:

https://your-lxc-ip:443

Default credentials (before you change them):

• Username: admin
• Password: Generated during installation (displayed at the end)

Step 2: Install Prometheus

Prometheus is our time-series database and monitoring system. It scrapes metrics from configured targets, stores them efficiently, and makes them available for querying.

Automated Installation Script

Save this script as install-prometheus.sh to automate the entire Prometheus installation:

#!/bin/bash
# Prometheus Installation Script
# Automates installation of Prometheus 3.8.0 on Linux

set -e  # Exit on any error

PROMETHEUS_VERSION="3.8.0"
PROMETHEUS_DIR="prometheus-${PROMETHEUS_VERSION}.linux-amd64"
PROMETHEUS_TAR="${PROMETHEUS_DIR}.tar.gz"
DOWNLOAD_URL="https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${PROMETHEUS_TAR}"

echo "=== Prometheus Installation Script ==="
echo "Installing Prometheus v${PROMETHEUS_VERSION}"
echo

# Create working directory
echo "[1/9] Creating working directory..."
mkdir -p ~/prometheus
cd ~/prometheus

# Download Prometheus
echo "[2/9] Downloading Prometheus..."
wget -q --show-progress "$DOWNLOAD_URL"

# Extract archive
echo "[3/9] Extracting archive..."
tar -xzf "$PROMETHEUS_TAR"

# Create Prometheus directories
echo "[4/9] Creating system directories..."
sudo mkdir -p /etc/prometheus /var/lib/prometheus

# Move binaries
echo "[5/9] Installing binaries..."
cd "$PROMETHEUS_DIR"
sudo mv prometheus promtool /usr/local/bin/

# Move configuration files
echo "[6/9] Installing configuration files..."
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ console_libraries/ /etc/prometheus/

# Create Prometheus user
echo "[7/9] Creating prometheus user..."
sudo useradd -rs /bin/false prometheus 2>/dev/null || echo "User 'prometheus' already exists"

# Set permissions
echo "[8/9] Setting permissions..."
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Create systemd service
echo "[9/9] Creating systemd service..."
sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd and start Prometheus
echo
echo "Reloading systemd and starting Prometheus..."
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

# Check status
echo
echo "=== Installation Complete ==="
echo
prometheus --version
echo
sudo systemctl status prometheus --no-pager

echo
echo "Prometheus is now running!"
echo "Access the web UI at: http://localhost:9090"
echo
echo "Useful commands:"
echo "  - Check status: sudo systemctl status prometheus"
echo "  - View logs: sudo journalctl -u prometheus -f"
echo "  - Restart: sudo systemctl restart prometheus"

Run the Installation

chmod +x install-prometheus.sh
./install-prometheus.sh

Verify Installation

Check that Prometheus is running and accessible:

# Check service status
sudo systemctl status prometheus

# Test the web interface
curl http://localhost:9090

Open your browser and navigate to http://your-lxc-ip:9090 to access the Prometheus web UI.

Step 3: Deploy PVE Exporter

The Proxmox VE Exporter allows Prometheus to scrape metrics from your Proxmox host, including CPU, memory, storage, and VM statistics.

Create Proxmox API User

On your Proxmox host (not the LXC container), run:

# Create the user
pveum user add pve-exporter@pve -password YourSecurePassword

# Grant read-only auditor permissions
pveum acl modify / -user pve-exporter@pve -role PVEAuditor

Remember to clear your command history after entering the password!

PVE Exporter Installation Script

Save this as install-pve-exporter.sh and customize the PROXMOX_HOST variable:

#!/bin/bash
# Proxmox PVE Exporter Installation Script
# Installs prometheus-pve-exporter as a systemd service

set -e  # Exit on any error

PVE_EXPORTER_PORT="9221"
CONFIG_DIR="/etc/prometheus"
CONFIG_FILE="${CONFIG_DIR}/pve.yml"
PROXMOX_HOST="192.168.1.253"  # Change this to your Proxmox IP

echo "=== Proxmox PVE Exporter Installation Script ==="
echo "Target Proxmox host: ${PROXMOX_HOST}"
echo

# Get credentials
echo "Enter the password you set for pve-exporter@pve: "
read -s PVE_PASSWORD
echo

# Install dependencies
echo "[1/8] Installing Python and pip..."
apt-get update
apt-get install -y python3 python3-pip python3-venv curl

# Create system user
echo "[2/8] Creating prometheus system user..."
if ! id -u prometheus &>/dev/null; then
    useradd --system --no-create-home --shell /bin/false prometheus
    echo "Created prometheus user"
else
    echo "User prometheus already exists"
fi

# Create virtual environment
echo "[3/8] Setting up Python virtual environment..."
mkdir -p /opt/prometheus-pve-exporter
python3 -m venv /opt/prometheus-pve-exporter

# Install exporter
echo "[4/8] Installing prometheus-pve-exporter..."
source /opt/prometheus-pve-exporter/bin/activate
pip install --upgrade pip
pip install prometheus-pve-exporter
deactivate

# Create config
echo "[5/8] Creating configuration files..."
mkdir -p "$CONFIG_DIR"

cat > "$CONFIG_FILE" < /etc/systemd/system/prometheus-pve-exporter.service <<'EOF'
[Unit]
Description=Prometheus Exporter for Proxmox VE
Documentation=https://github.com/prometheus-pve/prometheus-pve-exporter
After=network.target
Wants=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus-pve-exporter/bin/pve_exporter --config.file /etc/prometheus/pve.yml

[Install]
WantedBy=multi-user.target
EOF

# Start service
echo "[7/8] Starting prometheus-pve-exporter service..."
systemctl daemon-reload
systemctl enable prometheus-pve-exporter
systemctl start prometheus-pve-exporter

sleep 3

# Verify
echo
echo "[8/8] Verifying installation..."
systemctl status prometheus-pve-exporter --no-pager

echo
if curl -sf http://127.0.0.1:${PVE_EXPORTER_PORT}/pve | grep -q "pve_version_info"; then
    echo "✓ Metrics endpoint is working!"
    echo "✓ Successfully connected to Proxmox host"
else
    echo "✗ Warning: Could not fetch metrics"
    echo "Check logs: journalctl -u prometheus-pve-exporter -n 50"
fi

echo
echo "=== Installation Complete ==="
echo "Test: curl http://localhost:${PVE_EXPORTER_PORT}/pve"

Run the Installation

chmod +x install-pve-exporter.sh
./install-pve-exporter.sh

Configure Prometheus Scraping

Edit /etc/prometheus/prometheus.yml and add this job configuration:

scrape_configs:
  # Existing configs...
  
  - job_name: 'pve'
    static_configs:
      - targets:
          - 192.168.1.253  # Your Proxmox host IP
    metrics_path: /pve
    params:
      module: [default]
      cluster: ['1']
      node: ['1']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9221  # PVE exporter address

Reload Prometheus

sudo systemctl reload prometheus

Step 4: Install Grafana

Grafana transforms our Prometheus metrics into beautiful, interactive dashboards. It's the visualization layer of our monitoring stack.

Installation

Install Grafana Enterprise edition (free for personal use):

# Install dependencies
sudo apt-get install -y adduser libfontconfig1 musl

# Download Grafana
wget https://dl.grafana.com/grafana-enterprise/release/12.3.0/grafana-enterprise_12.3.0_19497075765_linux_amd64.deb

# Install the package
sudo dpkg -i grafana-enterprise_12.3.0_19497075765_linux_amd64.deb

# Start and enable service
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Verify Installation

# Check service status
sudo systemctl status grafana-server

# Test the web interface
curl http://localhost:3000

Initial Login

Access Grafana at http://your-lxc-ip:3000

Default credentials:

• Username: admin
• Password: admin

You'll be prompted to change the password on first login.

Add Prometheus Data Source

1. Click the gear icon (⚙️) in the left sidebar
2. Select "Data Sources"
3. Click "Add data source"
4. Select "Prometheus"
5. Set URL to: http://localhost:9090
6. Click "Save & Test"

Import Proxmox Dashboard

Import a pre-built dashboard for Proxmox monitoring:

1. Click the "+" icon in the left sidebar
2. Select "Import"
3. Enter dashboard ID: 10347
4. Click "Load"
5. Select your Prometheus data source
6. Click "Import"

Explore Your Metrics

The imported dashboard should now display live metrics from your Proxmox host including:

• CPU usage and load averages
• Memory utilization
• Storage capacity and I/O
• Network traffic
• VM and container status

Verification & Next Steps

Health Check Commands

Verify all services are running correctly:

# Check Wazuh
sudo systemctl status wazuh-manager
sudo systemctl status wazuh-dashboard

# Check Prometheus
sudo systemctl status prometheus

# Check PVE Exporter
sudo systemctl status prometheus-pve-exporter
curl http://localhost:9221/pve | head

# Check Grafana
sudo systemctl status grafana-server

Service URLs

Service	URL	Default Port
Wazuh Dashboard	https://your-lxc-ip	443
Prometheus	http://your-lxc-ip:9090	9090
PVE Exporter	http://your-lxc-ip:9221/pve	9221
Grafana	http://your-lxc-ip:3000	3000

Deploy Wazuh Agents

Now that your Watchtower is operational, deploy Wazuh agents to systems you want to monitor:

1. Log into Wazuh Dashboard
2. Navigate to "Agents" → "Deploy new agent"
3. Select your operating system
4. Copy the installation command
5. Run it on your target systems

Add More Exporters

Expand your monitoring capabilities with additional Prometheus exporters:

• Node Exporter: System metrics (CPU, memory, disk, network) for Linux hosts
• Windows Exporter: Metrics for Windows servers
• Blackbox Exporter: Probe endpoints over HTTP, HTTPS, DNS, TCP, and ICMP
• SNMP Exporter: Collect metrics from network devices

Enhance with Heimdall

Consider deploying Heimdall as a centralized dashboard to organize all your services in one place:

# Heimdall provides a beautiful landing page with links to all your services
# Perfect for managing multiple monitoring tools and infrastructure services

Maintenance Tips

• Regularly review Wazuh alerts and tune detection rules
• Set up alert notifications in Grafana for critical metrics
• Monitor disk usage on your log storage volume
• Keep services updated with security patches
• Create regular backups of your configuration files

Useful Resources

• Wazuh Documentation
• Prometheus Documentation
• Grafana Documentation
• PVE Exporter GitHub