122 lines
2.8 KiB
Markdown
122 lines
2.8 KiB
Markdown
# Homelab Log Aggregation Stack
|
|
|
|
Grafana Alloy + Loki + Grafana, configured for:
|
|
- **MikroTik RB5009** (and other network devices) via syslog
|
|
- **Docker container logs** on the host machine
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# (Optional) load secrets first if using the Vaultwarden secrets workflow
|
|
# ./secrets-load.sh docker/loki-stack .env
|
|
|
|
docker compose up -d
|
|
```
|
|
|
|
Grafana will be available at **http://\<host-ip\>:3098**
|
|
Default login: `admin` / `admin` — you will be prompted to change this.
|
|
|
|
---
|
|
|
|
## MikroTik RB5009 Configuration
|
|
|
|
In RouterOS (Winbox or SSH), run:
|
|
|
|
```routeros
|
|
# Create a remote logging action pointing at this Docker host
|
|
/system logging action
|
|
add name=remote-loki \
|
|
target=remote \
|
|
remote=<YOUR-DOCKER-HOST-IP> \
|
|
remote-port=514 \
|
|
bsd-syslog=yes \
|
|
syslog-facility=local0 \
|
|
syslog-severity=auto
|
|
|
|
# Send all log topics to Loki
|
|
/system logging
|
|
add action=remote-loki topics=all
|
|
```
|
|
|
|
To verify it's working, SSH into the RB5009 and run:
|
|
```routeros
|
|
/log print follow
|
|
```
|
|
...then in Grafana, open Explore → Loki and query `{source="network"}`.
|
|
You should see entries appearing within a few seconds.
|
|
|
|
---
|
|
|
|
## Useful LogQL Queries
|
|
|
|
**All RB5009 logs:**
|
|
```logql
|
|
{job="syslog", source="network"}
|
|
```
|
|
|
|
**RB5009 interface/link events only:**
|
|
```logql
|
|
{job="syslog", source="network"} |= "link"
|
|
```
|
|
|
|
**All logs from a specific Docker container:**
|
|
```logql
|
|
{job="docker", container="myapp"}
|
|
```
|
|
|
|
**Errors across all Docker containers:**
|
|
```logql
|
|
{job="docker"} |= "error" | logfmt | level="error"
|
|
```
|
|
|
|
**Everything in the last 24 hours, newest first:**
|
|
```logql
|
|
{job=~"syslog|docker"} | line_format "{{.source}} {{.container}} {{.message}}"
|
|
```
|
|
|
|
---
|
|
|
|
## File Layout
|
|
|
|
```
|
|
loki-stack/
|
|
├── docker-compose.yml
|
|
├── alloy/
|
|
│ └── config.alloy # Alloy pipeline config (syslog + Docker)
|
|
├── loki/
|
|
│ └── loki-config.yml # Loki storage and retention config
|
|
└── grafana/
|
|
└── provisioning/
|
|
└── datasources/
|
|
└── loki.yml # Auto-provisions Loki as default datasource
|
|
```
|
|
|
|
## Retention
|
|
|
|
Logs are kept for **90 days** by default. To change this, edit `loki/loki-config.yml`:
|
|
```yaml
|
|
limits_config:
|
|
retention_period: 30d # or 180d, etc.
|
|
```
|
|
Then restart Loki: `docker compose restart loki`
|
|
|
|
## Adding More Syslog Sources
|
|
|
|
Any device that can send syslog (UDP/TCP 514) will work automatically —
|
|
the `host` label will be set from the syslog hostname field, so you can
|
|
filter per-device in Grafana with `{host="my-device-hostname"}`.
|
|
|
|
---
|
|
|
|
## Integrating with the Vaultwarden Secrets Workflow
|
|
|
|
If you're using the `secrets-load.sh` script, store the Grafana admin
|
|
password as a custom field named `GF_SECURITY_ADMIN_PASSWORD` in a
|
|
Vaultwarden item called `docker/loki-stack`, then replace the hardcoded
|
|
value in `docker-compose.yml` with:
|
|
|
|
```yaml
|
|
env_file:
|
|
- .env
|
|
```
|