fix: security and monitoring improvements
buildbot/nix-eval Build done.

Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
This commit is contained in:
root
2026-03-25 06:02:17 -05:00
parent 8e12131608
commit 7fee882488
6 changed files with 107 additions and 5 deletions
+3 -3
View File
@@ -8,6 +8,9 @@ data:
scrape_interval: 10s
evaluation_interval: 10s
alerting:
alertmanagers: []
rule_files:
- /etc/prometheus/alerts.yml
scrape_configs:
# This scrapes endpoints on my servers
- job_name: node
@@ -20,7 +23,6 @@ data:
- "hosea.shire-zebra.ts.net:9100"
- "exodus.shire-zebra.ts.net:9100"
- "linode.shire-zebra.ts.net:9100"
- "vm-gitlab.shire-zebra.ts.net:9100"
- job_name: systemd
static_configs:
- targets:
@@ -31,7 +33,6 @@ data:
- "hosea.shire-zebra.ts.net:9558"
- "exodus.shire-zebra.ts.net:9558"
- "linode.shire-zebra.ts.net:9558"
- "vm-gitlab.shire-zebra.ts.net:9558"
- job_name: ping
static_configs:
- targets:
@@ -42,7 +43,6 @@ data:
- "hosea.shire-zebra.ts.net:9427"
- "exodus.shire-zebra.ts.net:9427"
- "linode.shire-zebra.ts.net:9427"
- "vm-gitlab.shire-zebra.ts.net:9427"
- job_name: kea
static_configs:
- targets: