Prometheus runs as a pod and only has Tailscale routes to the three
k8s nodes it runs on (isaiah, jeremiah, zeke) plus hosea. Scraping
genesis, exodus, and linode via *.shire-zebra.ts.net fails instantly
(~1ms, connection refused) because those Tailscale routes don't exist
inside the cluster network.
Fix by host type:
- k8s nodes (isaiah, jeremiah, zeke, hosea): keep Tailscale hostnames
- LAN hosts (genesis): use LAN IP 10.42.1.5
- Roaming/VPS (exodus, linode): use Tailscale IPs directly (100.x.x.x)
since they have no fixed LAN address but the k8s nodes do route
to other Tailscale IPs via the subnet router
Apply same logic to kea, dnsmasq (genesis-only jobs) and the
systemd/ping exporter jobs.
- Replace unpoller-credentials plain Secret with ExternalSecret
pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)
- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
as ap.thehellings.lan (discovered during network scan)
- Add blackbox exporter to genesis (port 9115, ICMP module)
for probing non-NixOS infrastructure that has no node_exporter:
OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
homeassistant, USW-Pro-HD-24 UniFi switch
- Add infra_ping scrape job to Prometheus using blackbox prober
targeting all non-NixOS LAN hosts via genesis blackbox exporter
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
root