Fix 1 - UnPoller 404 on UDM Pro:
Remove :8443 from UP_UNIFI_DEFAULT_URL. UniFi OS devices (UDM Pro,
UDM, UXG, CloudKey) use a different API path and do NOT use port 8443.
URL should be https://10.42.1.1 (no port); unpoller handles the
/proxy/network path internally for UniFi OS devices.
Fix 2 - Grafana dashboards showing invalid datasource:
Add uid = "prometheus" to the provisioned Prometheus datasource.
Without an explicit UID, Grafana auto-generates one on first run.
All dashboard JSON references datasource uid "prometheus" which
only works if the provisioning explicitly sets that UID.
- Replace unpoller-credentials plain Secret with ExternalSecret
pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)
- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
as ap.thehellings.lan (discovered during network scan)
- Add blackbox exporter to genesis (port 9115, ICMP module)
for probing non-NixOS infrastructure that has no node_exporter:
OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
homeassistant, USW-Pro-HD-24 UniFi switch
- Add infra_ping scrape job to Prometheus using blackbox prober
targeting all non-NixOS LAN hosts via genesis blackbox exporter
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
klaatu
root
greg