Prometheus runs as a pod and only has Tailscale routes to the three
k8s nodes it runs on (isaiah, jeremiah, zeke) plus hosea. Scraping
genesis, exodus, and linode via *.shire-zebra.ts.net fails instantly
(~1ms, connection refused) because those Tailscale routes don't exist
inside the cluster network.
Fix by host type:
- k8s nodes (isaiah, jeremiah, zeke, hosea): keep Tailscale hostnames
- LAN hosts (genesis): use LAN IP 10.42.1.5
- Roaming/VPS (exodus, linode): use Tailscale IPs directly (100.x.x.x)
since they have no fixed LAN address but the k8s nodes do route
to other Tailscale IPs via the subnet router
Apply same logic to kea, dnsmasq (genesis-only jobs) and the
systemd/ping exporter jobs.
Fix 1 - UnPoller 404 on UDM Pro:
Remove :8443 from UP_UNIFI_DEFAULT_URL. UniFi OS devices (UDM Pro,
UDM, UXG, CloudKey) use a different API path and do NOT use port 8443.
URL should be https://10.42.1.1 (no port); unpoller handles the
/proxy/network path internally for UniFi OS devices.
Fix 2 - Grafana dashboards showing invalid datasource:
Add uid = "prometheus" to the provisioned Prometheus datasource.
Without an explicit UID, Grafana auto-generates one on first run.
All dashboard JSON references datasource uid "prometheus" which
only works if the provisioning explicitly sets that UID.
- Replace unpoller-credentials plain Secret with ExternalSecret
pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)
- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
as ap.thehellings.lan (discovered during network scan)
- Add blackbox exporter to genesis (port 9115, ICMP module)
for probing non-NixOS infrastructure that has no node_exporter:
OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
homeassistant, USW-Pro-HD-24 UniFi switch
- Add infra_ping scrape job to Prometheus using blackbox prober
targeting all non-NixOS LAN hosts via genesis blackbox exporter
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
Somewhere in the definition of the Tailscale operator the configuraiton
changed from taking a basic-auth formatted secret to needing a
customized one. This makes that happen
Move to using kustomize'd helm charts to deploy it
Update the new version of the helm chart, which has very different
values structure
Move database into the db namespace
Upgrade the db to include vectorchord
Upgrade CNPG to now include barman cloud natively
Migrate databases to Barman Cloud plugin, completely
Eliminate yet another shell script step in applying my k8s config
root
greg