genesis, zeke, linode, and exodus were missing shire-zebra.ts.net
aliases in the genesis dnsmasq hosts file. Only hosea, isaiah,
jeremiah, gitlab, matrix, chronicles, and nas1 had them. Since
genesis is the DNS server for the whole network, Prometheus (and
everything else) couldn't resolve those four hostnames, causing
instant connection failures on scrape.
Also update exodus Tailscale IP from 100.80.99.48 to 100.70.99.91
(stale IP in both network.json and the hosts file).
With enforce_domain = true, Grafana redirects any request not matching
the configured domain (hosea.shire-zebra.ts.net) back to that hostname.
Since the nginx proxy serves Grafana at grafana.thehellings.lan, every
proxied request gets redirected to the Tailscale address, making the
proxy useless for non-Tailscale clients.
The domain setting is still correct for cookie scoping; enforce_domain
is only needed if direct port access is a concern, which is mitigated
by the firewall (port 3001 is not open on the LAN).
The greg.proxies module enables nginx but never opened the firewall,
so all proxied services (Jellyfin, Grafana, etc.) were unreachable
on the LAN. Add allowedTCPPorts [80 443] conditioned on the same
mkIf guard as the nginx virtualHosts config.
Add grafana.thehellings.lan (LAN) and grafana.home (Tailscale)
pointing at hosea's respective addresses, matching the pattern
used for jellyfin and other hosea-hosted services.
Add grafana.home and grafana.thehellings.lan nginx proxy entries
pointing at localhost:3001. This allows LAN clients (and Klaatu)
to reach Grafana without Tailscale or direct port access, working
around the enforce_domain restriction on hosea.shire-zebra.ts.net.
Fix 1 - UnPoller 404 on UDM Pro:
Remove :8443 from UP_UNIFI_DEFAULT_URL. UniFi OS devices (UDM Pro,
UDM, UXG, CloudKey) use a different API path and do NOT use port 8443.
URL should be https://10.42.1.1 (no port); unpoller handles the
/proxy/network path internally for UniFi OS devices.
Fix 2 - Grafana dashboards showing invalid datasource:
Add uid = "prometheus" to the provisioned Prometheus datasource.
Without an explicit UID, Grafana auto-generates one on first run.
All dashboard JSON references datasource uid "prometheus" which
only works if the provisioning explicitly sets that UID.
Move allowedTCPPorts for node/ping/systemd/blackbox exporters from
hosts/baseline.nix (applies to ALL hosts including linode) into
modules/nixos/home.nix (applies only to greg.home = true hosts).
This prevents ports 9100/9115/9427/9558 from being opened on linode,
which is internet-facing. All LAN hosts set greg.home = true and will
still get the ports opened; linode sets greg.home = false and will not.
- Replace unpoller-credentials plain Secret with ExternalSecret
pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)
- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
as ap.thehellings.lan (discovered during network scan)
- Add blackbox exporter to genesis (port 9115, ICMP module)
for probing non-NixOS infrastructure that has no node_exporter:
OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
homeassistant, USW-Pro-HD-24 UniFi switch
- Add infra_ping scrape job to Prometheus using blackbox prober
targeting all non-NixOS LAN hosts via genesis blackbox exporter
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
klaatuandGreg Hellings
greg