- manifests/gitea/dump-cronjob.yaml: Remove --skip-log, --skip-custom-dir,
--skip-db flags to make backup complete
- manifests/gitea/dump-cronjob.yaml: Replace NFS volume + cleanup container
with S3 upload to backup-gitea bucket using MinIO client (mc).
30-day lifecycle set via mc ilm. Uses minio_key/minio_secret from
existing gitea-config secret.
- hosts/unstable/hosea/default.nix: Replace raw services.restic.backups.albyhub
block with greg.backup.jobs.albyhub using the greg.backup module.
Remove manual age.secrets.restic-env and age.secrets.restic-pw entries
since the greg.backup module declares them.
Per review feedback:
- nebula module now defaults unsafeRoutes to [{route=10.42.0.0/16 via=10.157.0.2}]
so all regular nodes get home LAN routing automatically
- genesis overrides unsafeRoutes=[] since it IS the routing node (avoids loop)
- exodus and all k3s nodes (hosea, isaiah, jeremiah, zeke) simplified to
nebula.enable = true only, relying on the new default
Add a horizontal bar gauge showing the disk usage % of the 5 most-full
PersistentVolumes in the cluster, using kubelet volume stats already
scraped by Prometheus:
topk(5, kubelet_volume_stats_used_bytes
/ kubelet_volume_stats_capacity_bytes * 100)
Labels show namespace/PVC name. Color thresholds: green <70%, yellow
70–90%, red >90%. Placed at the bottom of the dashboard (y=46).
Bump dashboard version to 3.
With enforce_domain = true, Grafana redirects any request not matching
the configured domain (hosea.shire-zebra.ts.net) back to that hostname.
Since the nginx proxy serves Grafana at grafana.thehellings.lan, every
proxied request gets redirected to the Tailscale address, making the
proxy useless for non-Tailscale clients.
The domain setting is still correct for cookie scoping; enforce_domain
is only needed if direct port access is a concern, which is mitigated
by the firewall (port 3001 is not open on the LAN).
Add grafana.home and grafana.thehellings.lan nginx proxy entries
pointing at localhost:3001. This allows LAN clients (and Klaatu)
to reach Grafana without Tailscale or direct port access, working
around the enforce_domain restriction on hosea.shire-zebra.ts.net.
Fix 1 - UnPoller 404 on UDM Pro:
Remove :8443 from UP_UNIFI_DEFAULT_URL. UniFi OS devices (UDM Pro,
UDM, UXG, CloudKey) use a different API path and do NOT use port 8443.
URL should be https://10.42.1.1 (no port); unpoller handles the
/proxy/network path internally for UniFi OS devices.
Fix 2 - Grafana dashboards showing invalid datasource:
Add uid = "prometheus" to the provisioned Prometheus datasource.
Without an explicit UID, Grafana auto-generates one on first run.
All dashboard JSON references datasource uid "prometheus" which
only works if the provisioning explicitly sets that UID.
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
Make hosts read from directories, to limit manual changes
Rename vm-gitlab -> gitlab
A few reformatting and lint changes due to altered files
Create builder targets for Darwin builders
Greg Hellings
klaatu
greg