7 Commits

Include renames

Author	SHA1	Message	Date
Greg Hellings	7579b2b7a1	fix: Grafana network port	2026-03-25 12:24:03 -05:00
Greg Hellings	1a578ff032	fix: add grafana file and open Grafana port	2026-03-25 11:40:00 -05:00
root	ab6e1d24fe	feat: comprehensive observability stack ... - Add restic_backups and mattermost scrape jobs to Prometheus config - Add UnPoller deployment+service+secret for UniFi metrics (unpoller.yaml) - Add unifi scrape job targeting unpoller in-cluster - Add 4 Grafana dashboards as NixOS-managed etc files on hosea: - system-health: CPU, memory, disk, network per host - backup-health: restic backup freshness and duration - kubernetes: pod counts, restart rates, node resources - network: DNS queries, DHCP leases, UniFi ports, ping latency - Add grafana-api-token agenix secret placeholder on hosea - Open firewall ports 9100/9427/9558 for prometheus exporters in baseline.nix	2026-03-25 09:23:31 -05:00
root	7fee882488	fix: security and monitoring improvements ... Issue 1 - Grafana secret key: - Replace hardcoded '123456789' secret_key in hosea Grafana config with an agenix-managed secret ($__file interpolation) - Add age.secrets.grafana-secret-key declaration to hosea config - Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone) - NOTE: The .age file still needs to be created by running: agenix -e secrets/grafana-secret-key.age and entering a strong random secret (e.g. from 'pwgen -s 64 1') Issue 2 - Prometheus stale targets: - Remove retired vm-gitlab host from all three scrape jobs (node, systemd, ping) in manifests/monitoring/config.yaml - linode was already present in all scrape jobs Issue 3 - Prometheus alerting gaps: - Wire up rule_files in prometheus.yml pointing at alerts.yml - Add manifests/monitoring/alerts.yaml with alerting rules: * HostDown (node unreachable > 2m) - critical * HighCPULoad (>90% for 10m) - warning * LowDiskSpace (<10% free) - warning * CriticalDiskSpace (<5% free) - critical * HighMemoryUsage (>90% for 10m) - warning * ResticBackupStaleness (no success > 26h) - warning * PodCrashLooping - warning * PodNotReady (>10m) - warning - Mount alerts ConfigMap as separate volume subPath in deployment so prometheus.yml and alerts.yml are both accessible at /etc/prometheus/ without conflicts	2026-03-25 06:02:17 -05:00
Greg Hellings	46138b90c2	chore: enable transmission on hosea	2026-03-04 10:53:12 -06:00
Greg Hellings	61d4f4872e	chore: silence deprecation warning	2026-02-19 19:49:49 -06:00
Greg Hellings	e2181f124a	chore: dynamic hosts, builder targets ... Make hosts read from directories, to limit manual changes Rename vm-gitlab -> gitlab A few reformatting and lint changes due to altered files Create builder targets for Darwin builders	2026-02-17 16:11:29 -06:00