greg/nixos
1
0
Fork 1
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
1,537 Commits 12 Branches 0 Tags
4eebf07f6bfa42b3ca5658d34c6df3dc0f44ddfb
Commit Graph
8 Commits
Author SHA1 Message Date
root df2f9d69fe fix: address PR feedback + expand infrastructure coverage
buildbot/nix-eval Build done. (1 warning)
Details 
- Replace unpoller-credentials plain Secret with ExternalSecret
  pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
  via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)

- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
  as ap.thehellings.lan (discovered during network scan)

- Add blackbox exporter to genesis (port 9115, ICMP module)
  for probing non-NixOS infrastructure that has no node_exporter:
  OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
  homeassistant, USW-Pro-HD-24 UniFi switch

- Add infra_ping scrape job to Prometheus using blackbox prober
  targeting all non-NixOS LAN hosts via genesis blackbox exporter
 2026-03-25 09:25:16 -05:00
root ab6e1d24fe feat: comprehensive observability stack 
- Add restic_backups and mattermost scrape jobs to Prometheus config
- Add UnPoller deployment+service+secret for UniFi metrics (unpoller.yaml)
- Add unifi scrape job targeting unpoller in-cluster
- Add 4 Grafana dashboards as NixOS-managed etc files on hosea:
  - system-health: CPU, memory, disk, network per host
  - backup-health: restic backup freshness and duration
  - kubernetes: pod counts, restart rates, node resources
  - network: DNS queries, DHCP leases, UniFi ports, ping latency
- Add grafana-api-token agenix secret placeholder on hosea
- Open firewall ports 9100/9427/9558 for prometheus exporters in baseline.nix
 2026-03-25 09:23:31 -05:00
root 7fee882488 fix: security and monitoring improvements
buildbot/nix-eval Build done.
Details 
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
 2026-03-25 06:02:17 -05:00
Greg Hellings e0f18a4d47 Expand Longhorn volume sizes 2025-12-29 00:42:14 -06:00
Greg Hellings afbc6e4fef Increase disk size for Immich and Prometheus 2025-12-15 20:28:19 -06:00
Greg Hellings 572419b8d3 Enable node level monitoring 2025-11-23 01:12:20 -06:00
Greg Hellings 08b0ebf05f Fix Kubernetes access 
Give the pods the proper authorization
Clarify the jobs endpoint names
 2025-11-22 21:22:51 -06:00
Greg Hellings 82acff5857 Add Prometheus scraping to Kubernetes 2025-11-22 20:15:40 -06:00