greg/nixos
1
0
Fork 1
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
1,537 Commits 12 Branches 0 Tags
4eebf07f6bfa42b3ca5658d34c6df3dc0f44ddfb
Commit Graph
4 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
root 4eebf07f6b fix: scope prometheus exporter firewall rules to home hosts only
buildbot/nix-eval Build done.
Details 
Move allowedTCPPorts for node/ping/systemd/blackbox exporters from
hosts/baseline.nix (applies to ALL hosts including linode) into
modules/nixos/home.nix (applies only to greg.home = true hosts).

This prevents ports 9100/9115/9427/9558 from being opened on linode,
which is internet-facing. All LAN hosts set greg.home = true and will
still get the ports opened; linode sets greg.home = false and will not.
 2026-03-25 09:46:26 -05:00
root df2f9d69fe fix: address PR feedback + expand infrastructure coverage
buildbot/nix-eval Build done. (1 warning)
Details 
- Replace unpoller-credentials plain Secret with ExternalSecret
  pulling from Bitwarden item 15bfc957-5de5-49b2-ab6c-b41800e71564
  via bitwarden-login ClusterSecretStore (username + password)
- Pull UP_UNIFI_DEFAULT_USER from secret as well (was hardcoded)

- Add OpenWRT access point (10.42.0.3) to genesis DNS hosts file
  as ap.thehellings.lan (discovered during network scan)

- Add blackbox exporter to genesis (port 9115, ICMP module)
  for probing non-NixOS infrastructure that has no node_exporter:
  OpenWRT AP, Joel, pve1, chronicles, nas1, odoo, mattermost,
  homeassistant, USW-Pro-HD-24 UniFi switch

- Add infra_ping scrape job to Prometheus using blackbox prober
  targeting all non-NixOS LAN hosts via genesis blackbox exporter
 2026-03-25 09:25:16 -05:00
root ab6e1d24fe feat: comprehensive observability stack 
- Add restic_backups and mattermost scrape jobs to Prometheus config
- Add UnPoller deployment+service+secret for UniFi metrics (unpoller.yaml)
- Add unifi scrape job targeting unpoller in-cluster
- Add 4 Grafana dashboards as NixOS-managed etc files on hosea:
  - system-health: CPU, memory, disk, network per host
  - backup-health: restic backup freshness and duration
  - kubernetes: pod counts, restart rates, node resources
  - network: DNS queries, DHCP leases, UniFi ports, ping latency
- Add grafana-api-token agenix secret placeholder on hosea
- Open firewall ports 9100/9427/9558 for prometheus exporters in baseline.nix
 2026-03-25 09:23:31 -05:00
root 7fee882488 fix: security and monitoring improvements
buildbot/nix-eval Build done.
Details 
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
 2026-03-25 06:02:17 -05:00