Commit Graph
12 Commits
Author SHA1 Message Date
root 7fee882488 fix: security and monitoring improvements
buildbot/nix-eval Build done.
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
2026-03-25 06:02:17 -05:00
Greg Hellings e21ed89cfe fix: use upstream kiwix-server
buildbot/nix-eval Build done. (1 warning)
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.pkg-zim Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-icdm-root Build done.
buildbot/nix-build Build done.
2026-03-10 16:52:27 -05:00
Greg Hellings 01ce9d74da chore: update flake lock
buildbot/nix-eval Build done.
buildbot/nix-build Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-darwin Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-linux Build done.
2026-03-10 08:46:10 -05:00
Greg Hellings b1bfb337d6 chore: reduce eval workers on Jeremiah to reduce OOM 2026-03-04 10:53:12 -06:00
Greg Hellings d7c6f23280 chore: radarr installed 2026-03-04 10:53:12 -06:00
Greg Hellings 46138b90c2 chore: enable transmission on hosea 2026-03-04 10:53:12 -06:00
Greg Hellings 426a85e357 chore: remove GUI/steam from jeremiah 2026-02-19 19:56:15 -06:00
Greg Hellings e514982fb6 chore: remove proxmox from jeremiah 2026-02-19 19:55:13 -06:00
Greg Hellings c563b57974 opt: remove dead code 2026-02-19 19:50:28 -06:00
Greg Hellings 61d4f4872e chore: silence deprecation warning 2026-02-19 19:49:49 -06:00
Greg Hellings c6b973738b fix: paths to secrets files because of new folder structure
buildbot/nix-eval Build done.
2026-02-19 08:06:54 -06:00
Greg Hellings e2181f124a chore: dynamic hosts, builder targets
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-isaiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-linode Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-zeke Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-jeremiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-linux.nixos-nixos Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-gitlab Build done.
buildbot/nix-build Build done.
Make hosts read from directories, to limit manual changes
Rename vm-gitlab -> gitlab
A few reformatting and lint changes due to altered files
Create builder targets for Darwin builders
2026-02-17 16:11:29 -06:00