Commit Graph
69 Commits
Author SHA1 Message Date
Greg Hellings 985dd927c4 chore: move from attic to niks3
buildbot/nix-eval Build done.
buildbot/nix-build Build done.
buildbot/nix-effects Build done.
2026-05-12 21:25:24 -05:00
Greg Hellings 2932e9e335 chore: remove vm-gitlab which is no longer in use 2026-04-30 16:06:58 -05:00
Greg Hellings 14c7f34108 chore: lithic stuff
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-darwin Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-linux Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux Build done.
buildbot/nix-build Build done.
Update manifest chart versions / update-manifests (push) Has been cancelled
Update flake.lock / update-flake-lock (push) Has been cancelled
2026-04-08 08:00:35 -05:00
Greg Hellings 3847c7d92d fix: streamline Gitea runners 2026-04-05 16:51:31 -05:00
root 792217f640 feat: add Nebula mesh network overlay
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-isaiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-genesis Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-linode Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-jeremiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-zeke Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-hosea Build done.
buildbot/nix-build Build done.
Introduces a greg.nebula NixOS module and enables it across all managed
hosts for the nebula.thehellings.com overlay (CIDR: 10.157.0.0/16).

Architecture:
- linode: lighthouse + relay (public internet, UDP 4242)
- genesis: regular node + unsafe_routes router for 10.42.0.0/16 (home LAN)
- hosea, isaiah, jeremiah, zeke, exodus: regular nodes with unsafe_routes
  pointing to genesis to reach the home LAN

Changes:
- modules/nixos/nebula.nix: new greg.nebula module
  - isLighthouse / isRelay options
  - unsafeRoutes option (tun.unsafe_routes)
  - routesSubnet option: enables IP forwarding + nftables masquerade NAT
    on the gateway host (genesis) so Nebula peers reach 10.42.0.0/16
  - agenix secret reference per-host (secrets/nebula/<name>.key.age)
  - opens UDP/4242 in the firewall
- modules/nixos/default.nix: import nebula.nix
- hosts/unstable/linode/default.nix: greg.nebula.isLighthouse = true
- hosts/unstable/genesis/default.nix: greg.nebula.routesSubnet = "10.42.0.0/16"
- hosts/unstable/{hosea,isaiah,jeremiah,zeke,exodus}/default.nix:
  greg.nebula.enable = true with unsafeRoutes via genesis
- network.json: add nebulaIp field for each managed host
- secrets/secrets.nix: declare nebula/<host>.key.age entries
- secrets/nebula/README.md: full PKI bootstrap guide (CA, certs, agenix)
2026-03-28 23:24:52 -05:00
root ab6e1d24fe feat: comprehensive observability stack
- Add restic_backups and mattermost scrape jobs to Prometheus config
- Add UnPoller deployment+service+secret for UniFi metrics (unpoller.yaml)
- Add unifi scrape job targeting unpoller in-cluster
- Add 4 Grafana dashboards as NixOS-managed etc files on hosea:
  - system-health: CPU, memory, disk, network per host
  - backup-health: restic backup freshness and duration
  - kubernetes: pod counts, restart rates, node resources
  - network: DNS queries, DHCP leases, UniFi ports, ping latency
- Add grafana-api-token agenix secret placeholder on hosea
- Open firewall ports 9100/9427/9558 for prometheus exporters in baseline.nix
2026-03-25 09:23:31 -05:00
root 7fee882488 fix: security and monitoring improvements
buildbot/nix-eval Build done.
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
2026-03-25 06:02:17 -05:00
Greg Hellings c51e2e5810 fix for Genesis and Isaiah
buildbot/nix-eval Build done. (2 warnings)
buildbot/nix-build Build done.
buildbot/nix-effects Build done.
Get gitea-actions running on Isaiah
Fix adblock upate on Genesis
2025-12-30 20:21:16 -06:00
Greg Hellings 89a0a01aab Rekey secrets 2025-12-28 17:43:16 -06:00
Greg Hellings 2910ffce5b Rekey secrets 2025-12-26 23:29:28 -06:00
Greg Hellings 337accaeeb Enable buildbot-nix 2025-12-06 23:17:00 -06:00
Greg Hellings 459889d1d2 Add compose and attic configurations
Add compose directory to store docker/podman compose files for running
on the TrueNAS box
Add attic configurations to auto-listen on all my home boxes to upload
builds
Update nix-conf to support reading from the Attic server
2025-10-16 17:25:43 -05:00
Greg Hellings f9f1357b8e Rename of Jude to Zeke
Since Jude is a now a server, we rename it to Zeke, for Ezekiel
Update of tailscale to auto-register nodes
Update kubernetes to install the etcd application
Update Zeke IP address and DNS bindings
2025-07-02 12:01:27 -05:00
Greg Hellings e13bee750c Make gitlab use remote postgres 2025-06-11 21:29:20 -05:00
Greg Hellings d7f53e53a3 Expost postgres cluster
Expose the postgres cluster outside of the Kubernetes cluster
Give myself the option of connecting to postgres from my own machines,
so that I can hopefully leverage the improved performance
Update some file linting issues in Nix
Add necessary gitlab keys so it can be updated properly the next time I
deploy to it
Add DNS entries for postgres
Update the configuraiton of Traefik to expose the appropriate ports
2025-05-28 00:10:50 -05:00
Greg Hellings ce5f604636 External Secrets setup
Setup the external secrets helm charts
Convert hard coded secrets to external secrets
Add BitWarden CLI for serving external secrets
2025-05-26 17:10:01 -05:00
Greg Hellings 4cd0de0109 Isaiah gets new kubernetes runner 2025-05-20 22:51:42 -05:00
Greg Hellings 641c9fd7ac Rename Jeremiah key 2025-05-20 22:27:28 -05:00
Greg Hellings d8798bc17d Remove isaiah-podman and docker-auth gitlab keys 2025-05-20 22:13:31 -05:00
Greg Hellings 82f09cc33e Enable Kubernetes on Jeremiah 2025-05-19 22:01:49 -05:00
Greg Hellings fa302301af Isaiah spin up
Bring Isaiah back into commission, this time as the core of my k3s
cluster. Standing up the basic serverInit, and also creating the core
server components that are needed.
2025-05-19 20:10:21 -05:00
Greg Hellings 98efad2780 Rekey secrets to include gitlab 2025-04-14 15:04:54 -05:00
Greg Hellings 0607fc4f14 Enable pushing builds to S3 2025-01-21 16:51:04 -06:00
Greg Hellings 395bfc25bc Finish configuring vm-matrix 2024-11-20 00:11:12 -06:00
Greg Hellings f6fdeb1789 Mass rename of Myself to Isaiah 2024-11-18 21:11:17 -06:00
Greg Hellings a71eb7485a Completely reformat to satisfy nixfmt 2024-10-19 01:20:46 -05:00
Greg Hellings 229188214f Support for deadnix pre-commit check 2024-10-03 15:21:56 -05:00
Greg Hellings 78c802d32c A complete reformatting to match nixpkgs-fmt 2024-10-03 11:26:39 -05:00
Greg Hellings f9b4b7133b Make backups more fun with syncthing 2024-08-15 04:48:29 +00:00
Greg Hellings d80b7cb098 Add Jeremiah runner 2024-06-21 13:02:13 -05:00
Greg Hellings 7c0d5cf8cd Adding Jeremiah key 2024-06-20 17:41:11 -05:00
Greg Hellings 4cd66ec1b5 Add gitlab-runner to Linode 2024-05-01 19:46:34 +00:00
Greg Hellings 84b91d086d Update secrets 2024-04-21 15:03:39 -05:00
Greg Hellings f787fc93fa Add Jeremiah 2024-04-16 08:34:06 -05:00
Greg Hellings c729a5497a Add minio to Genesis 2024-04-03 08:32:31 -05:00
Greg Hellings cb5d1a7a83 Run VBox on Hosea 2024-04-02 14:07:13 -05:00
Greg Hellings 601068fc93 Rekey WITH Hosea 2024-04-02 14:02:41 -05:00
Greg Hellings 7714d18c73 Add hosea host 2024-04-01 14:07:49 -05:00
Greg Hellings 5139965bf4 Add hosea user 2024-04-01 14:05:54 -05:00
Greg Hellings 10b27c545f Merge branch 'main' of gh:greg-hellings/nixos-config 2024-03-25 15:20:47 -05:00
Greg Hellings f2afccf50b Lots of updates to try and get ACME working 2024-03-25 15:18:53 -05:00
Greg Hellings b536a7e3ef Move Dendrite to Isaiah 2024-02-27 14:42:19 -06:00
Greg Hellings 102463a783 Add dendrite key 2024-02-27 20:42:05 +00:00
Greg Hellings 61e49cbac5 Fix dendrite file 2024-02-27 18:58:31 +00:00
Greg Hellings 7dde6d256a CI running 2023-12-27 20:27:38 -06:00
Greg Hellings fb79feb58d Login from Windows finally! 2023-12-19 13:45:06 -06:00
Greg Hellings d0d7d25b90 QEmu and VirtualBox working 2023-12-17 19:54:28 -06:00
Greg Hellings 1afe353df7 Bring up the VM runners 2023-12-17 15:47:52 -06:00
Greg Hellings 69e6c42886 Gitlab and QEmu runner 2023-12-15 10:28:19 -06:00
Greg Hellings 65c35ae780 Add myself 2023-12-12 20:44:53 -06:00