Commit Graph
101 Commits
Author SHA1 Message Date
Greg Hellings 985dd927c4 chore: move from attic to niks3
buildbot/nix-eval Build done.
buildbot/nix-build Build done.
buildbot/nix-effects Build done.
2026-05-12 21:25:24 -05:00
Greg Hellings 2932e9e335 chore: remove vm-gitlab which is no longer in use 2026-04-30 16:06:58 -05:00
Greg Hellings 14c7f34108 chore: lithic stuff
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-darwin Build done.
buildbot/nix-build gitea:greg/nixos#checks.aarch64-linux Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux Build done.
buildbot/nix-build Build done.
Update manifest chart versions / update-manifests (push) Has been cancelled
Update flake.lock / update-flake-lock (push) Has been cancelled
2026-04-08 08:00:35 -05:00
Greg Hellings 3847c7d92d fix: streamline Gitea runners 2026-04-05 16:51:31 -05:00
Greg Hellings a1bef073cf fix: generate linode certs
buildbot/nix-eval Build done. (1 warning)
buildbot/nix-build Build done.
2026-04-01 17:20:46 -05:00
Greg Hellings 2bb0177ffa chore: add encrypted keys and unencrypted certs
buildbot/nix-eval Build done.
buildbot/nix-build Build started.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.hm-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-genesis Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-isaiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-jeremiah Build done.
2026-04-01 16:33:59 -05:00
root 792217f640 feat: add Nebula mesh network overlay
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-isaiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-genesis Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-linode Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-jeremiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-zeke Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-hosea Build done.
buildbot/nix-build Build done.
Introduces a greg.nebula NixOS module and enables it across all managed
hosts for the nebula.thehellings.com overlay (CIDR: 10.157.0.0/16).

Architecture:
- linode: lighthouse + relay (public internet, UDP 4242)
- genesis: regular node + unsafe_routes router for 10.42.0.0/16 (home LAN)
- hosea, isaiah, jeremiah, zeke, exodus: regular nodes with unsafe_routes
  pointing to genesis to reach the home LAN

Changes:
- modules/nixos/nebula.nix: new greg.nebula module
  - isLighthouse / isRelay options
  - unsafeRoutes option (tun.unsafe_routes)
  - routesSubnet option: enables IP forwarding + nftables masquerade NAT
    on the gateway host (genesis) so Nebula peers reach 10.42.0.0/16
  - agenix secret reference per-host (secrets/nebula/<name>.key.age)
  - opens UDP/4242 in the firewall
- modules/nixos/default.nix: import nebula.nix
- hosts/unstable/linode/default.nix: greg.nebula.isLighthouse = true
- hosts/unstable/genesis/default.nix: greg.nebula.routesSubnet = "10.42.0.0/16"
- hosts/unstable/{hosea,isaiah,jeremiah,zeke,exodus}/default.nix:
  greg.nebula.enable = true with unsafeRoutes via genesis
- network.json: add nebulaIp field for each managed host
- secrets/secrets.nix: declare nebula/<host>.key.age entries
- secrets/nebula/README.md: full PKI bootstrap guide (CA, certs, agenix)
2026-03-28 23:24:52 -05:00
Greg Hellings 1a578ff032 fix: add grafana file and open Grafana port
buildbot/nix-eval Build done. (1 warning)
buildbot/nix-build Build done.
buildbot/nix-effects Build done.
2026-03-25 11:40:00 -05:00
root ab6e1d24fe feat: comprehensive observability stack
- Add restic_backups and mattermost scrape jobs to Prometheus config
- Add UnPoller deployment+service+secret for UniFi metrics (unpoller.yaml)
- Add unifi scrape job targeting unpoller in-cluster
- Add 4 Grafana dashboards as NixOS-managed etc files on hosea:
  - system-health: CPU, memory, disk, network per host
  - backup-health: restic backup freshness and duration
  - kubernetes: pod counts, restart rates, node resources
  - network: DNS queries, DHCP leases, UniFi ports, ping latency
- Add grafana-api-token agenix secret placeholder on hosea
- Open firewall ports 9100/9427/9558 for prometheus exporters in baseline.nix
2026-03-25 09:23:31 -05:00
Greg Hellings 69ebf78d6c fix: add new agenix file
buildbot/nix-eval Build done. (1 warning)
buildbot/nix-build Build done.
2026-03-25 06:35:20 -05:00
root 7fee882488 fix: security and monitoring improvements
buildbot/nix-eval Build done.
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
  with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
    agenix -e secrets/grafana-secret-key.age
  and entering a strong random secret (e.g. from 'pwgen -s 64 1')

Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
  (node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs

Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
    * HostDown (node unreachable > 2m) - critical
    * HighCPULoad (>90% for 10m) - warning
    * LowDiskSpace (<10% free) - warning
    * CriticalDiskSpace (<5% free) - critical
    * HighMemoryUsage (>90% for 10m) - warning
    * ResticBackupStaleness (no success > 26h) - warning
    * PodCrashLooping - warning
    * PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
  so prometheus.yml and alerts.yml are both accessible at
  /etc/prometheus/ without conflicts
2026-03-25 06:02:17 -05:00
Greg Hellings 85a338220c Rekey with Exodus system key 2026-01-18 23:16:54 -06:00
Greg Hellings c51e2e5810 fix for Genesis and Isaiah
buildbot/nix-eval Build done. (2 warnings)
buildbot/nix-build Build done.
buildbot/nix-effects Build done.
Get gitea-actions running on Isaiah
Fix adblock upate on Genesis
2025-12-30 20:21:16 -06:00
Greg Hellings 8452caf7ac Rekey from Jeremiah
buildbot/nix-eval Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.hm-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-exodus Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-isaiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-genesis Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-hosea Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-jeremiah Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-linode Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-icdm-root Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-iso Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-zeke Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-vm-gitlab Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.nixos-proxmoxtemplate Build done.
buildbot/nix-build Build done.
buildbot/nix-build gitea:greg/nixos#checks.x86_64-linux.pkg-upgrade-pg-cluster Build done.
2025-12-29 13:36:53 -06:00
Greg Hellings 489bcb7e51 Update for new Gitea instance 2025-12-29 01:50:06 -06:00
Greg Hellings 89a0a01aab Rekey secrets 2025-12-28 17:43:16 -06:00
Greg Hellings 2910ffce5b Rekey secrets 2025-12-26 23:29:28 -06:00
Greg Hellings 20e6be10d0 Remove remote workers 2025-12-16 17:11:34 -06:00
Greg Hellings 337accaeeb Enable buildbot-nix 2025-12-06 23:17:00 -06:00
Greg Hellings 459889d1d2 Add compose and attic configurations
Add compose directory to store docker/podman compose files for running
on the TrueNAS box
Add attic configurations to auto-listen on all my home boxes to upload
builds
Update nix-conf to support reading from the Attic server
2025-10-16 17:25:43 -05:00
Greg Hellings 3f587cdf8e Use restic for backups
Now that I have a working Restic backup server, use that for backups
rather than rsyncing to Synology
2025-10-15 00:02:48 -05:00
Greg Hellings 16e98c44b8 Change backups to Restic 2025-10-13 21:09:27 -05:00
Greg Hellings f9f1357b8e Rename of Jude to Zeke
Since Jude is a now a server, we rename it to Zeke, for Ezekiel
Update of tailscale to auto-register nodes
Update kubernetes to install the etcd application
Update Zeke IP address and DNS bindings
2025-07-02 12:01:27 -05:00
Greg Hellings 676b100e99 Update gitlab-runner configs 2025-06-11 23:54:32 -05:00
Greg Hellings e13bee750c Make gitlab use remote postgres 2025-06-11 21:29:20 -05:00
Greg Hellings d7f53e53a3 Expost postgres cluster
Expose the postgres cluster outside of the Kubernetes cluster
Give myself the option of connecting to postgres from my own machines,
so that I can hopefully leverage the improved performance
Update some file linting issues in Nix
Add necessary gitlab keys so it can be updated properly the next time I
deploy to it
Add DNS entries for postgres
Update the configuraiton of Traefik to expose the appropriate ports
2025-05-28 00:10:50 -05:00
Greg Hellings 26d4e9b771 Configure Dendrite on kubernetes
Configure a matrix database
Configure Dendrite into Kubernetes
Update vps configuration to new settings
2025-05-26 23:37:24 -05:00
Greg Hellings ce5f604636 External Secrets setup
Setup the external secrets helm charts
Convert hard coded secrets to external secrets
Add BitWarden CLI for serving external secrets
2025-05-26 17:10:01 -05:00
Greg Hellings ffd2dd78bc Rework how gitlab-runners are defined 2025-05-21 22:25:11 -05:00
Greg Hellings 4cd0de0109 Isaiah gets new kubernetes runner 2025-05-20 22:51:42 -05:00
Greg Hellings 641c9fd7ac Rename Jeremiah key 2025-05-20 22:27:28 -05:00
Greg Hellings d8798bc17d Remove isaiah-podman and docker-auth gitlab keys 2025-05-20 22:13:31 -05:00
Greg Hellings 82f09cc33e Enable Kubernetes on Jeremiah 2025-05-19 22:01:49 -05:00
Greg Hellings fa302301af Isaiah spin up
Bring Isaiah back into commission, this time as the core of my k3s
cluster. Standing up the basic serverInit, and also creating the core
server components that are needed.
2025-05-19 20:10:21 -05:00
Greg Hellings 98efad2780 Rekey secrets to include gitlab 2025-04-14 15:04:54 -05:00
Greg Hellings 0607fc4f14 Enable pushing builds to S3 2025-01-21 16:51:04 -06:00
Greg Hellings 206d22aeb2 Update secrets to add new server 2024-11-20 00:18:52 -06:00
Greg Hellings 395bfc25bc Finish configuring vm-matrix 2024-11-20 00:11:12 -06:00
Greg Hellings f6fdeb1789 Mass rename of Myself to Isaiah 2024-11-18 21:11:17 -06:00
Greg Hellings a71eb7485a Completely reformat to satisfy nixfmt 2024-10-19 01:20:46 -05:00
Greg Hellings 229188214f Support for deadnix pre-commit check 2024-10-03 15:21:56 -05:00
Greg Hellings 78c802d32c A complete reformatting to match nixpkgs-fmt 2024-10-03 11:26:39 -05:00
Greg Hellings 4d4beaf620 Try to migrate minio to NAS
Try to move as much of the S3 load onto the NAS as I am able to manage,
incluing pointing s3.thehellings.lan towards the NAS
2024-08-16 22:46:05 -05:00
Greg Hellings f9b4b7133b Make backups more fun with syncthing 2024-08-15 04:48:29 +00:00
Greg Hellings 403a765244 Merge branch 'main' of gh:greg-hellings/nixos-config 2024-06-24 23:25:41 -05:00
Greg Hellings d80b7cb098 Add Jeremiah runner 2024-06-21 13:02:13 -05:00
Greg Hellings 8cfd27d6ca Update cache for vbox runner 2024-06-20 22:37:14 -05:00
Greg Hellings f94103e856 Add locally hosted S3 cache 2024-06-20 22:12:38 -05:00
Greg Hellings 7c0d5cf8cd Adding Jeremiah key 2024-06-20 17:41:11 -05:00
Greg Hellings 1b88076d97 Ceph on Jeremiah 2024-05-07 10:13:51 -05:00