- Remove builder2 (retired host): dangling network.json entry and
empty home/hosts/builder2 stub. Its old IP (10.42.1.17) is already
correctly owned by pve4.
- Remove the abandoned Ceph module (modules/nixos/ceph.nix) and its
unencrypted plaintext keyring files under secrets/. No host ever
enabled services.ceph-benaco; the keyrings were dead, unencrypted
credentials sitting in the repo.
- Normalize Darwin host identity: IVR and Lithic are the only two
physical Darwin machines. All other darwin/hosts/* names are now
symlinks to whichever of the two they represent, matching the DHCP
name variations nix-darwin sees depending on network:
gregory -> ivr
gregory.hellings-mbp -> ivr
MacBook-Prolocal -> ivr
gregs-MacBook-Pro-16-inch-Nov-2024 -> lithic
li -> lithic (pre-existing)
This lets each machine's config be maintained once regardless of
what hostname it currently advertises.
* keepalived does not work with Nebula VPN
* update Genesis firewall to allow passing through local traffic
* target all traffic directly to the LAN IP using genesis's routing
* Consolidate Linode into a single file
* Convert gitea and matrix to using Nebula connections
* Have Linode proxy to Nebula connections instead of Tailscale
* Update Acme to use DNS-01
* Update Flake to pull from branch that supports ACME 5.x client
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
Add compose directory to store docker/podman compose files for running
on the TrueNAS box
Add attic configurations to auto-listen on all my home boxes to upload
builds
Update nix-conf to support reading from the Attic server
Since Jude is a now a server, we rename it to Zeke, for Ezekiel
Update of tailscale to auto-register nodes
Update kubernetes to install the etcd application
Update Zeke IP address and DNS bindings
Expose the postgres cluster outside of the Kubernetes cluster
Give myself the option of connecting to postgres from my own machines,
so that I can hopefully leverage the improved performance
Update some file linting issues in Nix
Add necessary gitlab keys so it can be updated properly the next time I
deploy to it
Add DNS entries for postgres
Update the configuraiton of Traefik to expose the appropriate ports
Bring Isaiah back into commission, this time as the core of my k3s
cluster. Standing up the basic serverInit, and also creating the core
server components that are needed.
emily
Greg Hellings