Triggered by investigating a several-hour >10Mbps traffic spike to
linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h
before it crash-looped, but with 'option httplog' commented out and no
per-backend request logs, there was no way to attribute that traffic
to a specific backend, host, or client.
- linode: enable HAProxy httplog + defaults 'log global' (was
commented out) so every proxied HTTP request is now logged with
timing/status/bytes.
- linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live
per-backend/per-server connection and byte counters.
- linode: route nginx (Nextcloud's local vhost) access logs to
journald via syslog, since the read-only monitoring account has no
access to /var/log/nginx/*.
- linode: enable vnstat for historical per-interface bandwidth
tracking (5-min granularity) so a reported 'traffic was high for N
hours' can be confirmed/timestamped immediately instead of
reconstructed after the fact from journal timestamps.
- k3s manifests: enable Traefik access logging (JSON) — this is the
ingress layer HAProxy forwards :80 traffic to (git/matrix/immich),
and lacked any per-request visibility.
- hosts/baseline.nix (fleet-wide): add a journald rate limit
(2000 lines / 30s per unit). Found live while investigating that
uptime-kuma on 'kuma' was logging a Prometheus label-validation
error on every monitor beat (~100k lines/hour), which was itself
degrading journalctl responsiveness on that host during the
cross-host traffic scan.
Related but not otherwise addressed here: Nebula relay/handshake
churn on kuma's tunnel and the etcd read-latency warnings seen on
isaiah/zeke around the same incident window — noted for a future
investigation, not fixed by this PR.
pve1 is a static/DHCP-reserved Proxmox host at 10.42.0.4 (previously
mislabeled 'joel' in some places). 10.42.1.1 is the UDM Pro gateway
IP, not pve1 -- OPNsense was retired in favor of Ubiquiti. Removes
the stale duplicate PVE1 DHCP reservation at 10.42.1.1 and drops the
now-redundant 'joel' entry from network.json (consolidated into
pve1).
* Consolidate Linode into a single file
* Convert gitea and matrix to using Nebula connections
* Have Linode proxy to Nebula connections instead of Tailscale
* Update Acme to use DNS-01
* Update Flake to pull from branch that supports ACME 5.x client
- manifests/gitea/dump-cronjob.yaml: Remove --skip-log, --skip-custom-dir,
--skip-db flags to make backup complete
- manifests/gitea/dump-cronjob.yaml: Replace NFS volume + cleanup container
with S3 upload to backup-gitea bucket using MinIO client (mc).
30-day lifecycle set via mc ilm. Uses minio_key/minio_secret from
existing gitea-config secret.
- hosts/unstable/hosea/default.nix: Replace raw services.restic.backups.albyhub
block with greg.backup.jobs.albyhub using the greg.backup module.
Remove manual age.secrets.restic-env and age.secrets.restic-pw entries
since the greg.backup module declares them.
Greg Hellings
emily
greg
klaatu