emily
10cdf9408d
Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h before it crash-looped, but with 'option httplog' commented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client. - linode: enable HAProxy httplog + defaults 'log global' (was commented out) so every proxied HTTP request is now logged with timing/status/bytes. - linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live per-backend/per-server connection and byte counters. - linode: route nginx (Nextcloud's local vhost) access logs to journald via syslog, since the read-only monitoring account has no access to /var/log/nginx/*. - linode: enable vnstat for historical per-interface bandwidth tracking (5-min granularity) so a reported 'traffic was high for N hours' can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps. - k3s manifests: enable Traefik access logging (JSON) — this is the ingress layer HAProxy forwards :80 traffic to (git/matrix/immich), and lacked any per-request visibility. - hosts/baseline.nix (fleet-wide): add a journald rate limit (2000 lines / 30s per unit). Found live while investigating that uptime-kuma on 'kuma' was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour), which was itself degrading journalctl responsiveness on that host during the cross-host traffic scan. Related but not otherwise addressed here: Nebula relay/handshake churn on kuma's tunnel and the etcd read-latency warnings seen on isaiah/zeke around the same incident window — noted for a future investigation, not fixed by this PR.
Stands up my personal infrastructure in a Kubernetes environment.
First Thing First
In order to properly get things up and going, you will need to create a secret to allow the external secrets to log into BitWarden Password Manager. For obvious reasont this cannot be safely added to this repository. As such, it is suggested you create this manually.
kubectl create namespace bitwarden
kubectl create secret generic bitwarden-cli --namespace bitwarden --from-literal=BW_USERNAME=my_username --from-literal=BW_PASSWORD=my_password
# Alternative to the preceding line if you don't want the data in your shell history
kubectl create secret generic bitwarden-cli --namespace bitwarden --from-file=./BW_USERNAME.txt --from-file=./BW_PASSWORD.txt
# And yet an entirely other option, if I'm logged into my own systems
sudo cat /run/agenix/bw_secret | kubectl apply -f -
Now Configure Cluster Services
To apply this you need to install kubectl, kustomize, and helm. It can then by applied by simply invoking the command:
# Working directory is assumed to be the manifests directory
./apply.sh
Once the basic cluster stuff is setup, you can just apply this directory with
kubectl apply -k .