emily
10cdf9408d
Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h before it crash-looped, but with 'option httplog' commented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client. - linode: enable HAProxy httplog + defaults 'log global' (was commented out) so every proxied HTTP request is now logged with timing/status/bytes. - linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live per-backend/per-server connection and byte counters. - linode: route nginx (Nextcloud's local vhost) access logs to journald via syslog, since the read-only monitoring account has no access to /var/log/nginx/*. - linode: enable vnstat for historical per-interface bandwidth tracking (5-min granularity) so a reported 'traffic was high for N hours' can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps. - k3s manifests: enable Traefik access logging (JSON) — this is the ingress layer HAProxy forwards :80 traffic to (git/matrix/immich), and lacked any per-request visibility. - hosts/baseline.nix (fleet-wide): add a journald rate limit (2000 lines / 30s per unit). Found live while investigating that uptime-kuma on 'kuma' was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour), which was itself degrading journalctl responsiveness on that host during the cross-host traffic scan. Related but not otherwise addressed here: Nebula relay/handshake churn on kuma's tunnel and the etcd read-latency warnings seen on isaiah/zeke around the same incident window — noted for a future investigation, not fixed by this PR.
This is a unified repo to contain my personal configurations for NixOS machines.
How To Use This
Go through the normal process to setup a NixOS system during installation.
- Boot from an appropriate medium
- Parition the drives
- Mount them
- Before you generate out the configuration, clone this repoistory to your /etc/nixos folder
- Run the configuration generator. It should only genreate the hardware-configuration.nix file, which this repo gitignores
- Create a folder and file with the machine name in
hosts/<machine>/default.nix - Add
hosts/default.nixan appropriate entry for the machine you are building - Create a file
home/hosts/<machine>/default.nixwith the new machine name as well
Adding new hosts
To add a new host, create a folder in the directory hosts/ that matches the name of
the target system. Each host must contain, minimally, a default.nix file that serves
as the basis of configuring that host.
Test build a VM for your system
nom build ".#nixosConfigurations.<host>.config.system.build.vmexport QEMU_NET_OPTS="hostfwd=tcp::2221-:22"to export the SSH port./result/bin/run-<host>-vm