Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h before it crash-looped, but with option httplog commented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client after the fact.
linode: enable HAProxy httplog + defaults log global (was commented out) so every proxied HTTP request is now logged with timing/status/bytes.
linode: add a HAProxy stats listener on 127.0.0.1:8404 for live per-backend/per-server connection and byte counters.
linode: route nginx's (Nextcloud local vhost) access logs to journald via syslog, since the read-only monitoring account has no filesystem access to /var/log/nginx/*.
linode: enable vnstat for historical per-interface bandwidth tracking (5-min granularity) so a reported "traffic was high for N hours" can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps.
k3s manifests: enable Traefik access logging (JSON format). This is the ingress layer HAProxy forwards :80 traffic to (git/matrix/immich backends), and it previously had zero per-request visibility.
hosts/baseline.nix (fleet-wide): add a journald rate limit (2000 lines / 30s per unit). Found live while cross-host-scanning for related traffic spikes: uptime-kuma on the kuma host was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour, non-stop), which was itself degrading journalctl responsiveness on that host and would hamper investigating a real incident there.
Why
The original incident investigation had to reconstruct everything from systemd's coarse per-unit IPAccounting totals and journal line-count histograms because none of the actual proxy/ingress layers were logging requests. This closes that gap for future incidents.
Cross-host scan findings (informational, not fixed here)
While scanning the rest of the fleet for correlated spikes around the same window (~Aug 6 21:00 \u2013 Aug 7 22:00 UTC), also observed on the k3s nodes (isaiah/jeremiah/zeke): a large burst of etcd apply request took too long / waiting for ReadIndex response warnings, and heavy k3s/kubelet log volume tied to volume mount/unmount churn for a mariadb-operator pod reschedule. These look like a separate (or possibly upstream-causal) k3s-control-plane stress event overlapping the linode traffic spike, but I didn't change anything for them in this PR \u2014 flagging for a follow-up if you want it investigated further.
Testing
No local Nix toolchain in my sandbox (same caveat as the prior monitoring-account PR) \u2014 I hand-verified brace/paren balance on the edited .nix files and checked the YAML parses, but this hasn't been evaluated against the flake. Worth letting garnix CI run before merging.
## What
Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own `IPAccounting` confirmed ~121GB moved over ~19.6h before it crash-looped, but with `option httplog` commented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client after the fact.
- **linode**: enable HAProxy `httplog` + `defaults log global` (was commented out) so every proxied HTTP request is now logged with timing/status/bytes.
- **linode**: add a HAProxy `stats` listener on `127.0.0.1:8404` for live per-backend/per-server connection and byte counters.
- **linode**: route nginx's (Nextcloud local vhost) access logs to journald via syslog, since the read-only monitoring account has no filesystem access to `/var/log/nginx/*`.
- **linode**: enable `vnstat` for historical per-interface bandwidth tracking (5-min granularity) so a reported "traffic was high for N hours" can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps.
- **k3s manifests**: enable Traefik access logging (JSON format). This is the ingress layer HAProxy forwards `:80` traffic to (git/matrix/immich backends), and it previously had zero per-request visibility.
- **hosts/baseline.nix (fleet-wide)**: add a journald rate limit (2000 lines / 30s per unit). Found live while cross-host-scanning for related traffic spikes: `uptime-kuma` on the `kuma` host was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour, non-stop), which was itself degrading `journalctl` responsiveness on that host and would hamper investigating a real incident there.
## Why
The original incident investigation had to reconstruct everything from `systemd`'s coarse per-unit `IPAccounting` totals and journal line-count histograms because none of the actual proxy/ingress layers were logging requests. This closes that gap for future incidents.
## Cross-host scan findings (informational, not fixed here)
While scanning the rest of the fleet for correlated spikes around the same window (~Aug 6 21:00 \u2013 Aug 7 22:00 UTC), also observed on the k3s nodes (isaiah/jeremiah/zeke): a large burst of etcd `apply request took too long` / `waiting for ReadIndex response` warnings, and heavy `k3s`/`kubelet` log volume tied to volume mount/unmount churn for a `mariadb-operator` pod reschedule. These look like a separate (or possibly upstream-causal) k3s-control-plane stress event overlapping the linode traffic spike, but I didn't change anything for them in this PR \u2014 flagging for a follow-up if you want it investigated further.
## Testing
No local Nix toolchain in my sandbox (same caveat as the prior monitoring-account PR) \u2014 I hand-verified brace/paren balance on the edited `.nix` files and checked the YAML parses, but this hasn't been evaluated against the flake. Worth letting garnix CI run before merging.
Triggered by investigating a several-hour >10Mbps traffic spike to
linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h
before it crash-looped, but with 'option httplog' commented out and no
per-backend request logs, there was no way to attribute that traffic
to a specific backend, host, or client.
- linode: enable HAProxy httplog + defaults 'log global' (was
commented out) so every proxied HTTP request is now logged with
timing/status/bytes.
- linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live
per-backend/per-server connection and byte counters.
- linode: route nginx (Nextcloud's local vhost) access logs to
journald via syslog, since the read-only monitoring account has no
access to /var/log/nginx/*.
- linode: enable vnstat for historical per-interface bandwidth
tracking (5-min granularity) so a reported 'traffic was high for N
hours' can be confirmed/timestamped immediately instead of
reconstructed after the fact from journal timestamps.
- k3s manifests: enable Traefik access logging (JSON) — this is the
ingress layer HAProxy forwards :80 traffic to (git/matrix/immich),
and lacked any per-request visibility.
- hosts/baseline.nix (fleet-wide): add a journald rate limit
(2000 lines / 30s per unit). Found live while investigating that
uptime-kuma on 'kuma' was logging a Prometheus label-validation
error on every monitor beat (~100k lines/hour), which was itself
degrading journalctl responsiveness on that host during the
cross-host traffic scan.
Related but not otherwise addressed here: Nebula relay/handshake
churn on kuma's tunnel and the etcd read-latency warnings seen on
isaiah/zeke around the same incident window — noted for a future
investigation, not fixed by this PR.
greg
merged commit d1459a7f63 into main2026-08-09 21:33:55 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own
IPAccountingconfirmed ~121GB moved over ~19.6h before it crash-looped, but withoption httplogcommented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client after the fact.httplog+defaults log global(was commented out) so every proxied HTTP request is now logged with timing/status/bytes.statslistener on127.0.0.1:8404for live per-backend/per-server connection and byte counters./var/log/nginx/*.vnstatfor historical per-interface bandwidth tracking (5-min granularity) so a reported "traffic was high for N hours" can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps.:80traffic to (git/matrix/immich backends), and it previously had zero per-request visibility.uptime-kumaon thekumahost was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour, non-stop), which was itself degradingjournalctlresponsiveness on that host and would hamper investigating a real incident there.Why
The original incident investigation had to reconstruct everything from
systemd's coarse per-unitIPAccountingtotals and journal line-count histograms because none of the actual proxy/ingress layers were logging requests. This closes that gap for future incidents.Cross-host scan findings (informational, not fixed here)
While scanning the rest of the fleet for correlated spikes around the same window (~Aug 6 21:00 \u2013 Aug 7 22:00 UTC), also observed on the k3s nodes (isaiah/jeremiah/zeke): a large burst of etcd
apply request took too long/waiting for ReadIndex responsewarnings, and heavyk3s/kubeletlog volume tied to volume mount/unmount churn for amariadb-operatorpod reschedule. These look like a separate (or possibly upstream-causal) k3s-control-plane stress event overlapping the linode traffic spike, but I didn't change anything for them in this PR \u2014 flagging for a follow-up if you want it investigated further.Testing
No local Nix toolchain in my sandbox (same caveat as the prior monitoring-account PR) \u2014 I hand-verified brace/paren balance on the edited
.nixfiles and checked the YAML parses, but this hasn't been evaluated against the flake. Worth letting garnix CI run before merging.