a8ba8f247d9f47cb309059e2d3ca218affe3eb59
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 emily
|
6f63b38497 |
fix: correct too-aggressive HAProxy client keep-alive timeout from #39
DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly 15 minutes after PR #39 deployed. That PR's backend-side fix (option http-server-close on 'backend next') is confirmed working - journalctl/nginx access logs show a completely clean, uninterrupted request stream on the haproxy<->nginx leg through the exact failure timestamp. Root cause of the recurrence: PR #39 also added 'timeout http-keep-alive 30s' to defaults, intended as an unrelated tidy-up given maxconn=80. That value didn't account for client-side HTTP connection pooling: DAVx5 runs on OkHttp, which holds idle pooled connections open for up to 5 minutes by default before evicting them. With haproxy closing idle client-facing keep-alive connections after just 30s, any DAVx5 connection idle between 30s-300s got silently closed by haproxy while the client still considered it live - the client's next reuse attempt produced exactly the same class of error, just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of being fixed. Fix: - defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely above OkHttp's 300s (5min) idle-eviction default, so a client's own pool always evicts a stale connection before haproxy would. - backend next: add 'log-tag next' so this backend's haproxy log lines carry a distinct syslog tag ('journalctl -t next') instead of being interleaved with every other backend under the shared 'haproxy' tag - this specific incident took significant manual grep/awk work to isolate 'next' traffic from git/matrix/immich noise in the same log stream, which a dedicated tag eliminates going forward. Verified by comparing haproxy's own next/nextcloud access log lines (all showing normal termination, no CD/SD flags) against nginx's nginx_access journal (clean, continuous, no gap) across the exact 04:00:58 UTC failure window - confirming the backend-side legs were healthy and the failure had to be on the client<->haproxy leg instead. Could not run 'haproxy -c' locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as prior PRs in this series (#37, #38, #39). |
||
|
emily
|
2c3607f17c |
fix: prevent HAProxy from reusing stale keep-alive conns to nginx/Nextcloud
DAVx5 (CalDAV/CardDAV) on greg's phone was intermittently failing every sync type (CONTACTS/EVENTS/TASKS/RefreshCollectionsWorker) against next.thehellings.com with: java.io.IOException: unexpected end of stream Caused by: java.io.EOFException: \n not found: limit=0 This is the classic OkHttp/HTTP client signature of the far end silently closing a pooled keep-alive connection: the client reuses a socket it still believes is open, gets zero bytes back while reading response headers, and throws exactly this exception. Root cause: HAProxy's 'next' backend proxies to nginx on 127.0.0.1:8080, and HAProxy defaults to end-to-end keep-alive (both client- and server-side) unless told otherwise. nginx's keepalive_timeout is 65s, so any HAProxy<->nginx connection idle past that gets closed by nginx without HAProxy's knowledge. A request that lands on that now-dead pooled connection right after gets nothing back - surfacing to the client as a bare socket EOF while reading headers. The frontend's existing 'option http-server-close'/'http-keep-alive' pair only governs the client-facing side of HAProxy and does nothing for the HAProxy->nginx leg. Fix: - backend next: add 'option http-server-close' so HAProxy opens a fresh connection to nginx per request instead of pooling/reusing one. The backend is localhost, so the extra TCP handshake cost is negligible, and this removes the whole class of stale-connection EOF errors. - defaults: add 'timeout http-keep-alive 30s' to bound how long an idle client-facing keep-alive connection is held open. Previously unset, it fell back to 'timeout client' (500s) - unnecessarily long given maxconn is only 80, and tightens client-side connection churn to be more predictable too. Diagnosed by pulling the nginx_access journal (enabled in #37/#38) for the failing sync window and cross-referencing nginx's services.nginx.appendHttpConfig / generated nginx.conf keepalive settings against HAProxy's request-level defaults. Could not run 'haproxy -c'/'nginx -t' locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same as #38. |
||
|
emily
|
e4fe9e84bd |
fix: nginx syslog access_log tag must not contain a hyphen
The nginx_access syslog tag added in #37 (feat/emily-incident-logging) used tag=nginx-access. nginx's syslog sink only accepts alphanumeric characters and underscores in the tag field, so the generated nginx.conf failed its config test on linode: nginx: [emerg] syslog "tag" only allows alphanumeric characters and underscore in .../nginx.conf:114 Because nginx-pre-start failed, nginx.service crash-looped until it hit systemd's start-limit-hit and gave up entirely. Since Nextcloud is proxied through nginx (127.0.0.1:8080, fronted by haproxy's 'next' backend), this took next.thehellings.com down with a 503 from haproxy (phpfpm-nextcloud/postgresql/redis backends were all healthy and unaffected - purely an nginx config parse failure). Fix: use an underscore (nginx_access) instead of a hyphen. |
||
|
emily
|
10cdf9408d |
feat: enable request-level logging for bandwidth/traffic incident tracing
Triggered by investigating a several-hour >10Mbps traffic spike to linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h before it crash-looped, but with 'option httplog' commented out and no per-backend request logs, there was no way to attribute that traffic to a specific backend, host, or client. - linode: enable HAProxy httplog + defaults 'log global' (was commented out) so every proxied HTTP request is now logged with timing/status/bytes. - linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live per-backend/per-server connection and byte counters. - linode: route nginx (Nextcloud's local vhost) access logs to journald via syslog, since the read-only monitoring account has no access to /var/log/nginx/*. - linode: enable vnstat for historical per-interface bandwidth tracking (5-min granularity) so a reported 'traffic was high for N hours' can be confirmed/timestamped immediately instead of reconstructed after the fact from journal timestamps. - k3s manifests: enable Traefik access logging (JSON) — this is the ingress layer HAProxy forwards :80 traffic to (git/matrix/immich), and lacked any per-request visibility. - hosts/baseline.nix (fleet-wide): add a journald rate limit (2000 lines / 30s per unit). Found live while investigating that uptime-kuma on 'kuma' was logging a Prometheus label-validation error on every monitor beat (~100k lines/hour), which was itself degrading journalctl responsiveness on that host during the cross-host traffic scan. Related but not otherwise addressed here: Nebula relay/handshake churn on kuma's tunnel and the etcd read-latency warnings seen on isaiah/zeke around the same incident window — noted for a future investigation, not fixed by this PR. |
||
|
emily
|
3cba4cbd86 |
feat: add read-only emily monitoring account
Adds a new NixOS module (greg.monitoring-access) that provisions a dedicated, SSH-key-only 'emily' user account across all managed hosts. The account is intentionally minimal-privilege: - No password set (SSH key auth only) - Not a member of wheel, no sudo/sudo-rs rules - Only extra group membership is systemd-journal, granting read access to system logs for monitoring/analysis tasks - Authorized key lives in home/ssh/emily_authorized_keys, mirroring the existing pattern used for the greg account's authorized_keys This lets the Hermes agent (emily) log in read-only to inspect logs and system state when asked, without any ability to modify configuration, escalate privileges, or run destructive commands. Module is imported unconditionally in modules/nixos/default.nix like the other nixos modules, and defaults to enabled; it can be disabled per-host via greg.monitoring-access.enable = false if ever needed. |
||
|
emily
|
28977235a1 |
chore: clean up builder2, Ceph module, normalize Darwin host symlinks
- Remove builder2 (retired host): dangling network.json entry and
empty home/hosts/builder2 stub. Its old IP (10.42.1.17) is already
correctly owned by pve4.
- Remove the abandoned Ceph module (modules/nixos/ceph.nix) and its
unencrypted plaintext keyring files under secrets/. No host ever
enabled services.ceph-benaco; the keyrings were dead, unencrypted
credentials sitting in the repo.
- Normalize Darwin host identity: IVR and Lithic are the only two
physical Darwin machines. All other darwin/hosts/* names are now
symlinks to whichever of the two they represent, matching the DHCP
name variations nix-darwin sees depending on network:
gregory -> ivr
gregory.hellings-mbp -> ivr
MacBook-Prolocal -> ivr
gregs-MacBook-Pro-16-inch-Nov-2024 -> lithic
li -> lithic (pre-existing)
This lets each machine's config be maintained once regardless of
what hostname it currently advertises.
|
||
|
emily
|
fcb5a89727 |
fix: correct pve1 IP to 10.42.0.4, rename stale joel/opnsense refs
pve1 is a static/DHCP-reserved Proxmox host at 10.42.0.4 (previously mislabeled 'joel' in some places). 10.42.1.1 is the UDM Pro gateway IP, not pve1 -- OPNsense was retired in favor of Ubiquiti. Removes the stale duplicate PVE1 DHCP reservation at 10.42.1.1 and drops the now-redundant 'joel' entry from network.json (consolidated into pve1). |