DAVx5 (CalDAV/CardDAV) on greg's phone was intermittently failing every
sync type (CONTACTS/EVENTS/TASKS/RefreshCollectionsWorker) against
next.thehellings.com with:
java.io.IOException: unexpected end of stream
Caused by: java.io.EOFException: \n not found: limit=0
This is the classic OkHttp/HTTP client signature of the far end
silently closing a pooled keep-alive connection: the client reuses a
socket it still believes is open, gets zero bytes back while reading
response headers, and throws exactly this exception.
Root cause: HAProxy's 'next' backend proxies to nginx on
127.0.0.1:8080, and HAProxy defaults to end-to-end keep-alive (both
client- and server-side) unless told otherwise. nginx's
keepalive_timeout is 65s, so any HAProxy<->nginx connection idle past
that gets closed by nginx without HAProxy's knowledge. A request that
lands on that now-dead pooled connection right after gets nothing back
- surfacing to the client as a bare socket EOF while reading headers.
The frontend's existing 'option http-server-close'/'http-keep-alive'
pair only governs the client-facing side of HAProxy and does nothing
for the HAProxy->nginx leg.
Fix:
- backend next: add 'option http-server-close' so HAProxy opens a
fresh connection to nginx per request instead of pooling/reusing
one. The backend is localhost, so the extra TCP handshake cost is
negligible, and this removes the whole class of stale-connection EOF
errors.
- defaults: add 'timeout http-keep-alive 30s' to bound how long an
idle client-facing keep-alive connection is held open. Previously
unset, it fell back to 'timeout client' (500s) - unnecessarily long
given maxconn is only 80, and tightens client-side connection churn
to be more predictable too.
Diagnosed by pulling the nginx_access journal (enabled in #37/#38) for
the failing sync window and cross-referencing nginx's
services.nginx.appendHttpConfig / generated nginx.conf keepalive
settings against HAProxy's request-level defaults. Could not run
'haproxy -c'/'nginx -t' locally (no toolchain in the agent sandbox) -
recommend confirming via CI/garnix before merge, same as #38.
The nginx_access syslog tag added in #37 (feat/emily-incident-logging)
used tag=nginx-access. nginx's syslog sink only accepts alphanumeric
characters and underscores in the tag field, so the generated
nginx.conf failed its config test on linode:
nginx: [emerg] syslog "tag" only allows alphanumeric characters
and underscore in .../nginx.conf:114
Because nginx-pre-start failed, nginx.service crash-looped until it
hit systemd's start-limit-hit and gave up entirely. Since Nextcloud is
proxied through nginx (127.0.0.1:8080, fronted by haproxy's 'next'
backend), this took next.thehellings.com down with a 503 from haproxy
(phpfpm-nextcloud/postgresql/redis backends were all healthy and
unaffected - purely an nginx config parse failure).
Fix: use an underscore (nginx_access) instead of a hyphen.
Triggered by investigating a several-hour >10Mbps traffic spike to
linode. HAProxy's own IPAccounting confirmed ~121GB moved over ~19.6h
before it crash-looped, but with 'option httplog' commented out and no
per-backend request logs, there was no way to attribute that traffic
to a specific backend, host, or client.
- linode: enable HAProxy httplog + defaults 'log global' (was
commented out) so every proxied HTTP request is now logged with
timing/status/bytes.
- linode: add a haproxy 'stats' listener on 127.0.0.1:8404 for live
per-backend/per-server connection and byte counters.
- linode: route nginx (Nextcloud's local vhost) access logs to
journald via syslog, since the read-only monitoring account has no
access to /var/log/nginx/*.
- linode: enable vnstat for historical per-interface bandwidth
tracking (5-min granularity) so a reported 'traffic was high for N
hours' can be confirmed/timestamped immediately instead of
reconstructed after the fact from journal timestamps.
- k3s manifests: enable Traefik access logging (JSON) — this is the
ingress layer HAProxy forwards :80 traffic to (git/matrix/immich),
and lacked any per-request visibility.
- hosts/baseline.nix (fleet-wide): add a journald rate limit
(2000 lines / 30s per unit). Found live while investigating that
uptime-kuma on 'kuma' was logging a Prometheus label-validation
error on every monitor beat (~100k lines/hour), which was itself
degrading journalctl responsiveness on that host during the
cross-host traffic scan.
Related but not otherwise addressed here: Nebula relay/handshake
churn on kuma's tunnel and the etcd read-latency warnings seen on
isaiah/zeke around the same incident window — noted for a future
investigation, not fixed by this PR.
Adds a new NixOS module (greg.monitoring-access) that provisions a
dedicated, SSH-key-only 'emily' user account across all managed hosts.
The account is intentionally minimal-privilege:
- No password set (SSH key auth only)
- Not a member of wheel, no sudo/sudo-rs rules
- Only extra group membership is systemd-journal, granting read access
to system logs for monitoring/analysis tasks
- Authorized key lives in home/ssh/emily_authorized_keys, mirroring the
existing pattern used for the greg account's authorized_keys
This lets the Hermes agent (emily) log in read-only to inspect logs and
system state when asked, without any ability to modify configuration,
escalate privileges, or run destructive commands.
Module is imported unconditionally in modules/nixos/default.nix like
the other nixos modules, and defaults to enabled; it can be disabled
per-host via greg.monitoring-access.enable = false if ever needed.
- Remove builder2 (retired host): dangling network.json entry and
empty home/hosts/builder2 stub. Its old IP (10.42.1.17) is already
correctly owned by pve4.
- Remove the abandoned Ceph module (modules/nixos/ceph.nix) and its
unencrypted plaintext keyring files under secrets/. No host ever
enabled services.ceph-benaco; the keyrings were dead, unencrypted
credentials sitting in the repo.
- Normalize Darwin host identity: IVR and Lithic are the only two
physical Darwin machines. All other darwin/hosts/* names are now
symlinks to whichever of the two they represent, matching the DHCP
name variations nix-darwin sees depending on network:
gregory -> ivr
gregory.hellings-mbp -> ivr
MacBook-Prolocal -> ivr
gregs-MacBook-Pro-16-inch-Nov-2024 -> lithic
li -> lithic (pre-existing)
This lets each machine's config be maintained once regardless of
what hostname it currently advertises.
pve1 is a static/DHCP-reserved Proxmox host at 10.42.0.4 (previously
mislabeled 'joel' in some places). 10.42.1.1 is the UDM Pro gateway
IP, not pve1 -- OPNsense was retired in favor of Ubiquiti. Removes
the stale duplicate PVE1 DHCP reservation at 10.42.1.1 and drops the
now-redundant 'joel' entry from network.json (consolidated into
pve1).
emily