From 6f63b384971d86051f5e96dccf5af1703179b4c4 Mon Sep 17 00:00:00 2001 From: "Emily (Agent)" Date: Sun, 9 Aug 2026 23:09:31 -0500 Subject: [PATCH] fix: correct too-aggressive HAProxy client keep-alive timeout from #39 DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly 15 minutes after PR #39 deployed. That PR's backend-side fix (option http-server-close on 'backend next') is confirmed working - journalctl/nginx access logs show a completely clean, uninterrupted request stream on the haproxy<->nginx leg through the exact failure timestamp. Root cause of the recurrence: PR #39 also added 'timeout http-keep-alive 30s' to defaults, intended as an unrelated tidy-up given maxconn=80. That value didn't account for client-side HTTP connection pooling: DAVx5 runs on OkHttp, which holds idle pooled connections open for up to 5 minutes by default before evicting them. With haproxy closing idle client-facing keep-alive connections after just 30s, any DAVx5 connection idle between 30s-300s got silently closed by haproxy while the client still considered it live - the client's next reuse attempt produced exactly the same class of error, just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of being fixed. Fix: - defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely above OkHttp's 300s (5min) idle-eviction default, so a client's own pool always evicts a stale connection before haproxy would. - backend next: add 'log-tag next' so this backend's haproxy log lines carry a distinct syslog tag ('journalctl -t next') instead of being interleaved with every other backend under the shared 'haproxy' tag - this specific incident took significant manual grep/awk work to isolate 'next' traffic from git/matrix/immich noise in the same log stream, which a dedicated tag eliminates going forward. Verified by comparing haproxy's own next/nextcloud access log lines (all showing normal termination, no CD/SD flags) against nginx's nginx_access journal (clean, continuous, no gap) across the exact 04:00:58 UTC failure window - confirming the backend-side legs were healthy and the failure had to be on the client<->haproxy leg instead. Could not run 'haproxy -c' locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as prior PRs in this series (#37, #38, #39). --- hosts/unstable/linode/default.nix | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/hosts/unstable/linode/default.nix b/hosts/unstable/linode/default.nix index b8b538d..3f01426 100644 --- a/hosts/unstable/linode/default.nix +++ b/hosts/unstable/linode/default.nix @@ -178,10 +178,24 @@ in timeout server 1h # HAProxy defaults to end-to-end keep-alive (client AND server side) # unless a proxy overrides it. Bound how long an idle client-facing - # keep-alive connection is held: maxconn is only 80, and leaving - # this unset falls back to "timeout client" (500s), which is far - # longer than needed just to wait for a pipelined next request. - timeout http-keep-alive 30s + # keep-alive connection is held rather than falling back to + # "timeout client" (500s). + # + # CORRECTION (see #39): this was originally set to 30s as a + # tidy-up given maxconn=80, without considering client-side + # connection pooling behavior. That was too aggressive: DAVx5 (and + # OkHttp-based HTTP clients generally) keep idle pooled + # connections open for up to 5 minutes client-side before + # eviction. With a 30s haproxy-side timeout, any client connection + # idle between 30s-300s got silently closed by haproxy, and the + # client's next reuse of it produced exactly the "unexpected end + # of stream"/EOFException class of error this investigation + # started from - just relocated from the haproxy<->nginx leg + # (fixed in `backend next` below) to the client<->haproxy leg. + # Set comfortably above OkHttp's 300s default so a client's own + # pool eviction always happens first and haproxy is never the one + # to close a connection the client still thinks is good. + timeout http-keep-alive 6m listen gitsshd bind *:${toString sshPort} @@ -266,6 +280,7 @@ in backend next log global + log-tag next mode http balance roundrobin option accept-unsafe-violations-in-http-response -- 2.54.0