fix: correct too-aggressive HAProxy client keep-alive timeout from #39
buildbot/nix-eval Build done. (1 warning)
buildbot/nix-build Build done.

DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of
stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly
15 minutes after PR #39 deployed. That PR's backend-side fix (option
http-server-close on 'backend next') is confirmed working -
journalctl/nginx access logs show a completely clean, uninterrupted
request stream on the haproxy<->nginx leg through the exact failure
timestamp.

Root cause of the recurrence: PR #39 also added 'timeout
http-keep-alive 30s' to defaults, intended as an unrelated tidy-up
given maxconn=80. That value didn't account for client-side HTTP
connection pooling: DAVx5 runs on OkHttp, which holds idle pooled
connections open for up to 5 minutes by default before evicting them.
With haproxy closing idle client-facing keep-alive connections after
just 30s, any DAVx5 connection idle between 30s-300s got silently
closed by haproxy while the client still considered it live - the
client's next reuse attempt produced exactly the same class of error,
just relocated from the haproxy<->nginx leg to the client<->haproxy
leg instead of being fixed.

Fix:
- defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely
  above OkHttp's 300s (5min) idle-eviction default, so a client's own
  pool always evicts a stale connection before haproxy would.
- backend next: add 'log-tag next' so this backend's haproxy log
  lines carry a distinct syslog tag ('journalctl -t next') instead of
  being interleaved with every other backend under the shared
  'haproxy' tag - this specific incident took significant manual
  grep/awk work to isolate 'next' traffic from git/matrix/immich noise
  in the same log stream, which a dedicated tag eliminates going
  forward.

Verified by comparing haproxy's own next/nextcloud access log lines
(all showing normal termination, no CD/SD flags) against nginx's
nginx_access journal (clean, continuous, no gap) across the exact
04:00:58 UTC failure window - confirming the backend-side legs were
healthy and the failure had to be on the client<->haproxy leg instead.

Could not run 'haproxy -c' locally (no toolchain in the agent
sandbox) - recommend confirming via CI/garnix before merge, same
caveat as prior PRs in this series (#37, #38, #39).
This commit is contained in:
2026-08-09 23:09:31 -05:00
parent bc90eb34e4
commit 6f63b38497
+19 -4
View File
@@ -178,10 +178,24 @@ in
timeout server 1h
# HAProxy defaults to end-to-end keep-alive (client AND server side)
# unless a proxy overrides it. Bound how long an idle client-facing
# keep-alive connection is held: maxconn is only 80, and leaving
# this unset falls back to "timeout client" (500s), which is far
# longer than needed just to wait for a pipelined next request.
timeout http-keep-alive 30s
# keep-alive connection is held rather than falling back to
# "timeout client" (500s).
#
# CORRECTION (see #39): this was originally set to 30s as a
# tidy-up given maxconn=80, without considering client-side
# connection pooling behavior. That was too aggressive: DAVx5 (and
# OkHttp-based HTTP clients generally) keep idle pooled
# connections open for up to 5 minutes client-side before
# eviction. With a 30s haproxy-side timeout, any client connection
# idle between 30s-300s got silently closed by haproxy, and the
# client's next reuse of it produced exactly the "unexpected end
# of stream"/EOFException class of error this investigation
# started from - just relocated from the haproxy<->nginx leg
# (fixed in `backend next` below) to the client<->haproxy leg.
# Set comfortably above OkHttp's 300s default so a client's own
# pool eviction always happens first and haproxy is never the one
# to close a connection the client still thinks is good.
timeout http-keep-alive 6m
listen gitsshd
bind *:${toString sshPort}
@@ -266,6 +280,7 @@ in
backend next
log global
log-tag next
mode http
balance roundrobin
option accept-unsafe-violations-in-http-response