fix: correct too-aggressive HAProxy client keep-alive timeout from #39
DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly 15 minutes after PR #39 deployed. That PR's backend-side fix (option http-server-close on 'backend next') is confirmed working - journalctl/nginx access logs show a completely clean, uninterrupted request stream on the haproxy<->nginx leg through the exact failure timestamp. Root cause of the recurrence: PR #39 also added 'timeout http-keep-alive 30s' to defaults, intended as an unrelated tidy-up given maxconn=80. That value didn't account for client-side HTTP connection pooling: DAVx5 runs on OkHttp, which holds idle pooled connections open for up to 5 minutes by default before evicting them. With haproxy closing idle client-facing keep-alive connections after just 30s, any DAVx5 connection idle between 30s-300s got silently closed by haproxy while the client still considered it live - the client's next reuse attempt produced exactly the same class of error, just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of being fixed. Fix: - defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely above OkHttp's 300s (5min) idle-eviction default, so a client's own pool always evicts a stale connection before haproxy would. - backend next: add 'log-tag next' so this backend's haproxy log lines carry a distinct syslog tag ('journalctl -t next') instead of being interleaved with every other backend under the shared 'haproxy' tag - this specific incident took significant manual grep/awk work to isolate 'next' traffic from git/matrix/immich noise in the same log stream, which a dedicated tag eliminates going forward. Verified by comparing haproxy's own next/nextcloud access log lines (all showing normal termination, no CD/SD flags) against nginx's nginx_access journal (clean, continuous, no gap) across the exact 04:00:58 UTC failure window - confirming the backend-side legs were healthy and the failure had to be on the client<->haproxy leg instead. Could not run 'haproxy -c' locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as prior PRs in this series (#37, #38, #39).
This commit is contained in:
@@ -178,10 +178,24 @@ in
|
||||
timeout server 1h
|
||||
# HAProxy defaults to end-to-end keep-alive (client AND server side)
|
||||
# unless a proxy overrides it. Bound how long an idle client-facing
|
||||
# keep-alive connection is held: maxconn is only 80, and leaving
|
||||
# this unset falls back to "timeout client" (500s), which is far
|
||||
# longer than needed just to wait for a pipelined next request.
|
||||
timeout http-keep-alive 30s
|
||||
# keep-alive connection is held rather than falling back to
|
||||
# "timeout client" (500s).
|
||||
#
|
||||
# CORRECTION (see #39): this was originally set to 30s as a
|
||||
# tidy-up given maxconn=80, without considering client-side
|
||||
# connection pooling behavior. That was too aggressive: DAVx5 (and
|
||||
# OkHttp-based HTTP clients generally) keep idle pooled
|
||||
# connections open for up to 5 minutes client-side before
|
||||
# eviction. With a 30s haproxy-side timeout, any client connection
|
||||
# idle between 30s-300s got silently closed by haproxy, and the
|
||||
# client's next reuse of it produced exactly the "unexpected end
|
||||
# of stream"/EOFException class of error this investigation
|
||||
# started from - just relocated from the haproxy<->nginx leg
|
||||
# (fixed in `backend next` below) to the client<->haproxy leg.
|
||||
# Set comfortably above OkHttp's 300s default so a client's own
|
||||
# pool eviction always happens first and haproxy is never the one
|
||||
# to close a connection the client still thinks is good.
|
||||
timeout http-keep-alive 6m
|
||||
|
||||
listen gitsshd
|
||||
bind *:${toString sshPort}
|
||||
@@ -266,6 +280,7 @@ in
|
||||
|
||||
backend next
|
||||
log global
|
||||
log-tag next
|
||||
mode http
|
||||
balance roundrobin
|
||||
option accept-unsafe-violations-in-http-response
|
||||
|
||||
Reference in New Issue
Block a user