fix: correct too-aggressive HAProxy keep-alive timeout from #39

Conversation 0 Commits 1 Files Changed 1

+19 -4

1 Commits

Author	SHA1	Message	Date
emily	6f63b38497	fix: correct too-aggressive HAProxy client keep-alive timeout from #39 buildbot/nix-eval Build done. (1 warning) Details buildbot/nix-build Build done. Details DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly 15 minutes after PR #39 deployed. That PR's backend-side fix (option http-server-close on 'backend next') is confirmed working - journalctl/nginx access logs show a completely clean, uninterrupted request stream on the haproxy<->nginx leg through the exact failure timestamp. Root cause of the recurrence: PR #39 also added 'timeout http-keep-alive 30s' to defaults, intended as an unrelated tidy-up given maxconn=80. That value didn't account for client-side HTTP connection pooling: DAVx5 runs on OkHttp, which holds idle pooled connections open for up to 5 minutes by default before evicting them. With haproxy closing idle client-facing keep-alive connections after just 30s, any DAVx5 connection idle between 30s-300s got silently closed by haproxy while the client still considered it live - the client's next reuse attempt produced exactly the same class of error, just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of being fixed. Fix: - defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely above OkHttp's 300s (5min) idle-eviction default, so a client's own pool always evicts a stale connection before haproxy would. - backend next: add 'log-tag next' so this backend's haproxy log lines carry a distinct syslog tag ('journalctl -t next') instead of being interleaved with every other backend under the shared 'haproxy' tag - this specific incident took significant manual grep/awk work to isolate 'next' traffic from git/matrix/immich noise in the same log stream, which a dedicated tag eliminates going forward. Verified by comparing haproxy's own next/nextcloud access log lines (all showing normal termination, no CD/SD flags) against nginx's nginx_access journal (clean, continuous, no gap) across the exact 04:00:58 UTC failure window - confirming the backend-side legs were healthy and the failure had to be on the client<->haproxy leg instead. Could not run 'haproxy -c' locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as prior PRs in this series (#37, #38, #39).	2026-08-09 23:09:31 -05:00

Author

SHA1

Message

Date

emily

6f63b38497

fix: correct too-aggressive HAProxy client keep-alive timeout from #39

buildbot/nix-eval Build done. (1 warning)

Details

buildbot/nix-build Build done.

Details

DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of
stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly
15 minutes after PR #39 deployed. That PR's backend-side fix (option
http-server-close on 'backend next') is confirmed working -
journalctl/nginx access logs show a completely clean, uninterrupted
request stream on the haproxy<->nginx leg through the exact failure
timestamp.

Root cause of the recurrence: PR #39 also added 'timeout
http-keep-alive 30s' to defaults, intended as an unrelated tidy-up
given maxconn=80. That value didn't account for client-side HTTP
connection pooling: DAVx5 runs on OkHttp, which holds idle pooled
connections open for up to 5 minutes by default before evicting them.
With haproxy closing idle client-facing keep-alive connections after
just 30s, any DAVx5 connection idle between 30s-300s got silently
closed by haproxy while the client still considered it live - the
client's next reuse attempt produced exactly the same class of error,
just relocated from the haproxy<->nginx leg to the client<->haproxy
leg instead of being fixed.

Fix:
- defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely
  above OkHttp's 300s (5min) idle-eviction default, so a client's own
  pool always evicts a stale connection before haproxy would.
- backend next: add 'log-tag next' so this backend's haproxy log
  lines carry a distinct syslog tag ('journalctl -t next') instead of
  being interleaved with every other backend under the shared
  'haproxy' tag - this specific incident took significant manual
  grep/awk work to isolate 'next' traffic from git/matrix/immich noise
  in the same log stream, which a dedicated tag eliminates going
  forward.

Verified by comparing haproxy's own next/nextcloud access log lines
(all showing normal termination, no CD/SD flags) against nginx's
nginx_access journal (clean, continuous, no gap) across the exact
04:00:58 UTC failure window - confirming the backend-side legs were
healthy and the failure had to be on the client<->haproxy leg instead.

Could not run 'haproxy -c' locally (no toolchain in the agent
sandbox) - recommend confirming via CI/garnix before merge, same
caveat as prior PRs in this series (#37, #38, #39).

2026-08-09 23:09:31 -05:00

fix: correct too-aggressive HAProxy keep-alive timeout from #39 #40