Cross-checked haproxy's next/nextcloud backend log lines against nginx's nginx_access journal for the exact 04:00:58Z failure window: no CD/SD termination flags on the haproxy side, and a completely clean, uninterrupted request stream on the nginx side. The haproxy<->nginx leg fixed in #39 is healthy.
#39 also added timeout http-keep-alive 30s to defaults, framed as an unrelated tidy-up (maxconn is only 80). I didn't account for client-side connection pooling: DAVx5 runs on OkHttp, which by default holds idle pooled connections open for up to 5 minutes before evicting them.
With haproxy closing idle client-facing keep-alive connections after only 30s, any DAVx5 connection sitting idle between 30s-300s got silently closed by haproxy while the client still believed it was live. The client's next reuse attempt produced exactly the same error class - just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of actually being fixed.
Fix
defaults: raise timeout http-keep-alive from 30s to 6m - safely above OkHttp's 300s (5min) idle-eviction default, so the client always evicts a stale connection on its own before haproxy would close it out from under it.
backend next: add log-tag next so this backend's log lines carry a distinct syslog tag (journalctl -t next) instead of being interleaved with git/matrix/immich traffic under the shared haproxy tag. Diagnosing this specific incident required manually grepping/awk-ing the shared backend out of the combined stream - a dedicated tag removes that friction for the next incident.
Verification
Could not run haproxy -c locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as #37/#38/#39. After merge/deploy, recommend watching DAVx5 sync over several hours (long enough to span multiple idle periods >30s but <5min) to confirm the EOFException does not recur.
## Symptom
DAVx5 hit the exact same `unexpected end of stream`/`EOFException: limit=0` error again at `2026-08-10T04:00:58Z`, roughly 15 minutes after [#39](http://git.k3s.thehellings.lan/greg/nixos/pulls/39) deployed.
## The #39 backend fix is confirmed working
Cross-checked haproxy's `next/nextcloud` backend log lines against nginx's `nginx_access` journal for the exact `04:00:58Z` failure window: no `CD`/`SD` termination flags on the haproxy side, and a completely clean, uninterrupted request stream on the nginx side. The haproxy<->nginx leg fixed in #39 is healthy.
## Root cause of the recurrence: my own fix in #39
#39 also added `timeout http-keep-alive 30s` to `defaults`, framed as an unrelated tidy-up (`maxconn` is only 80). I didn't account for client-side connection pooling: DAVx5 runs on OkHttp, which by default holds idle pooled connections open for up to 5 minutes before evicting them.
With haproxy closing idle *client-facing* keep-alive connections after only 30s, any DAVx5 connection sitting idle between 30s-300s got silently closed by haproxy while the client still believed it was live. The client's next reuse attempt produced exactly the same error class - just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of actually being fixed.
## Fix
- `defaults`: raise `timeout http-keep-alive` from `30s` to `6m` - safely above OkHttp's 300s (5min) idle-eviction default, so the client always evicts a stale connection on its own before haproxy would close it out from under it.
- `backend next`: add `log-tag next` so this backend's log lines carry a distinct syslog tag (`journalctl -t next`) instead of being interleaved with git/matrix/immich traffic under the shared `haproxy` tag. Diagnosing this specific incident required manually grepping/awk-ing the shared backend out of the combined stream - a dedicated tag removes that friction for the next incident.
## Verification
Could not run `haproxy -c` locally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as #37/#38/#39. After merge/deploy, recommend watching DAVx5 sync over several hours (long enough to span multiple idle periods >30s but <5min) to confirm the EOFException does not recur.
DAVx5 (CalDAV/CardDAV) reported the exact same 'unexpected end of
stream' / EOFException error again at 2026-08-10T04:00:58Z, roughly
15 minutes after PR #39 deployed. That PR's backend-side fix (option
http-server-close on 'backend next') is confirmed working -
journalctl/nginx access logs show a completely clean, uninterrupted
request stream on the haproxy<->nginx leg through the exact failure
timestamp.
Root cause of the recurrence: PR #39 also added 'timeout
http-keep-alive 30s' to defaults, intended as an unrelated tidy-up
given maxconn=80. That value didn't account for client-side HTTP
connection pooling: DAVx5 runs on OkHttp, which holds idle pooled
connections open for up to 5 minutes by default before evicting them.
With haproxy closing idle client-facing keep-alive connections after
just 30s, any DAVx5 connection idle between 30s-300s got silently
closed by haproxy while the client still considered it live - the
client's next reuse attempt produced exactly the same class of error,
just relocated from the haproxy<->nginx leg to the client<->haproxy
leg instead of being fixed.
Fix:
- defaults: raise 'timeout http-keep-alive' from 30s to 6m, safely
above OkHttp's 300s (5min) idle-eviction default, so a client's own
pool always evicts a stale connection before haproxy would.
- backend next: add 'log-tag next' so this backend's haproxy log
lines carry a distinct syslog tag ('journalctl -t next') instead of
being interleaved with every other backend under the shared
'haproxy' tag - this specific incident took significant manual
grep/awk work to isolate 'next' traffic from git/matrix/immich noise
in the same log stream, which a dedicated tag eliminates going
forward.
Verified by comparing haproxy's own next/nextcloud access log lines
(all showing normal termination, no CD/SD flags) against nginx's
nginx_access journal (clean, continuous, no gap) across the exact
04:00:58 UTC failure window - confirming the backend-side legs were
healthy and the failure had to be on the client<->haproxy leg instead.
Could not run 'haproxy -c' locally (no toolchain in the agent
sandbox) - recommend confirming via CI/garnix before merge, same
caveat as prior PRs in this series (#37, #38, #39).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
DAVx5 hit the exact same
unexpected end of stream/EOFException: limit=0error again at2026-08-10T04:00:58Z, roughly 15 minutes after #39 deployed.The #39 backend fix is confirmed working
Cross-checked haproxy's
next/nextcloudbackend log lines against nginx'snginx_accessjournal for the exact04:00:58Zfailure window: noCD/SDtermination flags on the haproxy side, and a completely clean, uninterrupted request stream on the nginx side. The haproxy<->nginx leg fixed in #39 is healthy.Root cause of the recurrence: my own fix in #39
#39 also added
timeout http-keep-alive 30stodefaults, framed as an unrelated tidy-up (maxconnis only 80). I didn't account for client-side connection pooling: DAVx5 runs on OkHttp, which by default holds idle pooled connections open for up to 5 minutes before evicting them.With haproxy closing idle client-facing keep-alive connections after only 30s, any DAVx5 connection sitting idle between 30s-300s got silently closed by haproxy while the client still believed it was live. The client's next reuse attempt produced exactly the same error class - just relocated from the haproxy<->nginx leg to the client<->haproxy leg instead of actually being fixed.
Fix
defaults: raisetimeout http-keep-alivefrom30sto6m- safely above OkHttp's 300s (5min) idle-eviction default, so the client always evicts a stale connection on its own before haproxy would close it out from under it.backend next: addlog-tag nextso this backend's log lines carry a distinct syslog tag (journalctl -t next) instead of being interleaved with git/matrix/immich traffic under the sharedhaproxytag. Diagnosing this specific incident required manually grepping/awk-ing the shared backend out of the combined stream - a dedicated tag removes that friction for the next incident.Verification
Could not run
haproxy -clocally (no toolchain in the agent sandbox) - recommend confirming via CI/garnix before merge, same caveat as #37/#38/#39. After merge/deploy, recommend watching DAVx5 sync over several hours (long enough to span multiple idle periods >30s but <5min) to confirm the EOFException does not recur.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.