DAVx5 (CalDAV/CardDAV client on Android) intermittently fails syncing every collection type against next.thehellings.com with:
java.io.IOException: unexpected end of stream on https://next.thehellings.com/...
Caused by: java.io.EOFException: \n not found: limit=0
This is the classic signature of an HTTP client reusing a pooled keep-alive connection that the server already silently closed - it gets zero bytes back mid-read of the response headers.
Root cause
haproxy's next backend proxies to nginx on 127.0.0.1:8080 (which fronts phpfpm-nextcloud). HAProxy defaults to end-to-end keep-alive (both client- and server-facing sides) unless a proxy explicitly overrides it - the existing frontend-level option http-server-close/option http-keep-alive pair only governs the client-facing side of haproxy, not the haproxy->nginx leg.
nginx's generated config has keepalive_timeout 65;. Any pooled haproxy->nginx connection idle past 65s gets closed by nginx without haproxy knowing. If a request then lands on that now-dead pooled connection, the client sees a bare socket EOF while reading response headers - exactly this error.
Confirmed via journalctl -t nginx_access (enabled in #37/#38) for the failing sync window, cross-referenced against the generated /nix/store/*-nginx.conf keepalive settings on linode.
Fix
backend next: add option http-server-close so haproxy opens a fresh connection to nginx per request instead of pooling/reusing one. Backend is localhost, so the extra handshake is negligible cost, and it eliminates this whole class of stale-connection EOF.
defaults: add timeout http-keep-alive 30s to bound how long an idle client-facing keep-alive connection is held. Previously unset (fell back to timeout client, 500s) - unnecessarily long given maxconn 80.
Verification
Could not run haproxy -c/nginx -t locally (no toolchain in the agent sandbox) - please confirm CI/garnix validates the build before merge, same caveat as #38. After merge/deploy, recommend watching journalctl -t nginx_access/-u haproxy for a day and confirming DAVx5 sync stops throwing the EOFException.
## Symptom
DAVx5 (CalDAV/CardDAV client on Android) intermittently fails syncing every collection type against next.thehellings.com with:
```
java.io.IOException: unexpected end of stream on https://next.thehellings.com/...
Caused by: java.io.EOFException: \n not found: limit=0
```
This is the classic signature of an HTTP client reusing a pooled keep-alive connection that the server already silently closed - it gets zero bytes back mid-read of the response headers.
## Root cause
haproxy's `next` backend proxies to nginx on `127.0.0.1:8080` (which fronts phpfpm-nextcloud). HAProxy defaults to end-to-end keep-alive (both client- and server-facing sides) unless a proxy explicitly overrides it - the existing frontend-level `option http-server-close`/`option http-keep-alive` pair only governs the client-facing side of haproxy, not the haproxy->nginx leg.
nginx's generated config has `keepalive_timeout 65;`. Any pooled haproxy->nginx connection idle past 65s gets closed by nginx without haproxy knowing. If a request then lands on that now-dead pooled connection, the client sees a bare socket EOF while reading response headers - exactly this error.
Confirmed via `journalctl -t nginx_access` (enabled in #37/#38) for the failing sync window, cross-referenced against the generated `/nix/store/*-nginx.conf` keepalive settings on linode.
## Fix
- `backend next`: add `option http-server-close` so haproxy opens a fresh connection to nginx per request instead of pooling/reusing one. Backend is localhost, so the extra handshake is negligible cost, and it eliminates this whole class of stale-connection EOF.
- `defaults`: add `timeout http-keep-alive 30s` to bound how long an idle client-facing keep-alive connection is held. Previously unset (fell back to `timeout client`, 500s) - unnecessarily long given `maxconn 80`.
## Verification
Could not run `haproxy -c`/`nginx -t` locally (no toolchain in the agent sandbox) - please confirm CI/garnix validates the build before merge, same caveat as #38. After merge/deploy, recommend watching `journalctl -t nginx_access`/`-u haproxy` for a day and confirming DAVx5 sync stops throwing the EOFException.
DAVx5 (CalDAV/CardDAV) on greg's phone was intermittently failing every
sync type (CONTACTS/EVENTS/TASKS/RefreshCollectionsWorker) against
next.thehellings.com with:
java.io.IOException: unexpected end of stream
Caused by: java.io.EOFException: \n not found: limit=0
This is the classic OkHttp/HTTP client signature of the far end
silently closing a pooled keep-alive connection: the client reuses a
socket it still believes is open, gets zero bytes back while reading
response headers, and throws exactly this exception.
Root cause: HAProxy's 'next' backend proxies to nginx on
127.0.0.1:8080, and HAProxy defaults to end-to-end keep-alive (both
client- and server-side) unless told otherwise. nginx's
keepalive_timeout is 65s, so any HAProxy<->nginx connection idle past
that gets closed by nginx without HAProxy's knowledge. A request that
lands on that now-dead pooled connection right after gets nothing back
- surfacing to the client as a bare socket EOF while reading headers.
The frontend's existing 'option http-server-close'/'http-keep-alive'
pair only governs the client-facing side of HAProxy and does nothing
for the HAProxy->nginx leg.
Fix:
- backend next: add 'option http-server-close' so HAProxy opens a
fresh connection to nginx per request instead of pooling/reusing
one. The backend is localhost, so the extra TCP handshake cost is
negligible, and this removes the whole class of stale-connection EOF
errors.
- defaults: add 'timeout http-keep-alive 30s' to bound how long an
idle client-facing keep-alive connection is held open. Previously
unset, it fell back to 'timeout client' (500s) - unnecessarily long
given maxconn is only 80, and tightens client-side connection churn
to be more predictable too.
Diagnosed by pulling the nginx_access journal (enabled in #37/#38) for
the failing sync window and cross-referencing nginx's
services.nginx.appendHttpConfig / generated nginx.conf keepalive
settings against HAProxy's request-level defaults. Could not run
'haproxy -c'/'nginx -t' locally (no toolchain in the agent sandbox) -
recommend confirming via CI/garnix before merge, same as #38.
greg
merged commit bc90eb34e4 into main2026-08-10 03:24:40 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
DAVx5 (CalDAV/CardDAV client on Android) intermittently fails syncing every collection type against next.thehellings.com with:
This is the classic signature of an HTTP client reusing a pooled keep-alive connection that the server already silently closed - it gets zero bytes back mid-read of the response headers.
Root cause
haproxy's
nextbackend proxies to nginx on127.0.0.1:8080(which fronts phpfpm-nextcloud). HAProxy defaults to end-to-end keep-alive (both client- and server-facing sides) unless a proxy explicitly overrides it - the existing frontend-leveloption http-server-close/option http-keep-alivepair only governs the client-facing side of haproxy, not the haproxy->nginx leg.nginx's generated config has
keepalive_timeout 65;. Any pooled haproxy->nginx connection idle past 65s gets closed by nginx without haproxy knowing. If a request then lands on that now-dead pooled connection, the client sees a bare socket EOF while reading response headers - exactly this error.Confirmed via
journalctl -t nginx_access(enabled in #37/#38) for the failing sync window, cross-referenced against the generated/nix/store/*-nginx.confkeepalive settings on linode.Fix
backend next: addoption http-server-closeso haproxy opens a fresh connection to nginx per request instead of pooling/reusing one. Backend is localhost, so the extra handshake is negligible cost, and it eliminates this whole class of stale-connection EOF.defaults: addtimeout http-keep-alive 30sto bound how long an idle client-facing keep-alive connection is held. Previously unset (fell back totimeout client, 500s) - unnecessarily long givenmaxconn 80.Verification
Could not run
haproxy -c/nginx -tlocally (no toolchain in the agent sandbox) - please confirm CI/garnix validates the build before merge, same caveat as #38. After merge/deploy, recommend watchingjournalctl -t nginx_access/-u haproxyfor a day and confirming DAVx5 sync stops throwing the EOFException.