Large TUS uploads return 502 although the upload succeeded: tusd write deadline (120 s) expires during FinishUpload #3679

Describe the bug

Uploading a very large file (here 21.3 GB, part of a folder upload in the web client) ends in a 502 on the final TUS PATCH, and the web client reports the upload as failed. On the server the upload actually completed: storage-users logs UploadFinished and a 204, and postprocessing starts. Only the response to the client is lost.

The final PATCH stays open while FinishUpload runs. That took about 4 minutes here, longer than the write deadline tusd sets on the connection, so the 204 is written to a connection that has already been closed. The proxy gets EOF from storage-users and returns 502.

Cause

1. tusd v2 sets a write deadline of 2 × NetworkTimeout on every request, renewed after each successful body read (unrouted_handler.go L183 and L920). After the last byte of the final chunk is read, the deadline is not extended again.
2. reva does not set NetworkTimeout in its tusd.Config (pkg/rhttp/datatx/manager/tus/tus.go, still unset on main), so tusd's default of 60 s applies. The response must be written within 120 s of the last byte received.
3. On the final chunk, DecomposedFsSession.FinishUploadDecomposed (pkg/storage/pkg/decomposedfs/upload/upload.go) calls node.CalculateChecksums on the whole staged file before the response is sent. That function reads the file once more and computes sha1, md5 and adler32 one after another on a single goroutine (io.TeeReader chain in pkg/storage/pkg/decomposedfs/node/node.go). Its cost grows with file size and is limited by single-core hashing speed. Nothing bounds it.

Once finishing takes more than 120 s, the client always gets a 502.

Measurements on the affected server (21,290,612,585-byte file):

• Finishing took 237 s (last chunk written → UploadFinished), about 90 MB/s.
• CPU cost of hashing on this host (coreutils, 4 GiB from /dev/zero, scaled to the file size): sha1 ≈ 81 s, md5 ≈ 56 s, so ≈ 137 s before adler32. reva computes these one after another on one goroutine, so hashing alone is longer than the 120 s budget, whatever the storage.

These figures come from coreutils, not from profiling reva, so Go's hash speed may differ somewhat. Hashing is the only full pass over the file in the finish path, though.

Steps to reproduce

1. Use a server where hashing the file (sha1 + md5 + adler32 serially) takes more than about 120 s. On a small VPS core that is ~20 GB. Slow storage makes it happen at smaller sizes.
2. Upload one file of ~20 GB through the web client.
3. The last chunk's PATCH returns 502 after about 2+ minutes, and the client shows an error.
4. Reload: the file is there, at full size.

Expected behavior

The final PATCH returns 204 (or the client is otherwise told the upload succeeded), however long finishing takes.

Client and server disagree about the result

After the 502 the web client reports the upload as failed, but the server has committed the file, and it appears in the listing. The user is told an upload failed when it didn't. They may upload 21 GB again for nothing, or, on an overwrite, wrongly conclude the old version is still current.

Rolling back on the server would be the wrong fix: the server can't tell whether the client received the response. The client should check before reporting failure. TUS supports this: after a failed PATCH, a HEAD on the upload URL returns Upload-Offset, and Upload-Offset == Upload-Length means the upload is complete. Failing that, a PROPFIND on the target can show whether the file was committed. The web client does not check. In the second reproduction, the server logs show no further request for the upload (no HEAD, no retried PATCH) between the 502 at 05:11:24 UTC and 05:30. The client goes straight to "Unknown error". I haven't tested what a HEAD would return after a successful finish. If the upload resource is already gone (404), the client would need a WebDAV fallback to find out the upload succeeded.

Logs (trimmed; request ID 535e2dd2-…, upload ID 22d14ba6-…)

21:17:10.125 storage-users  PATCH ChunkWriteStart    offset=21290000000 maxSize=612585
21:17:10.345 storage-users  PATCH ChunkWriteComplete bytesWritten=612585
             ... ~237 s with no output for this request ...
21:21:07.321 postprocessing processing init          UploadID=22d14ba6-…
21:21:07.363 storage-users  PATCH UploadFinished     size=21290612585
21:21:07.364 storage-users  PATCH ResponseOutgoing   status=204
21:21:07.365 proxy          error happened in MultiHostReverseProxy  error="EOF"
21:21:07.366 proxy          access-log PATCH /data/tus/22d14ba6-… status=502 duration=237244ms

The PATCH chunks before it (10 MB each) all returned 204 in about 3 s.

Second reproduction with a generated 25 GiB file (large.bin, 26,843,545,600 bytes), from the browser HAR: the last 10 MB chunk returned 204 in 3.4 s. The final PATCH (Upload-Offset: 26840000000, Content-Length: 3545600) was sent in under 1 s, then waited 413.6 s and got a 502 with an empty body. The web client showed "Upload failed … Unknown error". After a reload the file is listed at its full size (26.8 GB). The sha1 that reva stored for it (user.oc.cs.sha1 = f6ab90cd71505b3f6ca3157f4d8e020b85b8a12a) matches the sha1 of the source file on the client, so the upload is complete and intact. Only the response was lost. The reverse proxy in front of OpenCloud (nginx, proxy_read_timeout 3600s, client_max_body_size 0, proxy_request_buffering off) logged no error for this request.

Possible fixes

• Short term: set NetworkTimeout in reva's tusd.Config, and preferably make it configurable (e.g. a STORAGE_USERS_DATA_SERVER_… env var). This raises the limit but doesn't remove it, and it also changes how long dead connections are kept.
• In tusd: extend the write deadline before calling FinishUpload (or let the store do it), so slow finalization can't use up the time left for the response.
• Remove the second read: compute the checksums while chunks are written (the existing TODO in upload.go points at this). Then finishing no longer depends on file size. Resumed uploads would need the hash state to be saved between chunks.
• Cheaper hashing: compute sha1, md5 and adler32 in parallel (one goroutine per hash, each fed the same buffers) instead of chaining them serially. This would roughly halve the finish time on multi-core hosts, but the time would still grow with file size.

Setup

• OpenCloud: opencloudeu/opencloud-rolling:8.1.0 (reva v2.51.0, tusd v2.10.0). reva main still leaves NetworkTimeout unset.
• Storage: posix driver; uploads staged on the same filesystem (btrfs on a loop-mounted image file on SMB/CIFS network storage)
• Reverse proxy: nginx → OpenCloud proxy on :9200
• Client: OpenCloud web (folder and single-file upload), tested in Firefox 153 on Windows. The failure is server-side timing, so the browser shouldn't matter.