Skip to content

TUS upload finalize fails for large (30+ GB) files on posix storage driver #2864

Closed
opencloud-eu/reva
#675
Closed
TUS upload finalize fails for large (30+ GB) files on posix storage driver#2864
opencloud-eu/reva
#675
Assignees
aduffeck
Labels
Priority:p2-highHigh priorityType:Bug

Description

@Tellervo89
Tellervo89
opened on May 22, 2026

Describe the bug

TUS uploads of very large files (≥ ~30 GB) fail at the finalize step on the posix storage driver. All bytes transfer successfully (the final TUS PATCH reaches Offset == Size), but the post-upload metadata commit fails with a context-cancel chain originating in the posix idcache jetstream KV operation, ending in failed to init new node with an empty path. The HTTP PATCH then returns 500 to the client.

Affected clients (all symptoms identical from the server's perspective):

  • Web UI (drag-and-drop upload) — shows file as "Finalizing… few seconds left" then fails with "1 of 1 item failed".
  • Desktop client (Linux) — fails the same file repeatedly.
  • rclone with vendor = infinitescale — fails identically.

A direct WebDAV PUT of the same file (bypassing TUS entirely) succeeds reliably:

curl -u USER:APPPASS -T bigfile.mp4 \
  "https://server/remote.php/dav/spaces/<space-id>/path/bigfile.mp4"
# → 201 Created, file appears immediately and is fully usable

So the issue is specific to the TUS upload finalize code path, not the byte transfer, the proxy, the disk, or the posix driver's general ability to store large files.

Steps to reproduce

  1. Deploy OpenCloud 7.0.0 with STORAGE_USERS_DRIVER=posix and STORAGE_SYSTEM_DRIVER=decomposed (standard docker-compose, antivirus disabled).
  2. Authenticate as a normal user.
  3. Upload a single ~30+ GB file via the web UI or desktop client into the personal space.
  4. Wait for the upload to reach 100% and enter "Finalizing".

Reproduces 100% of the time across multiple attempts with different files (verified with two distinct MP4 files of 32.1 GB and 57.5 GB).

Expected behavior

File is committed to the target location and visible in the user's space, exactly as occurs for smaller files and as occurs for a direct WebDAV PUT of the same file.

Actual behavior

  • HTTP PATCH (the final TUS chunk) returns 500 after ~166 seconds.
  • Response body: ERR_INTERNAL_SERVER_ERROR: open : no such file or directory.
  • Server-side upload session is left in Processing: true with Offset == Size, indefinitely (does not expire to the "expired" list, does not finalize, does not produce a visible file).
  • Subsequent retries create additional stuck sessions for the same file.

Setup

  • OpenCloud image: opencloudeu/opencloud-rolling:7.0.0
  • Deployment: docker-compose, single container, listening on 127.0.0.1:9200 on the host.
  • Reverse proxy: Nginx Proxy Manager in front, with client_max_body_size 0, proxy_buffering off, proxy_request_buffering off, proxy_read_timeout 12h, proxy_send_timeout 12h.
  • Storage driver: STORAGE_USERS_DRIVER=posix, STORAGE_SYSTEM_DRIVER=decomposed.
  • Antivirus: disabled
  • Host disk: Over 400 GB free at time of failure.
  • Host RAM: not at capacity during the failure.
  • TUS staging directory (/var/lib/opencloud/storage/users/uploads) is on the same filesystem as the user data root.

Relevant .env values:

STORAGE_USERS_DRIVER=posix
STORAGE_SYSTEM_DRIVER=decomposed
STORAGE_USERS_POSIX_ROOT=/var/lib/opencloud/storage
STORAGE_USERS_ID_CACHE_STORE=nats-js-kv

Server logs

Failure chain at the moment the final TUS PATCH lands:

{"level":"error","service":"storage-users","driver":"posix","error":"context canceled",
 "line":".../pkg/storage/fs/posix/idcache/idcache.go:215",
 "message":"error in jetstream kv operation, retrying"}

(several identical "context canceled" retries over ~5 seconds)

{"level":"error","service":"storage-users","driver":"posix","error":"context canceled",
 "line":".../pkg/storage/pkg/decomposedfs/upload/store.go:244",
 "message":"failed to cache id"}

{"level":"error","service":"storage-users","driver":"posix","path":"",
 "error":"open : no such file or directory",
 "line":".../pkg/storage/pkg/decomposedfs/upload/store.go:250",
 "message":"failed to init new node"}

{"level":"error","service":"storage-users","datatx":"tus","method":"PATCH",
 "path":"/14fae9ba-6b4a-4556-9b4c-84e156c2e471",
 "message":"open : no such file or directory",
 "line":".../pkg/rhttp/datatx/manager/tus/tus.go:282",
 "message":"InternalServerError"}

{"level":"info","service":"storage-users","datatx":"tus","method":"PATCH",
 "status":500,"body":"ERR_INTERNAL_SERVER_ERROR: open : no such file or directory\n"}

{"level":"error","service":"frontend",
 "error":"Patch \"http://localhost:9158/data/tus/14fae9ba-6b4a-4556-9b4c-84e156c2e471\": EOF",
 "message":"error doing PATCH request to data service"}

Total PATCH duration before the 500: 166.46 seconds (time_ns: 166460835118). The context canceled error in the idcache jetstream KV op precedes the path:"" returned to failed to init new node, so the empty path is a consequence of the cancellation, not the cause.

Upload session state after failure

After several failed attempts at the same files, opencloud storage-users uploads sessions --processing lists multiple stuck sessions per file, all with Offset == Size and Processing: true:

filename-example1.mp4 │ 34470813213 / 34470813213 │ Processing: true   (×6 stuck sessions)
filename-example2.mp4          │ 61697531888 / 61697531888 │ Processing: true   (×3 stuck sessions)

None ever transition out of processing; none ever expire to the expired list; --restart does not recover them; only --clean removes them.

Workaround

Direct WebDAV PUT (no TUS) finalises identically-sized files cleanly:

curl -u USER:APPPASS -T "/path/filename" \
  "https://server/remote.php/dav/spaces/<space>/path/filename"
# HTTP 201, file appears normally, posix storage driver commits metadata without issue

Activity

  1. edernunez61 commented on May 29, 2026

    @edernunez61
    edernunez61
    on May 29, 2026

    Open this in the main opencloud repo, they don't look at this repo very frequently. Though lately it seems like they don't look at any repo very frequently, there are so many data loss and corruption issues and failures like this. Just complete negligence of testing and quality control. Such a shame to see this project go so downhill.

    @tbsbdr @micbar

  2. micbar commented on May 29, 2026

    @micbar
    micbar
    on May 29, 2026
    Member

    @edernunez61

    I am respecting your opinion and also I must disagree. I am checking issues every day. Where do you see data loss? I have no ticket in mind where this happens.

    Please be assured that data loss or file corruption is always the highest priority.

    We are a small group of maintainers, we are also happy that so many people are using openCloud. Please feel free to look into the code or provide more detailed information.

  3. micbar
    transferred this issue fromopencloud-eu/opencloud-composeon May 29, 2026
  4. micbar
    added
    Type:Bug
    on May 29, 2026
  5. github-project-automation
    added this to OpenCloud Team Boardon May 29, 2026
  6. github-project-automation
    moved this to Qualification in OpenCloud Team Boardon May 29, 2026

  7. v-scharf commented on Jun 11, 2026

    @v-scharf
    v-scharf
    on Jun 11, 2026
    Contributor

    I can confirm it
    tested agains rc instance https://cloud.rc.opencloud.rocks/ 7.0.0-rc.0 rolling
    https://github.com/opencloud-eu/opencloud-compose with COMPOSE_FILE=docker-compose.yml:weboffice/collabora.yml:traefik/opencloud.yml:traefik/collabora.yml:idm/ldap-keycloak.yml:traefik/ldap-keycloak.yml:search/tika.yml:custom

    Result:

    • 500 error
    • uploads hangs -> you can find it here
    758f2ac5ffa8:/var/lib/opencloud/storage/users/uploads$ ls
1db34e68-cb21-4394-9aed-f65be34c2732       1db34e68-cb21-4394-9aed-f65be34c2732.info
758f2ac5ffa8:/var/lib/opencloud/storage/users/uploads$ cat 1db34e68-cb21-4394-9aed-f65be34c2732.info 
{"ID":"1db34e68-cb21-4394-9aed-f65be34c2732","Size":32212254720,"SizeIsDeferred":false,"Offset":0,"MetaData":{"dir":"/","expires":"1781188738","filename":"test.mp4","initiatorid":"ab7def6a-e561-4bf4-94b9-ed82578bcdf8","lockid":"","mtime":"1781101838.044","providerID":"0cb8dfe2-3e6d-48e6-bc4c-5cfa858aa6c5"},"IsPartial":false,"IsFinal":false,"PartialUploads":null,"Storage":{"Dir":"/","Idp":"https://keycloak.rc.opencloud.rocks/realms/openCloud","LogLevel":"info","NodeId":"26ce7309-c3fe-426e-93c6-1b9e287de4f1","NodeName":"test.mp4","NodeParentId":"27008e3d-40f5-4f6d-8abf-e80fe46b3a94","SpaceOwnerOrManager":"f40487cc-e893-1040-8b3b-0510e9a42b58","SpaceRoot":"27008e3d-40f5-4f6d-8abf-e80fe46b3a94","Type":"DecomposedFsStore","UserDisplayName":"dennis Ritchie","UserId":"f40487cc-e893-1040-8b3b-0510e9a42b58","UserName":"dennis","UserOpaque":"null","UserType":"primary"}}758f2ac5ffa8:/var/lib/opencloud/storage/users/uploads$ cat 1db34e68-cb21-4394-9aed-f65be34c2732
    • if you start upload same file again -> uploading starts from scratch and in the end -> same result
    Image

    logs:

    docker compose logs opencloud | grep 074bd218-2da8-4485-9c22-6d5423f84c3f
WARN[0000] The "DEFAULT_LANGUAGE" variable is not set. Defaulting to a blank string. 
opencloud-1  | {"level":"info","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","datatx":"tus","method":"PATCH","path":"/1db34e68-cb21-4394-9aed-f65be34c2732","requestId":"074bd218-2da8-4485-9c22-6d5423f84c3f","line":"github.com/tus/tusd/v2@v2.9.2/pkg/handler/unrouted_handler.go:194","time":"2026-06-11T08:45:06Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/rhttp/datatx/manager/tus/tus.go:282","message":"RequestIncoming"}
opencloud-1  | {"level":"info","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","datatx":"tus","method":"PATCH","path":"/1db34e68-cb21-4394-9aed-f65be34c2732","requestId":"074bd218-2da8-4485-9c22-6d5423f84c3f","id":"1db34e68-cb21-4394-9aed-f65be34c2732","line":"github.com/tus/tusd/v2@v2.9.2/pkg/handler/unrouted_handler.go:900","maxSize":8451595,"offset":32203803125,"time":"2026-06-11T08:45:06Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/rhttp/datatx/manager/tus/tus.go:282","message":"ChunkWriteStart"}
opencloud-1  | {"level":"info","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","datatx":"tus","method":"PATCH","path":"/1db34e68-cb21-4394-9aed-f65be34c2732","requestId":"074bd218-2da8-4485-9c22-6d5423f84c3f","id":"1db34e68-cb21-4394-9aed-f65be34c2732","line":"github.com/tus/tusd/v2@v2.9.2/pkg/handler/unrouted_handler.go:961","bytesWritten":8451595,"time":"2026-06-11T08:45:10Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/rhttp/datatx/manager/tus/tus.go:282","message":"ChunkWriteComplete"}
opencloud-1  | {"level":"error","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","driver":"posix","pid":1,"request-id":"074bd218-2da8-4485-9c22-6d5423f84c3f","error":"context canceled","time":"2026-06-11T08:46:37Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/storage/pkg/decomposedfs/upload/store.go:244","message":"failed to cache id"}
opencloud-1  | {"level":"error","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","driver":"posix","pid":1,"request-id":"074bd218-2da8-4485-9c22-6d5423f84c3f","path":"","error":"open : no such file or directory","time":"2026-06-11T08:46:37Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/storage/pkg/decomposedfs/upload/store.go:250","message":"failed to init new node"}
opencloud-1  | {"level":"error","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","datatx":"tus","method":"PATCH","path":"/1db34e68-cb21-4394-9aed-f65be34c2732","requestId":"074bd218-2da8-4485-9c22-6d5423f84c3f","id":"1db34e68-cb21-4394-9aed-f65be34c2732","message":"open : no such file or directory","time":"2026-06-11T08:46:37Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/rhttp/datatx/manager/tus/tus.go:282","message":"InternalServerError"}
opencloud-1  | {"level":"info","service":"storage-users","host.name":"758f2ac5ffa8","pkg":"rhttp","datatx":"tus","method":"PATCH","path":"/1db34e68-cb21-4394-9aed-f65be34c2732","requestId":"074bd218-2da8-4485-9c22-6d5423f84c3f","id":"1db34e68-cb21-4394-9aed-f65be34c2732","line":"github.com/tus/tusd/v2@v2.9.2/pkg/handler/unrouted_handler.go:1296","status":500,"body":"ERR_INTERNAL_SERVER_ERROR: open : no such file or directory\n","time":"2026-06-11T08:46:37Z","line":"github.com/opencloud-eu/reva/v2@v2.45.0/pkg/rhttp/datatx/manager/tus/tus.go:282","message":"ResponseOutgoing"}
opencloud-1  | {"level":"info","service":"proxy","proto":"HTTP/1.1","request-id":"074bd218-2da8-4485-9c22-6d5423f84c3f","traceid":"0f23109c918cabb9a2562023029f41bc","remote-addr":"95.117.230.255","method":"PATCH","status":500,"path":"/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOlsicmV2YSJdLCJleHAiOjE3ODExODg3MzgsImlhdCI6MTc4MTEwMjMzOCwidGFyZ2V0IjoiaHR0cDovL2xvY2FsaG9zdDo5MTU4L2RhdGEvdHVzLzFkYjM0ZTY4LWNiMjEtNDM5NC05YWVkLWY2NWJlMzRjMjczMiJ9.pthb4opZkFPEDnRpSQpInX6xsJNAh7ZHFVtYBsjPNhQ","duration":90602.630525,"bytes":0,"time":"2026-06-11T08:46:37Z","line":"github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34","message":"access-log"}
  8. v-scharf
    moved this from Qualification to Prio 2 in OpenCloud Team Boardon Jun 11, 2026
  9. v-scharf
    added
    Priority:p2-highHigh priority
    on Jun 11, 2026
  10. aduffeck
    self-assigned this
    on Jun 11, 2026
  11. rhafer
    moved this from Prio 2 to In Progress in OpenCloud Team Boardon Jun 11, 2026
  12. aduffeck
    closed this as completedin reva#675on Jun 11, 2026
  13. github-project-automation
    moved this from In Progress to Done in OpenCloud Team Boardon Jun 11, 2026

  14. tbsbdr commented on Jun 12, 2026

    @tbsbdr
    tbsbdr
    on Jun 12, 2026
    Member

    @edernunez61

    just complete negligence of testing and quality control.

    I disagree. also not a very helpful comment

  15. micbar commented on Jun 12, 2026

    @micbar
    micbar
    on Jun 12, 2026
    Member

    @aduffeck nice fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Priority:p2-highHigh priorityType:Bug

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions