Race condition when removing users

Describe the bug

related #1877

Steps to reproduce

run opencloud
run test several times USE_BEARER_TOKEN=true TEST_SERVER_URL=https://cloud.opencloud.test make test-acceptance-api BEHAT_FEATURE=tests/acceptance/features/apiSharingNgShareInvitation/shareInvitations.feature:1625

Expected behavior

no error

Actual behavior

error log:

{"level":"error","service":"storage-users","host.name":"dccd0e3abc1c","pkg":"rgrpc","traceid":"b78450a7eb199466805d9c4d430958cd","error":"node.Xattr /var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a user.oc.name: no data available","id":"1272256e-37f8-4a85-b2aa-3652d261cf94","time":"2025-11-19T13:09:28Z","line":"github.com/opencloud-eu/reva/v2@v2.39.3-0.20251113164418-9fd6b6864c10/pkg/storage/pkg/decomposedfs/spaces.go:317","message":"could not read node"}

{"level":"error","service":"storage-users","host.name":"dccd0e3abc1c","pkg":"rgrpc","traceid":"b78450a7eb199466805d9c4d430958cd","error":"node.Xattr /var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a user.oc.name: no data available","status":{"code":15,"message":"error listing spaces","trace":"b78450a7eb199466805d9c4d430958cd"},"filters":[{"type":2,"Term":{"Id":{"opaque_id":"77b1193c-84f6-43ae-b14d-a9bfa479de1d$1272256e-37f8-4a85-b2aa-3652d261cf94!1272256e-37f8-4a85-b2aa-3652d261cf94"}}},{"type":4,"Term":{"SpaceType":"+grant"}}],"time":"2025-11-19T13:09:28Z","line":"github.com/opencloud-eu/reva/v2@v2.39.3-0.20251113164418-9fd6b6864c10/internal/grpc/services/storageprovider/storageprovider.go:579","message":"failed to list storage spaces"}

Setup

Please describe how you started the server and provide a list of relevant environment variables or configuration files.

Details

OC_XXX=somevalue
OC_YYY=somevalue
PROXY_XXX=somevalue

Additional context

Add any other context about the problem here.

fix(decomposedfs): tolerate a space root with an unset name attribute

fix(decomposedfs): skip a space whose name attribute is unset

fix(decomposedfs): tolerate a space root with an unset name attribute

test(apiSpaces): a space admin can delete a space with no manager

test(apiSpaces): a space admin can delete a space with no manager (#3040

fix(decomposedfs): tolerate a space root with an unset name attribute

I am going to hide all the AI generated comments. They're completely bogus or at least not addressing the root cause of the issue.

{"level":"error","service":"storage-users","host.name":"dccd0e3abc1c","pkg":"rgrpc","traceid":"b78450a7eb199466805d9c4d430958cd","error":"node.Xattr /var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a user.oc.name: no data available","id":"1272256e-37f8-4a85-b2aa-3652d261cf94","time":"2025-11-19T13:09:28Z","line":"github.com/opencloud-eu/reva/v2@v2.39.3-0.20251113164418-9fd6b6864c10/pkg/storage/pkg/decomposedfs/spaces.go:317","message":"could not read node"}

Looking at the storage after such a message appeared there is some left over folder of a personal space (/var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a) containing a mlock file for the space root ( .oc-nodes/locks/<some-id>.mlock). The folder doesn't have any attributes set, so it's not only the missing name attribute. So adding code to tolerate space with a missing name attribute is just papering over a real problem. Such a space/folder shouldn't exist at all.

The issue is reproducible e.g bulk creating, logging in and deleting a couple of users. (It does not appear on every delete though. Out of 100 users I got this issue 5 times).

Some more root cause analysis:

Deleting a user causes the Personal Space to be deleted first
as space deletion is always a two step process the space is first disabled, then delete
disabling a space triggers an event to be sent, processing that event happen asynchronously and might happen after the final deletion of the space
processing that event, tries to read attributes the space root node
to do that it tries to acquires a lock on the node
if the node does not exist the hybrid metadata backend with happily create the basepath, metadata directory and lock file for the node <- this is the actual bug, locking a node (even the root node) on a non-exisiting space should fail

You're right: it's the async race.

Reproduced on 7.2.1 (posix, built-in IdM):

bulk-create users
log each in once (provisions the personal space)
delete them
a few per 80 leave a phantom storage/users/users/<id>/ with no xattrs, only a .oc-nodes/locks/<id>.mlock
grant-filtered ListStorageSpaces then logs no data available, could not read node, failed to list storage spaces (code 15)

Matches your ~5/100. The hybrid backend materialises a node when it locks an already-deleted space root; that is the layer to fix. #713 only tolerates the nameless node on read, not the cause; closing it. Fine to hide the earlier comments.

_{🤖 drafted with Claude Code, reviewed before submitting.}

I believe opencloud-eu/reva#722 is going to fix this.

Hm, with all the changes that happend in opencloud-eu/reva#722, it does not longer fix this issue :(

We need in an additional fix for the hybrid metadata backend.

fix(metadata/hybrid): Avoid resurrecting space root directory

Navigation Menu

Repository navigation

Race condition when removing users #1878

Describe the bug

Steps to reproduce

Expected behavior

Actual behavior

Setup

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Race condition when removing users #1878

Description

Describe the bug

Steps to reproduce

Expected behavior

Actual behavior

Setup

Additional context

Activity

michaelstingl commented on Jan 19, 2026

michaelstingl commented on Jun 27, 2026

michaelstingl commented on Jul 6, 2026

rhafer commented on Jul 7, 2026

rhafer commented on Jul 7, 2026

michaelstingl commented on Jul 7, 2026

rhafer commented on Jul 8, 2026

rhafer commented on Jul 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions