Skip to content

make datagateway optional in events - #778

Merged
butonic merged 1 commit into
mainfrom
make-tokens-in-events-optional
Aug 19, 2026
Merged

butonic merged 1 commit into
mainfrom
make-tokens-in-events-optional

Conversation

@butonic

@butonic butonic commented Aug 19, 2026 •

Copy link
Copy Markdown
Member

Part of opencloud-eu/opencloud#3289

Our current code actually only works with a datagateway. When a BytesReceived event is published the storage driver always wraps the DataProvider URL in a JWT and prepends the datagateway URL. This PR makes the datagateway optional.

Once this is merged I will rebase the opencloud PR and remove any config options for this.

We could delete the datagateway and token related code from the storage drivers later if we can assume that only internal services will consume events. Or replace the config option with an events "middleware"? nah ... just get rid of it.

🤔 wait ... In kubernetes the antivirus service lives in another pod. references to localhost won't work. Why do we even go through another HTTP download to read the bytes? Well because not all upload implementations might store the upload on disk. Ok, so reading via HTTP makes sense.

I still don't like that the storagedriver has to know its url, only so it can create a download url when emitting events. The dataprovider currently handles the datagateway wrapping if it is configured to not expose the data provider. But it currently cannot do that.

If we really need to wrap urls in events in the future we should use a custom events.Stream implementation to wrap the underlying stream. The storageproider can then inject this wrapper into the storage drivers:

	Stream interface {
		Publish(string, interface{}, ...events.PublishOption) error
		Consume(string, ...events.ConsumeOption) (<-chan events.Event, error)
	}

We could even always wrap the stream and let the storageprovider write the URL. The storage driver only sets the upload ID. Then it would not need to know the host and port of the storage provider. 🤩

@rhafer rhafer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but I can't really make sense of this PR. The commit message talks about events, but the codes only touches the URL() method. I suspect the result of URL() is used for sending some event. But it's not really clear. Some more context in the commit message would help.

The PR description/braindump, doesn't really help either.

@butonic butonic self-assigned this Aug 19, 2026
Our current code actually only works with a datagateway. When a BytesReceived event is published the storage driver always wraps the DataProvider URL in a JWT and prepends the datagateway URL. This PR makes that datagateway optional, which allows actually exposing the dataprovider to other services directly.

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic butonic changed the title make tokens in events optional make datagateway optional in events Aug 19, 2026
@butonic
butonic force-pushed the make-tokens-in-events-optional branch from 4845d1c to 164274c Compare August 19, 2026 11:54
@butonic
butonic requested a review from rhafer August 19, 2026 11:55
@butonic
butonic merged commit 76d14c1 into main Aug 19, 2026
18 of 19 checks passed
@butonic
butonic deleted the make-tokens-in-events-optional branch August 19, 2026 12:32
@openclouders openclouders mentioned this pull request Aug 19, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants