[Bug]: gcsEndpoint ignored by GcsUtilV1 GoogleCloudStorageReadChannel after Beam 2.75.0 upgrade

[kberezin-nshl]

What happened?

Beam 2.72.0 -> 2.75.0 regression.

--gcsEndpoint pipeline option is ignored by GcsUtilV1.

Full PipelineOptions args in my test:

--project=prj-unit-test
--gcsEndpoint=http://<fake-gcs-server>/storage/v1/
--credentialFactoryClass=org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory

Note that no experiments are passed.

Expected:
Beam reads/writes gs:// paths through the configured emulator endpoint.

Actual:
GcsUtilV1 read path calls real GCS:
GET https://storage.googleapis.com/storage/v1/b/result/o/...

Stack:

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://storage.googleapis.com/storage/v1/b/result/o/a%2Fb%2Fdocs%2Fd%2Farchive.zip?fields=contentEncoding,generation,size
{
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "location": "Authorization",
      "locationType": "header",
      "message": "The billing account for the owning project is disabled in state absent",
      "reason": "accountDisabled"
    }
  ],
  "message": "The billing account for the owning project is disabled in state absent"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:479)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:565)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
    at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.fetchInitialMetadata(GoogleCloudStorageReadChannel.java:226)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.<init>(GoogleCloudStorageReadChannel.java:176)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$3.<init>(GoogleCloudStorageImpl.java:696)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:695)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.open(GoogleCloudStorageImpl.java:664)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1.open(GcsUtilV1.java:606)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtilV1.open(GcsUtilV1.java:575)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.open(GcsUtil.java:194)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.open(GcsFileSystem.java:160)
<...>

Suspected cause:
After the gcs-connector 3.x compatibility changes, Beam constructs GoogleCloudStorageImpl through GoogleCloudStorageOptions / builder path, but the root URL/service path derived from GcsOptions#getGcsEndpoint() is not propagated into GoogleCloudStorageOptions (here).

Notice what is missing: no setStorageRootUrl(...), no setStorageServicePath(...), and no equivalent propagation from gcsOptions.getGcsEndpoint()

As a result, the Hadoop GCSIO read channel defaults to Storage.DEFAULT_ROOT_URL.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shunping]

Thanks for reporting this. Let me see if I can reproduce that.