[Task]: Beam tft extra depends on old version of tensorflow-transform that requires protobuf v4.

[tvalentyn]

What needs to happen?

We recently started to allow newer versions of GRPC. #37817.

I wanted to regenerate containers to start using newer versions of GRPC and clean up some tech debt we had in our code because grpc version was restricted, notably envoy-data-plane was set to use some really old version.

I ran into dependency hell in some of our test suites. The crux of it seems to be:

• our ml_test dependency requires tensorflow-transform
• tensorflow-transform doesn't support protobuf 6
• beam tft extra requires an even older tensorflow-transform that doesn't support protobuf 5. Our ml_tests suites (e.g. precommit XLang YAML) end up using protobuf 3 due to various dependency constraints.
• grpcio-status==1.78.0 requires protobuf 6: protobuf 5 will reach EOL at the end of March'26 .
• If we build Beam GRPC stubs with protobuf 5+, these stubs no longer work with protobuf 3: 531176c#r2932955415, in other words install pip install apache-beam[tft] will not work.

We need to upgrade to newer versions of TFT, which is currently blocked by tensorflow/transform#347 .

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[potiuk]

Just a comment here from Apache Airflow - we are trying to unsuspend Apache Beam Provider ( see apache/airflow#66952 ) and this one is the last blocker - so maybe there is a chance that next version of apache-beam will upgrade the envoy-data-plane to >=2.1.0 - that one uses a replacement for betterproto (betterproto2) that was the reason for the limits explained in the issue here.