kafka connect: wait through connect-gate cold start instead of failing

[simonlord]

What and why

Redpanda Cloud is adding connect-gate (redpanda-data/cloudv2#30703), a proxy in front of Kafka Connect that lets the workers scale to zero when idle. While the workers boot, the gate answers mutating calls with 503, a Retry-After header and a body like:

{"error_code":503,"reason":"kafka_connect_starting","phase":"pulling-image","retry_after_seconds":60,"estimated_wait_seconds":180,"message":"Kafka Connect is starting (pulling the Kafka Connect image). Retry in about 3m0s."}

Today Console turns that into a terminal red error (and the validate endpoint even returns HTTP 200 for it), so a user creating the first connector after a quiet period sees "Failed to validate Connector config: … (503)" with no hint that waiting a few minutes fixes it.

Backend

• A transport wrapper on the connect-client captures the gate's body and Retry-After per request (the client itself keeps only error_code and message), carried as connect.StartingError.
• REST handlers for create / validate / update / delete / pause / resume / restart / restart-task answer 503 with Retry-After and reason, phase, retryAfterSeconds, estimatedWaitSeconds in the body. Other errors are unchanged.
• Connect-RPC KafkaConnectService (v1, v1alpha2) maps 503 to CodeUnavailable with RetryInfo and ErrorInfo metadata instead of CodeInternal.
• Validate now propagates the upstream status like the other calls instead of hardcoding 200.

Frontend

• retryWhileKafkaConnectStarting retries a call while it fails with reason: kafka_connect_starting, honouring the retry hint (15 min budget), with a per-second countdown callback.
• Create wizard: validate and create wait through the cold start with a "Kafka Connect is starting… retrying in Ns" warning (Review step and the creating dialog). The Properties step's initial validation does the same; validate-on-change skips quietly while cold.
• Connector action confirm dialog (pause/resume/restart/delete/update): same waiting state, retry on its own.
• Fixes a pre-existing bug: a validate failure in the Review step threw past the wizard and left it on the loading skeleton; it now renders "Validation attempt failed".

Examples

REST, while the workers boot:

PUT /api/kafka-connect/clusters/redpanda/connector-plugins/org.apache.kafka.connect.mirror.MirrorSourceConnector/config/validate
HTTP/1.1 503 Service Unavailable
Retry-After: 60
{"statusCode":503,"message":"Kafka Connect is starting (pulling the Kafka Connect image). Retry in about 3m0s.","reason":"kafka_connect_starting","phase":"pulling-image","retryAfterSeconds":60,"estimatedWaitSeconds":180}

Connect-RPC: unavailable with RetryInfo{retry_delay: 60s} and ErrorInfo{reason: REASON_KAFKA_CONNECT_API_ERROR, metadata: {reason: kafka_connect_starting, phase: pulling-image, retry_after_seconds: 60}}.

UI recording: to be added from an integration cluster running connect-gate (cloudv2 integration pack 26.2.20261006181804 bundles this branch as console-unstable:sl-kafka-connect-starting-503-8480a75).

Tests: go test ./pkg/connect/... (transport capture, ordinary 503 untouched, header/default fallbacks), src/utils/kafka-connect-starting.test.ts.

🤖 Generated with Claude Code

[github-actions]

🚨 Registry drift detected

App: frontend · Scope: diff vs origin/master · Files: 8

	Count
⚠️ Outdated registry components	2
🛠 Locally-modified components	0
❓ Unknown to registry	0
🎨 Off-token palette colours	0
🔢 Ad-hoc utility classes	0

Components needing attention

Status	Component	Uses	Detail
⚠️ outdated	code-block	1×	installed 3.5.0 → latest 3.6.0
⚠️ outdated	stat	1×	installed 3.4.1 → latest 3.6.0

Refresh command:

bunx shadcn@latest add @redpanda/code-block @redpanda/stat --overwrite

Generated by lookout audit-changes.