fix xcom push not working in SparkKubernetesOperator

[kesem0811]

Fixes: #39184 where SparkKubernetesOperator tasks hang indefinitely because no sidecar is injected to read /airflow/xcom/return.json.

[potiuk]

Fixing #39184 is worthwhile — tasks hanging indefinitely because no sidecar is injected to read /airflow/xcom/return.json is a bad failure mode, since there is nothing in the logs pointing at the cause.

Two practical notes before I review the substance.

This branch is 1047 commits behind main and currently conflicting, so it needs a rebase.

More importantly, it overlaps with #68786 from you: both edit xcom_sidecar.py and tests/.../test_xcom_sidecar.py. That PR turns out to contain a genuine caller-mutation fix — add_xcom_sidecar was rebinding pod_cp.spec.volumes to the original pod's list, so the insert mutated the caller's pod despite the deepcopy — and it is a one-line change with tests around it.

My suggestion is to land #68786 first and rebase this one on top. It is the smaller and independently correct change, and this PR's PodDefaults restructuring (VOLUME_MOUNT_NAME, XCOM_SIDECAR_COMMAND, the ApiClient import) is much easier to rebase onto that one line than the reverse. Doing it the other way round means resolving the same test file twice.

Once the ordering is settled and this is rebased, I will review the SparkKubernetesOperator side properly.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[potiuk]

Thanks for adding XCom push to SparkKubernetesOperator; it's a useful feature. One blocking issue inline, plus a few smaller ones:

• Rebase needed. The add_xcom_sidecar change (pod_cp.spec.volumes = pod_cp.spec.volumes or []) is already on main via #72522, which also deep-copies the default volume and mount. Please drop that hunk when rebasing rather than taking this side, or the shared-default-object bug comes back.
• Volume serialization: PodDefaults.VOLUME.to_dict() produces Python-style keys (empty_dir) plus about 30 null fields, not the API form (emptyDir). The CRD drops the unknown key and the volume ends up with no source; it only works because Kubernetes defaults such a volume to an empty directory. ApiClient().sanitize_for_serialization(...), which the PR already uses for the mount, gives the right form.
• Sidecar resources: without resources configured on the connection, the Spark sidecar has no requests, while the regular pod operator's sidecar defaults to cpu 1m / memory 10Mi. In a namespace with a ResourceQuota on requests and no defaults, the driver pod is rejected.
• Security context: #69613 (merged after this PR was written) added the xcom_sidecar_container_security_context connection setting. The Spark path only passes image and resources, so after rebasing it ignores the security context, and the sidecar is rejected in namespaces enforcing the "restricted" pod security level.

---

Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

[potiuk]

This doesn't survive templates that use the documented top-level kubernetes section, as the spark_job_template in the docs does. CustomObjectLauncher.get_body() then sets spec["volumes"] and the driver's volumeMounts from that section (custom_object_launcher.py:244–255). That replaces the XCom volume and driver mount injected here, but keeps the sidecar, whose mount then points at a volume that no longer exists, so Kubernetes rejects the driver pod. The tests only use templates without a kubernetes section. Could you inject after get_body() has merged the kubernetes section, or merge into its lists, and add a test with a kubernetes section?