Detect Spark driver completion by container state when tracking via k8s API

[karenbraganz]

related: #67934

This PR tracks Spark job completion by container state instead of pod phase when track_driver_via_k8s_api=True. Sometimes the pod continues to run even after the driver container completes due to other sidecar containers. This PR makes driver completion detection more accurate by examining the container itself.

[karenbraganz]

I still need to test this out and write unit tests.

[karenbraganz]

This has passed all unit tests as well as a manual test that I ran.

[aaron-y-chen]

Should we also document the changes made in this PR in providers/apache/spark/docs/operators.rst?

[aaron-y-chen]

LGTM 😃

[uranusjr]

The original issue described a second problem

index 0 is not guaranteed to be the driver container in a multi-container pod.

This is reachable whenever driver_container is None and I think the PR does not address this? It’s fine to be fixed separately, but in that case the PR can’t close the issue outright.

[karenbraganz]

The original issue described a second problem

index 0 is not guaranteed to be the driver container in a multi-container pod.

This is reachable whenever driver_container is None and I think the PR does not address this? It’s fine to be fixed separately, but in that case the PR can’t close the issue outright.

I had initially decided to leave that part of the code as is because it only acts as a fallback when the driver container cannot be identified. However, now I am thinking it might be more accurate to just report the pod failure without the exact container status even in case of the fallback. Something like this:

if phase == "Succeeded":
    terminal_phase = phase
    break
if phase == "Failed" and not container_completed:
    raise RuntimeError(f"Spark application {app_id} failed (phase=Failed)")

Or we could print the statuses of all containers in the pod without assuming any specific one is the driver. @uranusjr what do you think?

I can create a separate PR for this. I have edited this PR description so that it dos not say that the issue is closed.