k8s spark operator compatibility

[blcksrx]

Changes in this PR:

• Added override_existing argument with default to the operator.
• Uses the K8S application uid instead of naming

closes: #32363

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[assaf-xm]

@blcksrx is there any progress with the review of this PR?
(this is a small PR)

[blcksrx]

@assaf-xm no review yet :(

[ruslanguns]

This PR is very important for us, we are start working with Airflow and Spark and this additional means a lot. I really encourage the reviewers to review this. Thanks in advanced:

FYI @jedcunningham @hussein-awala @josh-fell

[potiuk]

Can you rebase it please @blcksrx ? There are some conflicts.

[blcksrx]

@potiuk I see a lot of changes, It gonna check it during the weekend

[romsharon98]

I think it will be nice to add override_existing as templated field.
If it's not to much can u add it?

[blcksrx]

A major change happened in the operator actually, that makes it hard to do it!
The new change generates a new job name(at least what i understood from the code) and it needs collaboration of these changes commiters

[romsharon98]

A major change happened in the operator actually, that makes it hard to do it! The new change generates a new job name(at least what i understood from the code) and it needs collaboration of these changes commiters

isn't it just to add this line:

    template_fields: Sequence[str] = ("application_file", "namespace", "override_existing")

or I missing something?

[blcksrx]

This part is simple,the operator it self got completely changed, check it from the main branch

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.