Repository navigation
BUG: spark application can have multiple drivers at once #646
Description
Affected Stackable version
No response
Affected Apache Spark-on-Kubernetes version
No response
Current and expected behavior
Disclaimer, we're still in 25.3.0 with spark but currently upgrading to 25.11.0 - if you fixed that in between, just let me know but i didn't find a related issue fitting my problem
We found the cause for the situation of two or more same spark applications running at the same time. You can reproduce that by just deleting the spark-submit pod ungracefully (which is done in case of evictions or node outages) and even when gracefully shutting down the pod. After a couple of seconds, a new spark-submit pod is booting up, starting a new driver and the driver starts its executors. The problem: old driver and executors are still running. That leads to corrupted data, when to spark apps update the same dataset at the same time.
I expect a YARN like behaviour, when the submit is killed then a) the whole application should fail or b) the application should not be restarted and running application can continue its doing
Issue should be easily reproducible
Possible solution
No response
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None
Activity
[-]spark application can run multiple times because[/-][+]BUG: spark application can have multiple drivers at once[/+]razvan commentedon Jan 30, 2026
Thanks for the detailed explanation and sorry for the trouble.
a) is the correct fix but it's impossible to implement right now without a bigger change first. We'll have to drop the use of
Jobworkloads and move the management of thespark-submitpod to the operator.We'll implement b) as it's easy and hopefully helps a bit.
maxgruber19 commentedon Jan 30, 2026
Thank you for the quick reply as usual.
Both a) and b) are fine for us in the first place so we're looking forward for it anyways. Maybe a suggestion from my side, as it may not require so many changes: Isn't it possible to set the retry / backoff of the spark job to zero so that the job is not restarted that way? Maybe that's even less complicated
I would like to propose that the spark application is listed as UNKNOWN or something so that the submitter (custom airflow operator in our case) has the chance to decide what to do in that case. It would be a mess if the spark application continues but its status field is never updated in the end
maxgruber19 commentedon Jan 30, 2026
I see disabling the backoff is exactly what you did in your fix commit, thank you very much