Repository navigation
Spark options set by the user are not handled properly #431
Description
Affected Stackable version
all
Affected Apache Spark-on-Kubernetes version
all
Current and expected behavior
Spark properties that are set by both the user via spec.sparkConf and the operator are not merged properly.
Example: users that need to set Java proxy settings via spark.driver.extraJavaOptions cannot do so without overriding the same property set by the operator.
Possible solution
No response
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None
Activity
lfrancke commentedon Aug 7, 2024
Reported on Discord
sbernauer commentedon Aug 15, 2024
To me personally this sounds like expected behavior. The fields docs state
which it seems to do.
It think it's a very hard problem to correctly merge all the Spark properties in the world but let's see :)
I personally would consider this a feature request to make it easier to merge stuff from the user onto the operator generated ones - which sounds like a nice quality of live feature - if possible.
sbernauer commentedon Aug 15, 2024
I could not find it. Can you please paste the report here, so we can understand what exactly is going and maybe also conclude this is a bug?
lfrancke commentedon Sep 16, 2024
I can't find it either right now, sorry.