[GH-3398] Keep extra predicates on broadcast and nested KNN joins

[kalayciburak]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide.

Is this PR related to a ticket?

• Closes KNN joins silently drop additional ON conditions in broadcast and multi-conjunct plans #3398.

What changes were proposed in this PR?

KNN joins could drop additional ON conditions when broadcasting either side or when conditions contained nested conjunctions, producing different results depending on the physical strategy.

Reuse the residual condition already extracted by OptimizableJoinCondition and pass it unchanged to the regular, query-side broadcast, and object-side broadcast KNN executors. Remove the duplicate KNN-specific extractor. The original condition remains available in the regular plan for inspection.

Reject ST_KNN markers left in the residual of any detected spatial join during planning, including markers nested inside residual expressions. A physical KNN join implements only its selected predicate, so multiple KNN predicates are unsupported. If another spatial predicate is selected first, the error directs users to place ST_KNN first, preserving the existing predicate-selection rule while preventing a later per-pair evaluation failure.

Regression coverage checks independently selective predicates, mixed accepted/rejected pairs, null values, predicate ordering and nesting, both SQL table orders, both broadcast directions, three- and four-argument KNN, and filtering after nearest-neighbor selection. It also covers valid KNN-first spatial residuals and planning-time rejection of multiple-KNN conditions or KNN after another spatial predicate.

Document that residual ON predicates filter the selected neighbors without replacing rejected pairs, distinguish this from optimizer filter pushdown, and explain the single-KNN and spatial predicate ordering requirements.

How was this patch tested?

The branch now includes the latest master. Its merged MinIO test-image fix resolves the prior image-pull failure; this PR does not change those reader tests.

Java 17, Maven offline with cached dependencies:

• RED against the original PR: the nine new multiple-KNN planning-time rejection cases failed; the other 53 KNN tests passed.
• RED before the mixed-predicate guard: all six new planning-time rejection cases failed. A separate runtime reproduction confirmed the previous failure reached execution with KNN predicate is not supported.
• GREEN, Spark 3.5 / Scala 2.12: KnnJoinSuite, SpatialJoinSuite, BroadcastIndexJoinSuite, and SphereDistanceJoinSuite — 349 tests passed.
• GREEN after merging current master, Spark 3.5 / Scala 2.12: KnnJoinSuite — 74 tests passed.
• GREEN, Spark 4.0.2 / Scala 2.13: KnnJoinSuite — 74 tests passed.
• GREEN, Spark 4.1.1 / Scala 2.13: KnnJoinSuite — 74 tests passed.
• Spark 4 builds used separate source snapshots with matching KNN files. All three builds passed Spotless checks; git diff --check passed.

Commands, with JAVA_HOME set to Java 17 and SPARK_LOCAL_IP=127.0.0.1:

mvn -o -pl spark/common -am package \
  -DwildcardSuites=org.apache.sedona.sql.KnnJoinSuite,org.apache.sedona.sql.SpatialJoinSuite,org.apache.sedona.sql.BroadcastIndexJoinSuite,org.apache.sedona.sql.SphereDistanceJoinSuite \
  -Dtest=KnnJoinSuite -Dsurefire.failIfNoSpecifiedTests=false \
  -DfailIfNoTests=false -Dmaven.javadoc.skip=true

mvn -o -pl spark/common -am package -Dspark=4.0 -Dscala=2.13 -Dspark.version=4.0.2 \
  -DwildcardSuites=org.apache.sedona.sql.KnnJoinSuite \
  -Dtest=KnnJoinSuite -Dsurefire.failIfNoSpecifiedTests=false \
  -DfailIfNoTests=false -Dmaven.javadoc.skip=true

mvn -o -pl spark/common -am package -Dspark=4.1 -Dscala=2.13 \
  -DwildcardSuites=org.apache.sedona.sql.KnnJoinSuite \
  -Dtest=KnnJoinSuite -Dsurefire.failIfNoSpecifiedTests=false \
  -DfailIfNoTests=false -Dmaven.javadoc.skip=true

Did this PR include necessary documentation updates?

• Yes. Updated the nearest-neighbor SQL documentation with residual filtering semantics, the filter-pushdown caveat, and supported join-condition requirements.