docs: refresh stale issue references and normalize issue link format - #5062
Merged
Merged
Conversation
Audit the user guide and contributor guide for references to issues that have since been closed, and make the remaining issue and PR links use a single format. Thirteen places described a limitation that no longer exists, verified against the current serde and native code rather than the issue state alone: size() over MapType, try_mod / EvalMode.TRY, CAST(map AS map), spark.sql.legacy.castComplexTypesToString, replace with an empty search string, initcap, str_to_map with the legacy truncate flag, AVG(decimal) over a window, Percentile, the pyarrow-udf row round-trip, the datafusion-spark migration epic, the awslabs TPC-DS epic, and the interval type epic. Also drop or repoint links that pointed at a closed or unrelated tracker: the apache#4098 rows in expressions.md, translate, the TimestampNTZ scan note, and the from_utc_timestamp timezone-parser note. Convert bare URLs and bare #NNNN mentions to inline [#NNNN](url), keeping reference-style definitions where a file already used them, and point URLs at /pull/ where the number is a pull request.
andygrove
marked this pull request as ready for review
July 28, 2026 03:46
mbutrovich
approved these changes
Jul 28, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks @andygrove!
andygrove
added a commit
that referenced
this pull request
Jul 28, 2026
* refactor: unify shuffle configs under `spark.comet.shuffle.*` prefix
Shuffle-related configs were spread across four disjoint prefixes:
`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`,
`spark.comet.native.shuffle.*`, and `spark.comet.shuffle.*`. This
consolidates all shuffle configs under `spark.comet.shuffle.*` with
`.columnar.` and `.native.` sub-namespaces for mode-specific settings.
Every renamed key is registered via `.withAlternative(...)`, so
existing user configurations continue to work (a deprecation warning
is logged when an old key is read). Per-mode leaves that were
previously misplaced move to their honest namespace:
- `spark.comet.exec.shuffle.writeBufferSize` (native-only) becomes
`spark.comet.shuffle.native.writeBufferSize`.
- `spark.comet.shuffle.preferDictionary.ratio` (JVM-only) becomes
`spark.comet.shuffle.columnar.preferDictionary.ratio`.
The `spark.comet.columnar.shuffle.*` keys also fix Category 2 in
#4978 (dots-in-segment): `spill.threshold`,
`memory.factor`, and `batch.size` become `spillThreshold`,
`memoryFactor`, and `batchSize`.
Part of #4978.
* refactor: rename `.columnar.` sub-namespace to `.jvm.`
The JVM-side shuffle path in Comet is not the only columnar shuffle —
native shuffle is also columnar (both use Arrow). The distinguishing
axis is *where* the shuffle logic runs. Renaming the sub-namespace
to `.jvm.` aligns it with the mode value users already set:
`spark.comet.shuffle.mode = jvm`.
Keys renamed:
- spark.comet.shuffle.columnar.spillThreshold → spark.comet.shuffle.jvm.spillThreshold
- spark.comet.shuffle.columnar.memoryFactor → spark.comet.shuffle.jvm.memoryFactor
- spark.comet.shuffle.columnar.batchSize → spark.comet.shuffle.jvm.batchSize
- spark.comet.shuffle.columnar.preferDictionary.ratio → spark.comet.shuffle.jvm.preferDictionary.ratio
The pre-existing deprecated aliases (`spark.comet.columnar.shuffle.*`)
are untouched — they were never `.columnar.`-scoped under the new
prefix. Val names follow: `COMET_SHUFFLE_COLUMNAR_*` → `COMET_SHUFFLE_JVM_*`.
The Scala/Java class names (`CometColumnarShuffle`, `CometColumnarShuffleSuite`, etc.)
still use "columnar" and are left alone — that is a separate rename to
consider once the config vocabulary settles.
* chore: re-run prettier on shuffle docs
Table column alignment tweaks after the `.columnar.` → `.jvm.` key
rename. Content unchanged.
* style: apply spotless formatting to CometShuffleExternalSorter
* refactor: move maxBufferBytes under spark.comet.shuffle.native prefix
* test: update Spark diffs for renamed shuffle configs
Regenerate the Spark SQL test diffs to reference shuffle configs by their
string keys rather than CometConf constants, so the config rename in this
PR no longer breaks test compilation.
- spark.comet.exec.shuffle.enabled -> spark.comet.shuffle.enabled
- CometConf.COMET_EXEC_SHUFFLE_WITH_RANGE_PARTITIONING_ENABLED.key ->
"spark.comet.shuffle.native.partitioning.range.enabled"
* docs: refresh stale issue references and normalize issue link format (#5062)
Audit the user guide and contributor guide for references to issues that
have since been closed, and make the remaining issue and PR links use a
single format.
Thirteen places described a limitation that no longer exists, verified
against the current serde and native code rather than the issue state
alone: size() over MapType, try_mod / EvalMode.TRY, CAST(map AS map),
spark.sql.legacy.castComplexTypesToString, replace with an empty search
string, initcap, str_to_map with the legacy truncate flag, AVG(decimal)
over a window, Percentile, the pyarrow-udf row round-trip, the
datafusion-spark migration epic, the awslabs TPC-DS epic, and the
interval type epic.
Also drop or repoint links that pointed at a closed or unrelated tracker:
the #4098 rows in expressions.md, translate, the TimestampNTZ scan note,
and the from_utc_timestamp timezone-parser note.
Convert bare URLs and bare #NNNN mentions to inline [#NNNN](url), keeping
reference-style definitions where a file already used them, and point URLs
at /pull/ where the number is a pull request.
* docs: fix references to configuration keys that do not exist
Several docs referenced configuration keys that were never registered or were
renamed. Corrected the expression/operator opt-in prefix (spark.comet.expr.* ->
spark.comet.expression.* / spark.comet.operator.*), removed references to a
global allowIncompatible key that does not exist, and updated stale shuffle,
memory pool, and metrics keys.
comphead
pushed a commit
to comphead/arrow-datafusion-comet
that referenced
this pull request
Aug 26, 2026
…pache#5062) Audit the user guide and contributor guide for references to issues that have since been closed, and make the remaining issue and PR links use a single format. Thirteen places described a limitation that no longer exists, verified against the current serde and native code rather than the issue state alone: size() over MapType, try_mod / EvalMode.TRY, CAST(map AS map), spark.sql.legacy.castComplexTypesToString, replace with an empty search string, initcap, str_to_map with the legacy truncate flag, AVG(decimal) over a window, Percentile, the pyarrow-udf row round-trip, the datafusion-spark migration epic, the awslabs TPC-DS epic, and the interval type epic. Also drop or repoint links that pointed at a closed or unrelated tracker: the apache#4098 rows in expressions.md, translate, the TimestampNTZ scan note, and the from_utc_timestamp timezone-parser note. Convert bare URLs and bare #NNNN mentions to inline [#NNNN](url), keeping reference-style definitions where a file already used them, and point URLs at /pull/ where the number is a pull request.
comphead
pushed a commit
to comphead/arrow-datafusion-comet
that referenced
this pull request
Aug 26, 2026
…5063) * refactor: unify shuffle configs under `spark.comet.shuffle.*` prefix Shuffle-related configs were spread across four disjoint prefixes: `spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`, `spark.comet.native.shuffle.*`, and `spark.comet.shuffle.*`. This consolidates all shuffle configs under `spark.comet.shuffle.*` with `.columnar.` and `.native.` sub-namespaces for mode-specific settings. Every renamed key is registered via `.withAlternative(...)`, so existing user configurations continue to work (a deprecation warning is logged when an old key is read). Per-mode leaves that were previously misplaced move to their honest namespace: - `spark.comet.exec.shuffle.writeBufferSize` (native-only) becomes `spark.comet.shuffle.native.writeBufferSize`. - `spark.comet.shuffle.preferDictionary.ratio` (JVM-only) becomes `spark.comet.shuffle.columnar.preferDictionary.ratio`. The `spark.comet.columnar.shuffle.*` keys also fix Category 2 in apache#4978 (dots-in-segment): `spill.threshold`, `memory.factor`, and `batch.size` become `spillThreshold`, `memoryFactor`, and `batchSize`. Part of apache#4978. * refactor: rename `.columnar.` sub-namespace to `.jvm.` The JVM-side shuffle path in Comet is not the only columnar shuffle — native shuffle is also columnar (both use Arrow). The distinguishing axis is *where* the shuffle logic runs. Renaming the sub-namespace to `.jvm.` aligns it with the mode value users already set: `spark.comet.shuffle.mode = jvm`. Keys renamed: - spark.comet.shuffle.columnar.spillThreshold → spark.comet.shuffle.jvm.spillThreshold - spark.comet.shuffle.columnar.memoryFactor → spark.comet.shuffle.jvm.memoryFactor - spark.comet.shuffle.columnar.batchSize → spark.comet.shuffle.jvm.batchSize - spark.comet.shuffle.columnar.preferDictionary.ratio → spark.comet.shuffle.jvm.preferDictionary.ratio The pre-existing deprecated aliases (`spark.comet.columnar.shuffle.*`) are untouched — they were never `.columnar.`-scoped under the new prefix. Val names follow: `COMET_SHUFFLE_COLUMNAR_*` → `COMET_SHUFFLE_JVM_*`. The Scala/Java class names (`CometColumnarShuffle`, `CometColumnarShuffleSuite`, etc.) still use "columnar" and are left alone — that is a separate rename to consider once the config vocabulary settles. * chore: re-run prettier on shuffle docs Table column alignment tweaks after the `.columnar.` → `.jvm.` key rename. Content unchanged. * style: apply spotless formatting to CometShuffleExternalSorter * refactor: move maxBufferBytes under spark.comet.shuffle.native prefix * test: update Spark diffs for renamed shuffle configs Regenerate the Spark SQL test diffs to reference shuffle configs by their string keys rather than CometConf constants, so the config rename in this PR no longer breaks test compilation. - spark.comet.exec.shuffle.enabled -> spark.comet.shuffle.enabled - CometConf.COMET_EXEC_SHUFFLE_WITH_RANGE_PARTITIONING_ENABLED.key -> "spark.comet.shuffle.native.partitioning.range.enabled" * docs: refresh stale issue references and normalize issue link format (apache#5062) Audit the user guide and contributor guide for references to issues that have since been closed, and make the remaining issue and PR links use a single format. Thirteen places described a limitation that no longer exists, verified against the current serde and native code rather than the issue state alone: size() over MapType, try_mod / EvalMode.TRY, CAST(map AS map), spark.sql.legacy.castComplexTypesToString, replace with an empty search string, initcap, str_to_map with the legacy truncate flag, AVG(decimal) over a window, Percentile, the pyarrow-udf row round-trip, the datafusion-spark migration epic, the awslabs TPC-DS epic, and the interval type epic. Also drop or repoint links that pointed at a closed or unrelated tracker: the apache#4098 rows in expressions.md, translate, the TimestampNTZ scan note, and the from_utc_timestamp timezone-parser note. Convert bare URLs and bare #NNNN mentions to inline [#NNNN](url), keeping reference-style definitions where a file already used them, and point URLs at /pull/ where the number is a pull request. * docs: fix references to configuration keys that do not exist Several docs referenced configuration keys that were never registered or were renamed. Corrected the expression/operator opt-in prefix (spark.comet.expr.* -> spark.comet.expression.* / spark.comet.operator.*), removed references to a global allowIncompatible key that does not exist, and updated stale shuffle, memory pool, and metrics keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Several pages in the user guide and contributor guide link to issues that have since been closed, and in most of those cases the surrounding prose still describes the limitation the issue was filed for. A reader following the link lands on a closed issue and has no way to tell whether the documented behavior is still current.
A separate problem is that issue and PR references are written four different ways across these pages: inline
[#NNNN](url), bare URLs, bare#NNNNwith no link at all, and reference-style[#NNNN]:definitions. The bare forms are not clickable in the rendered docs.What changes are included in this PR?
Every issue and PR reference in
docs/source/user-guideanddocs/source/contributor-guidewas checked against its current state on GitHub. Where an issue was closed, the claim in the docs was re-verified against the serde and native code rather than taken from the issue state alone.Thirteen places described a limitation that no longer exists:
collection_funcs.mdsize()overMapTypefalls backCometSizereturnsCompatible()forArrayTypeandMapTypemath_funcs.mdCometRemainderrejectsEvalMode.TRY, sotry_modfalls backconversion_funcs.mdCAST(map AS map)falls backcast_map_to_map; support recurses into the key and value castsconversion_funcs.mdspark.sql.legacy.castComplexTypesToStringis not honouredCometCastand reportedUnsupportedstring_funcs.mdreplacemarksIncompatibleon a literal empty search stringCompatiblewith aNativeOptIn; codegen dispatch by defaultstring_funcs.mdinitcapis unconditionallyIncompatibleCompatiblewith aNativeOptInmap_funcs.mdstr_to_mapignoresspark.sql.legacy.truncateForEmptyRegexSplitIncompatible, routing to codegen dispatchwindow_funcs.mdAVG(decimal)over a window is dead on Spark 4.xCometWindowExec.convertunwraps theCast(Divide(...))shapeagg_funcs.mdPercentilereportsIncompatible, native path opt-inCompatible(), native by defaultpyarrow-udfs.mdCometArrowPythonRunnerconsumesColumnarBatchdirectlyroadmap.mdroadmap.mddatatypes.mdLinks that pointed at a closed or unrelated tracker were dropped or repointed: the ten
tracking #4098rows inexpressions.md(that issue tracked Spark 4.1 CI failures and never coveredkurtosis,skewness, or the Variant functions),translate(#4463, where the incompatibility is real but the issue is closed, so the reason is now stated inline), theTimestampNTZscan note (#4219), thefrom_utc_timestamptimezone-parser note (#2013, whose linked issue was "Add from_utc_timestamp support"), and themake_interval/try_make_intervalrows (#4540 to #5061).Seven closed-issue references were kept deliberately because they are historical or illustrative: the 0.9.1 jar-size incident in
release_process.md, the hash-aggregate epic inbug_triage.md(now explicitly labelled as closed), theto_jsoncode sample inadding_a_new_expression.mdthat mirrors live code instructs.scala, the twoquery ignore(...)syntax examples insql-file-tests.md, and the two audit notes whose workarounds are still present in the code.For formatting, 52 bare URLs and 27 bare
#NNNNmentions were converted to inline[#NNNN](url). Reference-style definitions were left in place inroadmap.mdandiceberg-spark-tests.md, which already used that style consistently. Cross-repo references use the[apache/arrow-rs#10028](url)form. 22 URLs were repointed from/issues/to/pull/where the number is a pull request. URLs inside fenced code blocks were left untouched, including theissues/NNNNplaceholder inadding_a_new_spark_version.md.How are these changes tested?
Documentation only, so there is no runtime behavior to test. Every claim that changed was checked against the code it describes (
CometSize,CometRemainder,CometCast,CometStringReplace,CometInitCap,CometStrToMap,CometWindowExec,CometPercentile,CometArrowPythonRunner). Link formatting was verified with a script that finds any GitHub issue or PR reference outside a fenced code block that is not an inline link or a reference definition; it reports zero remaining.prettier --checkpasses on both guides.