chore: fix stale docs and CI checks that were not running

[NoahKusaba]

Rationale for this change

I have noticed for a long time that Ballista's documentation has drifted from the code: wrong defaults, examples that no longer compile against the shipped API, and passages duplicated across pages that have since come to disagree with each other.

Chasing that turned up the same pattern in the checks that were supposed to
prevent it. Several were scoped by an explicit list that had gone stale, and four
were not running at all:

• Every intra-doc anchor link is dead on the published site. myst_heading_anchors
  was never set, so MyST emits no heading ids and all 14 file.md#heading links
  404. They resolve on GitHub, which is why nobody noticed. sphinx-build on
  main reports 16 warnings; this PR takes it to 0.
• docs/developer/ was never published. It sits outside the Sphinx source/
  tree, yet Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor #2038 and fix: repair AQE plan rewrites, remove a redundant shuffle, and enable adaptive query planning by default #2315 both added architecture docs to it.
• stale.yml has never run. ci: add take and stale #1488 committed it to the repository root, where
  GitHub does not look for workflows.
• run-rat.sh could pass without checking anything. It invoked python, not
  python3; with no python on PATH the checker never ran, left an empty report,
  and the script then printed "No unapproved licenses". I hit exactly this.

What changes are included in this PR?

71 files, +1088/−1303. Most of the net deletion is duplicated content collapsed to
a single source. Twelve commits, each separable.

Documentation correctness

• AQE was documented as experimental and off by default in three places.
  ballista.planner.adaptive.enabled defaults to true.
• Scheduler policy default is push-staged, documented as pull-staged.
  Event-loop buffer default is 1000, documented as 10000.
• ballista-cli has no spark-compat feature, but two guides told users to
  install and build with it. Both commands fail as written.
• The scheduler REST table was missing five endpoints and both health probes. The
  Prometheus list carried a failed metric that does not exist.
• Every doc that starts a second executor moved only --bind-port, colliding with
  the first executor's gRPC port.
• CLI usage blocks were clap 3 output missing --vcores, --color, --tui.
  Regenerated from the built binary.
• extending-components built a runtime with RuntimeConfig, removed in
  DataFusion 55, using helper names that had been renamed.
• The license badge pointed at datafusion-comet; adopter rows referenced images
  that are not in the repo.

Documentation rendering and structure

• Set myst_heading_anchors = 4, fixing the 14 dead cross-references. Also a short
  title overline in index.rst and a link to a directory rather than a document.
• Delete docs/developer/ after porting its Multi-Partition Tasks section and the
  SchedulerGrpc method table into the contributors guide. Its diagram was dropped
  rather than ported: it labels arrows ExecutePartition, not an RPC in the
  current proto.
• Pull the Rust examples from examples/examples/ with literalinclude rather
  than pasting them. All three copies had already drifted.
• Replace the hand-maintained config tables in the tuning guide with generated
  regions, so the existing config-docs gate covers them.
• Give ballista-core, ballista-scheduler and ballista-executor real READMEs.
  Each is a docs.rs landing page via include_str! and was four lines long.
• Standardise on "vcores" over "task slots" and "concurrency levels".
• Collapse cluster-start instructions from five copies to two canonical pages. That
  duplication is what produced the port collision above, in four places.

Change to the user-personas contract

Calling this out on its own, because user-personas.md is the contract PRs are
reviewed against and is append-only by its own terms. One persona's wording
changed, in Persona 2 (the Spark user):

• "one task per partition" became "tasks over partitions", and
  "executors with task slots" became "executors with vcores".
• "task-slot concurrency" became "vcore-bounded concurrency".

My reading is that this corrects a guarantee the page was already not providing
rather than withdrawing one. Multi-partition tasks landed in #2038, and
ballista.scheduler.max_partitions_per_task defaults to 0, so the scheduler
packs several partitions per task by default. The page was promising Spark's
strict one-task-per-partition dispatch, which Ballista had stopped doing.

The capability itself is not removed, and I added a line saying how to get it
back: set max_partitions_per_task to 1. Everything else that persona depends
on is untouched.

Flagging it because the page says a change like this needs explicit discussion
and sign-off rather than landing quietly, and a 69-file diff is exactly where it
could land quietly. Happy to revert the wording and leave the contract stale if
you would rather correct it in its own PR.

CI and repo hygiene

• Move stale.yml into .github/workflows/ and switch it on. It runs nightly,
  marks PRs with no activity for 60 days, and closes them 7 days later. Issues are
  left alone, and drafts are not exempt.
  Delete the root take.yml, a diverged duplicate of the live one.
• clippy named seven crates, leaving ballista-api-types, ballista-history and
  ballista-chaos unlinted. Use --workspace, keeping ballista-cli separate
  because its tui and web features target different platforms.
• ruff ran only inside python/, leaving twelve files unchecked including every
  dev/ script and docs/source/conf.py. Lint from the repository root; a new
  root ruff.toml covers what python/pyproject.toml does not. Fixes the three
  real findings this surfaced; the E402s in benchmark scripts are deliberate and
  ignored per-file.
• dev/rust_lint.sh, which CONTRIBUTING calls "run them all at once", ran four of
  the eight CI lint scripts.
• run-rat.sh now distinguishes "found unapproved files" from "did not run".
• docs/build.sh builds with -W --keep-going, so a broken cross-reference fails
  instead of shipping.
• Add examples/examples/** to the docs deploy trigger, now that the guide embeds
  those files.
• Drop Arrow-inherited entries from .gitignore and rat_exclude_files.txt, and a
  prettier exclusion for a CHANGELOG.md that does not exist. Fix pre-commit.sh,
  which referenced "apache arrow" and required an uninstalled nightly toolchain.
  Document dev/update_datafusion_versions.py, which nothing referenced.

Follow-ups from review

• Removed the duplicate SET s3.allow_http = true from custom-client.rs and
  from the three S3 integration tests, matching the earlier removal in the
  extending-components guide. It is dead: S3Options::set writes one field per
  key, nothing resets the config, and allow_http is read only inside
  s3_object_store_builder, which get_store calls from a fresh config read.
  Confirmed by running the S3 integration tests, which pass.
• Dropped the "Composition with in-flight DataFusion AQE" subsection from the
  architecture guide. It built on Parallel bounded RANGE-frame window functions without PARTITION BY (draft) datafusion#23026, still an open draft,
  and [PoC/Proposal] AQE-lite: change plan properties at runtime based on stats from pipeline breakers datafusion#23167, closed without merging. Moved to Design note: composing multi-partition tasks with plan-level AQE in DataFusion #2445. The
  multi-partition tasks section above it stays, since that describes dispatch we
  have. The one remaining draft link, in partitioned_bounded_window_agg.rs, is
  deliberate: it explains why that wrapper is temporary.
• The three S3 helper snippets in extending-components had drifted from the
  shipped functions: session_state_with_s3_support was missing its
  with_scalar_functions / with_aggregate_functions / with_window_functions
  chains, its use, and its full return type, while the prose claimed the
  snippets were the shipped implementations. They are now pulled from
  ballista/core/src/object_store.rs with literalinclude, anchored on doc
  comments rather than line numbers. Verified against the built HTML: the
  rendered block is byte-identical to the source slice.
• Consequently ballista/core/src/object_store.rs joins examples/examples/**
  on the docs deploy trigger. Every literalinclude target in the docs is now
  cross-checked against that path list.
• Removed the same non-existent failed metric from the
  PrometheusMetricsCollector rustdoc that was removed from the metrics guide.
  The counter is registered as job_failed_total; only the struct field is
  named failed. The header claimed 7 metrics while listing 8; doc list and
  registered names now both come to 7 and match exactly.

Keeping up with main

Merged main while this was in review. #2397 and #2434 conflicted with the REST
table and the generated config tables; resolved by keeping both sides and
regenerating the tables from source. Both also made docs this branch had already
corrected incomplete:

• feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json #2397 added a utoipa feature to ballista-core, now in both feature tables,
  and the scheduler's default rest-api feature now also serves the OpenAPI spec.
• feat(scheduler): stage a join's build side before deciding the join #2434 added build-side staging, on by default. It is now in the AQE lists in the
  tuning and architecture guides, and has a 55.0.0 upgrade entry, since it changes
  query plans.

Auditing those sections turned up three older inaccuracies:

• The Join Strategy section only described the static planner. Under AQE, the
  default, datafusion.optimizer.prefer_hash_join is not consulted; the join is
  chosen at runtime from broadcast_join_threshold_bytes and
  hash_join_max_build_partition_bytes. The section now covers both paths, and
  notes that the static planner only promotes hash joins to broadcast.
• Join reordering compares byte sizes first and falls back to row counts, not row
  counts alone.
• The REST API was described as an optional feature to enable, but rest-api is
  on by default.

Are there any user-facing changes?

No API, behaviour, or configuration-value changes.

Five .rs files are touched, none of them library logic:

File	Change
ballista/core/src/config.rs	one ConfigEntry description string
ballista/scheduler/src/config.rs	one rustdoc comment naming a TaskDistribution variant that does not exist
ballista/scheduler/src/metrics/prometheus.rs	one rustdoc line naming a metric that is not exported
examples/examples/custom-client.rs	drops a duplicate SET
examples/tests/object_store.rs	drops the same duplicate SET in three tests

Ten .py files are touched. Seven are reformat-only, which I checked by
comparing parsed ASTs before and after. The other three are the lint fixes
described above plus the one-line conf.py setting.

For contributors, lint scope widens and the docs build now fails on warnings. Both
are green as of this branch.

Notes for reviewers

1. docs/source/conf.py changes how every Markdown page renders. One line,
   site-wide effect.
2. Deleting docs/developer/ is the only irreversible change. Unique content
   was ported first.
3. stale.yml is now enabled, and will act on existing PRs. It has never run
   before, so its first nightly run sweeps the whole backlog. As of 2026-09-13, 8
   of the 45 open PRs have had no activity for 60 days, including 3 drafts. They
   would be marked stale and closed 7 days later unless someone comments. That is
   a project decision worth an explicit yes from a maintainer.
4. Persona 2's wording changed in the append-only personas contract. See the
   section above; this is the one change here that wants a deliberate yes or no.
5. The Join Strategy advice changed. It used to tell users to
   SET prefer_hash_join = true for hash joins. Under AQE, the default, that
   setting is ignored, so the section now says so and scopes that advice to AQE
   turned off.

Verification

Check	Result
sphinx-build -W	clean, 0 warnings (16 on main)
prettier@2.7.1 --check	matches
./dev/update_config_docs.sh --check	up to date
ruff check / ruff format --check, repo root	22 files, clean
cargo doc under RUSTDOCFLAGS="-D warnings"	passes
cargo test --doc	8 passed, 0 failed
apache-rat	clean on a git archive of the branch; pruned exclude list matches main's result
workflow YAML	16 parse
relative links and images	1970 checked, 0 broken
cargo clippy --workspace, -D warnings	passes, including the three newly covered crates
S3 integration tests, --features testcontainers	5 passed

How this was produced

Drafted with Claude Opus 5, starting from "scan all documentation and do a full
cleanup/correctness fix", then iterated into the structural and CI changes above.
Every factual claim was checked against the source tree rather than taken on the
model's word, and the table above is the gate it was held to. Reviewers should
still read the prose themselves.

[NoahKusaba]

The sensible thing would have been to split this up into many smaller PR's, but I don't want to spend much time on this, as I will be refocusing efforts on the iceberg-ballista integration, as the iceberg-datafusion crate has migrated to it's own repo.

This is clearly important work that should get done, but I'm hoping the other more experienced maintainers can focus their efforts on feature integrations to close the gap with Spark's performance on benchmarks.

Let me know how you would like me to proceed, or if we can just merge this as-is. I have read every line, and it makes sense to me.

-- I may have gotten a little overboard with also widening and cleaning up python linting. My hope is that fixing the docs can attract more people to Ballista.

[andygrove]

Thanks for digging into this. The four checks that were never actually running are a great find, especially run-rat.sh reporting "No unapproved licenses" when the checker had not run at all. I spot-checked the factual claims against the tree rather than taking them on trust, and most hold up: push-staged really is the default, prometheus-metrics really is not a default scheduler feature, ballista-cli really has no spark-compat, the added REST rows and both probes exist in api/routes.rs and api/health.rs, the coalesce and parallel-window keys exist and default to false, the port-collision fix is right against the executor defaults, and :lines: 18- lands exactly on the first use in both included examples.

No public API changes here, both .rs edits are string-only, and the one on task_distribution is a rustdoc line on a field with an explicit clap help, so --help output does not move. No plan shape or execution changes either, so no benchmark numbers needed for this one.

A few things inline before it goes in. The allow_http one is the only real blocker, the rest are small or process notes.

[andygrove]

I do not think this line should go. examples/examples/custom-client.rs still has it, and ballista/core/src/object_store.rs:205 rejects an http:// endpoint unless s3.allow_http is true. The line just above this sets s3.endpoint = 'http://localhost:9000', so with this removed the documented sequence fails against minio as written.

Can you put it back?

[NoahKusaba]

This is actually a redundant call!
If you check line 185, the ctx already calls "SET s3.allow_http = true".
custom-client.rs also has the redundant "SET s3.allow_http = true" as well.

I've removed it from the example to remove confusion (in my follow up commit). Let me know if that works for you?

I also verified the example runs and compiles without the duplicate.

[andygrove]

1000 is correct for this table since it is the CLI default at ballista/scheduler/src/config.rs:135, so no change needed here.

Worth knowing though that SchedulerConfig::default() at ballista/scheduler/src/config.rs:429 is still 10000, so anyone constructing the config in Rust rather than via the CLI gets a different number than this now claims. That is a pre-existing inconsistency in the code, not something you introduced, but it would be good to open a follow-up issue for it.

[NoahKusaba]

Should I just set the config.rs to 1000?

[andygrove]

Widening the scope is right, but this loses the inline annotations for everything it adds. Ruff emits paths relative to the working directory, so running from python/ with .. reports a finding in dev/foo.py as ../dev/foo.py, and GitHub cannot map that back to a file. The annotations quietly stop appearing for every file outside python/.

Running from the repository root instead should fix it and keep the per-file config resolution you describe in the comment.

[NoahKusaba]

AI Validation:

Thanks — I checked this by planting a deliberate F401 and reading the raw
workflow command. Ruff doesn't emit ../; with --output-format=github it
writes an absolute path into file=, which is what GitHub uses to place the
annotation.

For the same finding, file= is byte-identical across all three:

before  (cd python, `.`)   file=/…/python/python/ballista/jupyter.py
this PR (cd python, `..`)  file=/…/python/python/ballista/jupyter.py
root    (root, `.`)        file=/…/python/python/ballista/jupyter.py

So annotations behave exactly as they did before this PR. What does change with
the working directory is the human-readable text after the ::, which GitHub
ignores for placement — I suspect that's what you spotted in the log.

Happy to run from the root anyway if you prefer the shorter log paths, though
the cd python is load-bearing (it's how uv run --no-project finds the synced
venv), so it'd need to invoke python/.venv/bin/ruff directly.

[andygrove]

Agreed on moving to --workspace, the explicit list had clearly gone stale.

One side effect worth being aware of: --workspace --all-features now turns on ballista-core's force_hash_collisions while linting every other crate, which the old per-package invocations did not do. It is lint-only so no test behaviour changes, just noting it in case a confusing cfg-dependent warning shows up later.

[NoahKusaba]

Hey @milenkovicm there is a stale.yml file in the CI that I left disabled in this PR, which automatically closes old inactive PR's.
I noticed you were closing inactive PR's manually, would you want me to enable it here?

[milenkovicm]

Please do,I was wondering why is it not working

[NoahKusaba]

I cleaned up the comments, which were intended to explain changes to reviewers, but aren't really needed + resolved conflict and updated docs with latest merged changes. Also enabled the stale.yml file, per the discussion with milenkovicm.
The PR should be ready to merge, after approvals.

[NoahKusaba]

I'm getting the impression we don't feel comfortable shotgun merging alot of cleanups at once with AI yet.
I can look to split this up next week into smaller more manageable PR's next week if I have time.