Repository navigation
Stop DataFusionToolset from materializing full results for a capped query - #73384
Merged
Merged
Conversation
ColtenOuO
requested review from
Lee-W,
ashb,
gopidesupavan and
kaxil
as code owners
September 19, 2026 17:18
ColtenOuO
marked this pull request as draft
September 19, 2026 17:28
ColtenOuO
force-pushed
the
datafusion-toolset-limit-rows
branch
3 times, most recently
from
September 19, 2026 21:19
b9ce82c to
f9fef23
Compare
ColtenOuO
marked this pull request as ready for review
September 20, 2026 08:01
kaxil
reviewed
Sep 20, 2026
The bounded-results documentation still described the old behaviour, where the engine materialized the whole result before the toolset bounded it, so it now contradicted the code and the missing total_rows. The row limit also makes EXPLAIN fail, which is only reachable with allow_writes=True and is easier to meet in the docstring than in an error from the agent.
ColtenOuO
force-pushed
the
datafusion-toolset-limit-rows
branch
from
September 24, 2026 03:27
5200f44 to
53bdec9
Compare
Contributor
Author
|
Resolve the conflict :D |
Lee-W
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DataFusionToolset'squerytool returns at mostmax_rowsrows (default 50), but it ran the query unbounded and calledto_pydict()on the whole result before slicing, so a broad query likeSELECT * FROM big_tablecould exhaust worker memory. The query is now capped atmax_rows + 1rows with a DataFusionLIMITbefore anything is materialized; the extra row only signals truncation. What the agent sees is unchanged apart fromtotal_rows(see below).Changes
session_context.sql(sql).limit(max_rows + 1).to_pydict()and stop reportingtotal_rows. Errors are wrapped inQueryExecutionExceptionwith the same messageDataFusionEngine.execute_queryproduces, so the retry handling is unchanged.DataFusionEngine.execute_querygained amax_rowsargument only incommon-sql1.34.0, while this provider supports>=1.33.0. Applying the limit on the DataFrame keeps the change working on 1.33.0 without bumping the dependency.limit(max_rows + 1)call.Impact
Peak RSS of one
querycall on a local 3M-row parquet file (65 MiB),max_rows=50:SELECT *WHERE amount > 500ORDER BY amount DESCGROUP BY(20 rows)Before, memory grew linearly with the result size; after, it no longer depends on the result's row count.
Trade-off
total_rowsis no longer returned, since onlymax_rows + 1rows are read, and knowing the total would require a full scan or a secondCOUNT(*)query. An untruncated result loses nothing becauserow_countis already the total. For a truncated result, an agent that needs the count can runCOUNT(*), in line with the tool description andhint, which already tell it to aggregate in SQL rather than page through a truncated result.SQLToolsetalready omitstotal_rowswhenever the driver can't report a trustworthy total.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines