Skip to content

fix(mcp): isolate dashboard copies and clarify chart evidence - #45504

Merged
aminghadersohi merged 15 commits into
apache:masterfrom
fitzee:codex/mcp-eval-followups
Oct 21, 2026
Merged

aminghadersohi merged 15 commits into
apache:masterfrom
fitzee:codex/mcp-eval-followups

Conversation

@fitzee

@fitzee fitzee commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

SUMMARY

Refine MCP dashboard-copy safety, list-scope validation, and analytical guidance (implementation and tests at 831598148220a36a37c0d7a4a3890a75bff6ed59):

  • Deep-copy charts by default in duplicate_dashboard, including its compact tool description. Explicit duplicate_slices=false deliberately shares charts (test_compact_duplicate_description_keeps_default, test_duplicate_referencing_same_charts).
  • Reject unknown list-scope arguments rather than silently ignoring them; explain direct user-editor membership and creator/editor union (test_list_dashboards_rejects_unknown_arguments, test_list_dashboards_self_lookup_and_aliases).
  • Distinguish sampled output from input coverage. Ungrouped pivots retain ordinary completeness checks; row-collapsing pipelines conservatively report total_rows=null and truncated=true. Each layer uses its executed query limit, and longer completeness guidance appears once per dashboard (test_pivot_completeness_respects_groupby, test_multi_query_completeness_uses_each_executed_limit, test_dashboard_completeness_guidance_is_shared).
  • Use neutral number-format examples and evidence-based currency guidance (test_number_format_examples_are_currency_neutral, test_guided_chart_prompt_does_not_assume_currency).

Compatibility changes are documented in UPDATING.md and the MCP guide. No frontend or migration changes.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Not applicable: MCP contracts, guidance, and backend tests only.

TESTING INSTRUCTIONS

Validation at 831598148220a36a37c0d7a4a3890a75bff6ed59:

  • pytest across the touched MCP test modules and native surface selection: 906 passed. Command: PYTHONPATH=$PWD/superset-core/src:$PWD /tmp/pr45504-pivot-venv/bin/python -m pytest <touched-test-files> -m 'not manual and not llm_compliance' -q (11 files total, including native surface).
  • Targeted final regression set against the pre-follow-up source: 13 failed, 7 passed; with the follow-up: 20 passed. Preservation cases retain grouped-pivot uncertainty and production distribution-pipeline behavior.
  • Native inventory regenerated using SUPERSET_MCP_UPDATE_TOOL_INVENTORY=1 pytest tests/unit_tests/mcp_service/test_native_tool_surface.py: 38 passed.
  • Staged-file and all changed-branch-file pre-commit checks passed, including mypy, Ruff, and custom Pylint: pre-commit run and pre-commit run --files <changed-branch-files>.
  • Real-LLM checks not run: OpenRouter disabled; no real provider API calls.

Manual checks:

  1. Duplicate without duplicate_slices; expect independent chart IDs. Explicit false intentionally shares IDs.
  2. Inspect the compact tool description for the deep-copy default and explicit sharing option.
  3. Fetch an ungrouped 12-point timeseries under a 100-row limit; expect total_rows=12, truncated=false. Grouped pivots retain conservative input-coverage reporting.
  4. Request more rows than the server permits; inspect the executed fetch limit and per-layer completeness.

ADDITIONAL INFORMATION

Omitting duplicate_slices creates independent chart objects. Unknown list arguments fail validation. The response metadata is additive, but total_rows and truncated have a clarified meaning: unknown input coverage is conservative uncertainty, not observed truncation. Histogram computation and authorization are unchanged. Propagating raw sql_rowcount for exact input-cap detection remains a separate follow-up.

  • Has associated issue
  • Required feature flags
  • Changes UI
  • Includes DB Migration
  • Introduces new feature or API
  • Removes existing feature or API

@github-actions github-actions Bot added the doc Namespace | Anything related to documentation label Oct 19, 2026
@netlify

netlify Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for superset-docs-preview ready!

Name Link
🔨 Latest commit 8315981
🔍 Latest deploy log https://app.netlify.com/projects/superset-docs-preview/deploys/6ad8bcbee2159cdcafc5af47
😎 Deploy Preview https://deploy-preview-45504--superset-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@codeant-ai-for-open-source

codeant-ai-for-open-source Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

CodeAnt PR Risk: Low Risk

  • The PR appears safe to merge; no actionable issues were identified in the review.

Assessed commit: 831598148220

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.05%. Comparing base (4e7e2b7) to head (8315981).
⚠️ Report is 13 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #45504      +/-   ##
==========================================
+ Coverage   83.98%   84.05%   +0.06%     
==========================================
  Files        3055     3054       -1     
  Lines      201017   200913     -104     
  Branches    46949    46950       +1     
==========================================
+ Hits       168828   168875      +47     
+ Misses      28943    28800     -143     
+ Partials     3246     3238       -8     
Flag Coverage Δ
hive 34.92% <48.71%> (+0.03%) ⬆️
mysql 52.88% <48.71%> (+0.02%) ⬆️
postgres 52.89% <48.71%> (+0.03%) ⬆️
presto 36.65% <48.71%> (+<0.01%) ⬆️
python 87.32% <100.00%> (+0.12%) ⬆️
sqlite 52.65% <48.71%> (+0.02%) ⬆️
unit 81.57% <100.00%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gabotorresruiz gabotorresruiz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @fitzee, this is a good set of fixes and the four behaviours all hold up when I drive them.

I ran the affected suites plus the duplication, ownership filter and histogram paths against a real instance on this branch, and reverted each behaviour one at a time to confirm the cited tests pin it. Everything in the body checks out, including the merge base count. Comments inline, none of them blocking.

Holding off on a formal approval only because unit-tests is red and the branch is conflicting. The failure is test_native_inventory_report_matches_registered_tools and it reproduces unchanged at the merge base 3cf94c87, so it is inherited rather than yours: master already refreshed those generate_chart, update_chart and get_chart_preview byte counts in #45311, which is also the conflict. A catch up to master plus a regenerated inventory should clear both.

Comment thread superset/mcp_service/app.py Resolved
@github-actions github-actions Bot added the requires:rebase Requires rebasing on top of current master label Oct 20, 2026
@github-actions github-actions Bot removed the requires:rebase Requires rebasing on top of current master label Oct 21, 2026
Carry every executed query's pipeline privately through saved and rebuilt
JSON results into dashboard summaries. Do not infer input coverage from
post-processed output counts: affected layers return total_rows=null and
truncated=true conservatively, with the input fetch limit and an explanation.
Keep ordinary completeness checks for row-preserving pipelines and retain
the distribution warning for legacy results without execution metadata.

Post-processing registry audit:
- aggregate: grouping reduces rows to group aggregates.
- boxplot: grouping reduces rows to distribution statistics.
- histogram: input rows become bin counts per group.
- pivot: dimension values become columns and duplicate indexes aggregate.
- resample: coarser time bins can reduce rows.
- rolling: min_periods can slice leading rows from the result.
- prophet: forecast reconstruction replaces the input time index; output
  counts cannot establish input coverage.
- compare: adds/removes comparison columns, preserving input rows.
- contribution: computes cell shares, preserving input rows.
- cum: cumulative values preserve input rows.
- diff: differences preserve rows, including missing leading values.
- flatten: flattens column labels/resets the index without removing rows.
- geodetic_parse: appends parsed coordinate columns, preserving rows.
- geohash_decode: appends coordinate columns, preserving rows.
- geohash_encode: appends a geohash column, preserving rows.
- rank: assigns ranks without removing rows.
- rename: changes column labels without removing rows.
- select: selects/excludes/renames columns only, preserving rows.
- sort: reorders rows without removing them.
Unknown, custom and future operations are conservative by default. The
escape_separator/unescape_separator helpers are not registered DataFrame
operations.

Regression executes a generated timeseries/groupby pivot against 500 input
rows, producing 50 output rows, through saved and rebuilt query paths.
It fails against 377a908 and passes after.
Additional tests cover all registry classifications, custom operations,
mixed-query layers, empty leading layers, and the private metadata handoff.

@rebenitez1802 rebenitez1802 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: the deep-copy default, the strict self-scoped list schemas and the currency-neutral guidance all look right, and the authorization model is unchanged. Duplicating still needs Dashboard write plus editor on the source, the same capability as REST /copy/. I left two Medium inline suggestions worth taking before merge:

  1. pivot without a groupby is treated as row-collapsing, so most timeseries charts now report truncated=true.
  2. The compact tool description no longer mentions the new deep-copy default.

Smaller follow-ups (non-blocking):

  • 🟢 fetch_row_limit reports request.fetch_row_limit, not the limit that actually ran (query_context.queries[i].row_limit). They differ when SQL_MAX_ROW is lower than the request, and truncated can then be wrong too.
  • 🟢 The histogram/box-plot note branch only fires when operation metadata is missing. It then returns total_rows=1, truncated=false next to a note saying those values prove nothing. The two tests for it build ChartData without metadata, so they cover that fallback, not the production path.
  • 🟢 Stale descriptions on fetch_row_limit / total_rows in dashboard/schemas.py (they describe only a "capped fetch"), plus a capital "For" after a semicolon in the completeness note. The meaning change for truncated/total_rows deserves a line in UPDATING.md, since the PR calls the new fields "additive".
  • 🟢 A few $,.2f examples remain: chart/schemas.py Big Number y_axis_format and d3_number_format, and update_dataset_metric.py (sum_revenue with $,.2f).
  • 🟢 The ~370-char completeness_note repeats on every chart (~11 KB at max_charts=30). It could go once on DashboardData.

Preserve ordinary completeness for ungrouped pivots and use executed query limits for each layer. Keep distribution fallback coverage conservative, share population guidance once per dashboard, and document the count semantics. Retain the isolated copy default in compact tool descriptions and use currency-neutral number-format examples.
@aminghadersohi
aminghadersohi merged commit e882435 into apache:master Oct 21, 2026
74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc Namespace | Anything related to documentation size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants