ddl, disttask: import into integrate table mode by mileeeesW · Pull Request #61200 · pingcap/tidb

mileeeesW · 2025-05-20T03:24:32Z

What problem does this PR solve?

Issue Number: close #61199

Problem Summary:
import into need to integrate table mode for protect data read/write when import data.

What changed and how does it work?

alter table mode to Import when submit import into task and reset table mode to Normal when finish import data.
validate import into target table is empty when init executor.
If post process fail, using clean up factory to reset table mode to Normal.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Protect data consistency, DMLs are not allowed during the import into task running.

ti-chi-bot · 2025-05-20T03:24:42Z

Hi @River2000i. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-05-20T03:24:50Z

Hi @River2000i. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lance6716 · 2025-05-20T05:41:56Z

/ok-to-test

codecov · 2025-05-20T05:58:20Z

Codecov Report

❌ Patch coverage is 26.66667% with 77 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.8988%. Comparing base (cdff328) to head (5fe7c72).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61200        +/-   ##
================================================
+ Coverage   72.8032%   74.8988%   +2.0955%     
================================================
  Files          1795       1842        +47     
  Lines        487746     497140      +9394     
================================================
+ Hits         355095     372352     +17257     
+ Misses       111018     101783      -9235     
- Partials      21633      23005      +1372

Flag	Coverage Δ
integration	`49.0009% <16.1904%> (?)`
unit	`72.3585% <26.6666%> (+0.1752%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.1722% <ø> (+16.8454%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

joechenrh

New changes LGTM

joechenrh · 2025-08-06T05:44:38Z

+	require.Eventually(s.T(), func() bool {
+		err := s.tk.QueryToErr("SELECT * FROM import_into.table_mode;")
+		if err != nil {
+			s.ErrorContains(err, "Table table_mode is in mode Import")
+			return false
+		}
+		return true
+	}, 5*time.Second, 100*time.Millisecond)


Maybe we can extract it as a function like

func checkMode(tableName string, enterImportMode bool)

But it's fine to keep it

mileeeesW · 2025-08-06T06:28:43Z

/retest

ti-chi-bot · 2025-08-08T09:30:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, hawkingrei, joechenrh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,hawkingrei]
~~pkg/ddl/OWNERS~~ [D3Hunter]
~~pkg/disttask/OWNERS~~ [D3Hunter]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]
~~pkg/planner/OWNERS~~ [hawkingrei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-08-08T09:30:53Z

[LGTM Timeline notifier]

Timeline:

2025-08-06 05:54:43.658261744 +0000 UTC m=+420393.801022919: ☑️ agreed by joechenrh.
2025-08-08 09:30:52.51080714 +0000 UTC m=+651.545390368: ☑️ agreed by hawkingrei.

mileeeesW · 2025-08-11T05:35:55Z

/retest

mileeeesW · 2025-08-11T06:19:39Z

/retest

ti-chi-bot · 2025-08-11T07:35:48Z

@River2000i: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
non-block/pull-mysql-test-next-gen	`3353edb`	link	false	`/test pull-mysql-test-next-gen`
non-block/pull-integration-e2e-test-next-gen	`3353edb`	link	false	`/test pull-integration-e2e-test-next-gen`
non-block/pull-unit-test-next-gen	`2fd7a88`	link	false	`/test pull-unit-test-next-gen`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

mileeeesW · 2025-08-11T07:41:32Z

/retest

close pingcap#61199

mileeeesW added 4 commits May 15, 2025 16:16

set TableMode to Import when submit import into job

e1fe86e

set TableMode to Normal when clean up import into task

901b6f7

allow ADMIN CHECKSUM TABLE for import into checksum

0471516

check import into table is empty after alter table mode

a16ac09

ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. labels May 20, 2025

ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. sig/planner SIG: Planner labels May 20, 2025

ti-chi-bot Bot removed the do-not-merge/needs-tests-checked label May 20, 2025

add comment

7091f34

ti-chi-bot Bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels May 20, 2025

mileeeesW added 11 commits May 29, 2025 17:41

make bazel_prepare

1a63fe1

add failpoint

31e70f7

add failpoint for CheckImportIntoTableIsEmpty

9df4184

add debug test

748d62d

Merge remote-tracking branch 'upstream/master' into tableModeImportInto

250b75d

add test

528f88e

alter table mode in post process

6f023ba

mark import into task reset table mode in post process

1f15d61

add tet

0b88d47

fix comment

fdf3bd1

add test

4ea78e9

mileeeesW changed the title ~~[WIP]ddl, disttask: import into integrate table mode~~ ddl, disttask: import into integrate table mode Jun 5, 2025

address comment

2fd7a88

joechenrh approved these changes Aug 6, 2025

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 6, 2025

mileeeesW added 5 commits August 7, 2025 11:02

refactor test

e86de7c

fix comment

2d9e31b

refactor test

34eccb1

fix test

934b83d

fix test

712f537

mileeeesW requested a review from D3Hunter August 7, 2025 10:39

D3Hunter approved these changes Aug 8, 2025

View reviewed changes

hawkingrei approved these changes Aug 8, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 8, 2025

ti-chi-bot Bot added the approved label Aug 8, 2025

mileeeesW added 3 commits August 8, 2025 17:33

Merge remote-tracking branch 'upstream/master' into tableModeImportInto

e1834b1

Merge remote-tracking branch 'upstream/master' into tableModeImportInto

119bffe

lower log level

5fe7c72

ti-chi-bot Bot merged commit b6628ea into pingcap:master Aug 11, 2025

mileeeesW deleted the tableModeImportInto branch August 11, 2025 09:18

joechenrh pushed a commit to joechenrh/tidb that referenced this pull request Aug 18, 2025

ddl, disttask: import into integrate table mode (pingcap#61200)

1645932

close pingcap#61199

D3Hunter mentioned this pull request Feb 2, 2026

import into failed with [error="[executor:8173]PreCheck failed: target table is not empty"] when simulated one of tidb failure #65954

Closed

D3Hunter mentioned this pull request Feb 25, 2026

dxf/importinto: run table-empty check once per task #65970

Merged

1 of 12 tasks

D3Hunter mentioned this pull request Jun 15, 2026

importinto: integrate table mode for NextGen kernel #69182

Open

Navigation Menu

Repository navigation

ddl, disttask: import into integrate table mode - #61200

ddl, disttask: import into integrate table mode#61200
ti-chi-bot[bot] merged 111 commits into
pingcap:masterfrom
mileeeesW:tableModeImportInto

mileeeesW commented May 20, 2025 •

edited

Loading

Uh oh!

ti-chi-bot Bot commented May 20, 2025

Uh oh!

tiprow Bot commented May 20, 2025

Uh oh!

lance6716 commented May 20, 2025

Uh oh!

codecov Bot commented May 20, 2025 •

edited

Loading

Uh oh!

joechenrh left a comment

Uh oh!

joechenrh Aug 6, 2025

Uh oh!

mileeeesW commented Aug 6, 2025

Uh oh!

ti-chi-bot Bot commented Aug 8, 2025

Uh oh!

ti-chi-bot Bot commented Aug 8, 2025

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

ti-chi-bot Bot commented Aug 11, 2025 •

edited

Loading

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mileeeesW commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

ti-chi-bot Bot commented May 20, 2025

Uh oh!

tiprow Bot commented May 20, 2025

Uh oh!

lance6716 commented May 20, 2025

Uh oh!

codecov Bot commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joechenrh left a comment

Choose a reason for hiding this comment

Uh oh!

joechenrh Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

mileeeesW commented Aug 6, 2025

Uh oh!

ti-chi-bot Bot commented Aug 8, 2025

Uh oh!

ti-chi-bot Bot commented Aug 8, 2025

[LGTM Timeline notifier]

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

ti-chi-bot Bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mileeeesW commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mileeeesW commented May 20, 2025 •

edited

Loading

codecov Bot commented May 20, 2025 •

edited

Loading

ti-chi-bot Bot commented Aug 11, 2025 •

edited

Loading