importinto: integrate table mode for NextGen kernel #69182

Activity

1. D3Hunter

   added

   type/enhancementThe issue or PR belongs to an enhancement.The issue or PR belongs to an enhancement.

   component/import

   component/ddlThis issue is related to DDL of TiDB.This issue is related to DDL of TiDB.

   component/DXFDistributed eXecute Framework of tidbDistributed eXecute Framework of tidb

   on Jun 15, 2026
2. D3Hunter

   added a parent issue on Jun 15, 2026

   • DXF service on nextgen kernel #61702
3. ti-chi-bot

   added a commit that references this issue on Jun 17, 2026

   ddl: extract synchronous job submit primitive (#69184)

   ...

   d812ce2
4. ti-chi-bot

   added 4 commits that reference this issue on Jun 24, 2026

   ddl: extract table mode job submission helpers (#69423)

   ...

   bbf8075

   dxf: derive current keyspace when acquiring task runtime (#69434)

   ...

   42edcdf

   ddl: normalize scheduler job metadata (#69444)

   ...

   d13cf4e

   domain/crossks: support submit-only alter table mode (#69451)

   ...

   b38149d
5. wjhuang2016 commented on Aug 5, 2026

   

   wjhuang2016

   on Aug 5, 2026

   Member

   More actions

   I reproduced a data-integrity consequence of this missing table-mode fence on current master in a real Premium NextGen TCMS environment.

   Environment:

   • TiDB 8.0.11-TiDB-CLOUD.202608.0-16c97eb67f
   • Git commit 16c97eb67f9558f39535bf63b8c2ffe388fbe391
   • real PD/TiKV/CSE, default MDL enabled
   • no failpoint, source modification, process pause, or network/disk fault

   The target was empty and had a clustered primary key plus a unique index:

   CREATE TABLE t (
     id BIGINT PRIMARY KEY CLUSTERED,
     u BIGINT NOT NULL,
     payload VARCHAR(64) NOT NULL,
     UNIQUE KEY uk_u(u)
   );

   The CSV contained 50,000 primary keys twice. The second copy used a different u, so on_duplicate_key='capture' entered conflict cleanup. I then ran these two statements sequentially:

   IMPORT INTO t
   FROM 's3://<bucket>/dups/*.csv?...'
   WITH DETACHED, on_duplicate_key='capture';
   
   INSERT INTO t VALUES (30000, 9000040, 'foreground-committed');

   IMPORT INTO ... WITH DETACHED returned a pending job. The INSERT committed 39 ms after job creation and was immediately visible. When conflict cleanup completed:

   • the import failed with ErrChecksumMismatch (total_kvs: 1 vs 0);
   • a primary-key scan returned zero rows: the committed foreground row had been deleted;
   • FORCE INDEX(uk_u) WHERE u=9000040 returned error 8133, index-count:1 != record-count:0;
   • ADMIN CHECK TABLE t returned error 8223;
   • a later insert using u=9000040 returned error 1062 because the orphaned unique-index key remained.

   The same shape reproduced in three consecutive product-level runs. A matched run without foreground DML finished with 100000 conflicted rows, left both record and index scans empty, and passed ADMIN CHECK TABLE.

   There are two linked assumptions in the current path:

   1. NextGen performs the empty-table precheck but does not enter TableModeImport, so the target remains writable after admission.
   2. Conflict cleanup checks key existence through a lazily refreshed snapshot, buffers those keys, and later deletes them unconditionally in a new transaction. A foreground transaction can reuse the record key with a new unique-index value between those steps, so cleanup deletes the newer record and leaves its new index key behind.

   A realistic workflow trigger is an asynchronous import pipeline that starts writers after receiving the detached job ID instead of waiting for its terminal state. The documentation says not to run DML during import, so I would classify this as Major rather than Critical; however, the server currently accepts and commits the write, then loses it and leaves persistent physical corruption.

   Expected behavior is that DML is rejected for the full NextGen import lifetime. Conflict cleanup should also avoid deleting a row-key version whose ownership changed after it was observed.