Flaky test: TestStorageEnginesInSlowQuery in pkg/executor

Automated flaky test report.

Title: Flaky test: TestStorageEnginesInSlowQuery in pkg/executor

Repo: pingcap/tidb
Package: pkg/executor
Case: TestStorageEnginesInSlowQuery
Branch: master
Flaky Count: 14
Time Thresholded Count: 0
Latest Build: https://do.pingcap.net/jenkins/job/pingcap/job/tidb/job/pull_unit_test_next_gen/12067/

Window: 2026-02-27T00:00:00.000Z → 2026-03-06T00:00:00.000Z
Threshold: 100000 ms

Note: This issue aggregates all branches; branch-specific updates are posted as comments.

Automated flaky test report update.

Repo: pingcap/tidb
Branch: master
Package: pkg/executor
Case: TestStorageEnginesInSlowQuery
Flaky Count: 13
Time Thresholded Count: 0
Latest Build: https://do.pingcap.net/jenkins/job/pingcap/job/tidb/job/pull_unit_test_next_gen/11982/

Window: 2026-02-27T00:00:00.000Z → 2026-03-06T00:00:00.000Z
Threshold: 100000 ms

This test is among the top 10 flaky tests in the past week, directly blocking other PRs and affecting CI reliability.

Given its significant impact, it is assigned severity/critical and requires urgent attention.

@windtalker

Root cause analysis (locally verified)

This flaky is a timeout-in-teardown issue, not a result mismatch in TestStorageEnginesInSlowQuery.

The blocking path is in unistore MPP cleanup:

exchSenderExec.next() waits on <-tunnel.connectedCh in its defer path.
If downstream connection is never established (or task context is already canceled), this wait can block forever.
Then the MPP dispatch goroutine cannot reach reqCtx.finish() in executeMPPDispatch.
Server.Stop() keeps waiting for refCount == 0, so teardown hangs until global timeout.

Why this is flaky:

It depends on scheduling/timing order (connection-established vs task-finished/canceled), so it is nondeterministic and CI-load sensitive.

Local proof:

Added deterministic UT: TestExchSenderExecNextReturnsWhenCtxCanceledBeforeTunnelConnected.
Before fix: this UT fails (blocked on connectedCh).
After fix: this UT passes.
Related test TestStorageEnginesInSlowQuery also passes locally after the fix.

Proposed fix:

In exchSenderExec.next() defer cleanup, wait for either:
- tunnel.connectedCh, or
- mppCtx.Ctx.Done()
This prevents infinite blocking in tunnel cleanup and allows reqCtx.finish() to run.

Ownership suggestion:

This should be reviewed by unistore/MPP owners (pkg/store/mockstore/unistore/cophandler / MPP task+tunnel lifecycle), not only by slow query test owner.

Verification commands run locally:

make failpoint-enable
go test -run TestExchSenderExecNextReturnsWhenCtxCanceledBeforeTunnelConnected -tags=intest,deadlock ./pkg/store/mockstore/unistore/cophandler -count=1 -timeout=60s (fail before fix, pass after fix)
go test -run TestStorageEnginesInSlowQuery -tags=intest,deadlock ./pkg/executor -count=1 -timeout=300s
make failpoint-disable
make lint

Patch used for verification:

diff --git a/pkg/store/mockstore/unistore/cophandler/cop_handler_test.go b/pkg/store/mockstore/unistore/cophandler/cop_handler_test.go
index 79a3fa67d9..3dc98674ad 100644
--- a/pkg/store/mockstore/unistore/cophandler/cop_handler_test.go
+++ b/pkg/store/mockstore/unistore/cophandler/cop_handler_test.go
@@ -15,6 +15,7 @@
 package cophandler

 import (
+     "context"
      "errors"
      "fmt"
      "math"
@@ -22,6 +23,7 @@ import (
      "path/filepath"
      "slices"
      "testing"
+     "time"

      "github.com/pingcap/badger"
      "github.com/pingcap/badger/y"
@@ -36,6 +38,7 @@ import (
      "github.com/pingcap/tidb/pkg/store/mockstore/unistore/tikv/mvcc"
      "github.com/pingcap/tidb/pkg/tablecodec"
      "github.com/pingcap/tidb/pkg/types"
+     "github.com/pingcap/tidb/pkg/util/chunk"
      "github.com/pingcap/tidb/pkg/util/codec"
      "github.com/pingcap/tidb/pkg/util/collate"
      "github.com/pingcap/tidb/pkg/util/rowcodec"
@@ -545,6 +548,57 @@ func createTestDB(dbPath, LogPath string) (*badger.DB, error) {
      return badger.Open(opts)
 }

+type mockExchSenderChildExec struct{}
+
+func (*mockExchSenderChildExec) open() error                                   { return nil }
+func (*mockExchSenderChildExec) next() (*chunk.Chunk, error)                   { return nil, nil }
+func (*mockExchSenderChildExec) stop() error                                   { return nil }
+func (*mockExchSenderChildExec) getChildren() []mppExec                        { return nil }
+func (*mockExchSenderChildExec) getIntermediateFieldTypes() []*types.FieldType { return nil }
+func (*mockExchSenderChildExec) takeIntermediateResults() []*chunk.Chunk       { return nil }
+func (*mockExchSenderChildExec) getFieldTypes() []*types.FieldType             { return nil }
+func (*mockExchSenderChildExec) buildSummary() *tipb.ExecutorExecutionSummary  { return nil }
+
+func TestExchSenderExecNextReturnsWhenCtxCanceledBeforeTunnelConnected(t *testing.T) {
+     ctx, cancel := context.WithCancel(context.Background())
+     t.Cleanup(cancel)
+
+     tunnel := &ExchangerTunnel{
+             DataCh:      make(chan *tipb.Chunk, 1),
+             connectedCh: make(chan struct{}),
+             ErrCh:       make(chan error, 1),
+     }
+     t.Cleanup(func() {
+             select {
+             case <-tunnel.connectedCh:
+             default:
+                     close(tunnel.connectedCh)
+             }
+     })
+
+     exec := &exchSenderExec{
+             baseMPPExec: baseMPPExec{
+                     sctx:     flagsAndTzToSessionContext(0, time.UTC),
+                     mppCtx:   &MPPCtx{Ctx: ctx},
+                     children: []mppExec{&mockExchSenderChildExec{}},
+             },
+             tunnels: []*ExchangerTunnel{tunnel},
+     }
+
+     done := make(chan struct{})
+     go func() {
+             defer close(done)
+             _, _ = exec.next()
+     }()
+
+     cancel()
+     select {
+     case <-done:
+     case <-time.After(time.Second):
+             t.Fatal("exchSenderExec.next blocks on connectedCh even when context is canceled")
+     }
+}
+
 func BenchmarkExecutors(b *testing.B) {
      prepare := func(rows, limit int) (dagReq *tipb.DAGRequest, dagCtx *dagContext, clean func() error) {
              data, err := prepareTestTableData(rows, tableID)
diff --git a/pkg/store/mockstore/unistore/cophandler/mpp_exec.go b/pkg/store/mockstore/unistore/cophandler/mpp_exec.go
index a5d4412919..0be88b4289 100644
--- a/pkg/store/mockstore/unistore/cophandler/mpp_exec.go
+++ b/pkg/store/mockstore/unistore/cophandler/mpp_exec.go
@@ -949,9 +949,20 @@ func (e *exchSenderExec) toTiPBChunk(chk *chunk.Chunk) ([]tipb.Chunk, error) {
 }

 func (e *exchSenderExec) next() (*chunk.Chunk, error) {
+     var mppCtx context.Context
+     if e.mppCtx != nil {
+             mppCtx = e.mppCtx.Ctx
+     }
      defer func() {
              for _, tunnel := range e.tunnels {
-                     <-tunnel.connectedCh
+                     if mppCtx == nil {
+                             <-tunnel.connectedCh
+                     } else {
+                             select {
+                             case <-tunnel.connectedCh:
+                             case <-mppCtx.Done():
+                             }
+                     }
                      close(tunnel.ErrCh)
                      close(tunnel.DataCh)
              }

I'm assigning @windtalker to verify. Thanks.

Tha patch looks good to me, I'll create a PR based on this patch

*: fix flaky test TestStorageEnginesInSlowQuery (pingcap#66773)

*: fix flaky test TestStorageEnginesInSlowQuery (#66773) (#66891)

Automated reopen: flaky detected after this issue was closed.

Repo: pingcap/tidb
Branch: master
Package: pkg/executor
Case: TestStorageEnginesInSlowQuery
Issue closed_at: 2026-03-09T08:06:08.000Z
Build started_at: 2026-04-27T08:43:32.783Z
Build started_at source: jenkins.timestamp
Build: https://prow.tidb.net/jenkins/job/pingcap/job/tidb/job/pull_unit_test_next_gen/2452/

Window: 2026-04-27T00:00:00.000Z → 2026-04-28T00:00:00.000Z
Threshold: 100000 ms

Automated flaky test report update.

Repo: pingcap/tidb
Branch: master
Package: pkg/executor
Case: TestStorageEnginesInSlowQuery
Flaky Count: 1
Time Thresholded Count: 0
Latest Build: https://prow.tidb.net/jenkins/job/pingcap/job/tidb/job/pull_unit_test_next_gen/2452/

Window: 2026-04-27T00:00:00.000Z → 2026-04-28T00:00:00.000Z
Threshold: 100000 ms

@henrybw

The latest flaky is some query is not in the slow query

result.go:49: 
        	Error Trace:	pkg/testkit/result.go:49
        	            				pkg/executor/slow_query_sql_test.go:520
        	Error:      	Not equal: 
        	            	expected: "[1 0]\n"
        	            	actual  : ""
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1 @@
        	            	-[1 0]
        	            	 
        	Test:       	TestStorageEnginesInSlowQuery
        	Messages:   	sql:select storage_from_kv, storage_from_mpp from information_schema.slow_query where query like 'select%t_pointget%;', args:[]

@henrybw can you take a look at this?

(#68168)

executor: deflake TestStorageEnginesInSlowQuery (#68168)

executor: deflake TestStorageEnginesInSlowQuery (pingcap#68168)

Close it because it has not been reproduced for more than 1 month.

Navigation Menu

Repository navigation

Flaky test: TestStorageEnginesInSlowQuery in pkg/executor #66727

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Flaky test: TestStorageEnginesInSlowQuery in pkg/executor #66727

Description

Activity

ti-chi-bot commented on Mar 6, 2026

zanmato1984 commented on Mar 6, 2026

zanmato1984 commented on Mar 8, 2026

windtalker commented on Mar 8, 2026

ti-chi-bot commented on Apr 28, 2026

ti-chi-bot commented on Apr 28, 2026

windtalker commented on Apr 28, 2026

yinsustart commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions