Repository navigation
Conversation
Define 8 test dimensions (~41 scenarios) covering functional correctness, network errors, resource leaks, concurrency safety, stress tests, data integrity, graceful shutdown, and idempotency. Includes shared mock OCI registry infrastructure design. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
- Known bug tests use reverse assertions (CI stays green) - Retry-dependent network tests moved to //go:build slowtest - Remove untestable pull reader/tempdir leak scenarios - Push leak tests cover both success and error paths - Deduplicate against existing test coverage - Add GitHub issue references (#491, #492, #493, #494) - Add auth-error-retry as known design issue Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
9 tasks covering mock registry infrastructure, resource tracking, modelfile/pull/push/CLI/pb integration tests, slow tests, and stress tests. Each task has TDD-style steps with complete code. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…t coverage) Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…yWorkspace Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add pkg/backend/pull_integration_test.go with 11 integration tests that exercise the Pull function using MockRegistry (httptest) and mock Storage: Functional correctness: - HappyPath: manifest + 2 blobs, verify PushBlob/PushManifest calls - BlobAlreadyExists: StatBlob returns true, verify PushBlob skipped - ConcurrentLayers: 5 blobs in parallel, verify all stored Network errors: - ContextTimeout: latency + short deadline, verify context error - PartialResponse: FailAfterNBytes on blob, verify error - ManifestOK_BlobFails: 500 on specific blob, verify error Concurrency safety: - ConcurrentPartialFailure: 2 of 5 blobs fail, errgroup cancels Data integrity: - TruncatedBlob: FailAfterNBytes, digest validation catches mismatch - CorruptedBlob: wrong bytes under correct digest, caught by validateDigest Graceful shutdown: - ContextCancelMidDownload: cancel mid-flight, verify error Idempotency: - Idempotent: second pull with all blobs existing skips all writes Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…, dedup truncated/partial) - TestIntegration_Pull_Idempotent: remove unused delta variable and misleading comment; clarify that the key invariant is no storage writes on second pull (PushBlob/PushManifest not called), not a reduction in registry requests. - TestIntegration_Pull_TruncatedBlob: differentiate from PartialResponse by serving same-length wrong-byte content via AddBlobWithDigest instead of truncating the stream. This triggers digest-validation failure rather than a read error, and asserts the error message contains "digest". - TestIntegration_Pull_PartialResponse: add RequestCountByPath assertion to verify the blob endpoint was contacted at least once. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Add 8 integration tests for the Push workflow covering: - Functional correctness (happy path, blob-already-exists skip) - Network errors (manifest push failure via path fault) - Resource leak documentation (KnownBug reverse assertions for #491) - Data integrity (byte-level blob verification after push) - Graceful shutdown (context cancellation mid-upload) - Idempotency (second push skips existing blobs) The KnownBug tests use TrackingReadCloser with reverse assertions (AssertNotClosed) to document that PullBlob ReadClosers are never closed on either success or error paths. These will fail when #491 is fixed, signaling the assertions should be flipped. Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
…ation Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
… (//go:build stress) Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive integration testing suite for modctl, covering functional correctness, network resilience, concurrency, and stress scenarios. It includes a new mock OCI registry helper and resource tracking utilities. Review feedback identifies a logic error in a 'KnownBug' test that prevents it from failing when fixed, suggests renaming a misleadingly titled test, recommends optimizing the mock registry's performance by reducing mutex contention during I/O, and advises removing a redundant implementation plan file that duplicates the source code.
| @@ -0,0 +1,2222 @@ | |||
| # Integration Tests Implementation Plan | |||
There was a problem hiding this comment.
Please do not submit your AI's plan.md ;)
There was a problem hiding this comment.
Removed in 7dbe732 (both the plan and the design document under docs/superpowers). Sorry for the noise.
Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
The branch now includes main, where #491 (ReadCloser leak), #493 (disableProgress data race) and #494 (auth errors retried) are fixed. The reverse-assertion tests for those bugs fail on the merged tree, as designed. Flip them into regression tests: - TestKnownBug_Push_ReadCloserNotClosed_{SuccessPath,ErrorPath} become TestIntegration_Push_ReadCloserClosed_{SuccessPath,ErrorPath} and assert that Push closes every PullBlob ReadCloser. The error-path test no longer guards the assertion with a conditional. - TestKnownBug_DisableProgress_DataRace becomes TestIntegration_DisableProgress_ConcurrentAccess. - TestKnownBug_Pull_AuthErrorStillRetries becomes TestIntegration_Pull_AuthErrorFailsFast and asserts that a 401 fails in under 5 seconds. Review feedback: - Rename TestIntegration_Pull_TruncatedBlob to TestIntegration_Pull_CorruptedContentSameLength; the test serves a same-length body with wrong bytes, not a truncated body. - MockRegistry reads PATCH and PUT upload bodies into a local buffer before taking the registry mutex, so network I/O no longer serializes unrelated requests. - Remove the generated plan and design documents under docs/superpowers. Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
gofmt in Go 1.25 places //go:build lines before the leading comment block. Reformat the tagged test files so the gofmt linter stays clean. Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
Same fix as #679. The recordingStrategy helper merged from main uses io.Reader and io.Copy without importing io, which breaks go vet and golangci-lint on this branch. Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
…pb test The slowtest and stress suites are not run by CI and were broken: - TestSlow_Pull_RetryOnTransientError, TestSlow_Pull_RateLimited and TestSlow_Pull_RetryExhausted used a global fault. The first request of a pull is the manifest fetch, which runs outside the retry loop, so the pull failed immediately and the retry path was never exercised. Scope the faults to the layer blob path with PathFaults. - TestStress_Pull_RepeatedCycles counted about 5 goroutines per cycle. Every Pull builds its own http.Transport and leaves its idle keep-alive connections open; both ends of those connections live in the test process. Disable keep-alives on the mock server so the test measures real leaks only, and poll briefly for teardown. TestIntegration_DisableProgress_ConcurrentAccess deadlocked under -race: 10 goroutines replacing 10 bars in a tight loop hit an mpb deadlock between Abort(drop) and the decorator width sync of the render loop. The race the test guards is on disableProgress, which Add reads before it touches mpb, so use a unique bar name per call. Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
|
@bergwolf @imeoer Thanks for the review. Summary of the update:
PTAL. |
Summary
test/helpers/mockregistry.go) with fault injection (latency, partial response, status override, Nth-request failure, per-path faults) for httptest-based integration testingmainsince this PR was opened: ReadCloser leak (bug: Push leaks ReadCloser from PullBlob on both success and error paths #491), splitReader goroutine leak (bug: splitReader goroutine can block indefinitely if pipe reader is abandoned #492), disableProgress data race (bug: disableProgress global variable has data race (no atomic/mutex protection) #493), auth error retry (bug: retry.Do retries 401/403 auth errors instead of failing immediately #494)New Files (13 files)
test/helpers/mockregistry.go+_test.gotest/helpers/tracking.gopkg/modelfile/modelfile_integration_test.gopkg/backend/pull_integration_test.gopkg/backend/push_integration_test.gointernal/pb/pb_integration_test.gocmd/modelfile/generate_integration_test.gocmd/cli_integration_test.gopkg/backend/pull_slowtest_test.goslowtestpkg/backend/push_slowtest_test.goslowtestpkg/modelfile/modelfile_stress_test.gostresspkg/backend/pull_stress_test.gostressNotes from the review round
TestKnownBug_*reverse-assertion tests were flipped intoTestIntegration_*regression tests oncemain(merged in c38fe8d) fixed bug: Push leaks ReadCloser from PullBlob on both success and error paths #491, bug: disableProgress global variable has data race (no atomic/mutex protection) #493 and bug: retry.Do retries 401/403 auth errors instead of failing immediately #494.MockRegistryreads PATCH and PUT upload bodies before taking its mutex, so network I/O does not serialize unrelated requests.slowtestandstresssuites are not run by CI. They were broken in the first version of this PR (a global fault hit the manifest fetch, which is outside the retry loop, and the goroutine-leak test counted idle keep-alive connections). Both suites pass now; see the test plan.Pull/Pushbuilds its ownhttp.Transportinpkg/backend/remoteand never closes its idle keep-alive connections. The CLI exits after one operation, so this only matters for library users that call the backend repeatedly. The stress test disables keep-alives on the mock server to measure real goroutine leaks only.ProgressBar.Addon an existing name aborts and drops the old bar. Under-race, 10 goroutines replacing 10 bars in a tight loop deadlock mpb while it syncs decorator widths. The concurrent-access test uses unique bar names to avoid that path; production code replaces bars rarely.Test Plan
go test ./... -count=1— all default tests passgo test -race ./internal/pb/ ./pkg/backend/ ./test/helpers/— clean;go test -race -count=20 ./internal/pb/— cleango test ./pkg/backend/ -tags slowtest -timeout 900s— passes (about 150 s)go test ./pkg/modelfile/ ./pkg/backend/ -tags stress -timeout 600s— passesgolangci-lint run— 0 issues