Skip to content

[Bug]: Racing container build commands can fail on buildkit container cold start. #2001

Description

@jglogan

I have done the following

  • I have searched the existing issues
  • If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

When running experiments to parallelize the container build CLI integration tests, the first builds tend to fail because all are trying to start the builder, and the case of the builder existing simply errors out.

We should handle this error gracefully, and ensure that it's handled properly by parallelizing build tests.

Build lifecycle tests remain serial.

Problem description

With build tests moved to the parallel pass:

rm -rf ./test-logs ./test-temp ; make APP_ROOT=./test-data LOG_ROOT=./test-logs PRESERVE_KERNELS=true all test integration

Errors:

  Test testBuildExportTarToDirectory() recorded an issue at TestCLIBuilderTarExport.swift:65:17: Expectation failed: (result.status → 1) == 0
􀄵  build with tar export to directory should succeed
􀢄  Test testBuildExportTarToDirectory() recorded an issue at TestCLIBuilderTarExport.swift:67:17: Expectation failed: (FileManager.default → <NSFileManager: 0x10512afc0>).fileExists(atPath: expectedTar.string → "/Users/john/projects/jglogan/container/.test-scratch/testBuildExportTarToDirectory-4ac0172e/C3A8E163-F776-4CE6-AECE-35677254FE6E/exports/out.tar")
􀄵  tar file should exist at out.tar
􀢄  Test testBuildExportTarToDirectory() recorded an issue at TestCLIBuilderTarExport.swift:70:17: Expectation failed: (result.output → "").contains(expectedTar.string → "/Users/john/projects/jglogan/container/.test-scratch/testBuildExportTarToDirectory-4ac0172e/C3A8E163-F776-4CE6-AECE-35677254FE6E/exports/out.tar")
􀄵  output should reference out.tar
􁁛  Test testInitImageFlagInHelp() passed after 117.908 seconds.
􀢄  Test testBuildExportTarToDirectory() failed after 118.459 seconds with 3 issues.
􁁛  Test testStartIdempotent() passed after 118.709 seconds.
􀢄  Test testBuildExportTarMultipleRuns() recorded an issue at TestCLIBuilderTarExport.swift:96:17: Expectation failed: (r1.status → 1) == 0
􀄵  first build should succeed
􀢄  Test testBuildExportTarInvalidDest() recorded an issue at TestCLIBuilderTarExport.swift:121:17: Expectation failed: (result.error → "Warning! Running debug build. Performance may be degraded.
Error: internalError: "failed to create container" (cause: "exists: "container already exists: buildkit"")
").contains("dest field is required")
􀢄  Test testBuildExportTarMultipleRuns() recorded an issue at TestCLIBuilderTarExport.swift:97:17: Expectation failed: (FileManager.default → <NSFileManager: 0x10512afc0>).fileExists(atPath: exportDir.appending("out.tar").string → "/Users/john/projects/jglogan/container/.test-scratch/testBuildExportTarMultipleRuns-53e71eb8/B9AD16A3-DD1E-430E-8571-CE268A5E9993/exports/out.tar")
􁁛  Test testAnonymousVolumeMixedWithNamedVolume() passed after 118.811 seconds.
􀢄  Test testBuildExportTarInvalidDest() failed after 118.968 seconds with 1 issue.
􀢄  Test testBuildExportTarMultipleRuns() recorded an issue at TestCLIBuilderTarExport.swift:101:17: Expectation failed: (FileManager.default → <NSFileManager: 0x10512afc0>).fileExists(atPath: exportDir.appending("out.tar.1").string → "/Users/john/projects/jglogan/container/.test-scratch/testBuildExportTarMultipleRuns-53e71eb8/B9AD16A3-DD1E-430E-8571-CE268A5E9993/exports/out.tar.1")
􀄵  second tar should exist at out.tar.1
􁁛  Test testExecDetachProcessRunning() passed after 119.000 seconds.
􀢄  Test testBuildExportTarMultipleRuns() failed after 119.153 seconds with 3 issues.

Environment

- OS: 26
- Xcode: 26
- Container: main

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

builderIssues and features associated with container image building

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions