Skip to content

Isolate celery tests to separate container #50952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
May 22, 2025
Merged

Isolate celery tests to separate container #50952

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/run-unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ permissions:
contents: read
jobs:
tests:
timeout-minutes: 120
timeout-minutes: 35
Comment thread
potiuk marked this conversation as resolved.
Resolved
name: "\
${{ inputs.test-scope }}-${{ inputs.test-group }}:\
${{ inputs.test-name }}${{ inputs.test-name-separator }}${{ matrix.backend-version }}:\
Expand Down
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_testing_core-tests.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
e17e8722da2c099cb96f37dc51c279e0
de2875b9ee563468daf60d78527ff40c
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_testing_providers-tests.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
e38e20ca833d2fc03b5f6a8fb746f47c
71fa5f575e7628b431cb71a6ef749e00
5 changes: 3 additions & 2 deletions dev/breeze/src/airflow_breeze/commands/testing_commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@

GRACE_CONTAINER_STOP_TIMEOUT = 10 # Timeout in seconds to wait for containers to get killed

LOW_MEMORY_CONDITION = 8 * 1024 * 1024 * 1024
DEFAULT_TOTAL_TEST_TIMEOUT = 6500 # 6500 seconds = 1h 48 minutes
LOW_MEMORY_CONDITION = 8 * 1024 * 1024 * 1024 # 8 GB
DEFAULT_TOTAL_TEST_TIMEOUT = 30 * 60 # 30 minutes

logs_already_dumped = False

Expand Down Expand Up @@ -327,6 +327,7 @@ def _run_tests_in_pool(
"CLI",
"Serialization",
"Always",
"Providers[celery]",
Comment thread
potiuk marked this conversation as resolved.
Resolved
]
sort_key = {item: i for i, item in enumerate(sorting_order)}
# Put the test types in the order we want them to run
Expand Down
6 changes: 5 additions & 1 deletion dev/breeze/src/airflow_breeze/utils/selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,8 +965,12 @@ def _extract_long_provider_tests(current_test_types: set[str]):
In case of Providers, we need to replace it with Providers[-<list_of_long_tests>], but
in case of Providers[list_of_tests] we need to remove the long tests from the list.

In case of celery tests we want to isolate them from the rest, because they seem to be hanging
infrequently when running together with other tests

:param current_test_types: The set of test types to run
"""
long_tests = ["amazon", "google", "standard"]
long_tests = ["amazon", "celery", "google", "standard"]
for original_test_type in tuple(current_test_types):
if original_test_type == "Providers":
current_test_types.remove(original_test_type)
Expand Down
6 changes: 3 additions & 3 deletions dev/breeze/tests/test_selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,9 @@
ALL_PROVIDERS_SELECTIVE_TEST_TYPES_AS_JSON = json.dumps(
[
{
"description": "-amazon,googl...standard",
"test_types": "Providers[-amazon,google,standard] "
"Providers[amazon] Providers[google] Providers[standard]",
"description": "-amazon,celer...standard",
"test_types": "Providers[-amazon,celery,google,standard] "
"Providers[amazon] Providers[celery] Providers[google] Providers[standard]",
}
]
)
Expand Down