Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 28 additions & 12 deletions feature_integration_tests/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Feature Integration Tests

This directory contains Feature Integration Tests for the S-CORE project. It includes both Python test cases and Rust test scenarios to validate features work together.
This directory contains Feature Integration Tests for the S-CORE project. It includes Python test cases that orchestrate test scenarios implemented in Rust and C++ to validate that features work together correctly.

## Structure

Expand All @@ -9,32 +9,48 @@ This directory contains Feature Integration Tests for the S-CORE project. It inc
- `fit_scenario.py` — Base scenario class
- `requirements.txt` — Python dependencies
- `BUILD` — Bazel build and test definitions
- `tests/` — Test cases (e.g., orchestration with persistency)
- `test_scenarios/` — Location of test scenarios
- `rust/` — Rust-based integration test scenarios
- `src/` — Rust source code for test scenarios
- `BUILD` — Bazel build definitions
- `tests/` — Test cases organized by feature area
- `test_scenarios/` — Test scenario implementations
- `rust/` — Rust-based test scenarios
- `cpp/` — C++-based test scenarios
- `itf/` — Integration Test Framework tests (run on QEMU targets)
- `test_showcases.py` — Showcase validation tests
- `test_remote_logging.py` — Remote logging tests
- `test_ssh.py` — SSH connectivity tests
- `configs/` — Configuration files for ITF execution (DLT, QEMU bridge, etc.)

## Running Tests

### Python Test Cases
### Python Test Cases (scenario-based FIT)

Python tests are managed with Bazel and Pytest. To run the main test target:
Python tests are managed with Bazel and Pytest. To run all integration tests:

```sh
bazel test //feature_integration_tests/test_cases:fit
bazel test --config=linux-x86_64 //feature_integration_tests/test_cases:fit
```

### Rust Test Scenarios
To run specific test suites:

Rust test scenarios are defined in `test_scenarios/rust/src/scenarios`. Build and run them using Bazel:
```sh
bazel test //feature_integration_tests/test_cases:fit_rust
bazel test --config=linux-x86_64 //feature_integration_tests/test_cases:fit_cpp
```

### ITF Tests (QEMU-based)

ITF tests run on a QEMU target and require the `itf-qnx-x86_64` config:

```sh
bazel build //feature_integration_tests/test_scenarios/rust:rust_test_scenarios
bazel test --config=itf-qnx-x86_64 //feature_integration_tests/itf
```

### Test Scenarios

Test scenarios can be listed and run directly for debugging:

```sh
bazel run //feature_integration_tests/test_scenarios/rust:rust_test_scenarios -- --list-scenarios
bazel run --config=linux-x86_64 //feature_integration_tests/test_scenarios/cpp:cpp_test_scenarios -- --list-scenarios
```

## Updating Python Requirements
Expand Down
39 changes: 34 additions & 5 deletions feature_integration_tests/test_cases/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ load("@rules_python//python:pip.bzl", "compile_pip_requirements")
load("@score_tooling//python_basics:defs.bzl", "score_py_pytest")

# In order to update the requirements, change the `requirements.txt` file and run:
# `bazel run //feature_integration_tests/python_test_cases:requirements.update`.
# `bazel run //feature_integration_tests/test_cases:requirements.update`.
# This will update the `requirements.txt.lock` file.
# To upgrade all dependencies to their latest versions, run:
# `bazel run //feature_integration_tests/python_test_cases:requirements.update -- --upgrade`.
# `bazel run //feature_integration_tests/test_cases:requirements.update -- --upgrade`.
compile_pip_requirements(
name = "requirements",
srcs = [
Expand All @@ -36,10 +36,10 @@ compile_pip_requirements(

# Tests targets
score_py_pytest(
name = "fit",
srcs = glob(["tests/**/*.py"]) + [
],
name = "fit_rust",
srcs = glob(["tests/**/*.py"]),
args = [
"-m rust",
"--traces=all",
"--rust-target-path=$(rootpath //feature_integration_tests/test_scenarios/rust:rust_test_scenarios)",
],
Expand All @@ -55,3 +55,32 @@ score_py_pytest(
pytest_config = "//:pyproject.toml",
deps = all_requirements,
)

score_py_pytest(
name = "fit_cpp",
srcs = glob(["tests/**/*.py"]),
args = [
"-m cpp",
"--traces=all",
"--cpp-target-path=$(rootpath //feature_integration_tests/test_scenarios/cpp:cpp_test_scenarios)",
],
data = [
"conftest.py",
"fit_scenario.py",
"test_properties.py",
"//feature_integration_tests/test_scenarios/cpp:cpp_test_scenarios",
],
env = {
"RUST_BACKTRACE": "1",
},
pytest_config = "//:pyproject.toml",
deps = all_requirements,
)

test_suite(
name = "fit",
tests = [
":fit_cpp",
":fit_rust",
],
)
37 changes: 32 additions & 5 deletions feature_integration_tests/test_cases/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@
import pytest
from testing_utils import BazelTools

FAILED_CONFIGS = []


# Cmdline options
def pytest_addoption(parser):
Expand All @@ -33,14 +31,25 @@ def pytest_addoption(parser):
parser.addoption(
"--rust-target-name",
type=str,
default="//feature_integration_tests/rust_test_scenarios:rust_test_scenarios",
default="//feature_integration_tests/test_scenarios/rust:rust_test_scenarios",
help="Rust test scenario executable target.",
)
parser.addoption(
"--rust-target-path",
type=Path,
help="Rust test scenario executable target.",
)
parser.addoption(
"--cpp-target-name",
type=str,
default="//feature_integration_tests/test_scenarios/cpp:cpp_test_scenarios",
help="C++ test scenario executable target.",
)
parser.addoption(
"--cpp-target-path",
type=Path,
help="C++ test scenario executable target.",
)
parser.addoption(
"--build-scenarios",
action="store_true",
Expand All @@ -61,6 +70,18 @@ def pytest_addoption(parser):


# Hooks
def pytest_collection_modifyitems(items: list[pytest.Function]):
for item in items:
# Automatically mark tests parametrized with 'version' as 'cpp' or 'rust'.
# This allows -m cpp / -m rust to select the correct variant in each target.
if hasattr(item, "callspec") and "version" in getattr(item.callspec, "params", {}):
version = item.callspec.params["version"]
if version == "cpp":
item.add_marker(pytest.mark.cpp)
elif version == "rust":
item.add_marker(pytest.mark.rust)


@pytest.hookimpl(tryfirst=True)
def pytest_sessionstart(session):
try:
Expand All @@ -70,9 +91,15 @@ def pytest_sessionstart(session):

# Build Rust test scenarios.
print("Building Rust test scenarios executable...")
cargo_tools = BazelTools(option_prefix="rust", build_timeout=build_timeout)
rust_tools = BazelTools(option_prefix="rust", build_timeout=build_timeout)
rust_target_name = session.config.getoption("--rust-target-name")
cargo_tools.build(rust_target_name)
rust_tools.build(rust_target_name)

# Build C++ test scenarios.
print("Building C++ test scenarios executable...")
cpp_tools = BazelTools(option_prefix="cpp", build_timeout=build_timeout)
cpp_target_name = session.config.getoption("--cpp-target-name")
cpp_tools.build(cpp_target_name)

except Exception as e:
pytest.exit(str(e), returncode=1)
12 changes: 8 additions & 4 deletions feature_integration_tests/test_cases/fit_scenario.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
# SPDX-License-Identifier: Apache-2.0
# *******************************************************************************
import shutil
from collections.abc import Generator
from pathlib import Path
from typing import Generator

import pytest
from testing_utils import (
Expand Down Expand Up @@ -68,7 +68,10 @@ class FitScenario(Scenario):
"""

@pytest.fixture(scope="class")
def build_tools(self) -> BuildTools:
def build_tools(self, request: pytest.FixtureRequest) -> BuildTools:
# Use test parametrization when available; default to rust for non-parametrized tests.
if "version" in request.fixturenames:
return BazelTools(option_prefix=request.getfixturevalue("version"))
return BazelTools(option_prefix="rust")

def expect_command_failure(self, *args, **kwargs) -> bool:
Expand All @@ -87,9 +90,10 @@ def results(
) -> ScenarioResult:
result = self._run_command(command, execution_timeout, args, kwargs)
success = result.return_code == ResultCode.SUCCESS and not result.hang
if self.expect_command_failure() and success:
expect_failure = self.expect_command_failure()
if expect_failure and success:
raise RuntimeError(f"Command execution succeeded unexpectedly: {result=}")
if not self.expect_command_failure() and not success:
if not expect_failure and not success:
raise RuntimeError(f"Command execution failed unexpectedly: {result=}")
return result

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# *******************************************************************************
# Copyright (c) 2025 Contributors to the Eclipse Foundation
# Copyright (c) 2026 Contributors to the Eclipse Foundation
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
Expand All @@ -12,14 +12,17 @@
# *******************************************************************************

import json
from collections.abc import Generator
from pathlib import Path
from typing import Any, Generator
from typing import Any

import pytest
from fit_scenario import FitScenario, temp_dir_common
from test_properties import add_test_properties
from testing_utils import LogContainer

pytestmark = pytest.mark.parametrize("version", ["rust", "cpp"], scope="class")


@add_test_properties(
partially_verifies=["feat_req__persistency__multiple_kvs"],
Expand Down Expand Up @@ -51,8 +54,9 @@ def kvs_value_2(self) -> float:
def temp_dir(
self,
tmp_path_factory: pytest.TempPathFactory,
version: str,
) -> Generator[Path, None, None]:
yield from temp_dir_common(tmp_path_factory, self.__class__.__name__)
yield from temp_dir_common(tmp_path_factory, self.__class__.__name__, version)

@pytest.fixture(scope="class")
def test_config(
Expand Down
31 changes: 31 additions & 0 deletions feature_integration_tests/test_scenarios/cpp/BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# *******************************************************************************
# Copyright (c) 2026 Contributors to the Eclipse Foundation
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Apache License Version 2.0 which is available at
# https://www.apache.org/licenses/LICENSE-2.0
#
# SPDX-License-Identifier: Apache-2.0
# *******************************************************************************

load("@rules_cc//cc:defs.bzl", "cc_binary")

cc_binary(
name = "cpp_test_scenarios",
srcs = glob([
"src/**/*.cpp",
"src/**/*.h",
]),
includes = ["src"],
tags = ["manual"],
visibility = ["//visibility:public"],
deps = [
"@score_baselibs//score/json",
"@score_baselibs//score/mw/log:backend_stub_testutil",
"@score_persistency//:kvs_cpp",
"@score_test_scenarios//test_scenarios_cpp",
],
)
Loading
Loading