Skip to content

Enable core dump support from ITF - #697

Draft
NicolasFussberger wants to merge 3 commits into
eclipse-score:mainfrom
etas-contrib:core_dump_support
Draft

Enable core dump support from ITF#697
NicolasFussberger wants to merge 3 commits into
eclipse-score:mainfrom
etas-contrib:core_dump_support

Conversation

@NicolasFussberger

@NicolasFussberger NicolasFussberger commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

ITF adds the support for downloading core dumps.

However, the setup for core dumps is still for qemu / docker inside the module repo.

TimoSteuerwaldETAS
TimoSteuerwaldETAS reviewed Sep 24, 2026
View reviewed changes
Comment thread tests/README.md Outdated

For Linux, you'll have to set the core pattern on your **host** machine as this is shared to the docker container:

`sudo sysctl -w kernel.core_pattern="/tmp/core.%e.%p"` No newline at end of file

@TimoSteuerwaldETAS TimoSteuerwaldETAS Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to do this in automatic behavior. 😍

Save actual value to hidden file in repo (add this file to .gitignore), update value to for docker environment, execute tests and revert it again and delete file. If the execution of the tests is aborted via ctrl+c, the file still exists, so value is not needed to be updated on the next start.

What do you think?

If not, we should at least note, that the old value should be stored somewhere, to be able to restore the correct value later.

@NicolasFussberger NicolasFussberger Sep 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if this would work. I think you'll need a privileged container in the first place to adjust this host setting. On the pipeline this is probably not possible and it would influence other jobs running on the same VM/Host.
You would additionally need to create a pipeline-sepcific config to turn this off (core dumps are anyone not stored in the pipeline right now AFAIK).

@TimoSteuerwaldETAS TimoSteuerwaldETAS Sep 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but please move description to tests/integration/readme.md.

@NicolasFussberger NicolasFussberger Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@TimoSteuerwaldETAS

TimoSteuerwaldETAS commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Doesn't seem to work.

What I did
Added std::abort() in tests/integration/smoke/gtest_process.cpp:24

sudo sysctl -w kernel.core_pattern="/tmp/core.%e.%p"
bazel test --config=x86_64-linux //tests/integration/smoke:smoke

Got this output:

.
.
.
FAILED                                                                   [100%]
------------------------------ live log teardown -------------------------------
[2026-09-25 13:55:07.339] [INFO] [score.itf.plugins.core_dump] Core dump extraction enabled, writing to /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-25 13:55:07.339] [INFO] [score.itf.plugins.core_dump] Attempting core dump extraction to /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-25 13:55:07.365] [INFO] [score.itf.plugins.core_dump] Found 0 core files: []


=================================== FAILURES ===================================
__________________________________ test_smoke __________________________________

target = <score.itf.plugins.docker.DockerTarget object at 0x7761e7f12450>
setup_test = None
assert_test_results = <function assert_test_results.<locals>._assert at 0x7761e7eb11c0>
remote_test_dir = PosixPath('/tmp/tests/smoke')

    @add_test_properties(
        partially_verifies=[],
        test_type="interface-test",
        derivation_technique="explorative-testing",
    )
    def test_smoke(target, setup_test, assert_test_results, remote_test_dir):
        """
        Objective: Verifies the basic end-to-end lifecycle flow of the launch manager, including process startup, run target transitions, and execution state reporting.
    
        The launch manager starts with an initial run target. The control daemon activates the "Running" run target (starting the managed process), then transitions back to "Startup", and finally activates "Off".
        Expected Behaviour: All run target transitions complete successfully and all processes report running.
        """
    
>       run_test(
            target=target,
            binary_path=str(remote_test_dir / "launch_manager"),
            args=["-c", str(remote_test_dir / "etc/lifecycle_smoketest.bin")],
            cwd=str(remote_test_dir),
        )

tests/integration/smoke/smoke.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils/testing_utils/run_test.py:32: in run_test
    assert process.wait(timeout) == 0
           ^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <score.itf.plugins.docker.DockerAsyncProcess object at 0x7761e7f8fd10>
timeout_s = 15

    def wait(self, timeout_s: float = 15) -> int:
        """Block until the command finishes or *timeout_s* elapses.
    
        :param timeout_s: maximum seconds to wait.
        :return: exit code of the command.
        :raises RuntimeError: on timeout.
        """
        start_time = time.time()
        while self.is_running():
            if time.time() - start_time > timeout_s:
>               raise RuntimeError(
                    f"Waiting for process with PID [{self._pid}] to terminate timed out after {timeout_s} seconds"
                )
E               RuntimeError: Waiting for process with PID [22] to terminate timed out after 15 seconds

../score_itf+/score/itf/plugins/docker.py:101: RuntimeError
=============================== warnings summary ===============================
::test_smoke
  ../_main/tests/integration/smoke/smoke.py:19: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
    @add_test_properties(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.xml -
=========================== short test summary info ============================
FAILED ../score_itf+::test_smoke - RuntimeError: Waiting for process with PID...
======================== 1 failed, 1 warning in 16.69s =========================
================================================================================
INFO: Found 1 test target...
Target //tests/integration/smoke:smoke up-to-date:
  bazel-bin/tests/integration/smoke/smoke
INFO: Elapsed time: 19.261s, Critical Path: 18.59s
INFO: 5 processes: 1117 action cache hit, 1 internal, 4 linux-sandbox.
INFO: Build completed, 1 test FAILED, 5 total actions
//tests/integration/smoke:smoke                                          FAILED in 17.4s
  /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.log

Executed 1 out of 1 test: 1 fails locally.

@NicolasFussberger

NicolasFussberger commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Doesn't seem to work.

What I did Added std::abort() in tests/integration/smoke/gtest_process.cpp:24

sudo sysctl -w kernel.core_pattern="/tmp/core.%e.%p"
bazel test --config=x86_64-linux //tests/integration/smoke:smoke

Got this output:

.
.
.
FAILED                                                                   [100%]
------------------------------ live log teardown -------------------------------
[2026-09-25 13:55:07.339] [INFO] [score.itf.plugins.core_dump] Core dump extraction enabled, writing to /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-25 13:55:07.339] [INFO] [score.itf.plugins.core_dump] Attempting core dump extraction to /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-25 13:55:07.365] [INFO] [score.itf.plugins.core_dump] Found 0 core files: []


=================================== FAILURES ===================================
__________________________________ test_smoke __________________________________

target = <score.itf.plugins.docker.DockerTarget object at 0x7761e7f12450>
setup_test = None
assert_test_results = <function assert_test_results.<locals>._assert at 0x7761e7eb11c0>
remote_test_dir = PosixPath('/tmp/tests/smoke')

    @add_test_properties(
        partially_verifies=[],
        test_type="interface-test",
        derivation_technique="explorative-testing",
    )
    def test_smoke(target, setup_test, assert_test_results, remote_test_dir):
        """
        Objective: Verifies the basic end-to-end lifecycle flow of the launch manager, including process startup, run target transitions, and execution state reporting.
    
        The launch manager starts with an initial run target. The control daemon activates the "Running" run target (starting the managed process), then transitions back to "Startup", and finally activates "Off".
        Expected Behaviour: All run target transitions complete successfully and all processes report running.
        """
    
>       run_test(
            target=target,
            binary_path=str(remote_test_dir / "launch_manager"),
            args=["-c", str(remote_test_dir / "etc/lifecycle_smoketest.bin")],
            cwd=str(remote_test_dir),
        )

tests/integration/smoke/smoke.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/utils/testing_utils/run_test.py:32: in run_test
    assert process.wait(timeout) == 0
           ^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <score.itf.plugins.docker.DockerAsyncProcess object at 0x7761e7f8fd10>
timeout_s = 15

    def wait(self, timeout_s: float = 15) -> int:
        """Block until the command finishes or *timeout_s* elapses.
    
        :param timeout_s: maximum seconds to wait.
        :return: exit code of the command.
        :raises RuntimeError: on timeout.
        """
        start_time = time.time()
        while self.is_running():
            if time.time() - start_time > timeout_s:
>               raise RuntimeError(
                    f"Waiting for process with PID [{self._pid}] to terminate timed out after {timeout_s} seconds"
                )
E               RuntimeError: Waiting for process with PID [22] to terminate timed out after 15 seconds

../score_itf+/score/itf/plugins/docker.py:101: RuntimeError
=============================== warnings summary ===============================
::test_smoke
  ../_main/tests/integration/smoke/smoke.py:19: PytestExperimentalApiWarning: record_xml_attribute is an experimental feature
    @add_test_properties(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/sandbox/linux-sandbox/1137/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.xml -
=========================== short test summary info ============================
FAILED ../score_itf+::test_smoke - RuntimeError: Waiting for process with PID...
======================== 1 failed, 1 warning in 16.69s =========================
================================================================================
INFO: Found 1 test target...
Target //tests/integration/smoke:smoke up-to-date:
  bazel-bin/tests/integration/smoke/smoke
INFO: Elapsed time: 19.261s, Critical Path: 18.59s
INFO: 5 processes: 1117 action cache hit, 1 internal, 4 linux-sandbox.
INFO: Build completed, 1 test FAILED, 5 total actions
//tests/integration/smoke:smoke                                          FAILED in 17.4s
  /home/stt1rlz/.cache/bazel/_bazel_stt1rlz/219f19d5e240f8577853642c2a627748/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.log

Executed 1 out of 1 test: 1 fails locally.

Must be some additional host setting then that we need to consider additionally.
I tried the exact same use case for x86_64-linux config and worked locally (devcontainer):

[2026-09-30 14:03:54.174] [INFO] [score.itf.plugins.core_dump] Core dump extraction enabled, writing to /var/cache/bazel/625205d1e0aa59ca514fd964120d6a8a/sandbox/linux-sandbox/1131/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-30 14:03:54.174] [INFO] [score.itf.plugins.core_dump] Attempting core dump extraction to /var/cache/bazel/625205d1e0aa59ca514fd964120d6a8a/sandbox/linux-sandbox/1131/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps
[2026-09-30 14:03:54.209] [INFO] [score.itf.plugins.core_dump] Found 1 core files: ['/tmp/core.gtest_process.55']
[2026-09-30 14:03:54.209] [INFO] [score.itf.plugins.core_dump] Extracting core dump from /tmp/core.gtest_process.55 to /var/cache/bazel/625205d1e0aa59ca514fd964120d6a8a/sandbox/linux-sandbox/1131/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/integration/smoke/smoke/test.outputs/coredumps/tmp/core.gtest_process.55
[2026-09-30 14:03:54.214] [INFO] [score.itf.plugins.core_dump] Successfully extracted /tmp/core.gtest_process.55

This branch is waiting to be deployed

1 waiting deployment
workflow-approval — fdb8c036 Waiting Oct 1, 2026 by NicolasFussberger via Build and test unit-tests-arm64-qnx / approval #1152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pawelrutkaq pawelrutkaq Awaiting requested review from pawelrutkaq pawelrutkaq will be requested when the pull request is marked ready for review pawelrutkaq is a code owner
@ramceb ramceb Awaiting requested review from ramceb ramceb will be requested when the pull request is marked ready for review ramceb is a code owner
@MaciejKaszynski MaciejKaszynski Awaiting requested review from MaciejKaszynski MaciejKaszynski will be requested when the pull request is marked ready for review MaciejKaszynski is a code owner
@FScholPer FScholPer Awaiting requested review from FScholPer FScholPer will be requested when the pull request is marked ready for review FScholPer is a code owner
@antonkri antonkri Awaiting requested review from antonkri antonkri will be requested when the pull request is marked ready for review antonkri is a code owner
@anmittag anmittag Awaiting requested review from anmittag anmittag will be requested when the pull request is marked ready for review anmittag is a code owner
1 more reviewer
@TimoSteuerwaldETAS TimoSteuerwaldETAS TimoSteuerwaldETAS requested changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

LCM - Lifecycle & Health FT
Status: Backlog

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NicolasFussberger @TimoSteuerwaldETAS