feat: End to end performance tests by lurtz · Pull Request #257 · eclipse-score/inc_someip_gateway

lurtz · 2026-08-20T14:29:56Z

We need performance tests and profiling to figure out where we can improve the performance. This is only the first step and I would consider this far from being perfect.

I reworked the existing performance tests and made them work using bazel. We can also optionally run them using perf to create flamegraphs.

Having worked on the tests and seeing their CPU usage, I think there is room for improvement. I was expecting my CPU to be tortured, but it did not happen. I also see that dropped messages seem to be counted as delivered by the performance tests. I refrain from changing the C++ code too much to keep the PR at a reasonable size. Reworking the C++ code is a followup.

Closes #251

…e-tests

github-actions · 2026-08-20T14:31:17Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: 6cb6d690-db41-4944-a043-e9d89ab3be15
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 2 packages loaded
WARNING: Target pattern parsing failed.
ERROR: Skipping '//:license-check': no such target '//:license-check': target 'license-check' not declared in package '' defined by /home/runner/work/inc_someip_gateway/inc_someip_gateway/BUILD
ERROR: no such target '//:license-check': target 'license-check' not declared in package '' defined by /home/runner/work/inc_someip_gateway/inc_someip_gateway/BUILD
INFO: Elapsed time: 15.207s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

github-actions · 2026-08-20T14:37:40Z

Documentation preview for this pull request is available at:
pr-257: https://eclipse-score.github.io/inc_someip_gateway/pr-257/

…e-tests

github-actions · 2026-08-21T10:09:15Z

Coverage report for this pull request:

Status: success (workflow run)
Coverage data: coverage-report

The archive contains the HTML report, the raw LCOV data and the JUnit XML test results.

…e-tests

…nstalled

TimofteBogdan1900 · 2026-08-31T14:00:46Z

+
+The `.perf.data` recordings and `.perf.svg` flamegraphs are written beside `benchmarks.json`.
+
+The test starts and stops both daemon pairs and the echo server itself. The client and server use


if we use linux namespaces and this shouldn't be possible since the 3 processes are separated from the other 3 processes by the linux os

Linux Namespace is only used to separate tests from host environment and to have predefined environment (e.g. ports are available). All applications are running inside the same namespace.

I did not want to build too much on Linux Namespaces, because I still want to keep some possibility to launch the tests without bazel. I will consider if something can be done inside the Python Code.

my personal opninion i dont like not having this isolation between the 2 network stacks . if it was separate namespaces then we could have passed 1 mwcom json with the same 2 ids to both client and ipc benchmark app . now we need 4 service ids 7200,7203 ,7201, 7202 and 2 diff mwcom json files

I assumed this is the simpler solution. If we want to reuse the mwcom.json we need to put the benchmark processes and echo_server processes into separate Linux namespaces. I hope it works to connect both with a network bridge. This also gets closer to the docker-compose setup I am just removing.

IMHO the biggest benefit is that the current way is quite flexible. You can run on the same machine without containers or with containers. I assume it also works on real hardware or using QEMU. There is a reason I prefer to start with running everything on the same machine: I can then profile everything at once and see issues.

Global profiling is not implemented yet and I wonder if in this PR I should actually switch from per application profiling to global profiling.

TimofteBogdan1900 · 2026-08-31T14:15:09Z

+      - name: Run Linux perf tests
+        run: |
+          bazel test --config=perf-tests //tests/benchmarks/... --build_tests_only
+      - name: Run Linux perf tests (flamegraphs)


why do we need to run these performance tests on the gh pipeline ? only to see that to have some graphs ? the results cannot be used since this is not running on a real hardware

main purpose is that they do not bitrot and start to fail without us knowing. Github CI is I guess the least best place to get actual performance numbers from.

TimofteBogdan1900 · 2026-09-01T15:05:29Z

              << " -c/--configuration Specifies the configuration file\n"
+              << " -i/--ipc_channel Name of the IPC channel to gatewayd (default: "
+              << score::someip::kDefaultIpcChannelName << ")\n"
              << "\n";


both apps already have names (ipc_benchmarks and echo_server) so i propose meaningfull names instead of a and b

config/node_a_mw_com_config.json → config/bench_mw_com_config.json

config/node_b_mw_com_config.json → config/echo_mw_com_config.json

config/node_a_someip_config.json → config/bench_someip_config.json

config/node_b_someip_config.json → config/echo_someip_config.json

config/vsomeip_node_a.json → config/vsomeip_bench.json

config/vsomeip_node_b.json → config/vsomeip_echo.json

TimofteBogdan1900 · 2026-09-01T15:10:32Z

+NODE_A = NodeSpec(
+    name="benchmark_node_a",
+    ipc_channel="benchmark_ipc_a",
+    vsomeip_network="benchmark-node-a",


Could the nodes be named after the app each one hosts instead of a/b? NODE_A -> BENCH_NODE (runs ipc_benchmarks), NODE_B -> ECHO_NODE (runs echo_server), with name="bench" / "echo" so a failed run leaves bench_gatewayd.log rather than benchmark_node_a_gatewayd.log. Same for NODE_A_IP / NODE_B_IP in setup_network.sh.

TimofteBogdan1900 · 2026-09-01T15:13:15Z

+NODE_A_IP="127.0.0.2"
+NODE_B_IP="127.0.0.3"


NODE_A_IP / NODE_B_IP -> BENCH_IP / ECHO_IP if we go with the node rename.

TimofteBogdan1900 · 2026-09-01T15:13:16Z

+generate_someip_config_bin(
+    name = "node_a_config_file",
+    json = "config/node_a_someip_config.json",
+    output = "node_a_someip_config.bin",
+)
+
+generate_someip_config_bin(
+    name = "node_b_config_file",
+    json = "config/node_b_someip_config.json",
+    output = "node_b_someip_config.bin",
+)


These should follow the rename too: node_a_config_file -> bench_config_file, config/node_a_someip_config.json -> config/bench_someip_config.json, same for the _b / mw_com ones.

TimofteBogdan1900 · 2026-09-01T15:13:17Z

+)
+
 validate_json_schema_test(
    name = "benchmark_mw_com_config_schema_valid",


Nothing loads config/benchmark_mw_com_config.json any more — Drop it, the JSON, and the now-unused load on line 18?

replaced with validation of mw_com config for bench and echo node

TimofteBogdan1900 · 2026-09-01T16:00:17Z

+`e2e_benchmarks` runs the existing Google Benchmark client and echo server over the complete
+mw::com -> gatewayd -> someipd -> SOME/IP -> someipd -> gatewayd -> mw::com path. Both nodes run
+in the isolated network namespace configured by `tests/benchmarks/setup_network.sh`, at
+`127.0.0.2` and `127.0.0.3`.


The two throughput benchmarks looks to me they only write to a shm as fast as possible ?!
request_event.Send(std::move(pre_serialized_request));
, just send in a loop and never wait, so their messages/sec is how fast the client can write to shm

Can we say here which benchmark measures what? Otherwise that number ends up being quoted as SOME/IP traffic ?

I will change the logic. I did not look too much into the C++ applications because I assumed they would already be fine. However the more I look into it, it looks like fire+forget without checking if anything has been received. I was working under the assumption that the ipc_benchmark.cpp code checks the echoed data.

Clarification: The LatencyEcho test does that. The other are just fire and forget. I will change profiling to the LatencyEcho test.

TimofteBogdan1900 · 2026-09-01T16:23:33Z

+        benchmark_cmd = _perf_record_argv(benchmark_perf_data, benchmark_cmd)
+
+    try:
+        with node_a, node_b:


i get a passed test even with side b missing completely
xxx@yyyy:score_inc_someip_gateway$ bazel test --config=perf-tests --nocache_test_results //tests/benchmarks:e2e_benchmarks --test_output=summary
WARNING: Option 'local_ram_resources' is deprecated: --local_ram_resources is deprecated, please use --local_resources=memory= instead.
WARNING: Option 'local_cpu_resources' is deprecated: --local_cpu_resources is deprecated, please use --local_resources=cpu= instead.
WARNING: Option 'local_ram_resources' is deprecated: --local_ram_resources is deprecated, please use --local_resources=memory= instead.
WARNING: Option 'local_cpu_resources' is deprecated: --local_cpu_resources is deprecated, please use --local_resources=cpu= instead.
INFO: Analyzed target //tests/benchmarks:e2e_benchmarks (0 packages loaded, 13 targets configured).
INFO: Found 1 test target...
Target //tests/benchmarks:e2e_benchmarks up-to-date:
bazel-bin/tests/benchmarks/e2e_benchmarks
INFO: Elapsed time: 44.232s, Critical Path: 42.72s
INFO: 2 processes: 11 action cache hit, 2 linux-sandbox.
INFO: Build completed successfully, 2 total actions
//tests/benchmarks:e2e_benchmarks PASSED in 42.5s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

I will look into it

benchmark.cpp's main function always returned 0 / EXIT_SUCCESS. I fixed that now.

TimofteBogdan1900 · 2026-09-01T16:29:47Z

LGTM to me as a first try .i am done and submitted my review

…e-tests

The stress test will lose too much messages. I guess we need a better test, which does better rate limiting

…e-tests

TimofteBogdan1900 · 2026-09-03T11:52:38Z

@@ -20,15 +20,16 @@ ARG USERNAME=vscode
 # RENAME 'vscode' user to the specified username to satisfy the QNX license
 # We use usermod to change the login name and home directory


LGTM all my comments were adressed

lurtz added 6 commits August 19, 2026 15:10

make IPC address configurable

e39ffb2

end to end performance tests

c1ff53a

now with sandbox support

5dc5eda

simplified network_setup.sh

1a68c9d

remove manual label

c331ed9

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

688042b

…e-tests

lurtz added 3 commits August 20, 2026 14:56

document test result directory

ab278eb

add larger payload sizes to test

0713a7b

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

d2bbb49

…e-tests

lurtz added 8 commits August 21, 2026 11:14

first flamegraphs

4eaa28b

download flamegraph scripts instead of copy

46bae7b

also run tests/benchmarks via bazel

7370bb5

remove so much excercised intergration test code

0127746

reduce duplication

e7464bf

delete sloped code

a0c6712

move remaining content of e2e_perf to benchmarks

2682944

remove nodes export

416d216

lurtz commented Aug 21, 2026

View reviewed changes

Comment thread MODULE.bazel

lurtz added 8 commits August 25, 2026 10:22

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

8510ea0

…e-tests

remove obsolete code

688246a

Readd generic flamegraph creation

335db1e

install linux-tools

4005c37

add profiling of stress test

fda4d9c

do not fail devcontainer creation if matching linux-tools cannot be i…

66a1cd1

…nstalled

one configuration variant less

c9ec1a2

check in CI that perf tests stay working

7ee158a

lurtz requested a lite review from Copilot August 25, 2026 19:31

lurtz added 3 commits August 25, 2026 19:46

fix REUSE.toml

80d62bb

fix review findings

10d0e4a

allow perf in CI job

5aba3ef

lurtz marked this pull request as ready for review August 25, 2026 20:38

TimofteBogdan1900 self-requested a review August 31, 2026 13:48

TimofteBogdan1900 reviewed Aug 31, 2026

View reviewed changes

TimofteBogdan1900 reviewed Sep 1, 2026

View reviewed changes

lurtz added 14 commits September 2, 2026 13:18

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

31f1ef7

…e-tests

make rules_shell a dev_dependency

f552810

make surge command line arguments are terminated with nullptr

5a04e8b

format code

96b9a01

rename node a to bench and b to echo

91860a7

better use latency echo test for profiling

4ea07de

The stress test will lose too much messages. I guess we need a better test, which does better rate limiting

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

c8ed268

…e-tests

let benchmark process fail when it detects errors

e3a3ac5

little python simplification

d3daaed

encapsulate flamegraph creation with class

179c9ea

check if any test used SkipWithError()

71d3e27

Merge remote-tracking branch 'origin/main' into end-to-end-performanc…

118522d

…e-tests

kill processes using flamegraph manager

2cd4d92

move configuration files into subdirectories

b3fa0fc

TimofteBogdan1900 approved these changes Sep 3, 2026

View reviewed changes

lurtz enabled auto-merge September 3, 2026 11:53

opajonk approved these changes Sep 3, 2026

View reviewed changes

lurtz added this pull request to the merge queue Sep 3, 2026

Merged via the queue into eclipse-score:main with commit f701652 Sep 3, 2026
27 checks passed

lurtz deleted the end-to-end-performance-tests branch September 3, 2026 12:38


		The `.perf.data` recordings and `.perf.svg` flamegraphs are written beside `benchmarks.json`.

		The test starts and stops both daemon pairs and the echo server itself. The client and server use

		@@ -20,15 +20,16 @@ ARG USERNAME=vscode
		# RENAME 'vscode' user to the specified username to satisfy the QNX license
		# We use usermod to change the login name and home directory

		NODE_A_IP="127.0.0.2"
		NODE_B_IP="127.0.0.3"

Conversation

lurtz commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

TimofteBogdan1900 Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimofteBogdan1900 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimofteBogdan1900 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lurtz Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimofteBogdan1900 commented Sep 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lurtz commented Aug 20, 2026 •

edited

Loading

github-actions Bot commented Aug 20, 2026 •

edited

Loading

github-actions Bot commented Aug 21, 2026 •

edited

Loading

TimofteBogdan1900 Aug 31, 2026 •

edited

Loading

TimofteBogdan1900 Sep 1, 2026 •

edited

Loading

TimofteBogdan1900 Sep 1, 2026 •

edited

Loading

lurtz Sep 2, 2026 •

edited

Loading