Repository navigation
[codex] Add native Rust telemetry ingest pipeline - #38
Merged
Merged
Conversation
Ingest Rust ResultsCommit: cargo testCriterion benchmarks |
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark Compared To
|
| Metric | main | PR | Delta vs main |
|---|---|---|---|
| Requests/sec | 12089.91 | 2018.58 | -83.3% worse |
| Rows/sec | 120899.14 | 20185.80 | -83.3% worse |
| p50 latency | 4.86 ms | 31.06 ms | +539.4% worse |
| p95 latency | 7.93 ms | 35.11 ms | +342.7% worse |
| p99 latency | 10.87 ms | 37.87 ms | +248.3% worse |
| Export catch-up | 0.00 s | 0.03 s | +0.03 absolute worse |
| Max RSS | 31.42 MiB | 92.64 MiB | +194.8% worse |
| Max CPU | 0.00% | 71.40% | baseline was 0 |
| Avg CPU | 0.00% | 44.00% | baseline was 0 |
| Failures | 0 | 0 | same |
Benchmark setup: the same OTLP protobuf log workload is sent to both binaries. main runs the collector-forwarding path against a fake collector endpoint; the PR runs native Tinybird mode against a fake Tinybird endpoint. Both runs sample the ingest process RSS and CPU via ps.
PR load benchmark JSON
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 82,
"duration_seconds": 0.990795381,
"export_catchup_seconds": 0.025719573,
"request_rps": 2018.5802622347915,
"row_rps": 20185.802622347914,
"p50_ms": 31.064,
"p95_ms": 35.107,
"p99_ms": 37.867,
"max_rss_mb": 92.64453125,
"max_cpu_percent": 71.4,
"avg_cpu_percent": 44.0
}main load benchmark JSON
{
"ingest_mode": "forward",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 2000,
"duration_seconds": 0.165427149,
"export_catchup_seconds": 1.112e-06,
"request_rps": 12089.913971738702,
"row_rps": 120899.13971738702,
"p50_ms": 4.858,
"p95_ms": 7.93,
"p99_ms": 10.873,
"max_rss_mb": 31.421875,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
}cargo test
�[1m�[92m Updating�[0m crates.io index
�[1m�[92m Compiling�[0m maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
�[1m�[92m Finished�[0m `test` profile [unoptimized + debuginfo] target(s) in 6.68s
�[1m�[92m Running�[0m unittests src/lib.rs (target/debug/deps/maple_ingest-ed3692bd2d4d62b8)
running 8 tests
test telemetry::tests::hex_empty_for_zero_ids ... ok
test telemetry::tests::log_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::metric_encoder_matches_all_tinybird_datasource_shapes ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
�[1m�[92m Running�[0m unittests src/bin/load_test.rs (target/debug/deps/load_test-2400d1bca3e1f791)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
�[1m�[92m Running�[0m unittests src/main.rs (target/debug/deps/maple_ingest-d6da5bec54f731c7)
running 18 tests
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::d1_truthy_accepts_int_and_bool_self_managed ... ok
test tests::d1_response_parses_empty_results_as_no_match ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::d1_response_parses_failure_with_errors ... ok
test tests::d1_response_parses_success_with_rows ... ok
test tests::hash_is_deterministic ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::non_self_managed_goes_to_shared_pool ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::self_managed_degrades_to_shared_when_endpoint_unset ... ok
test tests::self_managed_goes_to_self_managed_pool_when_configured ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
�[1m�[92m Doc-tests�[0m maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
…ions
The PR comment was comparing tinybird-mode-PR against forward-mode-main and
labelling everything "-83% worse" — a meaningless cross-mode comparison. CPU
samples were also stuck at 0% because the base run finished before the 500ms
ps poller landed any sample.
Changes:
- Drop duplicate Rust (ingest) job from ci.yml; ingest-rust-tests.yml already
covers cargo check/test/build via path filter
- Run the load harness 3x on PR and 3x on base; report median + spread
- Probe whether the base branch supports the head's LOAD_TEST_INGEST_MODE
before running a baseline; skip baseline + render absolute-only table when
it doesn't (so the introducing PR no longer shows fake deltas)
- Sticky PR comment via gh api PATCH using a stable HTML marker; no more
comment spam on every push
- Strip ANSI escapes from embedded cargo test output (CARGO_TERM_COLOR=never)
- Drop misleading Max/Avg CPU rows from the comment table; ps -o %cpu is
lifetime-average on Linux and useless at sub-second scales
- Add LOAD_TEST_REPORT_PATH to write JSON to a file, removing the fragile
"find first { to last }" stdout parser
- Wire the existing Criterion microbench into CI and embed bencher output
in the comment
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makisuo
marked this pull request as ready for review
May 14, 2026 11:43
Contributor
|
This run croaked 😵 The workflow encountered an error before any progress could be reported. Please check the link below for details. |
Previously the ingest's Rust encoder and the Tinybird datasource schemas (packages/domain/src/tinybird/datasources.ts) were only loosely coupled: they asserted invariants independently, and a rename or jsonPath edit on either side would silently land rows in the wrong column (or drop them) with no test failure. Pin both sides against the same canonical JSON-key contract: - apps/ingest/src/telemetry.rs::tests::schema_contract holds the expected top-level JSON keys per ingested datasource (logs, traces, metrics_sum, metrics_gauge, metrics_histogram, metrics_exponential_histogram). New tests assert the encoder emits *exactly* that set — no missing, no extras. - packages/domain/src/tinybird/datasources.contract.test.ts introspects each datasource's schema via `getColumnJsonPath`, derives the top-level jsonPath roots, and asserts the same canonical set. If either side drifts, the failing side's test prints the diff and points at the paired file to update. Also fills coverage gaps that left non-logs paths under-tested: - pipeline_e2e_exports_traces_to_fake_tinybird: full pipeline through the fake Tinybird import endpoint for traces (logs already covered). - pipeline_e2e_exports_metrics_to_fake_tinybird: same for each of the four metric datasources, asserting the JSON keys per datasource. - custom_datasource_names_propagate_to_frames: confirms the INGEST_TINYBIRD_DATASOURCE_* overrides flow through to the frame's `datasource` field (which becomes the Tinybird `?name=` query param). - timestamps_match_clickhouse_datetime64_nine_format: enforces the `YYYY-MM-DD HH:MM:SS.fffffffff` shape ClickHouse `DateTime64(9)` expects across logs.timestamp, traces.start_time, and metric timestamps. - logs_severity_text_falls_back_to_mapped_number: pins the severity-number -> text fallback the encoder applies when the SDK only sent a numeric severity. - logs_use_observed_time_when_time_unix_nano_is_zero: pins the OTLP-spec observed_time fallback so the Timestamp column is never epoch. - metrics_summary_data_points_are_dropped: pins the intentional Summary drop, so adding a Summary datasource won't go unnoticed. Total: 10 new tests in the Rust ingest crate (`cargo test` 18 -> 28 in the telemetry mod) + 7 new tests in the TS domain package, all passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ingest Rust Test + Benchmark ResultsCommit: Load Benchmark —
|
| Metric | main (median) | PR (median) | Delta |
|---|---|---|---|
| Requests/sec | 13418.44 | 2855.56 | -78.7% worse |
| Rows/sec | 134184.39 | 28555.56 | -78.7% worse |
| p50 latency | 4.25 ms | 22.18 ms | +421.7% worse |
| p95 latency | 7.56 ms | 26.27 ms | +247.4% worse |
| p99 latency | 9.33 ms | 42.52 ms | +355.9% worse |
| Export catch-up | 0.000 s | 0.026 s | +0.03 worse |
| Max RSS | 31.23 MiB | 95.55 MiB | +205.9% worse |
| Failures | 0 | 0 | same |
Same code path on both sides (same LOAD_TEST_INGEST_MODE), so the delta column is meaningful. Numbers come from ubuntu-latest, which is noisy — treat single-digit-percent deltas as noise.
PR load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 59,
"duration_seconds": 0.717933979,
"export_catchup_seconds": 0.025613101,
"request_rps": 2785.7714755133493,
"row_rps": 27857.714755133493,
"p50_ms": 22.343,
"p95_ms": 26.274,
"p99_ms": 44.752,
"max_rss_mb": 95.546875,
"max_cpu_percent": 89.2,
"avg_cpu_percent": 52.900000000000006
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 58,
"duration_seconds": 0.700389083,
"export_catchup_seconds": 0.026486404,
"request_rps": 2855.555645489694,
"row_rps": 28555.55645489694,
"p50_ms": 22.182,
"p95_ms": 23.993,
"p99_ms": 42.518,
"max_rss_mb": 97.61328125,
"max_cpu_percent": 91.0,
"avg_cpu_percent": 45.5
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 54,
"duration_seconds": 0.647356189,
"export_catchup_seconds": 0.026381236,
"request_rps": 3089.489270334295,
"row_rps": 30894.89270334295,
"p50_ms": 20.291,
"p95_ms": 38.666,
"p99_ms": 41.1,
"max_rss_mb": 95.03515625,
"max_cpu_percent": 96.4,
"avg_cpu_percent": 56.5
}
]main load benchmark JSON (per-iteration)
[
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 2000,
"duration_seconds": 0.153418171,
"export_catchup_seconds": 9.11e-07,
"request_rps": 13036.265436901866,
"row_rps": 130362.65436901865,
"p50_ms": 4.453,
"p95_ms": 7.791,
"p99_ms": 9.411,
"max_rss_mb": 31.234375,
"max_cpu_percent": 0.0,
"avg_cpu_percent": 0.0
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 2000,
"duration_seconds": 0.14904863,
"export_catchup_seconds": 1.262e-06,
"request_rps": 13418.439337550437,
"row_rps": 134184.39337550435,
"p50_ms": 4.252,
"p95_ms": 7.563,
"p99_ms": 9.327,
"max_rss_mb": 31.3828125,
"max_cpu_percent": 16.6,
"avg_cpu_percent": 16.6
},
{
"ingest_mode": "tinybird",
"requests": 2000,
"successes": 2000,
"failures": 0,
"rows_sent": 20000,
"rows_exported": 20000,
"imports": 2000,
"duration_seconds": 0.133861778,
"export_catchup_seconds": 9.71e-07,
"request_rps": 14940.784665208916,
"row_rps": 149407.84665208915,
"p50_ms": 3.867,
"p95_ms": 5.927,
"p99_ms": 9.293,
"max_rss_mb": 31.1875,
"max_cpu_percent": 33.3,
"avg_cpu_percent": 33.3
}
]WAL-acked microbench (cargo bench --bench ingest_bench)
Compiling regex-automata v0.4.14
Compiling serde_urlencoded v0.7.1
Compiling num-traits v0.2.19
Compiling hyper-util v0.1.20
Compiling ppv-lite86 v0.2.21
Compiling hyper-rustls v0.27.7
Compiling hyper-timeout v0.5.2
Compiling axum v0.8.8
Compiling rand_chacha v0.9.0
Compiling rand v0.9.2
Compiling ahash v0.8.12
Compiling metrics v0.24.3
Compiling opentelemetry_sdk v0.31.0
Compiling reqwest v0.12.28
Compiling tonic v0.14.4
Compiling matchers v0.2.0
Compiling tracing-subscriber v0.3.22
Compiling tonic-prost v0.14.4
Compiling opentelemetry-proto v0.31.0
Compiling metrics-util v0.19.1
Compiling opentelemetry-http v0.31.0
Compiling either v1.15.0
Compiling rayon-core v1.13.0
Compiling chrono v0.4.43
Compiling opentelemetry-otlp v0.31.1
Compiling reqwest v0.13.2
Compiling metrics-exporter-prometheus v0.16.2
Compiling tracing-opentelemetry v0.32.1
Compiling half v2.7.1
Compiling crossbeam-deque v0.8.6
Compiling plotters-backend v0.3.7
Compiling ciborium-io v0.2.2
Compiling anstyle v1.0.14
Compiling clap_lex v1.1.0
Compiling clap_builder v4.6.0
Compiling ciborium-ll v0.2.2
Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
Compiling plotters-svg v0.3.7
Compiling itertools v0.10.5
Compiling futures-io v0.3.32
Compiling same-file v1.0.6
Compiling cast v0.3.0
Compiling walkdir v2.5.0
Compiling criterion-plot v0.5.0
Compiling futures v0.3.31
Compiling rayon v1.12.0
Compiling plotters v0.3.7
Compiling clap v4.6.1
Compiling ciborium v0.2.2
Compiling regex v1.12.3
Compiling tinytemplate v1.2.1
Compiling is-terminal v0.4.17
Compiling anes v0.1.6
Compiling oorandom v11.1.5
Compiling criterion v0.5.1
Finished `bench` profile [optimized] target(s) in 1m 20s
Running benches/ingest_bench.rs (target/release/deps/ingest_bench-12226c7892c2c0cd)
Gnuplot not found, using plotters backend
test ingest_accept/logs_10_rows_wal_ack ... bench: 366974 ns/iter (+/- 9715)
test ingest_accept/traces_10_spans_wal_ack ... bench: 395137 ns/iter (+/- 11364)
cargo test
Compiling hmac v0.12.1
Compiling regex v1.12.3
Compiling tinytemplate v1.2.1
Compiling is-terminal v0.4.17
Compiling num_cpus v1.17.0
Compiling oorandom v11.1.5
Compiling anes v0.1.6
Compiling dotenvy v0.15.7
Compiling criterion v0.5.1
Compiling rustls-webpki v0.103.9
Compiling tokio-rustls v0.26.4
Compiling hyper-rustls v0.27.7
Compiling rustls-platform-verifier v0.6.2
Compiling reqwest v0.12.28
Compiling reqwest v0.13.2
Compiling opentelemetry-http v0.31.0
Compiling opentelemetry-otlp v0.31.1
Compiling metrics-exporter-prometheus v0.16.2
Compiling tikv-jemallocator v0.6.1
Compiling maple-ingest v0.1.0 (/home/runner/work/maple/maple/apps/ingest)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 42s
Running unittests src/lib.rs (target/debug/deps/maple_ingest-ed3692bd2d4d62b8)
running 18 tests
test telemetry::tests::hex_empty_for_zero_ids ... ok
test telemetry::tests::log_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::logs_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::logs_severity_text_falls_back_to_mapped_number ... ok
test telemetry::tests::logs_use_observed_time_when_time_unix_nano_is_zero ... ok
test telemetry::tests::custom_datasource_names_propagate_to_frames ... ok
test telemetry::tests::metrics_summary_data_points_are_dropped ... ok
test telemetry::tests::metric_encoder_matches_all_tinybird_datasource_shapes ... ok
test telemetry::tests::metrics_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::sampling_keeps_errors_even_when_ratio_low ... ok
test telemetry::tests::timestamp_has_nano_precision ... ok
test telemetry::tests::timestamps_match_clickhouse_datetime64_nine_format ... ok
test telemetry::tests::trace_encoder_matches_tinybird_row_shape ... ok
test telemetry::tests::traces_emit_exactly_the_jsonpaths_declared_in_datasources_ts ... ok
test telemetry::tests::wal_round_trips_frame ... ok
test telemetry::tests::pipeline_e2e_exports_traces_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_metrics_to_fake_tinybird ... ok
test telemetry::tests::pipeline_e2e_exports_gzip_ndjson_to_fake_tinybird ... ok
test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s
Running unittests src/bin/load_test.rs (target/debug/deps/load_test-2400d1bca3e1f791)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/debug/deps/maple_ingest-d6da5bec54f731c7)
running 18 tests
test tests::cloudflare_log_record_maps_body_severity_and_attributes ... ok
test tests::cloudflare_ndjson_payload_parses_multiple_records ... ok
test tests::cloudflare_timestamps_support_rfc3339_unix_and_unix_nano ... ok
test tests::cloudflare_validation_payload_is_detected ... ok
test tests::d1_response_parses_failure_with_errors ... ok
test tests::d1_response_parses_empty_results_as_no_match ... ok
test tests::d1_truthy_accepts_int_and_bool_self_managed ... ok
test tests::d1_response_parses_success_with_rows ... ok
test tests::extract_ingest_key_returns_sentinel_literal_unchanged ... ok
test tests::enrichment_overwrites_tenant_fields ... ok
test tests::hash_is_deterministic ... ok
test tests::non_self_managed_goes_to_shared_pool ... ok
test tests::resolve_ingest_key_returns_none_when_hash_missing ... ok
test tests::resolve_ingest_key_returns_self_managed_true_when_active_settings_row ... ok
test tests::resolve_ingest_key_returns_self_managed_false_when_no_settings_row ... ok
test tests::self_managed_degrades_to_shared_when_endpoint_unset ... ok
test tests::self_managed_goes_to_self_managed_pool_when_configured ... ok
test tests::sentinel_token_matches_only_exact_literal ... ok
test result: ok. 18 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests maple_ingest
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
The base may not have a load_test binary at all (this PR introduces it). Building --bin load_test on the worktree checkout of main fails with "no bin target named load_test", which marks the step as failed even though the probe + benchmark steps only need the head's load_test binary pointed at the base's maple-ingest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
org_ingest_sampling_policiesmigration and Drizzle schema.mainand the PR, run the same OTLP protobuf load against both, and always comment a comparison table back to the PR.Testing
cargo test --manifest-path apps/ingest/Cargo.tomltarget/release/load_testin bothtinybirdandforwardmodes.Benchmark Reporting
Ingest Rust Tests.cargo test, PR native load results, andmaincollector-forwarding load results.main.Ingest Load Tests(workflow_dispatch).apps/ingest/LOAD_TESTS.md.