Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes

[Copilot]

Orphaned-event cleanup now uses one inclusive CreatedUtc cutoff from three days before each run for stack, project, and organization discovery and deletion. It still searches every versioned event index, including older occurrence dates ingested recently. This reduces historical aggregation and deletion work while preserving public cleanup entry points.

Search timeouts, failed shards, and unrecoverable delete failures fail cleanup. Health distinguishes active progress from successful completion and follows the eight-hour schedule. Logs distinguish missing parent IDs from deleted documents. Four response-validation tests now run without the Elasticsearch integration fixture.

Validation: merged latest main (deffcf185); Release build passed with zero warnings/errors; standalone response tests passed 4/4; local cleanup integration tests passed 23/23 and the adjacent 15,000-event cleanup test passed 1/1 with synthetic data; focused formatting and diff checks passed. Hosted API tests with coverage, Playwright E2E, client tests, version, application images, and the Elasticsearch image build all passed at 21b6d2b4a. The Copilot contributor agreement remains pending.

The rolling window intentionally leaves older orphans to retention after gaps longer than three days. Concurrent deletion conflicts remain best effort. Shard fan-out is unchanged, and production CPU improvement has not been measured. No API, WebSocket, configuration, serialization, or frontend contracts change.

Verification and implementation details

• One cutoff is captured per complete job run and reused for discovery and deletion; standalone public methods capture their own cutoff.
• Cleanup phases remain sequential to avoid multiplying cluster load. Retrying version conflicts would require refreshing the retained-index wildcard and can repeat deletions already performed by another phase.
• Existing regression coverage checks valid records, tenant isolation, volume, inclusive cutoff boundaries, old occurrence dates, missed runs, and health transitions. The four extracted response-test bodies are unchanged.
• dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --configuration Release --no-restore -m:1
• dotnet test --project tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-build --no-restore --configuration Release -- --filter-class Exceptionless.Tests.Jobs.CleanupOrphanedDataResponseTests

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ niemyjski
❌ Copilot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9531879f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0910ef8ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c247cbf45c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[niemyjski]

Follow-up feedback and thermo-nuclear audit complete.

Feedback classification:

• Ingest-time/index-scope thread: valid, fixed by b0910ef, now outdated/resolved. Current code queries the full versioned event pattern and filters by CreatedUtc; regression coverage includes a 30-day-old occurrence ingested now.
• Empty/no-recent-event health thread: valid, fixed by d11819a and resolved. Empty and historical-only cases are covered.
• Eight-hour schedule versus 65-minute health thread: valid, fixed by c9ebc8c and resolved. The health window is 9 hours, with exact-boundary coverage.
• Health updated before successful cleanup thread: valid, fixed by d0bcc2b and resolved. RCA was both premature _lastRun mutation and unchecked invalid Elasticsearch responses. Health is now stamped only after completion; cleanup search/delete/refresh responses are validated; delete races explicitly proceed through version conflicts. A connection-failure regression proves stale health remains unhealthy.
• Submitted-review bodies contained only review boilerplate; CLA and coverage comments are informational, not code findings.

Thermo-nuclear full-diff result: no remaining structural blocker. Production is 481 lines and tests 771; neither crosses 1k. The cleanup phases remain intentionally sequential to avoid multiplying Elasticsearch load, public wrappers preserve the existing API, and response validation is centralized in one canonical helper.

Verification:

• dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-restore --nologo --verbosity:minimal -m:1: passed, 0 warnings/errors.
• CleanupOrphanedDataJobTests: 20/20 passed.
• Focused coverage: 66/66 changed executable production lines covered.
• git diff --check: passed.
• Fresh CI on d0bcc2b: version, test-api, test-client, test-e2e, and docker-build all passed.

All four review threads are addressed and resolved; the PR is mergeable and current with origin/main. The external Copilot contributor CLA is the sole remaining gate.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9ebc8c1cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6136e0d9ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[niemyjski]

Final feedback, destructive-path, coverage, and thermo-nuclear audit completed on 5a53ae9c5.

Feedback classification:

• Ingest-time/index-scope, empty-run health, schedule mismatch, premature health success, incomplete Elasticsearch responses, misleading deletion telemetry, and no-success wording were valid and are fixed with focused regressions.
• The final active-run health finding was valid. RCA: the two-hour value is a renewable lock lease, not a maximum runtime. Active progress is now tracked separately from completed success; successful renewal keeps a running job healthy, while finally clears active state after failure/cancellation so progress cannot mask a failed run.
• The version-conflict retry suggestion remains intentionally not applied: integration reproduction proved normal sequential phase overlap reports conflicts for documents already deleted by an earlier phase. Retrying would require a full wildcard refresh and restore the cluster work this PR removes.
• Submitted-review bodies are boilerplate; the coverage-bot and CLA comments are informational. All nine review threads are resolved with zero open actionable threads.

Data-safety proof:

• Stack/project/organization tests assert exact retained and deleted event IDs, not only counts, preventing false positives where valid events are deleted and an equal number of orphans remain.
• Coverage includes valid parents, missing parents, tenant isolation, 15,000-event volume, inclusive cutoff boundaries, historical events outside the ingest window, old occurrence dates ingested recently, missed runs beyond the lookback, invalid/timed-out/partial Elasticsearch responses, health success/failure transitions, and fresh/stale/missing active progress.
• Discovery and deletion share one captured cutoff and the full versioned index wildcard; destructive phases remain sequential to avoid multiplying Elasticsearch load.

Thermo-nuclear result:

• No remaining correctness or structural blocker in the effective branch-vs-main diff.
• Production is 562 lines and focused tests are 969 lines; neither crosses 1,000.
• Public cleanup entry points and API/serialization/configuration contracts remain unchanged.

Verification:

• Build: passed, 0 warnings / 0 errors.
• Focused integration suite with coverage: 27/27 passed.
• Effective changed production coverage: 137/137 executable lines and 66/66 branches.
• git diff --check: clean.
• origin/main is an ancestor through merge-forward commit 82d671d08.
• Fresh CI run 30577192830 on 5a53ae9c5: version, test-client, test-api, test-e2e, and docker-build all passed.

The external Copilot contributor CLA remains the only expected non-code gate.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0333421b59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

This PR updates the backend orphaned-event cleanup job to bound Elasticsearch work to a rolling recent-ingest window (by CreatedUtc), aiming to eliminate periodic CPU spikes while keeping cross-index fan-out behavior intact. It also tightens failure handling for Elasticsearch responses and expands integration test coverage around the new cutoff and health semantics.

Changes:

• Scope orphaned stack/project/organization discovery and delete-by-query operations to events with CreatedUtc >= (now - 3 days), using a single cutoff captured per job run.
• Treat timed-out/invalid/partially failed Elasticsearch search and delete-by-query responses as job failures via centralized validation helpers.
• Update job health reporting to track the last successful completion and add regression tests for cutoff boundaries, missed runs, and failure/health behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs	Adds a rolling ingest cutoff to orphan cleanup queries, centralizes ES response validation, and adjusts health tracking to last successful run.
tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs	Adds/updates integration tests validating cutoff boundary behavior, health window timing, and failure handling.

Comments suppressed due to low confidence (2)

src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs:210

• This log line reports "orphaned events" but the value being logged (totalOrphanedEventCount) is incremented by the number of missing project IDs, not the number of deleted event documents. This makes the log message/structured field names misleading for operational analysis.

        _logger.LogInformation("Found {OrphanedEventCount} orphaned events from missing projects out of {ProjectIdCount} since {OrphanedEventCutoffUtc}", totalOrphanedEventCount, totalProjectIds, orphanedEventCutoffUtc);

src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs:277

• This log line reports "orphaned events" but the value being logged (totalOrphanedEventCount) is incremented by the number of missing organization IDs, not the number of deleted event documents. This makes the log message/structured field names misleading for operational analysis.

        _logger.LogInformation("Found {OrphanedEventCount} orphaned events from missing organizations out of {OrganizationIdCount} since {OrphanedEventCutoffUtc}", totalOrphanedEventCount, totalOrganizationIds, orphanedEventCutoffUtc);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4e43994c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9610e611cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T03:17:26.034367Z	21b6d2b	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Insulation	37%	35%	286	❌
Exceptionless.Web	85%	70%	8108	✔
Exceptionless.Core	77%	68%	10596	✔
Exceptionless.AppHost	38%	41%	147	❌
Summary	79% (26481 / 33491)	68% (12334 / 18107)	19137	✔