Skip to content

feat(engine): keep every packet for up to 7 days, one SQLite file a day - #458

Merged
gx1400 merged 2 commits into
mainfrom
feat/history-store
Oct 28, 2026
Merged

gx1400 merged 2 commits into
mainfrom
feat/history-store

Conversation

@gx1400

@gx1400 gx1400 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

First part of #314 (spec F1.41).

Store (PacketHistory, MIT Core):

  • What's kept: every packet the engine lists (received, sent, and same-source duplicates marked), with time, source, station or object, kind, position and the raw bytes.
  • Where: one SQLite file per UTC day in history/ under the data folder, through Microsoft.Data.Sqlite. WAL, synchronous=NORMAL, no pooling.
  • Writing: a bounded queue and a background writer that commits whatever has queued. Receiving never waits.
  • Rotation: once a minute, and when the settings change, days past the length are deleted, then the oldest days past the size limit.
  • Size warnings:
    • SizeLimitShortens, with "about N days fit", when the limit keeps fewer days than asked for. It's measured from whole days before they're trimmed.
    • TodayFull when today alone fills the limit. No more is kept until tomorrow.
    • The gateway also logs the warning once a day.
  • Robustness: an unreadable day file is renamed .invalid and named in the status. A .lock file keeps a second engine on the same folder out (it shows InUseElsewhere).
  • Deleting: turning history off and Start over delete it.

Settings:

  • AppSettings.History (HistorySettings(Days, MaxMegabytes), default 7 days within 1,024 MB), validated as settings.history-invalid.
  • [engine.history] days / max_megabytes in gateway.toml wins on every start, and Settings shows it without letting it be changed. The commented example is in deploy/linux/gateway.toml.
  • IEngineService.GetHistoryStatusAsync, operator only on a gateway.

UI: Settings › Data › History, above capture. A switch, days, size, what's kept ("7 days kept, 412 MB, since …"), and the warning. Turning it off asks once more, because it deletes what's kept.

Not in this PR: the restart replay, long tracks and older packets, and the dashboard's 7 days. Those are the next PRs in the stack.

Checks:

  • Tests: PacketHistoryTests (12) and HistoryUiTests (9), plus gateway access checks.
  • Release build, all tests and csharpier pass locally.
  • libe_sqlite3.so is in the linux-arm64 publish.

The packet history's store (F1.41 behaviors 1-6, 12, 13): every packet the
engine lists goes to a background writer and into history/<day>.db, so
receiving never waits. Once a minute the days past the length go, then the
oldest past the size limit; the status says when the limit keeps fewer
days than asked for, or when today alone filled it. Settings › Data ›
History sets the length (7 days) and size (1,024 MB); [engine.history] in
the gateway's configuration wins on every start. Turning it off, after a
second Save, and Start over delete it. An unreadable day is set aside, and
a second engine on the same folder keeps none.

Refs #314
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Package report

Vulnerable packages

No known vulnerabilities in direct or transitive packages.

Available package updates

Package Current version Available version Note
xunit.runner.visualstudio 3.1.5 4.0.1 Pinned in AprsMap.UI.Tests (see below)
xunit.v3.mtp-off 3.2.2 4.0.2 Pinned in AprsMap.UI.Tests (see below)

Pinned versions

These projects override the central version in Directory.Packages.props on purpose; updates for them are held back until the reason no longer applies.

Package Pinned version Project Reason
xunit.runner.visualstudio 3.1.5 AprsMap.UI.Tests Avalonia.Headless.XUnit 12.1 is built against xUnit v3 3.x and fails test discovery under 4.x (missing TestIntrospectionHelper overload). Pin this project only until Avalonia supports v4; see dependabot.yml.
xunit.v3.mtp-off 3.2.2 AprsMap.UI.Tests Avalonia.Headless.XUnit 12.1 is built against xUnit v3 3.x and fails test discovery under 4.x (missing TestIntrospectionHelper overload). Pin this project only until Avalonia supports v4; see dependabot.yml.

Checked at 67c0aac against nuget.org.

…for late sizes on Windows

The gateway's engine holds the packet history's files open, which Windows
won't delete; an asynchronous dispose can reach Dispose(true) before the
host stops. Windows also reports an open file's size late, so the days-fit
estimate is checked as a range.
@gx1400
gx1400 merged commit 9be3f74 into main Oct 28, 2026
12 checks passed
@gx1400
gx1400 deleted the feat/history-store branch October 28, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant