Repository navigation
feat(pnpr): keep metadata in libSQL, read through an embedded replica - #17583
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Micro-Benchmark ResultsLinux |
Same-run regression checkHEAD and main are measured on the same runner. The tolerance is 5% of the main median or 2 ms, whichever is larger. A regression requires both the median slowdown and the gap between the fastest retained HEAD sample and slowest retained main sample to exceed this tolerance, after trimming up to 10% from each tail. Suspected regressions are rerun once with target order reversed and only fail if reproduced. This conservative noise guard is not a statistical confidence interval. Inconclusive rows do not fail the check. They remain visible for performance review. Bencher retains absolute minimum timings for historical tracking; historical alerts do not gate this PR. Sequential measurements can still be affected by runner drift.
Same-run syscall checkEach install runs 2 more times per revision, untimed, under
Integrated-Benchmark Report (Linux)Commit: Compared against main at: Regular scenarios report direct and pnpr installs. The peer-heavy resolver scenario compares current Rust, main Rust, and TypeScript pnpm; the linked, injected, and Cargo workspace scenarios compare current Rust against main Rust. Bencher consumes pacquet@HEAD and pnpr@HEAD. The tables below show mean ± σ. Bencher tracks absolute minimum latency; PR regression checks compare HEAD and main samples from the same runner. Historical timing alerts do not gate PRs. Scenario: Isolated linker: fresh restore, cold cache + cold store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 2.35027352916,
"stddev": 0.15707005630300178,
"median": 2.29425268706,
"user": 1.9743862599999997,
"system": 1.44373102,
"min": 2.2272305565600004,
"max": 2.64675641456
},
{
"command": "pacquet@main",
"mean": 2.4161547154600003,
"stddev": 0.14003465675674567,
"median": 2.3789651760600004,
"user": 2.1029097599999997,
"system": 1.5517197200000001,
"min": 2.2430321705600003,
"max": 2.6329619065600003
},
{
"command": "pnpr@HEAD",
"mean": 1.4070767336599999,
"stddev": 0.12195899296735385,
"median": 1.3444693715599998,
"user": 1.7289994600000003,
"system": 1.3209152200000003,
"min": 1.26852658956,
"max": 1.5868794695600001
},
{
"command": "pnpr@main",
"mean": 1.41865129036,
"stddev": 0.119265869883788,
"median": 1.3817262935599999,
"user": 1.8635163599999998,
"system": 1.3908376199999999,
"min": 1.3084182465599998,
"max": 1.65969221256
}
]
}Scenario: Isolated linker: fresh restore, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.26774812688000005,
"stddev": 0.03528584626110027,
"median": 0.27603106728000004,
"user": 0.31052854,
"system": 0.6297571399999999,
"min": 0.21236748078,
"max": 0.31142187278000005
},
{
"command": "pacquet@main",
"mean": 0.23412796578,
"stddev": 0.01797904685379347,
"median": 0.22960638528,
"user": 0.27404654000000006,
"system": 0.56859164,
"min": 0.20950906378,
"max": 0.26546733378000004
},
{
"command": "pnpr@HEAD",
"mean": 0.24225661817999997,
"stddev": 0.019738487671810315,
"median": 0.23515171228,
"user": 0.26619984,
"system": 0.5723533399999999,
"min": 0.21889695878,
"max": 0.28958308078000006
},
{
"command": "pnpr@main",
"mean": 0.23152426338,
"stddev": 0.010536269430586439,
"median": 0.22624037028,
"user": 0.26196584,
"system": 0.5419087399999999,
"min": 0.22022202278,
"max": 0.25272377378000005
}
]
}Scenario: Isolated linker: repeat install, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.013057511657499996,
"stddev": 0.0012345800067114512,
"median": 0.01299450712,
"user": 0.007863228249999996,
"system": 0.00663086375,
"min": 0.01033811712,
"max": 0.01716703312
},
{
"command": "pacquet@main",
"mean": 0.013746503605207093,
"stddev": 0.0015573181405636298,
"median": 0.013685779120000002,
"user": 0.008531398343195263,
"system": 0.006727874556213016,
"min": 0.010588588120000002,
"max": 0.01752063512
},
{
"command": "pnpr@HEAD",
"mean": 0.012903762017321432,
"stddev": 0.0010371388998940488,
"median": 0.012836376120000001,
"user": 0.0077476907142857085,
"system": 0.006592629464285712,
"min": 0.01060219712,
"max": 0.01535910612
},
{
"command": "pnpr@main",
"mean": 0.01150444964991452,
"stddev": 0.0012324946259612975,
"median": 0.011369855620000001,
"user": 0.006648544957264961,
"system": 0.005859118974358972,
"min": 0.00956627312,
"max": 0.02174366112
}
]
}Scenario: Isolated linker: repeat install, cold cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.011755698302407405,
"stddev": 0.0013250962879117454,
"median": 0.011436828020000001,
"user": 0.006725662777777773,
"system": 0.006096652407407406,
"min": 0.009287287020000002,
"max": 0.015287840020000002
},
{
"command": "pacquet@main",
"mean": 0.0117708538148718,
"stddev": 0.0017941647672057024,
"median": 0.011556404020000001,
"user": 0.007066986495726497,
"system": 0.0056708011965811945,
"min": 0.009396804020000002,
"max": 0.03329446102
},
{
"command": "pnpr@HEAD",
"mean": 0.012167491740164614,
"stddev": 0.0017012085747999124,
"median": 0.011829212020000002,
"user": 0.007145329958847734,
"system": 0.006055038024691359,
"min": 0.009302096020000001,
"max": 0.020328548020000003
},
{
"command": "pnpr@main",
"mean": 0.012471110547777773,
"stddev": 0.001950108475739375,
"median": 0.012099806020000003,
"user": 0.0074745146296296295,
"system": 0.006124237777777778,
"min": 0.009655941020000001,
"max": 0.019071072020000003
}
]
}Scenario: Isolated linker: fresh install, cold cache + cold store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 2.5161767779600006,
"stddev": 0.033179165181082076,
"median": 2.5172677404600003,
"user": 2.3130892199999997,
"system": 1.5930076999999998,
"min": 2.44329552146,
"max": 2.55197577746
},
{
"command": "pacquet@main",
"mean": 2.50791574946,
"stddev": 0.023088201308936365,
"median": 2.5065485034600004,
"user": 2.3312655199999996,
"system": 1.5160543999999996,
"min": 2.46780404246,
"max": 2.55399973346
},
{
"command": "pnpr@HEAD",
"mean": 1.5731410078599999,
"stddev": 0.10454570672964712,
"median": 1.57599513546,
"user": 1.8186266200000003,
"system": 1.3550787,
"min": 1.39324721046,
"max": 1.76065779246
},
{
"command": "pnpr@main",
"mean": 1.4609020857600001,
"stddev": 0.07325200145340931,
"median": 1.4349659724600001,
"user": 1.81270022,
"system": 1.3073973,
"min": 1.37668455346,
"max": 1.5825519294600001
}
]
}Scenario: Isolated linker: fresh install, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.5338324308,
"stddev": 0.046195280725173565,
"median": 0.5266358191,
"user": 0.79750844,
"system": 0.9559890799999999,
"min": 0.4737621836,
"max": 0.6139991546
},
{
"command": "pacquet@main",
"mean": 0.5057418559,
"stddev": 0.021387632173500343,
"median": 0.5067870005999999,
"user": 0.7565603399999999,
"system": 0.9169418800000001,
"min": 0.4641114756,
"max": 0.5336508526
},
{
"command": "pnpr@HEAD",
"mean": 0.3308057887,
"stddev": 0.01118210093859611,
"median": 0.3303362971,
"user": 0.30293524000000005,
"system": 0.6480319800000001,
"min": 0.3179879326,
"max": 0.3540433886
},
{
"command": "pnpr@main",
"mean": 0.3155115596,
"stddev": 0.00954114623122776,
"median": 0.3151990086,
"user": 0.29096863999999995,
"system": 0.6354698799999999,
"min": 0.2997922786,
"max": 0.3275196976
}
]
}Scenario: Isolated linker: fresh install, cold cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 1.62446878662,
"stddev": 0.02368256551674043,
"median": 1.63218895662,
"user": 0.86144254,
"system": 0.8387753799999998,
"min": 1.58998516162,
"max": 1.65776817562
},
{
"command": "pacquet@main",
"mean": 1.6753243014199999,
"stddev": 0.025084542878074316,
"median": 1.6749809511200002,
"user": 0.9401493399999998,
"system": 0.9534007799999997,
"min": 1.6237989086200002,
"max": 1.71224688462
},
{
"command": "pnpr@HEAD",
"mean": 0.31210887162,
"stddev": 0.012081823910543403,
"median": 0.30982447162000004,
"user": 0.30562603999999993,
"system": 0.6066478799999999,
"min": 0.29689058562,
"max": 0.33032883462
},
{
"command": "pnpr@main",
"mean": 0.31853187802,
"stddev": 0.007699494974297146,
"median": 0.31730068362,
"user": 0.29515364,
"system": 0.6549324799999999,
"min": 0.30932148162,
"max": 0.33426405462000003
}
]
}Scenario: Isolated linker: fresh resolve, hot cache, offline
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.26498338652000003,
"stddev": 0.014697774233953817,
"median": 0.26492841042000004,
"user": 0.25604523999999995,
"system": 0.1826982,
"min": 0.23955539642,
"max": 0.28761677142000003
},
{
"command": "pacquet@main",
"mean": 0.2720467623200001,
"stddev": 0.02442182306600725,
"median": 0.26858064442,
"user": 0.26085714,
"system": 0.1921062,
"min": 0.23793837542000001,
"max": 0.31419325642
},
{
"command": "pnpr@HEAD",
"mean": 0.09371111888666668,
"stddev": 0.0024969115024503262,
"median": 0.09298094992,
"user": 0.03965047333333333,
"system": 0.017085899999999998,
"min": 0.09058305842000001,
"max": 0.10141696842000002
},
{
"command": "pnpr@main",
"mean": 0.09392987477483873,
"stddev": 0.002963159431947895,
"median": 0.09280750042000001,
"user": 0.036745249677419364,
"system": 0.017709596774193554,
"min": 0.09045561842000001,
"max": 0.10209240842000002
}
]
}Scenario: Isolated linker: peer-heavy resolve, hot cache, offline
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.2094095178555556,
"stddev": 0.016332827449604462,
"median": 0.2182179083,
"user": 0.22318481333333334,
"system": 0.04176326444444444,
"min": 0.1889849103,
"max": 0.23024258030000003
},
{
"command": "pacquet@main",
"mean": 0.22998267763333333,
"stddev": 0.010411792717755004,
"median": 0.22829499830000002,
"user": 0.24637081333333333,
"system": 0.04408582,
"min": 0.2152872063,
"max": 0.24606057530000003
},
{
"command": "pnpm@HEAD",
"mean": 4.336835884188888,
"stddev": 0.31819845392305013,
"median": 4.3204735573,
"user": 6.686554257777776,
"system": 0.40388482000000003,
"min": 3.8761828113,
"max": 4.9718565173
}
]
}Scenario: Isolated linker: linked-workspace resolve, hot cache, offline
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.10186351528702704,
"stddev": 0.014764271176127925,
"median": 0.10510092726,
"user": 0.09555517513513512,
"system": 0.06525362216216216,
"min": 0.07717730426000001,
"max": 0.13238601926000002
},
{
"command": "pacquet@main",
"mean": 0.08575271451000002,
"stddev": 0.014626053057368171,
"median": 0.07927940526,
"user": 0.07901109555555555,
"system": 0.05594943222222223,
"min": 0.06988354726,
"max": 0.13278227626
}
]
}Scenario: Isolated linker: injected workspace repeat install, hot cache + hot store
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.10125476626000002,
"stddev": 0.005490621989057935,
"median": 0.10005777046,
"user": 0.037780173333333326,
"system": 0.07100319999999999,
"min": 0.09333491346,
"max": 0.11284913146
},
{
"command": "pacquet@main",
"mean": 0.09678466549124999,
"stddev": 0.006261665810473412,
"median": 0.09293747846,
"user": 0.03576244624999998,
"system": 0.06813505624999999,
"min": 0.09042770146,
"max": 0.10946426846
}
]
}Scenario: Isolated linker: fresh restore, cold cache + cold store + cold pnpr
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 4.67880826918,
"stddev": 0.16654313448960814,
"median": 4.615114480379999,
"user": 2.6241430399999994,
"system": 1.7889284799999998,
"min": 4.504416540379999,
"max": 4.96509255238
},
{
"command": "pacquet@main",
"mean": 4.552266510279999,
"stddev": 0.13763826448353408,
"median": 4.501307304379999,
"user": 2.39608674,
"system": 1.7651564799999995,
"min": 4.40713458638,
"max": 4.82560339438
},
{
"command": "pnpr@HEAD",
"mean": 3.54517575828,
"stddev": 0.06528604299570451,
"median": 3.5471672188800003,
"user": 2.21024464,
"system": 1.5839012799999999,
"min": 3.43776197638,
"max": 3.67706668838
},
{
"command": "pnpr@main",
"mean": 3.602345303880001,
"stddev": 0.13709868886901344,
"median": 3.56790596138,
"user": 2.23011734,
"system": 1.6137452799999998,
"min": 3.46030315938,
"max": 3.8989712833800003
}
]
}Scenario: GVS linker: fresh restore, hot cache + hot storeSame install as the isolated fresh-restore hot/hot scenario, into the shared virtual store, the layout pnpm 12 installs into by default. Scenarios run on separate machines, so compare main and branch within each table.
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.1502716154047059,
"stddev": 0.01189308753136457,
"median": 0.14610081264,
"user": 0.19941134352941178,
"system": 0.1705629023529412,
"min": 0.13289105164,
"max": 0.17624790864
},
{
"command": "pacquet@main",
"mean": 0.14626616547333335,
"stddev": 0.012070532485068316,
"median": 0.14309578964,
"user": 0.17903902,
"system": 0.17359763111111112,
"min": 0.13365317463999998,
"max": 0.18721082264
},
{
"command": "pnpr@HEAD",
"mean": 0.16858684045250003,
"stddev": 0.012309887934378004,
"median": 0.16549267914,
"user": 0.19480451999999998,
"system": 0.18897745750000003,
"min": 0.15407009464,
"max": 0.19920794164
},
{
"command": "pnpr@main",
"mean": 0.16491977332750002,
"stddev": 0.010592793873108963,
"median": 0.16405871814,
"user": 0.20414814499999998,
"system": 0.16585745749999997,
"min": 0.14924312464,
"max": 0.18536893764
}
]
}Scenario: Store only: unlocked direct deps, cold cache + hot storeStore-only install (
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 1.33818664902,
"stddev": 0.05913818946830415,
"median": 1.32640017102,
"user": 0.7282128199999999,
"system": 0.30109184,
"min": 1.25685470402,
"max": 1.43955349002
},
{
"command": "pacquet@main",
"mean": 1.28072183552,
"stddev": 0.044979532583167704,
"median": 1.28230518802,
"user": 0.6456415200000001,
"system": 0.25880413999999996,
"min": 1.22835001102,
"max": 1.38350075202
}
]
}Scenario: Cargo: repeat install, hot cache + hot storeFrozen repeat install of a Cargo workspace (
BENCHMARK_REPORT.json{
"results": [
{
"command": "pacquet@HEAD",
"mean": 0.03902185305999999,
"stddev": 0.0037428100070243508,
"median": 0.03822686306,
"user": 0.028718663478260863,
"system": 0.022097249855072464,
"min": 0.03206414706,
"max": 0.04771765406
},
{
"command": "pacquet@main",
"mean": 0.0396038817725,
"stddev": 0.003760686667096805,
"median": 0.03888042406,
"user": 0.03019892,
"system": 0.0217057175,
"min": 0.03178082406,
"max": 0.05146687006
}
]
} |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #17583 +/- ##
==========================================
- Coverage 92.73% 92.72% -0.01%
==========================================
Files 2011 2014 +3
Lines 260008 260506 +498
==========================================
+ Hits 241108 241564 +456
- Misses 18900 18942 +42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
9163ad6 to
9538e13
Compare
bb53c81 to
774eb17
Compare
Nothing new recorded
TS CI / Test / ubuntu (24.0.0, 2) / Node 24 / chunk 2/3 · nothing new
How to read thisbash › curl ← ancestry as captured, then the process that acted └─ example[.]com ← what it reached Recorded by Garnet · Add it to your CI ↗ |
This comment has been minimized.
This comment has been minimized.
9538e13 to
600603c
Compare
1bc2274 to
d8d62c5
Compare
600603c to
972162f
Compare
d8d62c5 to
ad386c0
Compare
3d7f5b8 to
70e8fc8
Compare
1c5ff60 to
faa3293
Compare
70e8fc8 to
bfa5f06
Compare
faa3293 to
3bb5200
Compare
bfa5f06 to
3b0a0b9
Compare
3bb5200 to
c820060
Compare
b168ce5 to
d6ea9a9
Compare
c820060 to
aa7f334
Compare
This comment has been minimized.
This comment has been minimized.
d6ea9a9 to
f860860
Compare
f922337 to
65b8f36
Compare
f860860 to
6d5c9a7
Compare
|
Code review by qodo was updated up to the latest commit 80d47db |
PR Code Suggestions ✨Warning
No code suggestions found for the PR. |
…e clock Two readings of `Instant::now` can be equal on macOS, so a request read just after a sync started could tie with it and count as answered. The test now builds the request time from the sync's start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The merge queue run failed on macOS in @coderabbitai resume Written by an agent (Claude Code, claude-opus-5-5). |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
@coderabbitai review Written by an agent (Claude Code, claude-opus-5-5). |
|
Learnings used
Action performed
|
|
Code review by qodo was updated up to the latest commit 3b49c62 |
PR Code Suggestions ✨Warning
No code suggestions found for the PR.
|
Action not completed
|
Summary
Fourth of the SQL metadata adapter stack (pnpm/tasks#244, design in pnpm/rfcs#66 "Shared deployments"). Builds on #17577, #17578 and #17580, now merged.
Adds the libSQL metadata adapter, so several pnpr instances share one metadata database such as Turso. The schema and queries are the same as for SQLite.
Where each read goes. The shared layer decides freshness, and the adapter decides the source:
Currentis_package_owner, which now reads for update)CurrentBoundedsyncIntervaland no write of this process came after it; else sync, else primaryStorage::read_hosted_package_view_at_least)AtLeast(g)g(synced on demand), else primaryA replica read runs in one
BEGIN READONLYtransaction, which keeps it on the replica; a plainBEGINon a replica connection goes to the primary. Every write of this process marks the replica stale, so the next replica read syncs first, and an instance always reads its own writes. That includes release-note edits, which advance no generation. AnAtLeast(g)read also checks the catalog's generation in its snapshot; a replica found behind is synced once, and if still behind the read goes to the primary.Writes go to the primary as
BEGIN IMMEDIATE, one at a time per process. A busy primary (SQLITE_BUSY/SQLITE_LOCKED, including the Hrana error codes) is a retryable conflict, and the whole transaction runs again.Replica sync. The adapter syncs the replica itself every
syncIntervaland records when each sync finished, because libsql's background sync doesn't report that. That timestamp is what the freshness bound checks.syncInterval: 0is refused together with areplicaPath.Opening. The shared layer is synchronous on the blocking pool, so the libSQL database opens on first use (under a 5-minute startup deadline) and its statements wait for libsql's async client with
Handle::block_on. EveryStorageof a process that names the same settings shares one handle.Storage::from_configruns several times at startup, and a replica file must have one owner.Smaller changes:
catalog_generation/tag_historyare async. The tests that call them now await.execute_batchre-renders statements and drops the comma inSTRICT, WITHOUT ROWID.sql/transactions.rsand the view tosql/view.rs(file-length lint).pnpr-libsql:ConnectionPool::checkout. The mock primary now gives each Hrana stream its own connection, serves/v3/cursor, carries blobs, and reportsSQLITE_BUSY.Sqlite/Libsqlerror displays no longer say "Token database"/"Auth database".Not done here: a request header carrying the client's minimum generation (the RFC leaves the name open), and switching the cloud trial.
Squash Commit Body
Checklist
already linked to it solves it.
pnpm changeset) if this PR changes any publishedpackage. Keep it short and written for pnpm users — it becomes a release note.
Tests:
pnpm test:rustover everypnpr-*crate pluspnpm-registry-mock(1824 passed). New tests inmetadata/sql/tests/libsql.rsrun against the local primary that serves libsql's sync protocol. They cover: the primary shared by two instances; replica reads served locally (no statement reaches the primary) and lagging another instance's commit within the sync interval; current and at-least reads bypassing or syncing the replica; an instance reading its own commits and note edits from its replica; a replica past its interval syncing before it serves; one handle per process. Config tests cover the new keys. The existing SQLite conformance tests run against libSQL in the next PR.Written by an agent (Claude Code, claude-opus-5-5).
🤖 Generated with Claude Code
Summary by CodeRabbit