Skip to content

Improve commit diff efficiency and CU estimation #1918

Open
0 of 2 issues completed
Open

Description

@snawaz

AppliesTo: master (10ae5aae) / V1 (39e19138)

Scope

This issue tracks improvements to how the validator prepares account changes and how the Delegation Program (DLP) applies them. This is follow-up work and is not a priority for this week.

#1906 describes an existing failure where fragmented diffs exhaust the transaction’s compute budget. It includes examples where a diff is about 16 KB even though the account itself is less than 10 KB.

On master at 10ae5aae, every commit still receives the same estimate of 120,000 compute units (CU). We need better estimates without requesting the maximum 1.4 million CU for every transaction.

The work covers five areas:

  1. Estimate CU from the work being done.

    Account size alone does not tell us how expensive a commit will be. Applying many small changes can cost more than copying one continuous block. Estimate the cost using the delivery method, number of changed regions, bytes copied, and any resizing or transfers. Use DLP benchmarks to establish these estimates, include a safety margin, and compare them with actual consumption.

  2. Combine nearby changes into fewer diff segments.

    Each segment adds metadata and work for DLP. When two changes have a small unchanged gap between them, including that gap in one larger segment can be cheaper than sending two separate segments. Improve the diff generator to make this choice. This improvement can use the existing diff format.

  3. Use full account data when a diff is more expensive.

    A diff is not always the cheapest representation. Compare its encoded size and estimated execution cost with sending the full account, including the cost of uploading buffer accounts. Choose the cheaper supported approach instead of always using a diff when the base account is available.

  4. Handle insertions and deletions without sending unchanged data again.

    Inserting an item at the beginning of a vector shifts the existing items. Our current diff compares bytes at the same positions, so it can treat much of the shifted data as changed.

    Support patches that can reuse existing bytes at a different position. For example, changing [A, B, C] into [X, A, B, C] should allow us to send X and describe how to reuse A, B, and C. This requires a versioned patch format and changes in both DLP and the validator. Applying it must handle overlapping moves safely and stay within memory and CU limits.

  5. Recover when the requested CU budget was too small.

    After a confirmed failure caused by CU exhaustion, allow a limited retry with a higher budget or a cheaper representation. Separate work into transactions only where the intent allows it; accounts that must commit together must remain together. Do not treat every program failure as CU exhaustion, or resend a transaction whose outcome is still unknown.

Outcome

Commits use estimates supported by measurements. Fragmented changes and insertions can be represented more efficiently, and the validator chooses full account data when that is cheaper.

Validate the improvements with representative DLP benchmarks and validator tests, including the two-account failure in #1906 and insertions near the beginning of an account’s data. Record the changes in payload size and CU consumption so the benefits are clear.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions