Skip to content

feat(perf): sample async task terminal results without changing billing lifecycle #410

Closed

Description

@LIghtJUNction

Upstream source

Upstream adds performance samples when asynchronous video/music tasks reach SUCCESS or FAILURE, so task-based models contribute success-rate/latency data instead of remaining permanently without performance data.

Current LMM gap

LMM already has apps/api-go/pkg/perf_metrics and records synchronous relay samples through RecordRelaySample, but the async task polling paths do not currently emit performance samples.

The upstream patch is conceptually applicable, but should not be copied verbatim because LMM's task lifecycle has diverged around durable refunds and custom settlement. In particular:

  • apps/api-go/service/task_polling.go has LMM-specific terminal paths such as sweepTimedOutTasks and failTaskForPolling.
  • LMM's settleTaskBillingOnComplete and refund flow are different from upstream and must remain unchanged.
  • terminal writes are protected by UpdateWithStatus CAS; metrics must only be emitted by the CAS winner to avoid duplicate samples under overlapping pollers / multiple instances.

Relevant current files:

  • apps/api-go/pkg/perf_metrics/metrics.go
  • apps/api-go/model/task.go
  • apps/api-go/service/task_polling.go

model.Task already contains Group, SubmitTime, StartTime, FinishTime, Properties.OriginModelName and PrivateData.BillingContext.OriginModelName, so no schema migration is needed.

Suggested implementation

  1. Add a small RecordTaskResult(task, taskResult) helper to apps/api-go/pkg/perf_metrics.
    • model name: prefer BillingContext.OriginModelName when present, otherwise Properties.OriginModelName;
    • group: task.Group;
    • success: terminal status is SUCCESS;
    • latency: SubmitTime -> FinishTime;
    • optional throughput: only for successful tasks when the provider reports TotalTokens or CompletionTokens;
    • no user/task IDs or response payloads in metric keys.
  2. Emit the sample only after a successful terminal CAS transition. Do not emit on non-terminal updates or CAS loss.
  3. Cover LMM-specific terminal paths instead of introducing upstream's billing finalizer wholesale:
    • normal video/task terminal transition;
    • Suno terminal transition;
    • failTaskForPolling / missing upstream task id;
    • channel lookup failure that terminally fails a task;
    • timeout sweep (including an explicit decision whether legacy timeout rows should be sampled; if sampled, they must remain failure-only and must not alter legacy refund behavior).
  4. Keep all existing settlement/refund calls and their ordering/semantics intact. Metrics must be observational only.

Acceptance criteria

  • An async task contributes exactly one performance sample when it wins the durable transition to SUCCESS or FAILURE.
  • A competing poller that loses UpdateWithStatus does not emit another sample.
  • Non-terminal SUBMITTED / QUEUED / IN_PROGRESS updates emit no sample.
  • Successful token-reporting tasks populate throughput; tasks without token usage still populate success rate and latency.
  • Model-name fallback works for both current billing-context tasks and older tasks that only have Properties.OriginModelName.
  • Timeout / missing-task-id / channel lookup failure paths are covered deliberately rather than silently omitted.
  • Existing quota, subscription settlement, durable refund reconciliation, pricing, group routing, model price lock and /fast behavior are unchanged.
  • Focused tests cover success, failure, token/no-token, CAS-loser/no-duplicate, and at least one LMM-specific failure/timeout path.
  • go test ./pkg/perf_metrics ./service -count=1 (or the LMM-equivalent focused packages) passes; run go vet on touched packages.

Why issue first instead of a direct cherry-pick

The useful part is small, but the upstream patch also refactors terminal settlement into a shared finalizer. LMM's billing/refund lifecycle is fork-specific, so copying that refactor would create unnecessary financial-risk surface. This should be implemented as a metrics-only adaptation on top of the current LMM terminal/CAS paths.

Activity

  1. LIghtJUNction commented on Sep 20, 2026

    @LIghtJUNction
    MemberAuthor

    Upstream #7481 has now been merged to QuantumNous/new-api main as QuantumNous/new-api@3abbb81 (2026-09-20). Final upstream scope remains the same three files (pkg/perf_metrics/metrics.go, pkg/perf_metrics/metrics_test.go, service/task_polling.go) and still centralizes terminal sampling inside a billing finalizer. Keep this issue's LMM-specific adaptation plan unchanged: emit metrics only after the durable terminal CAS winner, but do not import upstream's settlement/refund refactor over LMM's custom billing lifecycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions