Skip to content

Align tool-result capture across AI adapters with Claude Agent #5154

Open
Open
Align tool-result capture across AI adapters with Claude Agent#5154
Labels
enhancementNew feature or requestfeaturenodeteam/ai-observability

Description

@gouveags
gouveags
opened on Sep 30, 2026

Is your feature request related to a problem?

Hey @marandaneto! Following your non-blocking follow-up on #5088, I’d like to align tool-result capture across Gemini, OpenAI Chat/Responses, direct Anthropic, and Claude Agent around Claude Agent’s structure-preserving approach.

In #5088, I went with a whole-result bound for Gemini: small results keep their structure, while oversized results become truncated text. That gives us a stronger limit on captured content, but it can also remove useful fields entirely.

For example:

const result = {
  body: '!'.repeat(6000),
  summary: '42 rows matched',
}

Claude shortens body while retaining summary and the object shape. Gemini’s truncated JSON-text preview ends inside body, so summary disappears. See the Claude formatter and Gemini’s whole-result bound.

Why I’d choose Claude’s approach

The implementation went through review and corrections by Bernat and Manoel in #4824. That included UTF-8 truncation and traversal safeguards, along with checks for error flags, multimodal content, full capture, and privacy. I haven’t found a reported regression caused by its per-string policy in the open and closed issues, or open, closed, and merged PRs I checked.

More importantly, this is an analytics tool. A field we discard might be the clue someone needs to understand why their agent chose an action, returned a bad answer, or failed. Capturing a smaller event isn’t much of a win if we’ve removed the information that made it useful.

My Gemini solution is more conservative about total content size, but after comparing the two, I’d favor preserving more of the result. I think accepting some remaining aggregate-size risk is a reasonable trade-off for more useful analytics. Ideally, the existing safeguards provide enough protection while we retain the evidence people need.

That risk is real: many small fields or repeated results can still create large events, and events exceeding transport limits may be dropped. The absence of a report doesn’t prove this cannot happen. I’d keep that limitation explicit rather than solve it now by discarding more content across every provider.

Describe the solution you’d like

  • Use Claude’s 5,000 UTF-8-byte per-string value cap for tool results in generation input. Preserve objects and arrays where the provider supplies them, without parsing arbitrary text into JSON
  • Keep the existing truncation marker outside that cap, allowing up to 5,015 bytes per retained string before JSON encoding. This is not a whole-event size guarantee
  • Share internal processing while preserving each provider’s message shapes, roles, linking IDs, error flags, and media metadata
  • Preserve Claude’s separate 200,000-byte per-string default for original tool-span output rather than reducing it to the history limit
  • Keep traversal safeguards and binary redaction. Normalization must preserve enough type information to redact supported binary values correctly
  • Preserve full capture’s bypass of binary redaction and string caps, with privacy taking precedence, following the Capture AI specification

For OpenAI and direct Anthropic, this adds a tool-result history cap to input paths that currently lack an equivalent one. For Gemini, it replaces whole-result truncation with per-string handling. Those are intentional changes to captured content and should be tested as such.

What I checked

I compared the formatter implementations, reviewed #4824’s history, searched related issues and PRs across states, and reproduced the example above locally against the existing formatters. Claude retained summary; Gemini did not. Neither changed the original input in that check. This wasn’t a live-provider or production-ingestion test.

Implementation and acceptance criteria

I’d start with characterization tests, then extract the shared processing and apply the agreed policy across the named provider paths.

The implementation should:

  • Preserve provider requests, responses, tool execution, and stream/control behavior
  • Preserve analytics roles, IDs, event relationships, and usage/cost reporting
  • Keep instrumentation failures from interrupting the application
  • Cover nested results, many small fields, UTF-8 boundaries, binary values, full capture/privacy, and nonmutation with cross-adapter tests
  • Verify the built package through applicable real-SDK recorder/replay paths as well as unit tests

Describe alternatives you’ve considered

Applying Gemini’s whole-result cap everywhere would provide a tighter content bound, but lose structure and later fields that Claude currently retains. I’d choose Claude’s approach for the product reasons above.

A conversation-wide budget is a separate follow-up. It needs a retention policy that preserves useful history and call/result pairing, rather than silently cutting off whatever exceeds a threshold.

Related sub-libraries

  • @posthog/ai

Additional context

This follows #5088 and the original Gemini issue #5085 without expanding their merge requirements. It proposes a shared capture policy, with no new public option or universal provider-message schema.

Agent context

Autonomy: Human-driven (agent-assisted).

I chose the proposed direction and the fidelity-versus-size trade-off. Codex assisted with repository and review-history research, local formatter checks, drafting, and technical/editorial review, using GitHub CLI and local scripts. No implementation is included in this issue. The working conversation contains unrelated private context, so I’m not sharing a session link.

Activity

  1. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsbot
    on Sep 30, 2026 – with GitHub Actions
    Contributor

    AI triage for #5154

    Conclusion: Valid cross-adapter @posthog/ai capture-policy enhancement; it is complementary to the open Gemini-specific work in #5088.
    Confidence: 90%
    Suggested labels: feature, enhancement, node, team/ai-observability

    Thanks—this is a well-scoped telemetry-fidelity enhancement rather than a duplicate of #5088. The current integrations have materially different input formatting paths, so the proposed policy should be introduced through characterization tests and only for provider-recognized tool-result payloads; arbitrary text should not be parsed as JSON.

    The key trade-off is explicit: a 5,000-byte cap per retained string preserves object/array structure and later fields, but is not an aggregate event-size limit. A coordinated change should preserve provider request behavior, roles, linking IDs, error metadata, media redaction, and full-capture/privacy precedence. Please keep the existing 200,000-byte handling for Claude Agent's original tool-span output separate from the smaller history cap.

    Before implementation, confirm the Capture AI specification remains the governing contract for full capture and privacy. Add adapter-level tests for structured/nested tool results, UTF-8 boundaries, typed binary values, full capture, privacy, and nonmutation; #5088 should remain independently reviewable until it is merged.

    Investigation

    • Claude Agent defines a 5,000-byte TOOL_RESULT_MAX_STRING_LENGTH; formatUserContent() applies it specifically to tool_result.content. Its truncate() path is UTF-8-aware, and existing tests cover an emoji boundary and expect the truncation marker outside the retained 5,000-byte prefix.
    • Claude Agent uses the normal 200,000-byte MAX_OUTPUT_SIZE default for other formatted content, so its tool-result history retention is already distinct from its general output/tool-span capture behavior.
    • Direct Anthropic captures sanitizeAnthropic(mergeSystemPrompt(...)), while OpenAI Chat captures sanitizeOpenAI(context.params.messages, ...). Neither path applies a tool-result-specific string cap.
    • OpenAI Responses sanitizes input and then formatOpenAIResponsesInput() selects content or text and applies toContentString(), which stringifies object/array content rather than preserving that structured content in the formatted input.
    • The current Gemini formatter handles text and inline-data parts but has no functionResponse branch. The linked fix(ai): capture Gemini tool history in generation input #5088 is therefore related work rather than behavior already present in this checkout.
    • Binary redaction recognizes Uint8Array/Buffer before ordinary object traversal. toJsonSafeValue() does not have an equivalent binary branch and recursively converts arbitrary objects, so shared processing must preserve binary identity until redaction has run.
    • Full capture bypasses the wrapper sanitizers, while captureAiGeneration() enforces privacy before traversing input or output. It also catches telemetry failures, preserving the requirement that instrumentation must not interrupt provider calls.
    • Existing Gemini tests verify inline binary data is redacted in captured telemetry without changing the request passed to the provider, and Claude Agent tests cover multimodal/full-capture/privacy behavior. Equivalent cross-adapter tool-result characterization coverage is not present.

    Relevant files: packages/ai/src/anthropic/index.ts, packages/ai/src/captureAiGeneration.ts, packages/ai/src/claude-agent-sdk/formatting.ts, packages/ai/src/gemini/index.ts, packages/ai/src/openai/telemetry.ts, packages/ai/src/sanitization.ts, packages/ai/src/sanitization/binary_content_redactor.ts, packages/ai/src/utils.ts, packages/ai/tests/binary_content_redactor.test.ts, packages/ai/tests/claude-agent-sdk.test.ts, packages/ai/tests/gemini.test.ts

    Fix assessment

    • Straightforward: no
    • Risk: medium
    • Reason: The desired policy is clear and adds no public option, but it crosses provider-specific input models with different current normalization behavior. A shared implementation must identify only recognized tool-result values, preserve each adapter's roles/IDs/error and media metadata, redact typed binary data before JSON-safe conversion, retain full-capture and privacy semantics, and avoid mutating provider requests.

    Fix PR skipped: feature requests require an explicit trusted fix command or mode: fix before opening a draft PR

    Pi session

    The resumable JSONL pi session for this run was saved to a private gist: https://gist.github.com/marandaneto/050bcf7b5db97b102fe8a05c6e86747b

    Download it locally, then fork it into your own pi session:

    pi --fork path/to/session.jsonl

    Saved session:

    Close proposal

    None. No issue close action was taken.

    Generated by posthog-watcher-action using pi.

  2. github-actions
    added
    enhancementNew feature or request
    feature
    node
    team/ai-observability
    on Sep 30, 2026
  3. gouveags
    changed the title [-]Proposal: align tool-result capture across AI adapters with Claude Agent[/-] [+]Align tool-result capture across AI adapters with Claude Agent[/+] on Sep 30, 2026

  4. dustinbyrne commented on Sep 30, 2026

    @dustinbyrne
    dustinbyrne
    on Sep 30, 2026
    Contributor

    thanks again @gouveags! we'll keep an eye out for your pull request once it's out of draft

  5. gouveags commented on Sep 30, 2026

    @gouveags
    gouveags
    on Sep 30, 2026
    ContributorAuthor

    Hey @dustinbyrne! Honestly, no need to thank me. I’ve been having a lot of fun working on PostHog's codebase. I use it all the time and love the product, so getting to help make it better is pretty cool.

    #5155 is out of draft now. I’d love to hear what you think when you get a chance to look. And if there’s anywhere else I can help out, let me know (feel free to tag me on open issues where I may be useful).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfeaturenodeteam/ai-observability

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions