Repository navigation
fix: close provider gaps after the AI SDK 7 upgrade - #251
Merged
Merged
Conversation
AppleFoundationModels.ts was added in #212 to reject countTokens on native builds without it. #248 removed that guard, so the wrapper only re-declared the native methods. Export the TurboModule directly, like the other native modules in the package and as 0.12.0 did. Also remove src/stream.ts, an unused stream helper still typed with LanguageModelV2StreamPart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Responses and streams reported 0 input and 0 output tokens, so AI SDK
usage, prepareStep thresholds and telemetry always saw zero.
- iOS 27 passes through the usage Foundation Models measures (input,
cached input, output and reasoning tokens), for the on-device model and
Private Cloud Compute
- iOS 26.4 counts the transcript before and after the response with the
on-device model's tokenizer
- otherwise usage is undefined instead of 0
Native generateText now resolves { content, usage } and onStreamComplete
carries usage, which changes the shape returned by
AppleFoundationModels.generateText.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
core-ai always reported 0 tokens and llama fell back to 0 when llama.rn returned no timings. AI SDK treats 0 as a measured count, so totals looked real. Report undefined instead, as adk does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@react-native-ai/utils summarizes history by calling countTokens on the model it is given, but no provider defined it, so it always fell back to four characters per token. Apple language models count with the on-device model's tokenizer (iOS 26.4+) and llama.rn models with their own tokenizer. AppleFoundationModels.countTokens stays available. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On iOS 26, stream snapshots have no transcript entries, so streamText never reported Apple tool calls. Read them from the session transcript instead, which already holds the tool call and output entries when the first text chunk arrives. streamText with Output.object threw "Streaming JSON responses is not yet supported." Native streams now send the snapshot's JSON. Partial snapshots are not prefixes of the final JSON (even key order changes), so the provider sends the complete object once the stream finishes. Schema streams also report reasoning and tool entries on iOS 27. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI SDK 7 names provider factories embeddingModel and rerankingModel and gives providers a specificationVersion. - every provider factory sets specificationVersion: 'v4' - apple and llama add embeddingModel, and llama adds rerankingModel; the published textEmbeddingModel and rerankModel stay as deprecated aliases - core-ai drops textEmbeddingModel, which was never published - mlc adds createMlcProvider, so mlc is a callable provider like the others; mlc.languageModel() keeps working Docs, README and skills use the new names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Llama token counting loses its instance receiver in history summarization, silently falling back to estimates.
1 open issue
What changed in this PR
This PR closes provider gaps after the AI SDK 7 upgrade, improving token accounting, Apple streaming, and provider API consistency.
Changes:
- Adds Apple usage reporting and model-level token counting; represents unknown usage as
undefined. - Supports Apple structured-output streaming and iOS 26 tool-call reporting, while simplifying native exports.
- Aligns provider factories with v4 naming and updates documentation and tests.
| File | Description |
|---|---|
| website/src/docs/llama/reranking.md | Uses rerankingModel in examples. |
| website/src/docs/llama/model-management.md | Documents embeddingModel. |
| website/src/docs/llama/getting-started.mdx | Updates the embedding factory name. |
| website/src/docs/llama/embeddings.md | Updates embedding examples. |
| website/src/docs/apple/generating.md | Documents streaming, usage, and native return changes. |
| website/src/docs/apple/embeddings.md | Uses embeddingModel in examples. |
| skills/react-native-ai/references/llama-provider.md | Updates embedding and reranking factory names. |
| skills/react-native-ai/references/apple-provider.md | Updates the embedding factory name. |
| README.md | Updates provider API examples. |
| packages/utils/README.md | Describes summarizer token counting. |
| packages/mlc/src/index.ts | Exports createMlcProvider. |
| packages/mlc/src/ai-sdk.ts | Adds a callable v4 provider factory. |
| packages/llama/src/ai-sdk.ts | Adds token counting, unknown usage, and v4 factory names. |
| packages/core-ai/src/ai-sdk.ts | Aligns factory naming and unknown usage. |
| packages/apple-llm/src/stream.ts | Removes the unused streaming helper. |
| packages/apple-llm/src/NativeAppleLLM.ts | Defines native usage and generation result types. |
| packages/apple-llm/src/index.ts | Exports the native module directly. |
| packages/apple-llm/src/AppleFoundationModels.ts | Removes the redundant native wrapper. |
| packages/apple-llm/src/ai-sdk.ts | Maps usage and adds token counting and structured streaming. |
| packages/apple-llm/src/__tests__/ai-sdk.test.ts | Tests usage, structured streaming, and token counting. |
| packages/apple-llm/ios/AppleLLMImpl.swift | Adds native usage and streaming transcript handling. |
| packages/apple-llm/ios/AppleLLM.mm | Forwards usage in stream completion events. |
| packages/adk/src/ai-sdk.ts | Declares the v4 provider specification. |
Review effort: Balanced
countPromptTokens took countTokens off the model and called it without a receiver. The llama.rn model reads this.context in countTokens, so it threw and the summarizer silently fell back to four characters per token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
countTokens on a Private Cloud Compute model counts with the on-device model's tokenizer, which may not match the server model, and needs the on-device model to be available. Calling it an approximation claimed more than is known. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes the gaps left in the providers after the AI SDK 7 upgrade: Apple reported 0 tokens for every call,
countTokenswasn't reachable from the model, Apple streams dropped tool calls on iOS 26 and threw for structured output, and the provider factories used pre-v7 names. Each commit can be reviewed on its own.Commits
AppleFoundationModels.tswas added in feat: graceful apple context window error handling, countTokens API #212 only to guardcountTokenson older native builds, and refactor(apple): drop redundant native method guards #248 removed that guard. The TurboModule is now exported directly, like the other native modules and as 0.12.0 did. Also removes the unusedsrc/stream.ts.Response.usage/Snapshot.usage(input, cached input, output and reasoning tokens), for the on-device model and Private Cloud Compute.undefinedinstead of0.apple().countTokens(text)andllama.languageModel(...).countTokens(text).@react-native-ai/utilsalready callsmodel.countTokensfor history summarization and fell back tolength / 4because no provider defined it.AppleFoundationModels.countTokensstays.session.transcriptwhile the response streams.streamTextwithOutput.objectno longer throws. Apple's partial JSON snapshots aren't prefixes of the final JSON (key order changes too), so the provider sends the complete object once generation finishes.specificationVersion: 'v4'.embeddingModel(apple, llama) andrerankingModel(llama) added; the publishedtextEmbeddingModel/rerankModelstay as deprecated aliases.textEmbeddingModel, which was never published.createMlcProvider();mlc.languageModel()keeps working.Breaking change
Native
generateTextnow resolves{ content, usage }instead of an array of parts, soAppleFoundationModels.generateTextreturns that object. The docs example is updated.Testing
bun lint,bun run typecheck, and package tests: apple 19 (6 new, includingstreamText+Output.objectthrough the real SDK), llama 35, core-ai 5.AppleLLMImpl.swifttypechecks against the iOS 27 device and simulator SDKs with Swift 6.4, so the#if compiler(>=6.4)branches are compiled.NativeAppleLLMspec.session.transcriptholds the tool call and output entries (same id) before the first text chunk streams.Not tested:
AppleLLM.mmchange hasn't been compiled.🤖 Generated with Claude Code