Skip to content

feat(session): accept images in prompts on iOS 27 - #42

Merged
corasan merged 1 commit into
mainfrom
feat/image-input
Oct 17, 2026
Merged

corasan merged 1 commit into
mainfrom
feat/image-input

Conversation

@corasan

@corasan corasan commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Closes #25.

respond, streamResponse, tokenCount, and the hooks now accept { text, images: [{ uri, label? }] } as well as a plain string. Images go to the model through Attachment(imageURL:). This works with structured output and streaming.

await session.respond(
  { text: 'What is on this receipt?', images: [{ uri: 'file:///…/receipt.png', label: 'receipt' }] },
  { schema: Receipt },
)
  • Only file:// URIs are accepted. Any other scheme, or a file that does not exist, rejects with the new INVALID_PROMPT code, and the error names the image index.
  • A model without vision rejects with UNSUPPORTED_CAPABILITY. On iOS 26, a prompt with images rejects with UNSUPPORTED_PLATFORM.
  • The native spec now takes a single NativePrompt { text, images } struct instead of string, so nitrogen output is regenerated.
  • A prompt with blank text and some images now reaches the model. Before, a blank prompt returned an empty answer without calling it.
  • ImageReference for tools is not included. Tools are defined by JSON schemas, so that needs its own design.

Verification

I ran the example app on the iOS 27 simulator (AFM 3 Core, vision capable) against a generated receipt image.

Case Result
respond with image Read the business name, total, and circle color correctly
streamResponse with image 25 chunks, correct description
Structured output { merchant: 'BLUE CAFE', total: 4.5 }
tokenCount 7 tokens text only, 138 with the image
Image with blank text Answered
https:// URI INVALID_PROMPT before reaching native
Missing file INVALID_PROMPT from native
Transcript with image Saved (191 KB for a 57 KB PNG) and restored. The restored session recalled the total

bun run typecheck, bun run test (92 pass), bun run test:swift (8 suites, including the new PromptImageFileTests), and biome all pass.

I did not run the iOS 26 runtime. The text path now goes through Prompt(text) instead of String, using API that has existed since iOS 26.0.

@corasan
corasan merged commit 1f7396f into main Oct 17, 2026
6 checks passed
@corasan
corasan deleted the feat/image-input branch October 17, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support image input in prompts (iOS 27)

1 participant