You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
My app crashes after exceeding the 4096 token context window of Apple Foundation Model #125
changed the title [-]My app crashes after exceeding the 4096 token context window of Apple Intelligence[/-][+]My app crashes after exceeding the 4096 token context window of Apple Foundation Model[/+]on Sep 19, 2025
Yeah, this is known limitations by Apple. I should’ve provided a safety mechanism around context overflow. Will look into it when back next week. For now, I’d suggest counting tokens manually.
Hey, there is no way to extend context window in apple foundational model here, its optimized for iPhone NPU. You have to manually handle context. Apple's framework provides a way to count tokens without running a full inference.
let currentTranscript = session.transcript
let pendingInput = userInput
// Count the tokens in the total potential context
let tokenCount = try await session.countTokens(for: currentTranscript + pendingInput)
if tokenCount > 3800 { // Leave 296 tokens for the model to "breathe"
try await pruneTranscript() // Drop the oldest messages
}
let response = try await session.respond(to: pendingInput)
You can consider couple of strategies:
Catch and handle error
Sliding Context Window - and erasing "old memory" in current session
Summarization - and forced memory conservation through sessions
My best recommendation is to implement sliding window it is quite easy and gives best results in many llm centric apps.
I think the most appropriate way at this point per Apple doc is to try/catch and detect context overflow error and then apply solution per your preference. Apple suggests starting over with a new session.
It is also worth saying that token estimations can vary depending on the language, so it would not be reliable to apply estimation as a part of the library.
I also don't think applying any context heuristic is in scope of the project, but for sure we can suggest some options in the docs.
Will work in the meantime on improving the documentation around the context overflow and how to handle this.
Activity
[-]My app crashes after exceeding the 4096 token context window of Apple Intelligence[/-][+]My app crashes after exceeding the 4096 token context window of Apple Foundation Model[/+]grabbou commentedon Sep 19, 2025
Yeah, this is known limitations by Apple. I should’ve provided a safety mechanism around context overflow. Will look into it when back next week. For now, I’d suggest counting tokens manually.
jadams777 commentedon Sep 20, 2025
Thank you. Codex estimates three characters per token, but says it's just a guess.
grabbou commentedon Sep 22, 2025
Yup, we don’t know how Apple tokenizes input so all we can do at this point is rough estimation too.
duochizhacai commentedon Jan 7, 2026
@grabbou any good new?
lech-kalinowski commentedon Jan 29, 2026
Hey, there is no way to extend context window in apple foundational model here, its optimized for iPhone NPU. You have to manually handle context. Apple's framework provides a way to count tokens without running a full inference.
You can consider couple of strategies:
Catch and handle error
Sliding Context Window - and erasing "old memory" in current session
Summarization - and forced memory conservation through sessions
My best recommendation is to implement sliding window it is quite easy and gives best results in many llm centric apps.
grabbou commentedon Jan 29, 2026
I think the most appropriate way at this point per Apple doc is to try/catch and detect context overflow error and then apply solution per your preference. Apple suggests starting over with a new session.
It is also worth saying that token estimations can vary depending on the language, so it would not be reliable to apply estimation as a part of the library.
I also don't think applying any context heuristic is in scope of the project, but for sure we can suggest some options in the docs.
Will work in the meantime on improving the documentation around the context overflow and how to handle this.