Skip to content

My app crashes after exceeding the 4096 token context window of Apple Foundation Model #125

Closed

Description

@jadams777

Questions:

  • Is there a way to increase the context window size?
  • Is there a programmatic way to prevent exceeding the available context window? This would prevent crashes.

Activity

  1. changed the title [-]My app crashes after exceeding the 4096 token context window of Apple Intelligence[/-] [+]My app crashes after exceeding the 4096 token context window of Apple Foundation Model[/+] on Sep 19, 2025
  2. grabbou commented on Sep 19, 2025

    @grabbou
    Contributor

    Yeah, this is known limitations by Apple. I should’ve provided a safety mechanism around context overflow. Will look into it when back next week. For now, I’d suggest counting tokens manually.

  3. jadams777 commented on Sep 20, 2025

    @jadams777
    Author

    Thank you. Codex estimates three characters per token, but says it's just a guess.

  4. grabbou commented on Sep 22, 2025

    @grabbou
    Contributor

    Yup, we don’t know how Apple tokenizes input so all we can do at this point is rough estimation too.

  5. duochizhacai commented on Jan 7, 2026

    @duochizhacai

    @grabbou any good new?

  6. lech-kalinowski commented on Jan 29, 2026

    @lech-kalinowski

    Hey, there is no way to extend context window in apple foundational model here, its optimized for iPhone NPU. You have to manually handle context. Apple's framework provides a way to count tokens without running a full inference.

    let currentTranscript = session.transcript
    let pendingInput = userInput
    
    // Count the tokens in the total potential context
    let tokenCount = try await session.countTokens(for: currentTranscript + pendingInput)
    
    if tokenCount > 3800 { // Leave 296 tokens for the model to "breathe"
        try await pruneTranscript() // Drop the oldest messages
    }
    
    let response = try await session.respond(to: pendingInput)
    

    You can consider couple of strategies:

    1. Catch and handle error

    2. Sliding Context Window - and erasing "old memory" in current session

    3. Summarization - and forced memory conservation through sessions

    My best recommendation is to implement sliding window it is quite easy and gives best results in many llm centric apps.

  7. grabbou commented on Jan 29, 2026

    @grabbou
    Contributor

    I think the most appropriate way at this point per Apple doc is to try/catch and detect context overflow error and then apply solution per your preference. Apple suggests starting over with a new session.

    It is also worth saying that token estimations can vary depending on the language, so it would not be reliable to apply estimation as a part of the library.

    I also don't think applying any context heuristic is in scope of the project, but for sure we can suggest some options in the docs.

    Will work in the meantime on improving the documentation around the context overflow and how to handle this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions