Add exact request budget preflight by baijumeswani · Pull Request #1101 · microsoft/foundry-local

Baiju Meswani (baijumeswani) · 2026-09-12T07:08:22Z

Summary

Add an exact request preflight API for chat requests in C, C++, and JavaScript.
Report prompt tokens, reserved output tokens, required tokens, the model context limit, whether the request fits, and the token deficit.
Use the same prompt rendering, tools, tokenizer, and output limit resolution as generation.
Keep the public Foundry Local API at version 2 for the next minor release.
Run JavaScript preflight asynchronously without blocking the extension host.
Capture request and session state before asynchronous work so later mutations do not change the result.
Keep URI file access, media preparation, prompt rendering, and tokenization off the JavaScript thread.
Use model.context_length as the context limit, with the existing legacy fallback.
Apply bounded output defaults when clients omit a limit: 2,048 tokens for text and 3,072 tokens for media.

Why

Toolkit needs an exact token count before generation so it can compact long coding conversations and retry safely.

The same budget checks also prevent oversized requests and unbounded output when clients such as GitHub Copilot BYOK omit an output limit.

vercel · 2026-09-12T07:08:27Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 19, 2026 8:39am UTC

Copilot

🟡 Changes recommended

The current implementation introduces an unsafe v2 ABI extension and a streaming option-update deadlock.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds exact chat request token-budget preflight across C, C++, and JavaScript while sharing generation preparation logic and preserving asynchronous state isolation.

Changes:

Adds versioned preflight APIs and asynchronous JavaScript bindings.
Unifies prompt preparation, media handling, context limits, and output-limit resolution.
Adds comprehensive snapshot, parity, lifecycle, ABI, and backend tests.

File summaries

File	Description
`sdk_v2/js/test/chat-session.test.ts`	Tests asynchronous preflight behavior.
`sdk_v2/js/src/session.ts`	Exposes the preflight API and result.
`sdk_v2/js/src/index.ts`	Exports the preflight type.
`sdk_v2/js/src/detail/native.ts`	Defines native preflight bindings.
`sdk_v2/js/README.md`	Documents JavaScript preflight usage.
`sdk_v2/js/native/src/session.h`	Declares the native addon method.
`sdk_v2/js/native/src/session.cc`	Implements asynchronous native preflight.
`sdk_v2/js/binding.gyp`	Adds private native headers.
`sdk_v2/cpp/test/internal_api/session_manager_test.cc`	Tests synchronization and session validation.
`sdk_v2/cpp/test/internal_api/item_test.cc`	Tests request snapshot and URI behavior.
`sdk_v2/cpp/test/internal_api/chat/search_options_test.cc`	Tests output-limit resolution.
`sdk_v2/cpp/test/internal_api/chat/request_budget_test.cc`	Tests budget arithmetic and limits.
`sdk_v2/cpp/test/internal_api/chat/dynamic_engine_chat_test.cc`	Tests Engine preflight parity and state.
`sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc`	Tests chat preflight behavior.
`sdk_v2/cpp/test/internal_api/chat_completions_converter_test.cc`	Tests OpenAI output-limit mapping.
`sdk_v2/cpp/test/internal_api/c_api_test.cc`	Tests ABI layout and private API.
`sdk_v2/cpp/test/CMakeLists.txt`	Registers request-budget tests.
`sdk_v2/cpp/src/util/file_uri.h`	Centralizes URI-backed file reads.
`sdk_v2/cpp/src/service/responses_handler.cc`	Atomically replaces request tools.
`sdk_v2/cpp/src/node_private_api.h`	Defines the Node preflight bridge.
`sdk_v2/cpp/src/items/message_item.h`	Removes the old clone helper.
`sdk_v2/cpp/src/items/message_item.cc`	Uses shared request-item cloning.
`sdk_v2/cpp/src/items/item.h`	Declares deep request-item cloning.
`sdk_v2/cpp/src/items/item.cc`	Implements cloning and audio reads.
`sdk_v2/cpp/src/items/image_item.cc`	Reuses common file reading.
`sdk_v2/cpp/src/items/audio_item.h`	Exposes audio byte loading.
`sdk_v2/cpp/src/inferencing/session/tool_registry.h`	Adds atomic tool replacement.
`sdk_v2/cpp/src/inferencing/session/tool_registry.cc`	Implements replacement and normalization.
`sdk_v2/cpp/src/inferencing/session/session.h`	Adds state locking and preflight operation support.
`sdk_v2/cpp/src/inferencing/session/session.cc`	Synchronizes session options and validation.
`sdk_v2/cpp/src/inferencing/session/request.h`	Adds immutable request snapshots.
`sdk_v2/cpp/src/inferencing/generative/genai_config.h`	Widens context limits to 64-bit.
`sdk_v2/cpp/src/inferencing/generative/genai_config.cc`	Parses 64-bit context limits.
`sdk_v2/cpp/src/inferencing/generative/chat/search_options.h`	Defines canonical output-limit resolution.
`sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc`	Enforces bounded limits and overflow checks.
`sdk_v2/cpp/src/inferencing/generative/chat/request_budget.h`	Declares request-budget results.
`sdk_v2/cpp/src/inferencing/generative/chat/request_budget.cc`	Computes checked request budgets.
`sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.h`	Defines reusable prepared prompts.
`sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.cc`	Implements text and media preparation.
`sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.h`	Adds prepared-prompt Engine APIs.
`sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.cc`	Reuses prepared Engine prompts.
`sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.h`	Adds prepared-prompt generator APIs.
`sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc`	Reuses prepared generator inputs.
`sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc`	Enforces resolved Engine output limits.
`sdk_v2/cpp/src/inferencing/generative/chat/chat_session.h`	Exposes native chat preflight.
`sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc`	Shares snapshot preparation with generation.
`sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h`	Adds prepared-prompt abstraction.
`sdk_v2/cpp/src/exports.def`	Exports the private Node getter.
`sdk_v2/cpp/src/contracts/chat_completions_converter.h`	Clarifies catalog-default behavior.
`sdk_v2/cpp/src/contracts/chat_completions_converter.cc`	Stops inferring output limits from catalog metadata.
`sdk_v2/cpp/src/c_api.cc`	Implements public and private preflight entry points.
`sdk_v2/cpp/src/c_api_types.h`	Adds preflight-version validation.
`sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h`	Implements the C++ wrapper call.
`sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h`	Adds the C++ preflight surface.
`sdk_v2/cpp/include/foundry_local/foundry_local_c.h`	Adds the C ABI result and function.
`sdk_v2/cpp/docs/CppPortGuide.md`	Documents the private Node bridge.
`sdk_v2/cpp/docs/CodingConventionsAndStandards.md`	Updates export conventions.
`sdk_v2/cpp/CMakeLists.txt`	Builds the new prompt and budget sources.

Review details

Files reviewed: 58/58 changed files
Comments generated: 2
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Share prompt preparation with generation, expose the version 2 native preflight operation, and add thin C++, JavaScript, Rust, Python, and C# bindings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4c092375-f5b0-46e7-8d35-d6a8b94ee50e

Copilot

Copilot review overview

🟡 Changes recommended

ABI compatibility, JavaScript-thread deadlock risks, and incorrect context and output-limit policies remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 2

Open (4)

Add API versioning before accessing new function pointers · New
Avoid blocking JavaScript thread on generation mutex · New
Use model.context_length as the authoritative context limit · New
Apply default output token limits consistently · New

Resolved since last review (2)

Taking request_mutex_ here can deadlock the synchronous JS session.setOptions() call during…
This appends a public vtable entry to the already-current API version 2. A client built with this…

+  FL_API_STATUS(Session_CreateRequestPreflight, _In_ const flSession* session, _In_ const flRequest* request,
+                _Outptr_ flRequestPreflight** out_preflight);


+  CallCheckedVoid(env, [&]() {
+    preflight = std::make_shared<foundry_local::RequestPreflight>(
+        impl_->CaptureRequestPreflight(*request));


+    request_.reset();
+    auto prepared = PrepareChatRequest(request, transcript_, base_session_options_, chat_session_options_,
+                                       tool_definitions_, model_info_, model_, logger_, message_preparer_);
+    const auto context_limit = static_cast<int64_t>(GetModelMaxContextLength(model_.GetGenAIConfig()));


+  if (backend_kind == ChatBackendKind::kEngine && !has_media) {
+    return std::max<int64_t>(0, context_limit_tokens - prompt_tokens);
+  }
+
+  return GetDefaultMaxOutputTokens(has_media);


Copilot AI balanced review requested due to automatic review settings September 12, 2026 07:08

Copilot started reviewing on behalf of Baiju Meswani (baijumeswani) September 12, 2026 07:08 View session

Copilot AI reviewed Sep 12, 2026

View reviewed changes

Comment thread sdk_v2/cpp/include/foundry_local/foundry_local_c.h Outdated

Comment thread sdk_v2/cpp/src/inferencing/session/session.cc Outdated

Baiju Meswani (baijumeswani) force-pushed the baijumeswani/request-budget-preflight branch from 127dd9b to eb5d133 Compare September 19, 2026 08:39

Baiju Meswani (baijumeswani) requested a balanced review from Copilot September 20, 2026 06:10

Copilot started reviewing on behalf of Baiju Meswani (baijumeswani) September 20, 2026 06:11 View session

Copilot AI reviewed Sep 20, 2026

View reviewed changes

Navigation Menu

Add exact request budget preflight - #1101

Add exact request budget preflight#1101
Baiju Meswani (baijumeswani) wants to merge 1 commit into
mainfrom
baijumeswani/request-budget-preflight

Baiju Meswani (baijumeswani) commented Sep 12, 2026

Uh oh!

vercel Bot commented Sep 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		FL_API_STATUS(Session_CreateRequestPreflight, _In_ const flSession* session, _In_ const flRequest* request,
		_Outptr_ flRequestPreflight** out_preflight);

Conversation

Baiju Meswani (baijumeswani) commented Sep 12, 2026

Summary

Why

Uh oh!

vercel Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Sep 12, 2026 •

edited

Loading