Skip to content

feat(core): in-process mistral.rs local runtime for qwen3.5-1b - #46

Merged
senamakel merged 9 commits into
mainfrom
feat/local-llm
Mar 27, 2026
Merged

feat(core): in-process mistral.rs local runtime for qwen3.5-1b#46
senamakel merged 9 commits into
mainfrom
feat/local-llm

Conversation

@senamakel

@senamakel senamakel commented Mar 27, 2026

Copy link
Copy Markdown
Member

Summary

  • Added a new core-owned local AI runtime in rust-core using mistral.rs with qwen3.5-1b defaults.
  • Implemented background bootstrap, model download/checksum handling, warmup, status tracking, and resident in-memory inference.
  • Added core RPC methods for local AI status/download/summarization/suggested questions.
  • Wired Tauri commands and frontend wrappers to expose local AI debug/interaction controls.
  • Switched Tauri thread suggestion generation and context compaction summarization to use the local core runtime (with fallback behavior).

Problem

  • Prior flow relied on external local-model orchestration assumptions (e.g., Ollama-style separation), which does not satisfy the requirement that LLM logic run inside rust-core for background execution.
  • We needed an in-process local model path that can be initialized once, reused cheaply, and exposed through Tauri only as control/debug endpoints.

Solution

  • Introduced local_ai configuration schema and module in rust-core.
  • Implemented LocalAiService to:
    • load GGUF via mistralrs::GgufModelBuilder
    • keep Model resident in memory
    • run one warmup request after load
    • provide summarize + suggest APIs
    • download model artifacts when configured
  • Added JSON-RPC handlers:
    • openhuman.local_ai_status
    • openhuman.local_ai_download
    • openhuman.local_ai_summarize
    • openhuman.local_ai_suggest_questions
  • Triggered background bootstrap at core startup and integrated local summarization into compaction flow.
  • Added Tauri command proxies and frontend usage in Settings/Home/thread suggestion paths.

Testing

  • yarn -s compile
  • cargo check --manifest-path src-tauri/Cargo.toml
  • Other checks run (list commands)
    • cargo fmt --all
    • cargo check -p openhuman-core
    • cargo check -p OpenHuman
    • yarn -s tsc --noEmit
    • pre-push hooks (yarn format:check, yarn lint, yarn compile) during push
  • Manual validation completed (list scenarios)

Impact

  • Desktop/Tauri path now supports an in-process core local model runtime for background summarization/suggestion tasks.
  • First-time setup has heavier dependency/build and model artifact requirements (GGUF artifact path/download config).
  • No new external model service requirement for this path.

Breaking Changes

  • None
  • Yes (describe clearly, including migration steps)

Related

  • Issue(s): N/A
  • Follow-up PR(s)/TODOs:
    • Add explicit model source/bootstrap UX for qwen3.5-1b GGUF download/configuration.
    • Add integration tests for local AI RPC methods and fallback behavior.

@senamakel
feat(core): add in-process mistral.rs local AI runtime (qwen3.5-1b)
4d02db4
@senamakel
style: apply pre-push formatting for local-ai changes
2877468
@senamakel
feat: add Local Model panel and navigation support
38e8384 
- Introduced a new Local Model panel for managing local AI runtime, including status monitoring and download options.
- Updated settings navigation to include 'local-model' route.
- Enhanced AIPanel and Home components to integrate local model management features.
- Improved error handling in Tauri commands for local AI operations.
- Updated subproject reference in skills.
@senamakel
fix: enhance error handling in local model management
7b1ef6b 
- Improved error handling mechanisms in the Local Model panel to provide clearer feedback during AI runtime operations.
- Updated relevant components to ensure consistent error reporting and user notifications.
- Refactored code to streamline error management processes across local model features.
@senamakel
feat(local-ai): improve local runtime bootstrap, status UX, and qwen …
1d5a91f 
…defaults
@senamakel
Merge remote-tracking branch 'origin/main' into feat/local-llm
46c5257
@senamakel
feat(core): introduce CoreRunMode for process management
65c0e67 
- Added CoreRunMode enum to support in-process and child process execution modes.
- Updated CoreProcessHandle to manage tasks and child processes based on the selected run mode.
- Enhanced process spawning logic to differentiate between in-process server and dedicated core binary execution.
- Improved error handling for process readiness checks in both run modes.
@senamakel
refactor: clean up unused imports and streamline runtime module exports
2d134b7 
- Removed unused HashMap import from runtime.rs to enhance code clarity.
- Updated runtime module exports to only include necessary components, simplifying the interface for consumers.
- Changed variable name in llm_generator.rs for clarity without altering functionality.
@senamakel
refactor: streamline runtime module exports by consolidating imports
1341873
@senamakel
senamakel marked this pull request as ready for review March 27, 2026 23:44
@senamakel
senamakel merged commit 0f2530b into main Mar 27, 2026
8 of 14 checks passed
@senamakel
senamakel deleted the feat/local-llm branch March 28, 2026 17:46
AusAgentSmith pushed a commit to AusAgentSmith/openhuman that referenced this pull request May 23, 2026
@senamakel
feat(core): in-process mistral.rs local runtime for qwen3.5-1b (tinyh…
1dc0e98 
…umansai#46)

* feat(core): add in-process mistral.rs local AI runtime (qwen3.5-1b)

* style: apply pre-push formatting for local-ai changes

* feat: add Local Model panel and navigation support

- Introduced a new Local Model panel for managing local AI runtime, including status monitoring and download options.
- Updated settings navigation to include 'local-model' route.
- Enhanced AIPanel and Home components to integrate local model management features.
- Improved error handling in Tauri commands for local AI operations.
- Updated subproject reference in skills.

* fix: enhance error handling in local model management

- Improved error handling mechanisms in the Local Model panel to provide clearer feedback during AI runtime operations.
- Updated relevant components to ensure consistent error reporting and user notifications.
- Refactored code to streamline error management processes across local model features.

* feat(local-ai): improve local runtime bootstrap, status UX, and qwen defaults

* feat(core): introduce CoreRunMode for process management

- Added CoreRunMode enum to support in-process and child process execution modes.
- Updated CoreProcessHandle to manage tasks and child processes based on the selected run mode.
- Enhanced process spawning logic to differentiate between in-process server and dedicated core binary execution.
- Improved error handling for process readiness checks in both run modes.

* refactor: clean up unused imports and streamline runtime module exports

- Removed unused HashMap import from runtime.rs to enhance code clarity.
- Updated runtime module exports to only include necessary components, simplifying the interface for consumers.
- Changed variable name in llm_generator.rs for clarity without altering functionality.

* refactor: streamline runtime module exports by consolidating imports
senamakel referenced this pull request in senamakel/openhuman Jul 10, 2026
@senamakel
fix(agent): repair Motion A test-debt + bump tinyagents to v1.8.0 (#49 …
85f7c4e 
…merged)

The lib-test target (cargo test --lib) did not compile: Motion A renamed
ParentExecutionContext.provider -> turn_model_source (and the AgentBuilder
field), but ~8 agent_orchestration/harness test modules still built the struct
with the old field. No PR existed before tinyhumansai#4769, so this was never CI-tested;
CI rust-core-coverage compiles the full lib-test target and would fail here.

Wrap each stale site in TurnModelSource::new(provider) and fix the AgentBuilder
default-test field access. Sites: subagent_runner/ops_tests, agent_orchestration
{ops_tests, agent_teams/runtime_tests, workflow_runs/engine_tests, tools/
(close_subagent, spawn_parallel_agents_tests, spawn_worker_thread, tools_e2e_tests)}.

Also bump vendor/tinyagents to 7c6e81a (tinyagents#49 merged onto v1.8.0 main:
OpenAiModel native-tool/vision toggles + baked provider options; carries v1.8.0's
#45 tool-call-arg recovery + #46 microcompact gate). Cargo.lock 1.7.1 -> 1.8.0;
the `version = "1.7"` requirement admits 1.8.

Verified: lib-test target type-checks with 0 errors (reaches codegen); full
compile/run delegated to CI (contended dev box can't link the full test binary
in-budget). Crate: 61 openai unit tests pass on 7c6e81a.

Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB
YellowSnnowmann added a commit to YellowSnnowmann/openhuman that referenced this pull request Aug 19, 2026
@YellowSnnowmann
Ratchet the kernel floor down for the shed the bump delivers; track
01b7814 
the engine-module rename in the memory allowlist

Two CI findings on the tinymemory bump, both the bump doing its job:

1. Feature-Gate Smoke: "kernel floor IMPROVED but was not ratcheted —
   flows resolves 264 names, limit still 268". The profile got SMALLER:
   tinymemory#43 (§D2) moved `axum` from tinymemory-core's normal
   dependencies to dev-only (its four references were all inside `mod
   tests`), and that takes the whole tail out of openhuman's always-on
   kernel profile -- axum, axum-core, axum-macros, matchit, httpdate,
   serde_path_to_error leave (-6). Two enter, tinymemory-sources and
   tinymemory-sync, the crates tinyhumansai#46/tinyhumansai#48 split relocated code into, so
   packages stay flat. Net -4 names. `scripts/kernel-floor.limits` is
   lowered 268 -> 264 with the history entry written in the file's own
   terms (measured on CI Linux; macOS resolves 265 per the documented
   skew), and the dep-sim calibration assertion moves with it. Verified
   by diffing the resolved crate set old-pin vs new-pin: exactly the six
   out, two in, listed above.

2. Core Coverage: three `bypass_allowlist` tests failed. tinymemory#18
   §C1 renamed `core/src/tinycortex/` to `core/src/engine/`, so the two
   allowlist entries for the engine seam pointed at a file that no
   longer exists; and §B1 added `core/src/sync/pipelines/host.rs`, the
   engine-free sync runner -- an adapter over `MemoryClient` shaped
   exactly like the engine seam, with the same two needles. Entries
   moved to the new path and added for the new seam with the same
   justifications, each re-verified against the code: every
   `from_workspace_dir` hit in both files sits inside an inline
   `#[cfg(test)]` module, and `client_if_ready` is the production seam
   beneath the contract, not above it. All six scanner tests pass,
   including the one that fails on any entry that no longer matches.

12,093 tests passed on CI before these; the three scanner tests now join
them.
senamakel pushed a commit to nocstah/openhuman that referenced this pull request Sep 11, 2026
@YellowSnnowmann
Ratchet the kernel floor down for the shed the bump delivers; track\n…
8c40b9c 
…the engine-module rename in the memory allowlist\n\nTwo CI findings on the tinymemory bump, both the bump doing its job:\n\n1. Feature-Gate Smoke: "kernel floor IMPROVED but was not ratcheted —\n flows resolves 264 names, limit still 268". The profile got SMALLER:\n tinymemory#43 (§D2) moved `axum` from tinymemory-core's normal\n dependencies to dev-only (its four references were all inside `mod\n tests`), and that takes the whole tail out of openhuman's always-on\n kernel profile -- axum, axum-core, axum-macros, matchit, httpdate,\n serde_path_to_error leave (-6). Two enter, tinymemory-sources and\n tinymemory-sync, the crates tinyhumansai#46/tinyhumansai#48 split relocated code into, so\n   packages stay flat. Net -4 names. `scripts/kernel-floor.limits` is\n   lowered 268 -> 264 with the history entry written in the file's own\n   terms (measured on CI Linux; macOS resolves 265 per the documented\n   skew), and the dep-sim calibration assertion moves with it. Verified\n   by diffing the resolved crate set old-pin vs new-pin: exactly the six\n   out, two in, listed above.\n\n2. Core Coverage: three `bypass_allowlist` tests failed. tinymemory#18\n   §C1 renamed `core/src/tinycortex/` to `core/src/engine/`, so the two\n   allowlist entries for the engine seam pointed at a file that no\n   longer exists; and §B1 added `core/src/sync/pipelines/host.rs`, the\n   engine-free sync runner -- an adapter over `MemoryClient` shaped\n   exactly like the engine seam, with the same two needles. Entries\n   moved to the new path and added for the new seam with the same\n   justifications, each re-verified against the code: every\n   `from_workspace_dir` hit in both files sits inside an inline\n   `#[cfg(test)]` module, and `client_if_ready` is the production seam\n   beneath the contract, not above it. All six scanner tests pass,\n   including the one that fails on any entry that no longer matches.\n\n12,093 tests passed on CI before these; the three scanner tests now join\nthem.\n
senamakel added a commit to nocstah/openhuman that referenced this pull request Sep 11, 2026
@senamakel
feat(core): in-process mistral.rs local runtime for qwen3.5-1b (tinyh…
b31f093 
…umansai#46)\n\n* feat(core): add in-process mistral.rs local AI runtime (qwen3.5-1b)\n\n* style: apply pre-push formatting for local-ai changes\n\n* feat: add Local Model panel and navigation support\n\n- Introduced a new Local Model panel for managing local AI runtime, including status monitoring and download options.\n- Updated settings navigation to include 'local-model' route.\n- Enhanced AIPanel and Home components to integrate local model management features.\n- Improved error handling in Tauri commands for local AI operations.\n- Updated subproject reference in skills.\n\n* fix: enhance error handling in local model management\n\n- Improved error handling mechanisms in the Local Model panel to provide clearer feedback during AI runtime operations.\n- Updated relevant components to ensure consistent error reporting and user notifications.\n- Refactored code to streamline error management processes across local model features.\n\n* feat(local-ai): improve local runtime bootstrap, status UX, and qwen defaults\n\n* feat(core): introduce CoreRunMode for process management\n\n- Added CoreRunMode enum to support in-process and child process execution modes.\n- Updated CoreProcessHandle to manage tasks and child processes based on the selected run mode.\n- Enhanced process spawning logic to differentiate between in-process server and dedicated core binary execution.\n- Improved error handling for process readiness checks in both run modes.\n\n* refactor: clean up unused imports and streamline runtime module exports\n\n- Removed unused HashMap import from runtime.rs to enhance code clarity.\n- Updated runtime module exports to only include necessary components, simplifying the interface for consumers.\n- Changed variable name in llm_generator.rs for clarity without altering functionality.\n\n* refactor: streamline runtime module exports by consolidating imports\n
@M3gA-Mind M3gA-Mind mentioned this pull request Oct 8, 2026
Merged
12 tasks done
senamakel added a commit that referenced this pull request Oct 23, 2026
@senamakel @medullabot
docs(plans): record windows ci failure in security plan
7306448 
Add a follow-up note to the security plan documenting the first PR #46 Windows Rust job failure in secret_cipher_native_contract at mode 13, where ExecuteSecretCipher2 exceeded the fixture's 5-second bus timeout. The cause is not yet classified as a repeatable defect or a runner timing issue, so the note records the observation while other platforms and a second CI run are still in progress.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
senamakel added a commit that referenced this pull request Oct 23, 2026
@senamakel @medullabot
docs(plans): record windows ci timeout fix for pr #46
cd6b0e8 
The security plan now notes that coverage-instrumented cipher fixtures used 50 ms backend and 5-second bus deadlines, and that test-only timeout adjustments were pushed to PR #46 as d8707a0. It also records the second Windows coverage failure and that the focused local test cannot compile under Rust 1.99 due to an existing needs_drop assertion, so the hosted run must verify the change.

Auto-committed-on: cuttlefish
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@senamakel senamakel mentioned this pull request Oct 23, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@senamakel