feat: split screen intelligence from accessibility and add dedicated settings by senamakel · Pull Request #50 · tinyhumansai/openhuman

senamakel · 2026-03-28T19:18:10Z

Summary

Introduced screen_intelligence as a first-class feature area while preserving existing openhuman.accessibility_* RPC compatibility.
Split Rust core module ownership so screen intelligence now lives under openhuman/screen_intelligence, with openhuman/accessibility as a compatibility re-export.
Added screen-intelligence settings update flow (openhuman.update_screen_intelligence_settings) and config fields for policy mode and allowlist/denylist behavior.
Kept accessibility and screen intelligence separate in settings UI by restoring Accessibility naming and adding a dedicated Screen Intelligence panel/route.
Wired screen captures to active-window policy filtering and memory persistence under screen_intelligence category.

Problem

The initial implementation mixed accessibility automation and screen intelligence into a single surfaced feature, creating product confusion and coupling.
Screen intelligence needed to become a configurable core feature with independent policy controls for capture targeting.
We needed this without breaking existing accessibility command paths already used by frontend and Tauri command bridges.

Solution

Created a dedicated openhuman::screen_intelligence module and moved implementation ownership there.
Kept backward compatibility by re-exporting from openhuman::accessibility and leaving openhuman.accessibility_* RPC methods in place.
Added and connected openhuman.update_screen_intelligence_settings to persist and apply runtime config updates.
Implemented policy controls (all_except_blacklist / whitelist_only) with allowlist + denylist matching.
Added a standalone Screen Intelligence settings panel and route, while restoring Accessibility panel/menu labeling and behavior.

Testing

yarn -s compile
cargo check --manifest-path src-tauri/Cargo.toml
Other checks run (list commands)
Commands: npx prettier --check ., npx eslint ., npx tsc --noEmit, cargo fmt --manifest-path src-tauri/Cargo.toml, cargo check --manifest-path src-tauri/Cargo.toml
Manual validation completed (list scenarios)

Impact

Runtime impact: desktop-focused screen intelligence controls and capture behavior now have their own feature boundary.
Compatibility: existing accessibility RPC command names remain functional; no immediate client migration required.
Architecture: clearer separation between accessibility permissions/session controls and screen intelligence policy/config ownership.
Data impact: screen-intelligence summaries are persisted under screen_intelligence memory category.

Breaking Changes

None
Yes (describe clearly, including migration steps)

- Reformatted code in `core_server.rs` for better readability, including consistent indentation and line breaks. - Enhanced the clarity of function calls and JSON handling by spreading parameters across multiple lines. - Improved overall structure and maintainability of the dispatch and settings view response functions.

…IntelligenceConfig - Renamed AccessibilityAutomationConfig to ScreenIntelligenceConfig across the codebase for improved clarity. - Updated related structures and functions to utilize the new configuration, including default values and additional fields for enhanced functionality. - Ensured backward compatibility by adjusting references in the AccessibilityEngine and related modules. - Enhanced the AppContext structure to include window bounds, improving state management for UI components.

… functionality - Added a new Screen Intelligence module to manage window capture policies, vision summaries, and memory ingestion. - Implemented settings for enabling/disabling features, configuring capture policies, and managing allowlists/deny lists. - Updated the accessibility panel to include Screen Intelligence options and integrated permission requests for screen recording and accessibility. - Enhanced the core server to handle updates to Screen Intelligence settings and ensure proper state management across the application. - Refactored related components and Redux state management to support the new features, improving user experience and functionality.

Claude Agents updated

…settings (tinyhumansai#50) * refactor(core_server): improve code formatting and readability - Reformatted code in `core_server.rs` for better readability, including consistent indentation and line breaks. - Enhanced the clarity of function calls and JSON handling by spreading parameters across multiple lines. - Improved overall structure and maintainability of the dispatch and settings view response functions. * feat(accessibility): refactor AccessibilityAutomationConfig to ScreenIntelligenceConfig - Renamed AccessibilityAutomationConfig to ScreenIntelligenceConfig across the codebase for improved clarity. - Updated related structures and functions to utilize the new configuration, including default values and additional fields for enhanced functionality. - Ensured backward compatibility by adjusting references in the AccessibilityEngine and related modules. - Enhanced the AppContext structure to include window bounds, improving state management for UI components. * feat(screen-intelligence): introduce Screen Intelligence settings and functionality - Added a new Screen Intelligence module to manage window capture policies, vision summaries, and memory ingestion. - Implemented settings for enabling/disabling features, configuring capture policies, and managing allowlists/deny lists. - Updated the accessibility panel to include Screen Intelligence options and integrated permission requests for screen recording and accessibility. - Enhanced the core server to handle updates to Screen Intelligence settings and ensure proper state management across the application. - Refactored related components and Redux state management to support the new features, improving user experience and functionality. * chore: apply rust fmt and fix blocking eslint warning

…avior The upstream merge replaced the old cron_agent_job_local_provider_offline_trips_ halt_guard test with cron_agent_job_short_loopback_send_error_stays_retryable — i.e. upstream decided an offline local provider STAYS RETRYABLE (it may be transiently starting up), rather than tripping the halt guard. My earlier cron loopback work went the opposite way (surface the connection- refused errno so the offline signal trips the halt guard). With the local-runtime flip routing lmstudio crate-native and tinyagents#50 (error_source_chain) surfacing the errno, run_agent_job's raw now carried "connection refused (os error N)", so is_local_provider_unreachable_failure returned true and the new upstream test failed. Defer to upstream: - run_agent_job raw back to e.to_string() (drop the {e:#} cause-chain surfacing). - Remove the is_non_retryable loopback fast-fail branch. - Re-pin vendor/tinyagents to 7c6e81a (before #50), so the crate-native local error does not surface the errno the classifier trips on. #50 stays a good crate improvement but is deliberately not consumed here. The local-runtime crate cutover itself (which needs #49's toggles, not #50) is unchanged. Resolves the sole failure after the upstream merge. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…ative Extends the BYOK cloud cutover to cover EVERY configured cloud slug except the managed OpenhumanJwt entry. try_create_cloud_slug_chat_model now: - Codex OAuth (Bearer + using_oauth) -> crate OpenAiModel on the Responses API (with_responses_api_primary), with the codex account/originator headers, user-agent, client_version query param, and max_output_tokens omitted. - non-codex openai + custom slugs -> crate OpenAiModel Chat Completions. The legacy host's rare chat-404 -> /v1/responses fallback is NOT replicated (the crate has responses-primary, not fallback); chat completions is these slugs' primary path. Enabled by the crate /v1/responses port (tinyagents#51); host pinned at 8e57665 (= #49 toggles + #51 Responses, with #50 error-source-chain reverted via #52 so an offline local provider stays retryable). CrateOpenAiConfig / build_crate_openai_model gain responses_api_primary / responses_omit_max_output_tokens / extra_query_params / user_agent knobs. create_chat_model no longer constructs a host OpenAiCompatibleProvider for any managed/local/cloud role; the remaining users of compatible*.rs are the direct create_chat_provider callers (follow-up). Lib compiles clean. LIVE per-provider validation (real openai /responses + codex OAuth + each cloud wire) is deferred to a dedicated tinyagents run. Claude-Session: https://claude.ai/code/session_01U8NDGbt9tKj443VquzycLB

…ts updated\n

…settings (tinyhumansai#50)\n\n* refactor(core_server): improve code formatting and readability\n\n- Reformatted code in `core_server.rs` for better readability, including consistent indentation and line breaks.\n- Enhanced the clarity of function calls and JSON handling by spreading parameters across multiple lines.\n- Improved overall structure and maintainability of the dispatch and settings view response functions.\n\n* feat(accessibility): refactor AccessibilityAutomationConfig to ScreenIntelligenceConfig\n\n- Renamed AccessibilityAutomationConfig to ScreenIntelligenceConfig across the codebase for improved clarity.\n- Updated related structures and functions to utilize the new configuration, including default values and additional fields for enhanced functionality.\n- Ensured backward compatibility by adjusting references in the AccessibilityEngine and related modules.\n- Enhanced the AppContext structure to include window bounds, improving state management for UI components.\n\n* feat(screen-intelligence): introduce Screen Intelligence settings and functionality\n\n- Added a new Screen Intelligence module to manage window capture policies, vision summaries, and memory ingestion.\n- Implemented settings for enabling/disabling features, configuring capture policies, and managing allowlists/deny lists.\n- Updated the accessibility panel to include Screen Intelligence options and integrated permission requests for screen recording and accessibility.\n- Enhanced the core server to handle updates to Screen Intelligence settings and ensure proper state management across the application.\n- Refactored related components and Redux state management to support the new features, improving user experience and functionality.\n\n* chore: apply rust fmt and fix blocking eslint warning\n

@lid

Takes tinychannels #50 (ChannelMessage.sender_name, the sender's display name; a WhatsApp LID-only sender stays <lid>@lid instead of +<lid>) and #47 (tokio-tungstenite 0.29 -> 0.30). - The gitlink and the compiled registry pin move together. The record points at the v0.1.11 release; each of its 11 platform digests is copied from the release's published checksum.toml, which matches GitHub's own asset digest. The release covers the same 11 platforms. - The gitlink now sits on the tag, so the source/module skew (13 commits past v0.1.10, tinychannels#49 among them) is gone and its exemption in module-pin-exemptions.json is removed. - sender_name: None at the 38 ChannelMessage literals that name every field; the 4 that spread from another message keep working. - Lockfiles: the three tinychannels crates 0.1.10 -> 0.1.11; tinychannels now depends on tokio-tungstenite 0.30.0, added beside the 0.29.0 that openhuman, axum and tinyliveagents keep, with tungstenite 0.30.0. Their dependencies already resolve in the lock (rand 0.10.2, sha1 0.11.0).

senamakel added 4 commits March 28, 2026 11:55

chore: apply rust fmt and fix blocking eslint warning

5cd0432

senamakel merged commit a0c3089 into tinyhumansai:main Mar 28, 2026
2 of 5 checks passed

AusAgentSmith pushed a commit to AusAgentSmith/openhuman that referenced this pull request May 23, 2026

Merge pull request tinyhumansai#50 from graycyrus/main

f959299

Claude Agents updated

senamakel mentioned this pull request Jul 11, 2026

feat(inference): flip openai/codex + custom cloud slugs to crate-native OpenAiModel #4782

Merged

senamakel mentioned this pull request Aug 14, 2026

Upgrade tinyflows to 0.8: host-owned checkpointer, engine decoupled from tinyagents, five new node kinds #5543

Merged

8 of 12 tasks

senamakel added a commit to nocstah/openhuman that referenced this pull request Sep 11, 2026

Merge pull request tinyhumansai#50 from graycyrus/main\n\nClaude Agen…

062b44d

…ts updated\n

M3gA-Mind mentioned this pull request Oct 8, 2026

chore: bump tinymemory to v1.23.9 and tinychannels to v0.1.11 #7130

Merged

12 tasks done

senamakel mentioned this pull request Oct 11, 2026

[core-shrink] integrations: move composio contract tables and payload types into tinyconnectors #7413

Closed

4 tasks

senamakel mentioned this pull request Oct 19, 2026

Epic: tinystoragedrivers as a hardened static primitive (types + API + per-primitive crates, engines as features) #8131

Closed

29 tasks done

Navigation Menu

Repository navigation

feat: split screen intelligence from accessibility and add dedicated settings - #50

feat: split screen intelligence from accessibility and add dedicated settings#50
senamakel merged 4 commits into
tinyhumansai:mainfrom
senamakel:feat/screen-intelligence

senamakel commented Mar 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

senamakel commented Mar 28, 2026

Summary

Problem

Solution

Testing

Impact

Breaking Changes

Related

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant