Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
518c52c
test(multimodal): expect zero files when disabled
senamakel Oct 16, 2026
e8af402
fix(multimodal): preserve zero max_files as hard-disable in effective…
senamakel Oct 16, 2026
2f9a8a2
feat(multimodal): add media request module
senamakel Oct 16, 2026
d03be14
test(multimodal): add request tests
senamakel Oct 16, 2026
3c20e3b
test(multimodal): build requests via struct literal and inline text e…
senamakel Oct 16, 2026
f0eb6cc
fix(multimodal): clamp max_files to at least one
senamakel Oct 16, 2026
d63e9a6
fix(multimodal): preserve zero max_files as hard disable
senamakel Oct 16, 2026
af5f06c
fix(multimodal): enforce remote fetch policy and preserve hard disable
senamakel Oct 16, 2026
2d16afc
docs(multimodal): document the media request model and host seams
senamakel Oct 16, 2026
b92bca7
fix(multimodal): reject media anywhere in the request when files are …
senamakel Oct 16, 2026
5c019bd
feat(multimodal): bound historical media resolution with host fallback
senamakel Oct 16, 2026
ed276bb
test(multimodal): cover history budget, scope isolation, and safe fal…
senamakel Oct 16, 2026
d4f6f22
test(multimodal): import ImageRef in request tests
senamakel Oct 16, 2026
8175c50
refactor(multimodal): re-export history fallback and tidy request for…
senamakel Oct 16, 2026
a4f8a2f
docs(multimodal): document history fallback and model profile seams
senamakel Oct 16, 2026
0e5a7b8
docs(multimodal): clarify host-owned decisions and counting rules
senamakel Oct 16, 2026
8f69549
test(multimodal): gate archive fallback test behind feature flag
senamakel Oct 16, 2026
a5c0aae
fix(multimodal): recover historical media failures per attachment
senamakel Oct 16, 2026
3dc7686
test(multimodal): cover historical media fallback recovery
senamakel Oct 16, 2026
e334aca
test(multimodal): cover fatal source failure in latest turn
senamakel Oct 16, 2026
1b68bf3
test(multimodal): rename request binding in history fallback test
senamakel Oct 16, 2026
b61b99a
fix(multimodal): apply history budget per media reference
senamakel Oct 16, 2026
8936a00
fix(multimodal): truncate attachment context to the configured limit
senamakel Oct 16, 2026
65865b6
test(multimodal): cover unicode-safe truncation of attachment context
senamakel Oct 16, 2026
6248bf3
refactor(multimodal): rename historical media block budget to items
senamakel Oct 16, 2026
d02cdd0
style: reformat multimodal request tests
senamakel Oct 16, 2026
a82dd14
test(multimodal): compare request fields individually in normalizatio…
senamakel Oct 16, 2026
8ee64e5
docs(multimodal): document request preparation and fallback limits
senamakel Oct 16, 2026
b5d72e6
fix(multimodal): apply context limit to file payloads
senamakel Oct 16, 2026
f952cd8
fix(multimodal): correct stale comment on media counting
senamakel Oct 16, 2026
4bfa239
feat(multimodal): carry host annotations alongside resolved media
senamakel Oct 16, 2026
075260e
feat(multimodal): carry resolver annotations into resolved media
senamakel Oct 16, 2026
2c94ca6
refactor(multimodal): build file context per attachment
senamakel Oct 16, 2026
901a239
test(multimodal): cover per-attachment file fallback limits
senamakel Oct 16, 2026
da3a286
refactor(multimodal): extract request scope factory type alias
senamakel Oct 16, 2026
49a8421
docs(multimodal): document resolver return type
senamakel Oct 16, 2026
cf554d4
docs: add host adoption map for 482
senamakel Oct 16, 2026
9812498
docs(host-adoption): fix attachment provider path in adoption map
senamakel Oct 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 28 additions & 12 deletions crates/tinyagents-harness/src/multimodal/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ message text.

A user attaches a picture or a document; somewhere between the text box and
the provider that attachment has to become bytes the model can read —
validated, size-capped, MIME-checked, and rendered into the message. This
module is that pipeline, minus every decision that belongs to a host.
validated, size-capped, MIME-checked, and rendered into the message. The
multimodal module provides that generic pipeline while leaving source access,
scope, selected-model facts, and other host policy behind explicit seams.
Attachments travel as markers inside message text rather than a parallel
structured field, so they survive every hop a host already has (persistence,
summarisation, delegation) without each of those learning about attachments.
Expand Down Expand Up @@ -45,6 +46,18 @@ format contributes a header naming it plus a content hash.
[`compose_multimodal_message`] (renders the final provider-bound message),
and supporting helpers (`truncate_chars`, `sha256_prefix`, `format_size`,
`escape_attr`).
- [`request`] — [`MediaRequestModel`], a `ChatModel` decorator that prepares a
disposable request, checks counts before reads, preserves per-call scope,
applies a bounded historical-media-item budget, dispatches native media only
when both model and transport support it, and renders character-bounded
text/archive/image fallbacks. The host supplies [`MediaSourceResolver`] for
authorized source access, [`MediaHistoryFallback`] for its safe history
behavior, and may supply [`MediaImageReadout`] plus selected-model
[`ModelProfile`] facts. A resolver returns [`ResolvedMedia`] so host-authored
safe path or recovery annotations stay beside native media and in fallbacks.
Its synchronous trusted-request callback may consume request metadata or
normalize host markers on the disposable copy before counting; it must not
resolve sources or perform other I/O.
- [`error`] — [`MultimodalError`] and its [`Result`] alias; every variant
carries the offending `input` verbatim so a multi-attachment turn's failure
is attributable.
Expand All @@ -66,23 +79,26 @@ format contributes a header naming it plus a content hash.
| `archive/zip_admission.rs` | Allocation-free ZIP preflight and admitted reader. |
| `png.rs` | Optional lossless PNG optimization. |
| `payload.rs` | `FilePayload`, message composition, truncation, hashing. |
| `request.rs` | `MediaRequestModel` and host scope/resolver/fallback seams. |
| `error.rs` | `MultimodalError`, `Result`. |
| `mod_tests.rs`, `resolve_tests.rs`, `archive/ops_tests.rs`, `png_tests.rs`, `blob_stash_tests.rs` | Marker, MIME, size, path, intake, archive, PNG and stash behavior. |
| `mod_tests.rs`, `resolve_tests.rs`, `request_tests.rs`, `archive/ops_tests.rs`, `png_tests.rs`, `blob_stash_tests.rs` | Marker, MIME, size, path, intake, request preparation, archive, PNG and stash behavior. |

## Operational constraints

- **Count before resolving.** `FileLimits::files_disabled` and the per-turn
count caps must be checked against the raw markers, before any read
happens — a cap enforced after the fetch is not a cap.
- **Check the sentinel before the clamp.** `max_files == 0` means *none*;
`FileLimits::effective` clamps it up to `1`. Consulting only the clamped
value would admit one attachment from a source that asked for zero.
- What stays with the host, deliberately: the `TextExtractor` implementation (which parser, if any, and its
timeout), the attachment stash (where bytes live between ingress and
dispatch), and message-level marker counting (only the host knows its
message type). This module never decides which local paths may be read —
that is `FileLimits::files_disabled`'s lever, not a filesystem allowlist
here.
- **Preserve the hard disable.** `max_files == 0` means *none* and remains
zero in `FileLimits::effective`. `MediaRequestModel` checks it before asking
the host source resolver to read anything.
- Final image readout and archive fallback context is capped using
`max_extracted_text_chars` and Unicode scalar values, then marked when
truncated.
- What stays with the host, deliberately: the `TextExtractor` implementation
(which parser, if any, and its timeout), the attachment stash (where bytes
live between ingress and dispatch), image-readout model choice, source
authorization, access scope, selected-model profile, and safe historical
placeholders. `MediaRequestModel` never decides which local paths may be read.
- Opt-in remote image and file URLs pass TinyTools' URL and DNS guard. The
resolver pins the vetted addresses in its own direct HTTP client and refuses
redirects. It uses 30-second request and 10-second connection timeouts.
Expand Down
29 changes: 15 additions & 14 deletions crates/tinyagents-harness/src/multimodal/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@
//!
//! ## The one asymmetry worth knowing
//!
//! `max_files == 0` is a **hard-disable sentinel**, and it is checked before
//! the clamp rather than after. The clamp lifts `0` to `1`, so a caller that
//! only consulted [`FileLimits::effective`] would admit a single file marker
//! from a source that asked for none. [`FileLimits::files_disabled`] is the
//! check that must run first; [`resolve`](super::resolve) does exactly that.
//! `max_files == 0` is a **hard-disable sentinel**. It remains zero in the
//! effective limits, so every caller can enforce the same hard stop before
//! resolving a source.

use serde::{Deserialize, Serialize};

Expand Down Expand Up @@ -76,9 +74,8 @@ impl ImageLimits {
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default)]
pub struct FileLimits {
/// Maximum files accepted in one turn. Clamped to `1..=16` — except `0`,
/// which is the hard-disable sentinel checked by
/// [`FileLimits::files_disabled`] before any clamp applies.
/// Maximum files accepted in one turn. Clamped to `1..=16`; `0` disables
/// file resolution outright.
pub max_files: usize,
/// Maximum size of a single file, in MiB. Clamped to `1..=50`.
pub max_file_size_mb: usize,
Expand Down Expand Up @@ -128,10 +125,15 @@ pub fn default_allowed_file_mime_types() -> Vec<String> {

impl FileLimits {
/// Clamp configured values into runtime bounds:
/// `(max_files, max_file_size_mb, max_extracted_text_chars)`.
/// `(max_files, max_file_size_mb, max_extracted_text_chars)`. A zero file
/// count is preserved as the hard-disable setting.
pub fn effective(&self) -> (usize, usize, usize) {
(
self.max_files.clamp(1, 16),
if self.max_files == 0 {
0
} else {
self.max_files.clamp(1, 16)
},
self.max_file_size_mb.clamp(1, 50),
self.max_extracted_text_chars.clamp(1_000, 200_000),
)
Expand All @@ -144,10 +146,9 @@ impl FileLimits {

/// `true` when file markers are disabled outright.
///
/// The sentinel exists so a host can refuse `[FILE:…]` resolution for a
/// turn whose text came from somewhere it does not trust, without editing
/// the operator's own allowlist. It must be consulted **before**
/// [`FileLimits::effective`], whose clamp lifts `0` to `1`.
/// The sentinel lets a host refuse media resolution for an untrusted turn
/// without editing the operator's own allowlist. It is also preserved by
/// [`FileLimits::effective`].
pub fn files_disabled(&self) -> bool {
self.max_files == 0
}
Expand Down
26 changes: 16 additions & 10 deletions crates/tinyagents-harness/src/multimodal/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,17 @@
//!
//! ## What stays with the host
//!
//! Four things, each because it depends on the host's own runtime, storage, or
//! threat model rather than on anything universal:
//! Host decisions remain host-owned because they depend on its runtime,
//! storage, or threat model:
//!
//! | Host owns | Why |
//! | --- | --- |
//! | The `reqwest::Client` | retained in resolver signatures for compatibility; remote fetches use a DNS-pinned direct client with redirects disabled |
//! | [`TextExtractor`] | which document parser (if any) a host carries, and how long it may run |
//! | The stash policy | which storage scope holds bytes between ingress and dispatch, how large it may grow, how long they live ([`BlobAttachmentStash`] is the mechanism) |
//! | Message-level counting | only the host knows what its message type is |
//! | The source resolver and scope | only the host can authorize a path, fetch remote data, and bind a request to trusted access context |
//! | The selected model profile | catalog facts and transport serialization support are independent inputs |
//! | Historical placeholders and image readout | safe path hints and non-vision model choice are host policies |
//!
//! The stash deserves a note, because the split is not obvious. A host that
//! persists conversations should **not** persist a multi-megabyte `data:` URI:
Expand All @@ -62,13 +64,12 @@
//!
//! Two rules, both learned the hard way:
//!
//! 1. **Count before resolving.** [`config::FileLimits::files_disabled`] and the
//! per-turn caps are checked against the raw markers, before any read
//! happens. A cap enforced after the fetch is not a cap.
//! 2. **Check the sentinel before the clamp.** `max_files == 0` means *none*,
//! and [`config::FileLimits::effective`] clamps it to 1. Consulting only the
//! clamped value admits exactly one attachment from a source that asked for
//! zero.
//! 1. **Count before resolving.** [`request::MediaRequestModel`] checks latest
//! turn caps and its bounded history budget against raw markers before any
//! source resolver is called. A cap enforced after the fetch is not a cap.
//! 2. **Preserve the hard disable.** `max_files == 0` means *none* and stays
//! zero in [`config::FileLimits::effective`]. The request decorator checks
//! this before calling a host resolver.

pub mod archive;
pub mod config;
Expand All @@ -78,6 +79,7 @@ pub mod markers;
pub mod mime;
pub mod payload;
pub mod png;
pub mod request;
pub use png::optimize_png_lossless;
pub mod blob_stash;
pub mod resolve;
Expand All @@ -101,6 +103,10 @@ pub use markers::{
text_has_image_placeholders,
};
pub use payload::{FilePayload, compose_multimodal_message, sha256_prefix, truncate_chars};
pub use request::{
MediaHistoryFallback, MediaImageReadout, MediaRequestConfig, MediaRequestModel,
MediaSourceResolver, ResolvedMedia,
};
pub use resolve::{
NoTextExtractor, TextExtractor, resolve_attachment, resolve_file, resolve_image,
};
Expand Down
4 changes: 2 additions & 2 deletions crates/tinyagents-harness/src/multimodal/mod_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -353,8 +353,8 @@ fn the_disable_sentinel_survives_the_clamp() {
assert!(disabled.files_disabled());
assert_eq!(
disabled.effective().0,
1,
"the clamp lifts 0 to 1 — which is exactly why files_disabled exists"
0,
"zero files must remain disabled after applying runtime limits"
);
assert!(!FileLimits::default().files_disabled());
}
Expand Down
Loading
Loading