feat: vision-language stack batches 1-2 - contrastive encoders and vision foundations (25 models) - #869
Merged
Conversation
…sion foundations (25 models) Implements Batches 1 and 2 of issue #273 Vision-Language Stack: Batch 1 - Contrastive Vision-Language Encoders (16 models): - OpenCLIP, SigLIP (fixed build errors in existing files) - ALIGN (EfficientNet), BASIC (CoAtNet), LiT, FLIP - EVA-CLIP, MetaCLIP, DFN-CLIP, CLIPA, LLM2CLIP - DeCLIP, RegionCLIP, BiomedCLIP, MedCLIP, RemoteCLIP Batch 2 - Vision Encoders / Foundation Models (9 models): - ViT, DINOv2, DINOv3, SAM, InternViT - SigLIP-SO, RADIOv2.5, Perception Encoder, Florence-2 Infrastructure: - VisionLanguageModelBase<T> with dual ONNX/native mode - IContrastiveVisionLanguageModel<T>, IVisualEncoder<T>, ITextEncoder<T> - ContrastiveEncoderOptions, VisionEncoderOptions base classes - VisionLanguageEnums (ViTVariant, PoolingStrategy, etc.) - LayerHelper methods: CreateDefaultOpenCLIPLayers, CreateDefaultSigLIPLayers, CreateDefaultALIGNLayers, CreateDefaultBASICLayers, CreateDefaultViTLayers, CreateDefaultFlorence2Layers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a comprehensive Vision-Language stack: new interfaces/enums, extensive LayerHelper factories, and dozens of model classes with corresponding Options for contrastive encoders, pure vision encoders, foundational/fusion VLMs, generative/Q-Former VLMs, instruction-tuned/chat VLMs, grounding/referring models, document OCR/VLMs, and image editing VLMs. Dual-mode (native/ONNX), serialization, tokenization, training, and metadata supported. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant Model as Contrastive VLM
participant Tokenizer
participant ImgEnc as ImageEncoder (ONNX/Native)
participant TxtEnc as TextEncoder (ONNX/Native)
participant Sim as Similarity
User->>Model: ComputeSimilarity(image, text)
Model->>ImgEnc: EncodeImage(image)
Model->>Tokenizer: Tokenize(text)
Tokenizer-->>Model: token ids
Model->>TxtEnc: EncodeText(tokens)
ImgEnc-->>Model: image embedding
TxtEnc-->>Model: text embedding
Model->>Sim: Cosine(image,text) with temperature
Sim-->>Model: score
Model-->>User: similarity score
sequenceDiagram
autonumber
actor User
participant Gen as Generative VLM
participant Tokenizer
participant Vision as Vision Encoder (ONNX/Native)
participant Decoder as Text Decoder
participant Out as Output
User->>Gen: GenerateFromImage(image, prompt?)
Gen->>Vision: EncodeImage(image)
alt prompt provided
Gen->>Tokenizer: Tokenize(prompt)
Tokenizer-->>Gen: token ids
end
Gen->>Decoder: Decode(vision features, tokens?)
Decoder-->>Gen: logits/tokens
Gen->>Out: Postprocess
Out-->>User: generated text/tokens
Estimated code review effort🎯 5 (Critical) | ⏱️ ~180 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements Batches 1 and 2 of the Vision-Language Stack (issue #273), adding 25 new vision-language models organized into two categories: contrastive vision-language encoders and standalone vision encoders. The implementation includes comprehensive infrastructure with base classes, interfaces, option classes, and enums to support dual-mode operation (ONNX inference and native training).
Changes:
- Added
VisionLanguageModelBase<T>base class with ONNX and native training support - Implemented 16 contrastive encoder models (OpenCLIP, SigLIP, ALIGN, BASIC, LiT, FLIP, EVA-CLIP, MetaCLIP, DFN-CLIP, CLIPA, LLM2CLIP, DeCLIP, RegionCLIP, BiomedCLIP, MedCLIP, RemoteCLIP)
- Implemented 9 vision encoder models (ViT, DINOv2, DINOv3, SAM, InternViT, SigLIP-SO, RADIOv2.5, Perception Encoder, Florence-2)
- Added interfaces (IVisualEncoder, ITextEncoder, IContrastiveVisionLanguageModel) and comprehensive configuration option classes
Reviewed changes
Copilot reviewed 58 out of 58 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| VisionLanguageModelBase.cs | Base class providing ONNX/native dual-mode support, image preprocessing, and utility methods |
| IVisualEncoder.cs, ITextEncoder.cs, IContrastiveVisionLanguageModel.cs | Core interfaces for encoder functionality |
| VisionLanguageEnums.cs | Enums for model variants, loss types, datasets, and architecture options |
| ContrastiveEncoderOptions.cs, VisionEncoderOptions.cs | Base option classes with common hyperparameters |
| 25 model implementation files | Contrastive encoders (Batch 1) and vision encoders (Batch 2) with dual-mode support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 49
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/Helpers/LayerHelper.cs`:
- Around line 22549-22612: The CNN-stage loop inside CreateDefaultBASICLayers
currently emits DenseLayer stacks as placeholders for MBConv CNN stages; replace
that block (the for loop over cnnStages that yields
DenseLayer<T>(visionEmbeddingDim, visionFfnDim,...), LayerNormalizationLayer<T>,
DenseLayer<T>(visionFfnDim, visionEmbeddingDim,...), etc.) with real
convolutional/MBConv stage implementations (e.g., instantiate and yield an
MBConvBlock/CoAtNetConvStage or call a new helper like
CreateCoAtNetCNNStage/BuildMBConvStage that returns ILayer<T> instances)
ensuring you preserve the same input/output embedding dims (visionEmbeddingDim),
maintain normalization/dropout semantics (keep LayerNormalizationLayer<T> and
DropoutLayer<T> placements or incorporate them into the MBConv block), and adapt
any intermediate channel changes (visionFfnDim) to actual conv
expansion/pointwise conv channels so the downstream transformer stages and final
Dense projection remain compatible.
- Around line 22497-22547: The CreateDefaultALIGNLayers method currently
implements the ALIGN vision encoder as a dense-layer "approximation" (see the
vision encoder block that yields DenseLayer<T> repeatedly and uses
visionFfnDim), which is blocking because it is not a real EfficientNet‑B7;
replace that dense-only stack with a proper EfficientNet‑B7 implementation
(MBConv blocks, SE modules, correct conv/stride/pool stages) or delegate to an
existing EfficientNet helper (e.g., call an EfficientNetHelper.CreateB7Layers or
introduce an MBConvBlock/SEBlock class and assemble the EfficientNet stages
inside CreateDefaultALIGNLayers), keep the final projection to projectionDim,
and remove/update the misleading comment about a dense approximation so the code
reflects the real model construction.
In `@src/VisionLanguage/Encoders/ALIGN.cs`:
- Around line 304-309: The Dispose(bool disposing) override currently only sets
_disposed and calls base.Dispose(disposing) but does not release ONNX resources;
update Dispose(bool disposing) to, when disposing is true and not already
disposed, dispose and null out any ONNX-related fields (for example _session,
_onnxModel or similar members used to hold the ONNX inference session/resources)
before setting _disposed and calling base.Dispose(disposing) so the ONNX native
resources are properly released.
- Around line 281-297: ForwardVisionEncoder and ForwardTextEncoder currently
split Layers using Layers.Count / 2 which ignores the configured counts; change
them to use the configured values (use _options.NumVisionLayers for the vision
loop end and _options.NumVisionLayers/_options.NumTextLayers to compute the text
loop start/end as appropriate) and validate that _options.NumVisionLayers +
_options.NumTextLayers == Layers.Count (throw or handle mismatch). Update
ForwardVisionEncoder (replace visionLayerEnd = Layers.Count / 2 with
visionLayerEnd = _options.NumVisionLayers) and ForwardTextEncoder (set
textLayerStart = _options.NumVisionLayers and iterate for _options.NumTextLayers
layers) so the encoder split honors the options.
In `@src/VisionLanguage/Encoders/BASIC.cs`:
- Around line 178-179: The Dispose implementation in BASIC.cs currently only
sets _disposed and calls base.Dispose but does not release ONNX resources;
update the Dispose(bool disposing) override in the OnnxImageEncoder and
OnnxTextEncoder classes to, when disposing is true and not already disposed,
call Dispose on any IDisposable ONNX fields (e.g. _session, _inferenceSession,
_sessionOptions, _allocator or similarly named fields that hold OnnxRuntime
resources), set those fields to null, then set _disposed = true and call
base.Dispose(disposing); ensure you reference the Dispose(bool disposing) method
and the exact ONNX field names present in OnnxImageEncoder and OnnxTextEncoder
when implementing the changes.
- Around line 175-176: The ForwardVisionEncoder and ForwardTextEncoder functions
use a hardcoded midpoint (Layers.Count / 2) which breaks when NumVisionLayers
and NumTextLayers are configured separately; change them to use the configured
layer counts (e.g., Options.NumVisionLayers and Options.NumTextLayers or the
class properties that hold those values). Specifically set end = NumVisionLayers
(or Options.NumVisionLayers) in ForwardVisionEncoder and start = NumVisionLayers
(or Options.NumVisionLayers) / or alternatively start = Layers.Count -
NumTextLayers if only NumTextLayers is available in ForwardTextEncoder so the
loops iterate exactly over the vision and text layer ranges; update any related
comments/tests that assumed a 50/50 split.
In `@src/VisionLanguage/Encoders/BiomedCLIP.cs`:
- Line 31: In ZeroShotClassify, guard against non-positive _options.Temperature
before using it for division: check _options.Temperature (used as temp) and if
it's <= 0 either throw a clear ArgumentException (including the invalid value)
or replace it with a safe positive fallback (e.g., Math.Epsilon) before
computing logits; update the logic around temp/CosineSimilarity and Softmax to
use the validated value so you never divide by zero or a negative temperature.
- Around line 28-30: In ONNX mode EncodeText (and by extension EncodeTexts and
ComputeSimilarity) can return raw token IDs when OnnxTextEncoder is null; update
EncodeText to detect IsOnnxMode && OnnxTextEncoder is null (or missing
TextEncoderModelPath) and throw a clear exception (e.g.,
InvalidOperationException) explaining the ONNX text encoder is not loaded and
TextEncoderModelPath is required, so callers cannot silently get non-embeddings;
ensure EncodeTexts and ComputeSimilarity rely on the same guarded EncodeText so
they inherit the protection.
In `@src/VisionLanguage/Encoders/CLIPA.cs`:
- Around line 23-29: The class allows ONNX mode without a text encoder which
makes EncodeText return invalid native-layer outputs; add a fail-fast guard so
that when IsOnnxMode (or _useNativeMode == false) and no OnnxTextEncoder is
configured you throw a clear exception during construction or immediately in
EncodeText; specifically update the CLIPA constructor(s) to validate
_options.TextEncoderModelPath/OnnxTextEncoder when _useNativeMode is false (or
add a check at the start of EncodeText) and throw an InvalidOperationException
(or ArgumentException) referencing TextEncoderModelPath/OnnxTextEncoder and
IsOnnxMode so callers cannot run EncodeText in ONNX mode without a text ONNX
model.
- Around line 38-40: The CLIPA-specific fields InitialImageSize and
ReducedResolutionFraction are not persisted: update SerializeNetworkSpecificData
to write _options.InitialImageSize and _options.ReducedResolutionFraction (after
the existing writes) and update DeserializeNetworkSpecificData to read them back
in the same order and assign to _options.InitialImageSize and
_options.ReducedResolutionFraction; ensure the read/write ordering matches and
keep existing logic that initializes OnnxImageEncoder/OnnxTextEncoder and builds
ModelMetadata in GetModelMetadata.
In `@src/VisionLanguage/Encoders/CLIPAOptions.cs`:
- Around line 25-38: CLIPAOptions currently exposes InitialImageSize,
InitialSequenceLength and ReducedResolutionFraction without validation; add
guards to ensure InitialImageSize and InitialSequenceLength are > 0 and
ReducedResolutionFraction is > 0 and <= 1. Implement the validation either in
the property setters of CLIPAOptions (throw ArgumentOutOfRangeException with
clear messages) or add a Validate() method called during model initialization
that checks these three properties and throws on invalid values; reference the
property names InitialImageSize, InitialSequenceLength, and
ReducedResolutionFraction when adding the checks so callers can find and fix
offending inputs.
In `@src/VisionLanguage/Encoders/ContrastiveEncoderOptions.cs`:
- Around line 111-118: Add input validation to the Temperature property on the
ContrastiveEncoderOptions class: change the auto-property to a property with a
custom setter that throws an ArgumentOutOfRangeException (or ArgumentException)
if a value <= 0 is assigned, preserving the default of 0.07; ensure the
exception message clearly states that Temperature must be greater than zero and
include the attempted value to aid debugging (reference:
ContrastiveEncoderOptions and the Temperature property).
In `@src/VisionLanguage/Encoders/DeCLIP.cs`:
- Around line 43-44: The methods ForwardVisionEncoder and ForwardTextEncoder
incorrectly assume the split is at Layers.Count/2; replace the hardcoded
half-split with the actual configured split index (e.g., use the existing
configuration field or property that indicates the number of vision layers such
as visionLayerCount, VisionEncoderLayers, or a TextStartIndex) so the vision
encoder runs layers [0 .. visionLayerCount-1] and the text encoder runs layers
[visionLayerCount .. Layers.Count-1]; update ForwardVisionEncoder and
ForwardTextEncoder to read that configuration, validate the split is within
bounds (0 <= split <= Layers.Count) and use it when iterating, and ensure any
constructors or initialization set the correct split value.
In `@src/VisionLanguage/Encoders/DFNCLIP.cs`:
- Around line 24-30: The DFNCLIP class can run in ONNX mode without a configured
text encoder, causing EncodeText to call uninitialized native layers; update the
code to fail fast by adding a guard: in the DFNCLIP constructor that sets
_useNativeMode = false (ONNX mode) verify _options.TextEncoderModelPath is
non-empty and that the file exists (same pattern used for image encoder) and
throw ArgumentException/FileNotFound if not, or alternatively add a check at the
start of EncodeText that if IsOnnxMode && OnnxTextEncoder == null throw
InvalidOperationException indicating a missing text ONNX model; reference
DFNCLIP constructor, _options.TextEncoderModelPath, OnnxTextEncoder, IsOnnxMode,
and EncodeText.
In `@src/VisionLanguage/Encoders/DINOv3.cs`:
- Around line 47-48: The override Dispose(bool disposing) currently only sets
_disposed and calls base.Dispose; modify it to also release ONNX-related
resources when disposing is true by disposing any fields like _onnxSession (or
_inferenceSession/_onnxRunner) and related disposable members, nulling them
afterward, and ensuring this happens before calling base.Dispose(disposing);
keep the existing _disposed guard and only dispose ONNX resources when
disposing==true to follow the dispose pattern.
In `@src/VisionLanguage/Encoders/EVACLIP.cs`:
- Around line 88-89: EVACLIP.Dispose(bool) currently only sets _disposed and
calls base.Dispose(disposing) but does not dispose ONNX resources; update the
Dispose(bool disposing) method in class EVACLIP to check disposing and, if true,
call Dispose() (or null-check + Dispose()) on OnnxImageEncoder and
OnnxTextEncoder instances (and any other ONNX fields) before setting _disposed
and calling base.Dispose(disposing), then null out those fields to avoid
double-dispose; ensure you reference the existing _disposed flag and the
class-level encoder fields (OnnxImageEncoder, OnnxTextEncoder) so resources are
released when the object is disposed.
- Around line 85-86: The layer-splitting uses Layers.Count/2 instead of the
configured counts; update ForwardVisionEncoder and ForwardTextEncoder to use the
options' NumVisionLayers and NumTextLayers boundaries: set the vision loop end
to Options.NumVisionLayers (or the configured NumVisionLayers) and set the text
loop start to that same NumVisionLayers (or compute start as Layers.Count -
Options.NumTextLayers) so the vision encoder (ForwardVisionEncoder) iterates
only the first NumVisionLayers and the text encoder (ForwardTextEncoder)
iterates only the final NumTextLayers; reference the existing Layers collection
and the options properties (NumVisionLayers / NumTextLayers / Options) to locate
and change the loop bounds.
In `@src/VisionLanguage/Encoders/FLIP.cs`:
- Around line 72-74: The one-line implementations of EncodeImage, EncodeText,
and EncodeTexts hinder debugging; refactor each into multi-line bodies that 1)
call ThrowIfDisposed, 2) assign intermediate variables (e.g., var p =
PreprocessImage(image) in EncodeImage, var t = TokenizeText(text) in
EncodeText), 3) branch on IsOnnxMode and use
OnnxImageEncoder.Run/OnnxTextEncoder.Run or
ForwardVisionEncoder/ForwardTextEncoder accordingly, 4) apply L2Normalize to the
result, and 5) in EncodeTexts use an explicit for-loop that calls EncodeText for
each entry and assigns to the e array so breakpoints can be set on each step.
Ensure method names EncodeImage, EncodeText, EncodeTexts, PreprocessImage,
TokenizeText, OnnxImageEncoder, OnnxTextEncoder, ForwardVisionEncoder,
ForwardTextEncoder, L2Normalize and ThrowIfDisposed are used exactly as in the
diff.
- Around line 111-112: The current ForwardVisionEncoder and ForwardTextEncoder
use Layers.Count/2 to split the model which breaks when vision/text layer counts
differ; update them to use the explicit counts from FLIPOptions (NumVisionLayers
and NumTextLayers) or a stored split index: in ForwardVisionEncoder use end =
options.NumVisionLayers (or the stored visionLayerCount) and iterate 0..end-1,
and in ForwardTextEncoder use start = options.NumVisionLayers (or compute start
= Layers.Count - options.NumTextLayers if that pattern is used) and iterate
start..Layers.Count-1 so the layer partitioning matches the configured
NumVisionLayers/NumTextLayers.
- Around line 44-49: The constructor contains multiple statements crammed onto
single lines (setting base.ImageSize/base.ImageChannels/base.EmbeddingDim,
validating/assigning imageEncoderModelPath, creating OnnxImageEncoder, and
conditionally validating/creating OnnxTextEncoder) which hurts readability;
refactor the block so each logical operation is on its own line: set
base.ImageSize, base.ImageChannels, and base.EmbeddingDim on separate lines;
perform the null/whitespace check and File.Exists check for
imageEncoderModelPath on their own lines before assigning
_options.ImageEncoderModelPath; instantiate OnnxImageEncoder on its own line;
and expand the conditional that reads _options.TextEncoderModelPath (tp) into a
multi-line if block that validates File.Exists and then instantiates
OnnxTextEncoder, keeping use of the existing symbols (base.ImageSize,
base.ImageChannels, base.EmbeddingDim, imageEncoderModelPath,
_options.ImageEncoderModelPath, OnnxImageEncoder, _options.TextEncoderModelPath,
OnnxTextEncoder) unchanged.
- Around line 86-91: FLIP is missing random patch masking: use the serialized
_options.MaskingRatio inside ForwardVisionEncoder to randomly keep a subset of
patch embeddings during training (when IsTrainingMode is true) and process only
those through the transformer layers; at inference (when IsTrainingMode is
false) process all patches. Update ForwardVisionEncoder to 1) perform patch
embedding as before, 2) compute numToKeep = ceil((1 - _options.MaskingRatio) *
totalPatches), 3) sample indices (per-sample or batched) to select the kept
patches and a boolean mask for reconstruction if needed, 4) pass only the
selected patch embeddings into the existing Layers pipeline (the one filled via
InitializeLayers / LayerHelper<T>.CreateDefaultOpenCLIPLayers) and ensure
outputs are reassembled to full-patch order if downstream expects full-length
outputs; keep InitializeLayers/CreateDefaultOpenCLIPLayers unchanged except
ensure they accept variable sequence lengths produced by masking.
In `@src/VisionLanguage/Encoders/FLIPOptions.cs`:
- Around line 26-47: The MaskingRatio and UnmaskedTuningEpochs properties need
defensive range validation to prevent invalid external inputs: replace the
auto-properties in FLIPOptions with backing fields and validated setters that
throw ArgumentOutOfRangeException when values are outside allowed bounds
(MaskingRatio must be > 0 and < 1; UnmaskedTuningEpochs must be >= 0 — or >= 1
if UseUnmaskedTuning semantics require at least one epoch), keep the existing
defaults (0.5 and 1), and include clear exception messages naming the property
(MaskingRatio, UnmaskedTuningEpochs) so callers know the valid range.
In `@src/VisionLanguage/Encoders/Florence2.cs`:
- Around line 31-32: The Florence2 constructors are compressed into single lines
which hurts readability; split each constructor body into multiple statements on
separate lines so initialization and checks are explicit: in the
Florence2(NeuralNetworkArchitecture<T>, string modelPath, ...) constructor,
assign _options, set _useNativeMode, set
base.ImageSize/ImageChannels/EmbeddingDim on separate lines, validate modelPath
with separate if checks that throw ArgumentException/FileNotFoundException, set
_options.ModelPath, instantiate OnnxModel and call InitializeLayers each on
their own lines; in the Florence2(NeuralNetworkArchitecture<T>,
Florence2Options?, IGradientBasedOptimizer...) constructor, similarly expand
assignments for _options, _useNativeMode, _optimizer (defaulting to new
AdamWOptimizer<T,...>(this)), set base.ImageSize/ImageChannels/EmbeddingDim, and
call InitializeLayers on its own line to improve clarity.
- Around line 48-49: The Dispose(bool disposing) override in class Florence2
currently only sets _disposed and calls base.Dispose; update it to dispose the
OnnxModel when disposing is true: inside Dispose(bool disposing) check if
(_disposed) return; then if (disposing && OnnxModel != null) call
OnnxModel.Dispose() and set OnnxModel = null; finally set _disposed = true and
call base.Dispose(disposing) to ensure managed ONNX resources are released.
Ensure you reference the existing Dispose(bool disposing), _disposed and
OnnxModel symbols when making the change.
- Line 35: The encoder boundary uses Layers.Count / 2 which assumes a split that
may be wrong for Florence-2; update EncodeImage to use the configured encoder
depth (e.g. _options.NumLayers) instead of Layers.Count/2, set encoderEnd =
_options.NumLayers (ensuring _options is available and validated) and iterate i
from 0 to encoderEnd - 1 over Layers[i].Forward(c), also add a guard that
encoderEnd is within Layers.Count to avoid out-of-range access.
In `@src/VisionLanguage/Encoders/LiT.cs`:
- Around line 149-150: The Dispose(bool disposing) override currently only
toggles _disposed and calls base.Dispose(disposing) but does not release ONNX
resources; update Dispose(bool disposing) to, when disposing is true and
_disposed is false, dispose any ONNX runtime objects used by this class (e.g.,
InferenceSession/OrtSession or similar fields used for model inference) before
calling base.Dispose(disposing), set those fields to null, and keep the existing
_disposed guard; reference the Dispose(bool disposing) method and any
ONNX-related fields (e.g., the session/onnx client member names) to locate where
to add the disposal logic.
- Around line 109-124: The current InitializeLayers uses
LayerHelper<T>.CreateDefaultOpenCLIPLayers which instantiates new OpenCLIP
layers instead of preserving LiT's pretrained ViT structure; update
InitializeLayers to prefer using Architecture.Layers when present (already done)
but when falling back, create LiT-specific vision/text layers or load the
pretrained ViT/LiT backbone instead of OpenCLIP defaults—e.g., replace the
CreateDefaultOpenCLIPLayers call with a LayerHelper<T>.CreateDefaultLiTLayers or
a loader like LayerHelper<T>.LoadPretrainedViTLayers (using
_options.VisionEmbeddingDim, ProjectionDim, NumVisionLayers, NumVisionHeads,
DropoutRate) and ensure any available Architecture.PretrainedWeights or
Architecture.Backbone is used to populate weights so the ViT backbone structure
and pretrained params are preserved.
- Around line 146-147: The bug is that ForwardVisionEncoder and
ForwardTextEncoder independently compute the split, risking an
off-by-one/inconsistent split; compute the midpoint once and use it for both
sides so the split is identical: in ForwardVisionEncoder and ForwardTextEncoder
calculate int mid = Layers.Count / 2 (or use a named variable like mid/end) and
iterate vision layers 0..mid-1 and text layers mid..Layers.Count-1 (i.e., set
start = mid in ForwardTextEncoder), ensuring both methods reference the same
midpoint calculation (functions: ForwardVisionEncoder, ForwardTextEncoder and
expression Layers.Count / 2).
- Around line 127-128: The Train method currently updates all layers, ignoring
the LiT freeze option; modify Train and UpdateParameters to respect
_options.FreezeVisionEncoder by skipping parameter updates and backward passes
for vision encoder layers—identify vision encoder layers via their class/type or
a flag on each layer (e.g., VisionEncoderLayer or layer.IsVision) and in Train
only call Backward and let the optimizer UpdateParameters for layers when
_options.FreezeVisionEncoder is false for those layers; similarly, in
UpdateParameters when _useNativeMode is true, skip slicing/applying parameters
for frozen vision layers so only text-encoder parameters are updated.
In `@src/VisionLanguage/Encoders/LLM2CLIP.cs`:
- Around line 23-29: The ONNX path can produce invalid text embeddings when no
text encoder is configured; in the ONNX constructor (the
LLM2CLIP(NeuralNetworkArchitecture<T>, string imageEncoderModelPath, ...) that
sets _useNativeMode = false) validate that _options.TextEncoderModelPath is
non-empty and the file exists and initialize OnnxTextEncoder (throw
ArgumentException/FileNotFoundException otherwise), and also add a defensive
check in EncodeText: if (IsOnnxMode && OnnxTextEncoder is null) throw
InvalidOperationException("ONNX text encoder not configured"); this ensures we
fail fast instead of returning normalized token IDs.
In `@src/VisionLanguage/Encoders/MedCLIP.cs`:
- Around line 24-30: The ONNX-mode path can return invalid text embeddings
because EncodeText falls back to uninitialized native layers when
OnnxTextEncoder is null; update the MedCLIP constructor that sets _useNativeMode
= false (the one taking imageEncoderModelPath) to validate
_options.TextEncoderModelPath/is Onnx mode: if IsOnnxMode (or _useNativeMode ==
false) and _options.TextEncoderModelPath is null/empty or OnnxTextEncoder is
null after attempting to load, throw an ArgumentException/FileNotFoundException
to fail fast; alternatively, add a guard at EncodeText (method EncodeText) to
throw a clear InvalidOperationException when IsOnnxMode && OnnxTextEncoder is
null instead of falling back to ForwardTextEncoder, and mention the related
symbols (_useNativeMode, OnnxTextEncoder, EncodeText, the MedCLIP ctor that sets
_useNativeMode) so reviewers can locate and apply the change.
- Around line 39-41: GetModelMetadata includes _options.Domain but
SerializeNetworkSpecificData/DeserializeNetworkSpecificData do not persist it,
so add persistence: in SerializeNetworkSpecificData write the domain (e.g., as
an int or string) after the other options, and in DeserializeNetworkSpecificData
read that value and set _options.Domain accordingly (parse or cast back to the
Domain enum). Update the DeserializeNetworkSpecificData flow that initializes
OnnxImageEncoder/OnnxTextEncoder to occur after you restore _options.Domain if
any branching depends on it; reference SerializeNetworkSpecificData,
DeserializeNetworkSpecificData, _options.Domain, and GetModelMetadata to locate
the changes.
In `@src/VisionLanguage/Encoders/MedCLIPOptions.cs`:
- Around line 33-49: MedCLIPOptions currently exposes auto-properties that allow
invalid hyperparameters; replace the auto-properties for SemanticMatchingWeight
and EntitySimilarityThreshold with backing fields and property setters that
validate values are within [0.0, 1.0] and throw ArgumentOutOfRangeException with
a clear message if not; keep UseEntityExtraction as a bool (no change) but
ensure any future mutation points use the validated properties; update any code
that constructs MedCLIPOptions to rely on these setters so invalid
weights/thresholds are rejected early.
In `@src/VisionLanguage/Encoders/MetaCLIP.cs`:
- Line 32: The ZeroShotClassify method uses _options.Temperature (assigned to
temp) without validating it before dividing logits by temp; add a guard at the
start of ZeroShotClassify to ensure _options.Temperature (temp) is > 0 and
handle invalid values by either throwing an ArgumentException with a clear
message or substituting a safe default (e.g., 1.0) before performing
CosineSimilarity/Softmax operations; update any related references (temp,
_options.Temperature) so the division by temp cannot occur when temp <= 0.0.
- Around line 29-31: EncodeText currently allows IsOnnxMode to be true while
OnnxTextEncoder is null and silently falls back to ForwardTextEncoder, producing
invalid embeddings; update EncodeText to validate ONNX configuration: if
IsOnnxMode is true and OnnxTextEncoder is null (or TextEncoderModelPath is
unset), throw an informative exception requiring a valid Text ONNX path or
disable ONNX mode, so callers of EncodeText/EncodeTexts/ComputeSimilarity cannot
proceed with uninitialized text encoder; include the symbols IsOnnxMode,
OnnxTextEncoder, TextEncoderModelPath, EncodeText and ForwardTextEncoder in the
error message to aid debugging.
In `@src/VisionLanguage/Encoders/MetaCLIPOptions.cs`:
- Around line 36-39: Rename the public property UseSubStringMatching to
UseSubstringMatching in MetaCLIPOptions and update its XML doc summary
accordingly; then update all call sites, tests, and serializer mappings to use
the new name. If binary/consumer-compatibility is required, add a short-lived
obsolete shim property named UseSubStringMatching that forwards to the new
UseSubstringMatching (mark with [Obsolete] and update any serialization
attributes as needed) before removing the shim in a future release.
In `@src/VisionLanguage/Encoders/OpenCLIP.cs`:
- Around line 382-401: Both ForwardTextEncoder and Forward use a hardcoded
midpoint (Layers.Count / 2) instead of the configured split; update them to use
_options.NumVisionLayers directly: in ForwardTextEncoder remove the fallback to
Layers.Count/2 and start at textLayerStart = _options.NumVisionLayers > 0 ?
_options.NumVisionLayers : 0 (or simply _options.NumVisionLayers), and in
Forward set visionLayerEnd = _options.NumVisionLayers so the vision loop runs i
from 0 to visionLayerEnd-1; ensure you reference the _options.NumVisionLayers
field when computing the split and keep the existing loop logic using
Layers[i].Forward(current).
- Around line 413-418: The Dispose(bool disposing) override currently only sets
_disposed and calls base.Dispose; update it to also dispose the ONNX resources:
when disposing is true, if _onnxImageEncoder != null call
_onnxImageEncoder.Dispose() and set it to null, and likewise for
_onnxTextEncoder (OnnxImageEncoder and OnnxTextEncoder fields), ensuring
null-checks and swallowing or rethrowing exceptions as appropriate; keep the
existing _disposed guard and call base.Dispose(disposing) as before.
In `@src/VisionLanguage/Encoders/RADIOv25.cs`:
- Around line 42-44: The code currently exposes _options.TeacherModels in
GetModelMetadata but SerializeNetworkSpecificData/DeserializeNetworkSpecificData
do not persist it, causing provenance loss; update SerializeNetworkSpecificData
to write the collection (write its length followed by each string, preserving
null/empty semantics) and update DeserializeNetworkSpecificData to read the
length and rebuild the same concrete collection type used by
_options.TeacherModels (e.g., List<string> or string[]) and assign it back to
_options.TeacherModels before any code that relies on it (e.g., before OnnxModel
creation); keep using the existing SerializeNetworkSpecificData and
DeserializeNetworkSpecificData methods and the same reader/writer pattern for
other fields.
In `@src/VisionLanguage/Encoders/RegionCLIP.cs`:
- Around line 39-41: GetModelMetadata reports _options.Domain but the value
isn't persisted; update SerializeNetworkSpecificData to write _options.Domain
(e.g., as an int or string) and update DeserializeNetworkSpecificData to read
that value and assign it back to _options.Domain (with a safe parse/default if
the stored value is missing or unrecognized). Modify the existing methods in
RegionCLIP (SerializeNetworkSpecificData and DeserializeNetworkSpecificData) to
include this additional write/read step while keeping current writes/reads
ordering consistent and ensuring backward compatibility when older serialized
blobs lack the domain field.
- Around line 24-30: The ONNX path can return invalid text embeddings because
when running in ONNX mode (IsOnnxMode/_useNativeMode==false) and OnnxTextEncoder
is null, EncodeText falls back to uninitialized native layers; update the guard
by either (preferred) validating in the ONNX constructor
RegionCLIP(NeuralNetworkArchitecture<T>, string imageEncoderModelPath, ...) to
require a non-empty TextEncoderModelPath and create OnnxTextEncoder (throw
ArgumentException/FileNotFound if missing), or (alternatively) add a fast-fail
in EncodeText that throws when IsOnnxMode && OnnxTextEncoder == null to prevent
returning token IDs; reference RegionCLIP constructor, OnnxTextEncoder,
EncodeText, _useNativeMode/IsOnnxMode and InitializeLayers when implementing the
check.
In `@src/VisionLanguage/Encoders/RemoteCLIP.cs`:
- Around line 39-41: The GroundSampleDistance value on _options is included in
ModelMetadata but not persisted; update SerializeNetworkSpecificData to call
writer.Write(_options.GroundSampleDistance) and update
DeserializeNetworkSpecificData to read it back (e.g.,
_options.GroundSampleDistance = reader.ReadDouble()) using the matching numeric
type (double) and keep the read/write order consistent with the other fields in
SerializeNetworkSpecificData/DeserializeNetworkSpecificData.
- Around line 24-30: The ONNX path can produce invalid text embeddings because
when running in ONNX mode (RemoteCLIP constructed with imageEncoderModelPath)
EncodeText falls back to native layers that aren't initialized if
OnnxTextEncoder is null; fix by validating/guarding: in the
RemoteCLIP(architecture, imageEncoderModelPath, ...) constructor enforce that
_options.TextEncoderModelPath is non-empty and the file exists (throw
ArgumentException/FileNotFoundException) or alternatively update EncodeText to
check if IsOnnxMode && OnnxTextEncoder is null and throw an
InvalidOperationException requiring a text encoder path; reference
RemoteCLIP(NeuralNetworkArchitecture<T>, string imageEncoderModelPath,...),
EncodeText, IsOnnxMode, OnnxTextEncoder, and TextEncoderModelPath.
In `@src/VisionLanguage/Encoders/SigLIP.cs`:
- Line 69: The field _useNativeMode in class SigLIP is only assigned in
constructors but not marked readonly; change its declaration to "private
readonly bool _useNativeMode" to prevent future mutations and signal intent, and
ensure all constructors that set _useNativeMode continue to assign it during
construction (no other code changes needed).
- Around line 260-272: The Train method uses a null-conditional call
(_optimizer?.UpdateParameters(Layers)) which can silently skip parameter
updates; change this to explicitly require an optimizer: before calling
UpdateParameters verify _optimizer is non-null and throw a clear
InvalidOperationException (or similar) if it's null, then call
_optimizer.UpdateParameters(Layers); reference the Train method and the
_optimizer field (constructor should still default to AdamWOptimizer) so the
invariant is enforced rather than silently ignored.
- Around line 391-407: ForwardVisionEncoder and ForwardTextEncoder currently
split Layers by Layers.Count/2 which is brittle and silently misaligns when
layer counts are uneven or custom architectures are used; fix by computing and
storing an explicit boundary when layers are initialized (e.g., set a private
field like visionLayerEnd or visionLayerCount inside
InitializeLayers/CreateDefaultSigLIPLayers or when Architecture.Layers is
assigned), then have ForwardVisionEncoder use that stored boundary and
ForwardTextEncoder start from it; additionally add a sanity check in
InitializeLayers to validate the sum of vision+text layers equals Layers.Count
(or throw/emit an error) to fail fast on mismatches.
- Around line 360-364: During deserialization where OnnxImageEncoder and
OnnxTextEncoder are re-instantiated (the block checking _useNativeMode and using
_options.ImageEncoderModelPath / _options.TextEncoderModelPath to create new
OnnxModel<T>), add explicit file-existence validation before constructing
OnnxModel<T>; if the path is null/empty or File.Exists(path) is false, throw a
clear, descriptive exception (or set the corresponding
OnnxImageEncoder/OnnxTextEncoder to null and log an error) so users get a
meaningful error instead of an ONNX runtime exception; update the logic around
_useNativeMode, OnnxImageEncoder, and OnnxTextEncoder accordingly.
In `@src/VisionLanguage/Encoders/ViT.cs`:
- Around line 43-45: The ViT variant stored in _options.Variant is shown in
GetModelMetadata but not persisted; update SerializeNetworkSpecificData to write
the Variant (as its underlying integer or use
writer.Write((int)_options.Variant)) and update DeserializeNetworkSpecificData
to read and assign it back (cast the int to the enum type for _options.Variant)
before any logic that depends on it (e.g., the OnnxModel construction); modify
the SerializeNetworkSpecificData and DeserializeNetworkSpecificData methods to
include these read/write operations for the Variant enum.
In `@src/VisionLanguage/VisionLanguageModelBase.cs`:
- Around line 101-107: The NormalizeImage method in VisionLanguageModelBase
currently assumes a channels-first single-image [C,H,W] layout and iterates only
up to mean.Length, which corrupts batched inputs or channel-mismatched tensors;
update NormalizeImage to first validate the tensor shape and channel count
(accept either [C,H,W] or [N,C,H,W]), verify mean.Length and std.Length exactly
equal the channel dimension, and fail-fast by throwing an ArgumentException with
a clear message on mismatch; for batched inputs loop over the channel axis (not
mean.Length) and apply per-channel normalization across all spatial locations
and batch items; also update the XML docs to state the expected layouts and the
CLIP default mean/std values used.
…per architectures Add ViLBERT, LXMERT, VisualBERT, UNITER, Oscar, VinVL, ViLT, METER, and BridgeTower foundational vision-language fusion models with IVisionLanguageFusionModel interface, FoundationalVLMOptions base class, and 4 new LayerHelper methods for dual-stream, single-stream, cross-modal, and bridge fusion architectures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 23
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/Helpers/LayerHelper.cs`:
- Around line 22664-22673: The comment claiming "Spatial window attention (odd
layers) / channel group attention (even layers)" is inaccurate because the loop
yields the same MultiHeadAttentionLayer<T> every iteration; either implement the
DaViT dual-attention behavior by alternating between a spatial-window attention
implementation and a channel-group attention implementation (introduce or use
distinct classes/functions, e.g., SpatialWindowAttentionLayer<T> and
ChannelGroupAttentionLayer<T>, and toggle based on i % 2), or change the comment
to state this is a simplified uniform multi-head attention stack; update
references around MultiHeadAttentionLayer<T>, LayerNormalizationLayer<T>,
DenseLayer<T>, DropoutLayer<T>, and the numEncoderLayers loop accordingly so the
code and comment match.
In `@src/VisionLanguage/Foundational/BridgeTower.cs`:
- Around line 27-53: BridgeTower.cs duplicates most logic from ViLBERT; extract
shared functionality into a generic base (e.g., FoundationalFusionModelBase<T,
TOptions>) that holds the common fields (_options, _tokenizer, _optimizer,
_useNativeMode, _disposed), common ctor behavior, InitializeLayers, EncodeImage,
FuseImageText, ComputeMatchingScore, Predict, Train, UpdateParameters,
serialization, Dispose and ThrowIfDisposed, and expose abstract/virtual hooks
for the options type, the layer factory (e.g., CreateDefaultFusionLayers) and
metadata strings; then refactor BridgeTower<T> to inherit that base, pass
BridgeTowerOptions as TOptions, implement the specific layer factory call
(CreateDefaultBridgeFusionLayers) and override metadata values
(Name/Description/FusionType) so only options, layer creation and metadata
remain in the subclass.
- Line 37: In EncodeImage replace the hardcoded end = Layers.Count / 3 logic
with a deterministic calculation based on explicit layer roles: add or use an
existing property (e.g. EncoderLayerCount or BridgeLayerCount) or derive the
encoder region by inspecting layer metadata (e.g. layer.Role or layer.Name) and
compute end = EncoderLayerCount (or end = Layers.Count - BridgeLayerCount); then
iterate only over those encoder layers (Layers[0..end]) so the division is not
implicit; update any constructors/config to set
EncoderLayerCount/BridgeLayerCount and document the new property.
- Line 43: The Train method currently runs a single forward/backward pass but
neither reports the loss nor enforces gradient clipping; modify the
Train(Tensor<T> input, Tensor<T> expected) implementation (references: Train,
Predict, LossFunction.CalculateDerivative, Layers, Backward,
_optimizer.UpdateParameters, SetTrainingMode) to (1) compute the scalar loss
from the output and expected (e.g., via LossFunction.Calculate or summing the
vector returned by CalculateDerivative reverse-engineered to loss) and return it
(change the signature or provide an out/return value), (2) apply gradient
clipping to per-layer gradients before calling _optimizer.UpdateParameters (clip
by norm or value on tensors produced/held by each Layer after Backward), and (3)
optionally log or expose the loss (e.g., via a returned value or an event) so
callers get training feedback.
- Line 31: The _tokenizer field in class BridgeTower is dead code: it's
instantiated in both BridgeTower constructors but never used; either remove the
_tokenizer field and its instantiation from the constructors, or wire it into
the text-processing path by using _tokenizer in the methods that handle textual
input (e.g., the class methods that encode, tokenize, or build text
embeddings/captions) so text is actually tokenized; locate the BridgeTower
constructors and any text-input/encoding methods and either delete the unused
_tokenizer creation and related using/imports or replace the placeholder text
handling with calls into _tokenizer (ensuring the field is non-null where used).
- Around line 38-39: FuseImageText and ComputeMatchingScore currently ignore the
text parameter; fix by encoding the text and supplying it to the bridge layers
instead of only passing the image tensor. Specifically, in FuseImageText, call
the text encoder (or tokenizer + text embedding routine) to produce a text
Tensor, then feed both the preprocessed image tensor (PreprocessImage/image
variable) and the text tensor into the bridge fusion path (use Layers/bridge
layer API to accept/merge both modalities or concatenate/stack them before
calling l.Forward), preserve OnnxModel branch by ensuring the OnnxModel.Run
receives both inputs in the expected shape when IsOnnxMode is true, and update
ComputeMatchingScore to compute the score from the fused multimodal tensor
returned by FuseImageText (not solely from image values) so text contributes to
the final similarity.
In `@src/VisionLanguage/Foundational/LXMERT.cs`:
- Around line 37-38: FuseImageText currently ignores the text parameter and so
does ComputeMatchingScore; use the existing _tokenizer to tokenize the input
text, pass token ids through the language encoder layers (the text encoder
methods/fields in this class or similar), compute text feature tensor and then
perform cross-modal attention/interaction between the preprocessed image tensor
(p) and the text feature tensor using the model's cross-attention layers (or
Layers if they contain multimodal blocks) to produce the fused Tensor<T> in
FuseImageText; update ComputeMatchingScore to call the corrected FuseImageText
and compute the scalar matching score from that fused representation (using
NumOps conversions as before).
- Line 36: The EncodeImage method currently computes the vision layer range with
int end = Layers.Count / 3 which is fragile; change the design to explicitly
track vision/text/cross-modal boundaries (e.g., add a private field like
_visionLayerEnd and set it during InitializeLayers or when calling
LayerHelper<T>.CreateDefaultCrossModalFusionLayers) or split Layers into
separate collections (VisionLayers, TextLayers, CrossModalLayers), then update
EncodeImage to iterate only over the VisionLayers (or up to _visionLayerEnd) and
remove the implicit /3 calculation so layer reorganization won't break encoding.
- Around line 28-51: The class is unreadable because many constructors and
methods are compressed into single-line statements; expand each compressed
member (notably the two constructors LXMERT(...), InitializeLayers, EncodeImage,
FuseImageText, ComputeMatchingScore, Predict, Train, UpdateParameters,
SerializeNetworkSpecificData, DeserializeNetworkSpecificData, CreateNewInstance,
ThrowIfDisposed, and Dispose) into properly formatted multi-line blocks with
clear statements, braces, and line breaks while preserving existing logic,
control flow, exception checks and referenced symbols (OnnxModel, Layers,
_options, _useNativeMode, _optimizer, NumOps, Tensor<T>, etc.); ensure method
bodies are split across lines for readability and debugging (e.g., assign
intermediate variables on separate lines, expand loops and conditionals) and
keep public APIs and behavior unchanged.
In `@src/VisionLanguage/Foundational/METER.cs`:
- Around line 37-38: FuseImageText currently ignores the text parameter; update
FuseImageText (and ensure ComputeMatchingScore uses it via FuseImageText) to
tokenize and encode the input text with the existing tokenizer (initialized via
ClipTokenizerFactory.CreateSimple), produce a text tensor/embedding, and pass
both image and text tensors into OnnxModel.Run as a named-input dictionary when
IsOnnxMode is true; for the non-ONNX path, incorporate the text embedding into
the fusion pipeline (e.g., combine/concatenate/embed it into the initial tensor
`c` before looping over `Layers` or call a layer that consumes both image and
text) so both fusion paths use the tokenized text. Ensure you reference the
existing symbols FuseImageText, ComputeMatchingScore, OnnxModel.Run, and the
tokenizer field when making the change.
In `@src/VisionLanguage/Foundational/Oscar.cs`:
- Line 26: Oscar<T>.FuseImageText currently ignores the text parameter; update
it to actually tokenize and encode the text and fuse it with image embeddings:
use the existing _tokenizer to convert the text input, run the token ids through
the model's text encoding pipeline (the text encoder/text layers used elsewhere
in this class or named methods like EncodeText/ProcessText), obtain text
embeddings, compute image embeddings as now done, and then pass both embeddings
into the fusion component/layer (the same fusion layers used by other fusion
models or named FuseLayers/FuseEmbeddings in this class) to produce the final
fused representation; ensure method signature and return type remain unchanged
and mirror the same fix in all other IVisionLanguageFusionModel implementations
(LXMERT, VisualBERT, VinVL, ViLT, ViLBERT, METER, UNITER, BridgeTower).
- Around line 37-38: FuseImageText currently ignores the text parameter; update
it to tokenize/encode the text (use the existing tokenizer instance) and include
the resulting text tensor when running the ONNX model by calling OnnxModel.Run
with the named-input dictionary (image and text tensors). In native mode,
compute a text embedding from the tokenizer output and fuse it with the
preprocessed image tensor (e.g., concatenate along the channel/feature
dimension) before feeding through Layers (the loop that calls l.Forward on c),
so both modalities are consumed; update ComputeMatchingScore to continue using
FuseImageText as the fused representation.
In `@src/VisionLanguage/Foundational/OscarOptions.cs`:
- Line 28: Add an inline clarifying comment next to the VisionDim assignment in
the OscarOptions class/property so future maintainers understand the 2054
choice; update the line containing "VisionDim = 2054" to include a comment such
as "// 2048 (visual features) + 6 (bbox/tag dimensions)" (or equivalent wording)
to document the 2048 base visual feature dimension plus the 6 extra bbox/tag
dimensions.
In `@src/VisionLanguage/Foundational/UNITER.cs`:
- Around line 37-38: FuseImageText currently ignores the text parameter (causing
image-only behavior); fix by tokenizing/encoding the text into a tensor and
feeding it into the model: inside FuseImageText (and used by
ComputeMatchingScore) call your tokenizer to produce a text tensor, then if
IsOnnxMode and OnnxModel is not null pass both the preprocessed image tensor (p)
and the text tensor as named inputs to OnnxModel.Run; otherwise create a proper
cross-modal input (e.g., concatenate or otherwise combine the image tensor p and
the text tensor along the model's expected dimension) and then run the
Layer.Forward loop (the Layers collection) on that combined tensor; preserve
existing lifecycle checks like ThrowIfDisposed and ensure tensor shapes/dtype
match model expectations before calling OnnxModel.Run or Layers.Forward.
In `@src/VisionLanguage/Foundational/ViLBERT.cs`:
- Line 42: The Train method (Train in ViLBERT.cs) currently runs
forward/backward but doesn't produce or expose the loss or training metrics;
change Train to compute the loss value using LossFunction.Calculate (or the same
loss used for CalculateDerivative) from the network output `o` and `expected`,
return either the scalar loss or a small training result struct (e.g., loss,
gradient norm) and accumulate it when called by a loop; additionally, before
calling _optimizer.UpdateParameters(Layers) compute gradient norm from `gt` and
apply optional gradient clipping (e.g., clamp `gt` if above a threshold) and
invoke any learning rate scheduler hook (call a provided scheduler method or
expose a callback) via existing members (SetTrainingMode, Layers, _optimizer,
LossFunction, Tensor<T>.FromVector) so the method both updates parameters and
returns/records the loss and metric info for monitoring.
- Line 30: The _tokenizer field is instantiated in the constructors but never
used; update FuseImageText and ComputeMatchingScore to call _tokenizer.Tokenize
(or the appropriate tokenize method on ITokenizer) to preprocess text input
before embedding/matching, ensure you handle the nullable _tokenizer (throw or
fallback if null) and pass the tokenized input into the existing text-encoding
or fusion pipeline (e.g., replace raw text parameters with tokenized output in
FuseImageText and ComputeMatchingScore), and remove the field or its
initialization only if you intentionally decide to not use tokenization.
- Line 36: EncodeImage currently uses the magic expression Layers.Count / 3 to
decide the vision endpoint which is fragile; change this to a deterministic,
documented source: add a VisionLayerCount property to the model/options that is
set when layers are created (e.g., by
LayerHelper<T>.CreateDefaultDualStreamFusionLayers) or, alternatively, compute
the vision endpoint by scanning layer metadata/tags (e.g., a Layer.Tag or
Layer.Type on each entry in Layers) rather than dividing by 3; update
EncodeImage to use that explicit count (or the scanned index) and add a short
comment documenting the invariant and throw a clear exception if the expected
vision layer boundary cannot be found.
- Around line 28-35: The file has many collapsed statements making the ViLBERT
constructor, field declarations (e.g., private readonly ViLBERTOptions _options,
_optimizer, _tokenizer, _useNativeMode, _disposed) and property implementations
hard to read; reformat by expanding each field declaration, constructor
parameter checks and assignments, OnnxModel creation, tokenizer initialization
and InitializeLayers() call onto separate well-indented lines inside both
ViLBERT constructors (the one taking modelPath and the native-mode overload),
and place each property (EmbeddingDimension, FusionEmbeddingDim,
MaxSequenceLength, IVisualEncoder members) on its own line — keep logic
unchanged but improve readability with standard C# line breaks and indentation
around methods and blocks.
- Around line 37-38: FuseImageText currently ignores the text parameter; update
FuseImageText to tokenize the text via the _tokenizer, run the resulting tokens
through the text encoder layers (e.g., TextEncoderLayers or TextLayers), obtain
a text embedding, and fuse it with the image representation (e.g., via a
FusionModule or concatenation + fusion Layers) before returning the fused
Tensor; ensure the OnnxModel path also accepts/uses tokenized text when
IsOnnxMode is true and adjust ComputeMatchingScore to rely on the fused
multimodal tensor returned from FuseImageText so the text input actually affects
the score.
In `@src/VisionLanguage/Foundational/ViLT.cs`:
- Around line 37-38: FuseImageText currently ignores the text parameter; update
it to tokenize and encode the text, then fuse the resulting text tensor with the
image tensor before running ONNX or the layered forward pass. Specifically,
inside FuseImageText call the existing text tokenizer/encoder utility (e.g.,
TokenizeText/EncodeText or add one) to produce a Tensor<T> textTensor, handle
the ONNX branch by passing both image and text tensors as model inputs to
OnnxModel.Run, and in the CPU path modify the fusion pipeline so the Layers
sequence consumes a fused tensor (e.g., concat/cross-attention of p and
textTensor) instead of just the image; also ensure ComputeMatchingScore still
operates on the returned fused tensor. Ensure ThrowIfDisposed, PreprocessImage,
and any shape/dtype checks are preserved and update any method signatures or
helper methods used to construct the fused tensor.
In `@src/VisionLanguage/Foundational/VinVL.cs`:
- Around line 37-38: FuseImageText currently ignores the text parameter; call
the tokenizer to encode the input text (respecting MaxSequenceLength) and
produce a text embedding tensor, then combine that embedding with the image
tensor from PreprocessImage (e.g., concatenation along channel/feature dim or
via a cross-attention fusion op) before passing to the model pipeline; in ONNX
mode pass both image and text inputs to OnnxModel.Run (or construct a single
fused tensor if the ONNX graph expects that) and in native mode fuse the tensors
and feed the result through Layers in FuseImageText so ComputeMatchingScore uses
a true multimodal fused tensor; ensure the tokenizer instance and
MaxSequenceLength are used and update ComputeMatchingScore to compute the score
from the multimodal fused output.
In `@src/VisionLanguage/Foundational/VisualBERT.cs`:
- Around line 37-38: The FuseImageText and ComputeMatchingScore methods ignore
the text input; implement text encoding and fuse it with image features before
running layers or ONNX: add a private EncodeText(string) that uses the existing
_tokenizer and _options.MaxSequenceLength to produce a Tensor<T> (convert token
ids with NumOps.FromInt32 into a Tensor<T>), then update FuseImageText to call
EncodeText(text) and either concatenate the image tensor and text tensor into
the multimodal input consumed by Layers (or pass both as a dictionary to
OnnxModel.Run when IsOnnxMode), and ensure ComputeMatchingScore uses the fused
output from FuseImageText; also throw if _tokenizer is null.
---
Duplicate comments:
In `@src/Helpers/LayerHelper.cs`:
- Around line 22549-22612: The CNN half of CreateDefaultBASICLayers incorrectly
uses DenseLayer<T> placeholders for MBConv/CNN stages; replace the placeholder
sequence inside the first loop (currently yielding DenseLayer<T>,
LayerNormalizationLayer<T>, DenseLayer<T>, LayerNormalizationLayer<T>, optional
DropoutLayer<T>) with real convolutional/MBConv building blocks (e.g., an
MBConvBlock/ConvBlock class or composed Conv2D, DepthwiseConv, SqueezeExcite,
and pointwise Conv layers) so the vision encoder actually implements CoAtNet CNN
stages; update the loop to instantiate and yield the correct MBConv/ConvBlock
type(s), ensure their input/output dims match visionEmbeddingDim and
visionFfnDim semantics, and remove or repurpose the placeholder DenseLayer<T>
usages accordingly.
- Around line 22497-22547: The CreateDefaultALIGNLayers implementation currently
approximates EfficientNet MBConv blocks by stacking DenseLayer<T> (see
DenseLayer<T> usages inside the vision loop and the comment "MBConv-style
block"), which is not production-ready; replace that dense-only vision path with
a proper CNN/MBConv implementation: remove the DenseLayer-based
expand/depthwise/project sequence in the vision loop and instead instantiate
real convolutional blocks (e.g., an MBConvBlock/Conv2D/DepthwiseConv2D sequence
or a provided EfficientNetBlock class) that accept and output tensor shapes
consistent with visionEmbeddingDim and visionFfnDim, apply correct normalization
and activation (e.g., Squeeze-and-Excitation, Swish/GELU as appropriate), and
ensure any DropoutLayer<T> remains applied; add or reuse an MBConvBlock<T> type
and update CreateDefaultALIGNLayers to yield those MBConv blocks and a final
convolutional projection to projectionDim instead of DenseLayer projections.
…i-modal processing All 9 foundational VLM models now properly: - Tokenize text input via ITokenizer (was completely ignored before) - Track layer boundaries (_projectionLayerEnd, _visionLayerEnd, _textLayerEnd) - Use architecture-appropriate forward passes (single-stream, dual-stream, cross-modal, bridge) - Compute cosine similarity for matching scores (was using meaningless sigmoid on sum) Single-stream (VisualBERT, UNITER, Oscar, VinVL, ViLT): project image, then process jointly Dual-stream (ViLBERT, METER): separate vision/text encoders, then co-attention fusion Cross-modal (LXMERT): relationship encoder, language encoder, cross-modality encoder Bridge (BridgeTower): vision/text encoders with bridge cross-attention layers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 78 out of 78 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (7)
src/VisionLanguage/Foundational/VisualBERT.cs:1
FuseImageTextignores thetextparameter in both ONNX and native modes, so the method cannot produce a fused image-text representation. Tokenize the text and incorporate it into the fusion path (e.g., concatenate visual + text tokens for single-stream models, or use cross-attention layers that accept both tensors); for ONNX, pass both image and token inputs into the session using named inputs rather than running only on the image tensor.
src/VisionLanguage/Foundational/VisualBERT.cs:1- In ONNX mode,
OnnxModel.Run(textTokens)suggests the same ONNX graph is being executed for text-only input, while earlier calls run the model for image input; this is likely to fail at runtime if the ONNX model expects image+text inputs (or expects only image inputs). Consider using separate ONNX models for image/text encoders, or (for a fusion model) provide both image and text inputs in a single ONNX invocation and extract the appropriate outputs.
src/VisionLanguage/Foundational/ViLBERT.cs:1 FuseImageTextcomputestextOutbut never uses it (fusion layers only receivevisionOut). As written, there is no cross-modal interaction, so the result cannot depend on the provided text. To fix this, the fusion layers (or the fusion call) need to accept both modality tensors (e.g., cross-attention blocks that consume queries from one stream and keys/values from the other), or you need an explicit concatenation/joint representation that includes both streams before running the fusion blocks.
src/VisionLanguage/Foundational/LXMERT.cs:1- The cross-modality encoder section never consumes
textOut, so the 'fused' output is effectively vision-only. If the design is a true cross-attention/cross-modal encoder, the fusion layers must be able to attend over bothvisionOutandtextOut(or a combined sequence) to produce a text-conditioned output.
src/VisionLanguage/Foundational/UNITER.cs:1 - Like other single-stream fusion implementations in this PR,
FuseImageTextignores thetextargument entirely (and returnsOnnxModel.Run(p)in ONNX mode). This makes UNITER's fusion API effectively vision-only. Incorporate tokenized text into the model forward path (concat/joint sequence or cross-attention that takes both modalities) and ensure ONNX invocation supplies both image + text inputs.
src/VisionLanguage/Foundational/UNITEROptions.cs:1 - Fix capitalization typo in the acronym expansion: 'TExt' should be 'Text'.
src/VisionLanguage/Encoders/RemoteCLIP.cs:1 - Multiple
usingdirectives are combined on a single line, which hurts readability and creates noisy diffs for future changes. Split these into oneusingper line to match typical C# conventions and improve maintainability.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/VisionLanguage/Foundational/BridgeTower.cs`:
- Around line 89-94: The code is assuming Architecture.Layers can be split into
exact thirds when setting _visionLayerEnd and _textLayerEnd, which is fragile;
update the block that handles Architecture.Layers so it first checks for
explicit boundary configuration on Architecture (e.g.
Architecture.VisionLayerEnd / Architecture.TextLayerEnd or similar properties)
and uses those if provided, otherwise validate that Layers.Count is divisible by
3 (Layers.Count % 3 == 0) before computing _visionLayerEnd = Layers.Count / 3
and _textLayerEnd = Layers.Count * 2 / 3; if neither explicit boundaries exist
nor the count is divisible by 3, throw a clear ArgumentException indicating that
Architecture.Layers must either include explicit boundaries or have a length
divisible by 3 so callers fix their input.
In `@src/VisionLanguage/Foundational/LXMERT.cs`:
- Around line 88-93: The current logic assumes custom Architecture.Layers follow
a 1:1:1 split by computing _visionLayerEnd = Layers.Count / 3 and _textLayerEnd
= Layers.Count * 2 / 3, which is fragile; update the initializer that handles
Architecture.Layers to determine encoder boundaries explicitly (instead of
thirds) by either: 1) using per-layer metadata or type annotations on each layer
(e.g., a Layer.Role/LayerType property) to compute _visionLayerEnd and
_textLayerEnd by finding the last vision and last text layer indices in Layers,
or 2) falling back to explicit counts provided on Architecture (e.g.,
Architecture.VisionLayerCount and Architecture.TextLayerCount) and validating
they sum to Layers.Count, and if neither metadata nor explicit counts are
present throw a clear argument exception; adjust the code that currently touches
Layers.AddRange(Architecture.Layers) to perform these checks and set
_visionLayerEnd/_textLayerEnd accordingly.
In `@src/VisionLanguage/Foundational/ViLBERT.cs`:
- Around line 107-112: ComputeDualStreamBoundaries assumes internal layout (6 vs
5 layers per block and projection placement) from
LayerHelper<T>.CreateDefaultDualStreamFusionLayers and does not validate against
the actual Layers collection; update it to derive block size and projection
presence from the created Layers (or a trusted LayerHelper constant/API) instead
of hardcoded 5/6, compute _visionLayerEnd and _textLayerEnd based on actual
Layers sequence or metadata, and then validate both indices against Layers.Count
(throw a clear exception or clamp/adjust with an explanatory message) so
boundary errors are detected early; reference ComputeDualStreamBoundaries,
LayerHelper<T>.CreateDefaultDualStreamFusionLayers, and the Layers collection
when making these changes.
- Around line 130-131: SerializeNetworkSpecificData currently omits the layer
boundary fields causing deserialized instances to have _visionLayerEnd and
_textLayerEnd at 0; update SerializeNetworkSpecificData to write _visionLayerEnd
and _textLayerEnd after the existing options, and update
DeserializeNetworkSpecificData to read those two integers and assign them to
_visionLayerEnd and _textLayerEnd (or set them and then call InitializeLayers())
so that subsequent calls to EncodeImage, FuseImageText, and ComputeMatchingScore
operate over the correct layer ranges.
- Around line 48-68: The FuseImageText method currently ignores the text
modality: fix the ONNX branch to pass both the preprocessed image tensor and the
tokenized text tensor to OnnxModel.Run (or to the ONNX input dictionary expected
by the model) instead of only passing p, and in the native path initialize fused
from both visionOut and textOut (not just visionOut) so the fusion layers
actually attend between modalities—i.e., change the fusion loop to feed each
fusion Layer with the combined representation (or call the fusion Layer API that
accepts both tensors) using the variables visionOut, textOut, fused and Layers
so subsequent Layers[i].Forward consumes both modalities rather than only the
vision representation.
---
Duplicate comments:
In `@src/VisionLanguage/Foundational/BridgeTower.cs`:
- Around line 50-69: FuseImageText currently discards textOut before final
fusion; change it so the final fusion layers operate on a combined vision+text
tensor instead of only visionOut: after computing visionOut and textOut in
FuseImageText, construct a merged tensor (e.g., concat along the token/sequence
dimension or use a helper like MergeVisionAndText/CreateFusedInput) and set
fused = merged before running the loop over the remaining Layers; also update
the ONNX branch (OnnxModel.Run) to accept both image and text (or throw a clear
NotSupportedException if the ONNX model cannot take text) so text is not
ignored.
In `@src/VisionLanguage/Foundational/METER.cs`:
- Around line 48-61: FuseImageText currently discards textOut and only forwards
visionOut through the fusion layers; update FuseImageText so the fusion stage
receives both modalities: after computing visionOut (via PreprocessImage and
vision layers) and textOut (via TokenizeText and text layers), create a combined
fusion input (e.g., a pair/concatenation/batched tensor as expected by your
fusion Layers) and pass that combined tensor into the loop that runs from
_textLayerEnd to Layers.Count so the cross-attention blocks operate on both
vision and text; also ensure the OnnxModel branch (OnnxModel.Run) accepts and is
called with both the preprocessed image tensor and the tokenized text tensor (or
raise/handle unsupported ONNX mode) so ONNX inference doesn't ignore text.
In `@src/VisionLanguage/Foundational/Oscar.cs`:
- Around line 49-64: FuseImageText currently ignores the text parameter:
tokenize the input text using the existing TokenizeText method, obtain text
token embeddings (or projection) compatible with image features, concatenate the
token embeddings (and any object tags) with imageProj at the single-stream
point, then feed the combined sequence through the remaining transformer Layers
starting at _projectionLayerEnd; also ensure ONNX branch (IsOnnxMode /
OnnxModel.Run) receives both image and token inputs (or the prebuilt fused
tensor) instead of only the image so the ONNX path mirrors native behavior;
update FuseImageText to call PreprocessImage, TokenizeText, perform any
necessary projection or padding to match fusion dim, concatenate into c, and
pass c through Layers (and into OnnxModel) accordingly.
In `@src/VisionLanguage/Foundational/UNITER.cs`:
- Around line 49-64: FuseImageText currently ignores the text input: update
FuseImageText to tokenize and embed the text, build text token tensors and
attention/mask tensors, concatenate the text embeddings with the projected image
embeddings (imageProj) along the sequence dimension before feeding the joint
transformer layers (the loop from _projectionLayerEnd to Layers.Count), and
ensure the resulting combined tensor is passed through those Layers to produce
cross-modal outputs; additionally, handle IsOnnxMode/OnnxModel by either
invoking an ONNX entrypoint that accepts both image and text inputs or by
throwing a clear error if ONNX mode cannot accept text, and reuse existing
helpers such as PreprocessImage, the tokenizer/embedding utilities,
_projectionLayerEnd, Layers, and any mask construction utilities when
implementing the fix.
In `@src/VisionLanguage/Foundational/ViLBERT.cs`:
- Line 125: The Train method currently discards the computed loss—update the
Train signature (e.g., public override double Train(...)) to return a numeric
loss, compute the loss via the existing LossFunction (call
LossFunction.Calculate or equivalent using o.ToVector() and
expected.ToVector()), and return that scalar after the backward/update steps;
ensure callers of ViLBERT.Train are updated to handle the returned loss and keep
the existing checks and optimizer update logic (symbols: Train, Predict,
LossFunction.Calculate/CalculateDerivative, Layers, _optimizer,
SetTrainingMode).
- Around line 28-31: The fields and method on this line are compressed and need
to be split for readability: break out the private readonly ViLBERTOptions
_options and implement public override ModelOptions GetOptions() => _options; on
its own line; similarly place private readonly IGradientBasedOptimizer<T,
Tensor<T>, Tensor<T>>? _optimizer; private readonly ITokenizer? _tokenizer;
private bool _useNativeMode; private bool _disposed; private int
_visionLayerEnd; and private int _textLayerEnd; each on separate lines,
preserving their modifiers and types and keeping the GetOptions() method
signature and body intact (references: _options, GetOptions(), _optimizer,
_tokenizer, _useNativeMode, _disposed, _visionLayerEnd, _textLayerEnd).
- Around line 91-96: When Architecture.Layers is supplied, avoid the hardcoded
Layers.Count/3 pattern: either compute boundaries using the existing
ComputeDualStreamBoundaries() helper or require explicit boundary fields on
Architecture (e.g., VisionLayerEnd/TextLayerEnd) and validate them; update the
block that currently calls Layers.AddRange(Architecture.Layers) to (1) if
Architecture exposes explicit boundaries, set _visionLayerEnd/_textLayerEnd from
those and validate 0 < _visionLayerEnd < _textLayerEnd < Layers.Count, else (2)
if no explicit boundaries but Layers.Count is divisible by 3 call the same logic
as ComputeDualStreamBoundaries to derive thirds; throw a clear exception if
neither condition is met or validations fail so custom architectures cannot
silently use incorrect boundaries.
In `@src/VisionLanguage/Foundational/ViLT.cs`:
- Around line 49-64: FuseImageText currently processes only image patches; to
fix it, tokenize and embed the input text, align/embed token dimension to the
fusion dimension, concatenate the text token embeddings with the projected image
patches, and then run the concatenated sequence through the transformer layers
after _projectionLayerEnd. Specifically: call the existing text
tokenizer/embedding utility (e.g., a Tokenize/EmbedText method or TextEmbedding
layer) to get textEmb for the input string, ensure textEmb has the same
embedding dim as imageProj (apply a linear projection if needed), concatenate
along the sequence dimension (e.g., sequence = Concat(imageProj, textEmb)), and
replace the current second loop so Layers[_projectionLayerEnd..end].Forward is
invoked on that concatenated sequence; maintain the existing
Onnx/PreprocessImage/ThrowIfDisposed logic and preserve shape/checks (padding,
type/position embeddings) when building the concatenated input.
In `@src/VisionLanguage/Foundational/VinVL.cs`:
- Around line 49-64: FuseImageText currently ignores the text parameter: ensure
text is tokenized and concatenated with the image features before running the
model both in ONNX and native paths. Use TokenizeText(text) to produce token
tensors, obtain image features via PreprocessImage(image) and the projection
steps (imageProj using _projectionLayerEnd and Layers), then concatenate tokens
with imageProj along the sequence dimension to form a joint input for the
transformer layers (the same Layers slice after _projectionLayerEnd). In ONNX
mode call OnnxModel.Run with the concatenated input instead of p; in native mode
feed the concatenated tensor through Layers from _projectionLayerEnd to
Layers.Count (like variable c) so text participates in the single-stream fusion;
preserve ThrowIfDisposed() behavior and existing return type Tensor<T>.
…ypes Implements 16 generative vision-language models for issue #273: - Q-Former Bridge: InstructBLIP, BLIP-3 - Encoder-Decoder: GIT, CoCa, PaLI, PaLI-X, PaLI-3 - Perceiver Resampler: OpenFlamingo, IDEFICS, IDEFICS2, IDEFICS3 - Causal Multimodal: KOSMOS-1, KOSMOS-2 - Unified Generation: Emu, Emu2, Emu3 Each model has proper layer boundary computation, tokenization, architecture-specific generation pipeline, and serialization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 26
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/Helpers/LayerHelper.cs`:
- Around line 22910-22912: Guard against numBridgeLayers being zero or negative
in the bridge interval and related checks: in the code paths that compute
bridgeInterval (currently using expressions like numVisionLayers /
numBridgeLayers and numTextLayers / numBridgeLayers) add a precheck for
numBridgeLayers <= 0 and set bridgeInterval to a safe default (e.g., 1) or
short-circuit the bridge logic so no division occurs; update the corresponding
checks referenced around the same area (the checks at the other occurrences you
noted) to use this safe value or to skip bridge processing when numBridgeLayers
<= 0, and ensure all uses (the current bridgeInterval assignment and the later
divisions) reference the same guarded value or branch to avoid divide-by-zero.
- Around line 23238-23247: The comment is misleading because the loop yields
vanilla Transformer pieces (MultiHeadAttentionLayer<T>,
LayerNormalizationLayer<T>, DenseLayer<T> with geluActivation) rather than
LLaMA-specific components; fix by either removing the "LLaMA" label or replacing
the loop with a new CreateLlamaDecoderLayers() helper that yields LLaMA-correct
building blocks: use an RMSNormLayer<T> instead of LayerNormalizationLayer<T>,
replace the GELU FFN with a SwiGLU implementation (or a SwiGLUDense layer)
instead of DenseLayer<T> + geluActivation, ensure the attention implementation
supports RoPE positional encoding and enforces causal masking in
MultiHeadAttentionLayer<T> (or supply a LlamaCausalAttentionLayer<T>), and keep
optional DropoutLayer<T> behavior; update the call site to iterate
CreateLlamaDecoderLayers(decoderDim, decoderFfnDim, numHeads, dropoutRate) if
choosing the helper approach.
- Around line 22378-22386: LayerHelper<T> and its factory methods (e.g.,
CreateDefaultOpenCLIPLayers) are public but should be internal implementation
details; change the visibility: either make the LayerHelper<T> type internal or,
if other public members require it to remain public, change the visibility of
all CreateDefault* factory methods (CreateDefaultOpenCLIPLayers,
CreateDefaultSigLIPLayers, CreateDefaultALIGNLayers, CreateDefaultBASICLayers,
CreateDefaultViTLayers, CreateDefaultFlorence2Layers,
CreateDefaultBridgeFusionLayers, etc.) to internal so these factories are not
exposed in the public API used by AiModelBuilder/AiModelResult. Ensure
signatures and callers within the assembly are updated accordingly.
- Around line 22983-23010: The numQueryTokens parameter in
CreateDefaultQFormerGenerativeLayers is unused; either remove it from the method
signature if query tokens are managed externally (e.g., in Blip2NeuralNetwork),
or initialize learnable query tokens inside this layer stack using an existing
embedding-type layer (e.g., add an EmbeddingLayer<T>(numQueryTokens, qFormerDim)
or similar at the start of the Q-Former section and wire that embedding output
into the subsequent cross-attention layers); update function signature and any
callers if removing the param, or add the EmbeddingLayer initialization and
ensure its outputs are fed into the Q-Former MultiHeadAttention sequence (refer
to CreateDefaultQFormerGenerativeLayers, numQueryTokens, and qFormerDim to
locate where to change).
- Around line 23075-23083: The decoder loop currently yields only a non-causal
MultiHeadAttentionLayer<T> and thus lacks the required cross-attention to vision
features; replace the single MultiHeadAttentionLayer<T> with a causal
self-attention step followed by a CrossAttentionLayer<T> that conditions on
visual features (use the existing CrossAttentionLayer<T> type), preserving
surrounding LayerNormalizationLayer<T>(decoderDim) and feed‑forward
DenseLayer<T> blocks and applying DropoutLayer<T> as before; do not attempt to
pass a nonexistent isCausal parameter to MultiHeadAttentionLayer<T>—instead
instantiate or configure a causal self-attention layer (or apply a causal mask
if supported) then yield new CrossAttentionLayer<T>(decoderDim, decoderDim,
numHeads) to implement the cross‑attention to vision features.
- Around line 22483-22490: Create the MultiHeadAttentionLayer for the SigLIP
text encoder using the existing constructor signature in
CreateDefaultSigLIPLayers() (e.g., new
MultiHeadAttentionLayer<T>(textEmbeddingDim, textEmbeddingDim, numTextHeads)),
do not pass a nonexistent isCausal parameter, and immediately set its
UseCausalMask property to true (assign to a local variable like textAttention,
set textAttention.UseCausalMask = true) before yielding it; this ensures causal
masking is enabled for the text attention while keeping the rest of the yielded
layers (LayerNormalizationLayer, DenseLayer, DropoutLayer) unchanged.
- Around line 23176-23198: The vision encoder currently clamps the head count
when constructing MultiHeadAttentionLayer<T>; remove the ternary clamp (numHeads
> 16 ? 16 : numHeads) so the vision loop uses the caller-provided numHeads value
when instantiating MultiHeadAttentionLayer<T>. In the decoder loop, after
creating each MultiHeadAttentionLayer<T>(decoderDim, decoderDim, numHeads), set
its UseCausalMask property to true (e.g., layer.UseCausalMask = true) to enforce
causal masking on the causal transformer decoder; do this for every decoder
attention layer created so the decoder is actually causal.
In `@src/VisionLanguage/Encoders/VisionLanguageEnums.cs`:
- Around line 3-14: The public enums (e.g., ModelPrecision) rely on implicit
integer values which can break binary serialization if members are inserted
later; update every enum in this file to assign explicit integer values to each
member (preserve current ordering by assigning sequential integers starting at 0
for the existing members) so each symbol (for example ModelPrecision.Float32,
Float16, BFloat16) has a fixed numeric value for stable persisted artifacts.
- Around line 16-43: The enum ViTVariant mixes ViT models with non‑ViT backbones
(EfficientNetB7, CoAtNet); update the API docs or split the enum so it's
accurate—either rename ViTVariant to ImageEncoderVariant (and change the enum
summary to "Specifies the image encoder/backbone variant used as the image
encoder") and adjust individual XML summaries to note which are ViT vs non‑ViT,
or create a new enum (e.g., ImageBackbone) for non‑ViT entries and leave
ViTVariant containing only ViT members; ensure references to ViTVariant,
EfficientNetB7, and CoAtNet in comments and code are updated accordingly.
In `@src/VisionLanguage/Generative/BLIP3.cs`:
- Around line 39-52: The GenerateFromImage method tokenizes the prompt into
promptTokens but never uses it, so you must feed the text tokens into the
Q-Former / decoder pipeline to enable cross-modal conditioning: after
TokenizeText(prompt) produce the text embeddings (e.g., via EmbedText or
TextEncoder) and pass them into the Q-Former processing (use qFormerOut and the
layers in the _visionLayerEnd.._qFormerLayerEnd range) by invoking the Q-Former
layer forward calls with both vision and text inputs (or call a helper like
ApplyTextConditioning/CombineCrossAttention) so that Layers[i].Forward for the
Q-Former range consumes prompt embeddings (and any attention masks) instead of
only qFormerOut; ensure the downstream decoder layers (the loop after
_qFormerLayerEnd) receive the conditioned qFormerOut.
In `@src/VisionLanguage/Generative/CoCa.cs`:
- Around line 40-52: GenerateFromImage currently tokenizes the prompt into
promptTokens (via TokenizeText) but never uses them; fix by converting
promptTokens into text embeddings and passing those embeddings into the
multimodal decoder so decoder layers perform cross-attention on both image
features (encoderOut) and text embeddings. Specifically: after var promptTokens
= TokenizeText(prompt) call, produce textEmb = EmbedTextTokens(promptTokens) (or
the existing text embedding routine) and then modify the decoder loop (the
Layers iteration from _encoderLayerEnd to Layers.Count) to call the decoder
layer forward method with both the image-feature input (output/encoderOut) and
textEmb as the cross-attention/key-value input (e.g., Layers[i].Forward(output,
textEmb) or the decoder variant used in this codebase); ensure the OnnxModel
branch remains unchanged or supports conditioned inputs if required.
In `@src/VisionLanguage/Generative/Emu.cs`:
- Around line 41-58: In GenerateFromImage the promptTokens produced by
TokenizeText(prompt) are computed but never used; fix this by converting
promptTokens into the decoder input and conditioning the multimodal decoder on
them: call the model's text-embedding/token-to-hidden converter (the same
component used for text-only inputs) to produce token embeddings from
promptTokens, then concatenate or appropriately interleave those token
embeddings with visionOut (the EVA-CLIP projection) to form decoderIn before
running the decoder loop (the for loop using Layers from index _visionLayerEnd
to _decoderLayerEnd); ensure you also create/update any required
attention/masking or positional embeddings for the combined sequence so the
subsequent decoder Layers see the text+vision context (leave the
IsOnnxMode/OnnxModel branch unchanged).
In `@src/VisionLanguage/Generative/Emu2.cs`:
- Around line 40-57: GenerateFromImage currently tokenizes the prompt into
promptTokens but never uses it; modify GenerateFromImage to incorporate the
tokenized prompt into the model input stream (using the model's interleaved
inputs design) instead of discarding promptTokens. Specifically: keep the
TokenizeText(prompt) result as a Tensor (promptTokens) and merge/interleave or
concatenate it with the vision output at the boundary between the vision encoder
and decoder (around _visionLayerEnd / before processing decoder layers in
Layers[_visionLayerEnd]), so the decoder layers (used in decoderOut and
subsequent output computation) receive the prompt-conditioned input; ensure null
prompt remains handled and that any shape/position ids or attention mask logic
required for interleaving is applied when combining promptTokens with visionOut.
In `@src/VisionLanguage/Generative/Emu3.cs`:
- Around line 49-50: The prompt tokens created by TokenizeText(prompt) are being
discarded; preserve the promptTokens and merge them with the visual tokens to
form a single unified token sequence for autoregressive generation per Emu3
design. Locate where TokenizeText(prompt) is called in Emu3.cs and
concatenate/interleave promptTokens with the visual token sequence (e.g.,
visualTokens or whatever variable holds image tokens) using the model's required
ordering, then pass that merged token sequence into the subsequent
generation/forward code path (the same place that currently uses only visual
tokens) so text and visual tokens share the unified vocabulary. Ensure the
variable promptTokens is not scoped inside a throwaway block and is available to
the generation routine.
In `@src/VisionLanguage/Generative/GIT.cs`:
- Around line 40-54: The GenerateFromImage method currently tokenizes a prompt
into promptTokens but never uses them, so decoder conditioning is missing;
modify GenerateFromImage to pass promptTokens into the decoder path so Layers in
the decoder (Layers indexed from _encoderLayerEnd to Layers.Count) receive both
encoderOut and the tokenized prompt (use TokenizeText(prompt) output) — e.g.,
prepare a decoder input/conditioning structure from promptTokens and feed it
into the decoder layers' Forward calls (or call a decoder-specific method such
as Layers[i].Forward(decoderInput, promptTokens) or merge promptTokens into the
initial output before the decoder loop) so the decoder actually conditions on
the text prompt. Ensure null prompt still results in the original behavior.
In `@src/VisionLanguage/Generative/InstructBLIP.cs`:
- Around line 33-34: The two InstructBLIP constructors are written as single
dense lines; refactor each ctor (the public InstructBLIP(...) overloads) into
multiple readable statements on separate lines: assign _options, set
_useNativeMode and _optimizer (when present), set base.ImageSize,
base.ImageChannels, base.EmbeddingDim, validate modelPath (first overload) with
the ArgumentException/FileNotFoundException checks on their own lines, set
_options.ModelPath, instantiate OnnxModel (first overload), create _tokenizer
via ClipTokenizerFactory.CreateSimple, and call InitializeLayers(); keep the
same logic and ordering but split into clear statements and add simple inline
comments if helpful to improve readability.
- Around line 78-87: The current InitializeLayers method uses a fragile 1:1:1
heuristic (Layers.Count / 3 and Layers.Count * 2 / 3) when Architecture.Layers
is provided; change it to require or read explicit layer counts/boundaries from
the configuration instead of assuming equal thirds: update InitializeLayers to
check for a provided boundary (e.g., Architecture.LayerBoundaries or new options
fields like VisionLayerCount, QFormerLayerCount, DecoderLayerCount), validate
that these counts sum to Layers.Count, set _visionLayerEnd and _qFormerLayerEnd
from those values, and fall back to the old heuristic only with a clear warning
log if no explicit boundaries are present; ensure ComputeQFormerBoundaries is
only called when relying on created default layers and keep the branch for
Layers.AddRange(Architecture.Layers) to use the validated boundaries.
- Around line 48-76: GenerateFromImage currently tokenizes the prompt into
promptTokens but never uses it; fix by converting promptTokens into text
embeddings (e.g., via the existing text embedding/text encoder method you have)
and feeding those embeddings into both the Q-Former and the decoder pipeline so
the instruction conditions visual features and generation: for the Q-Former
stage (layers indexed _visionLayerEnd.._qFormerLayerEnd) pass or concatenate the
text embeddings with vision features (instead of using qFormerOut alone) so
cross-attention sees the prompt, and for the decoder stage (layers from
_qFormerLayerEnd..Layers.Count) initialize decoderInput as the combination of
qFormer output and the prompt embeddings (or otherwise inject them into decoder
layer inputs/attention) rather than just using qFormerOut; use the symbols
TokenizeText, promptTokens, qFormerOut, decoderInput, and Layers to locate where
to add calls to your text embedding/encoder and where to merge embeddings into
the forward passes.
- Around line 28-36: The InstructBLIP class currently packs multiple field
declarations and constructor statements onto single lines; split each field
(e.g., _options, _optimizer, _tokenizer, _useNativeMode, _disposed,
_visionLayerEnd, _qFormerLayerEnd) onto its own line and reformat both
constructors (the one taking modelPath and the one taking optimizer) so each
assignment, validation (e.g., string.IsNullOrWhiteSpace(modelPath),
File.Exists), property set (ImageSize, ImageChannels, EmbeddingDim), and method
call (OnnxModel instantiation, ClipTokenizerFactory.CreateSimple,
InitializeLayers) appears on its own line with normal braces and indentation;
also place each property implementation (EmbeddingDimension,
MaxGenerationLength, DecoderEmbeddingDim, and explicit interface members) on
separate lines to follow standard C# style and improve readability.
In `@src/VisionLanguage/Generative/KOSMOS1.cs`:
- Around line 41-55: GenerateFromImage currently tokenizes the prompt into
promptTokens and then discards them; you must interleave those text tokens with
the visual token sequence (visionOut / p) before running the causal transformer
decoder. Specifically, after TokenizeText(prompt) produce a combined token
sequence that alternates (or otherwise interleaves as KOSMOS requires) visual
token embeddings from visionOut with the promptTokens embeddings/ids, replace
the decoder input (currently "output" initialized to visionOut) with this
combined sequence, and then feed that combined sequence into Layers[i].Forward
for i from _visionLayerEnd to Layers.Count; also handle null prompts by skipping
interleaving and ensure types/shapes align between PreprocessImage/visionOut and
TokenizeText outputs.
In `@src/VisionLanguage/Generative/KOSMOS2.cs`:
- Around line 57-68: InitializeLayers currently sets _visionLayerEnd using a
fragile heuristic Layers.Count / 3 when Architecture.Layers is provided; change
this to honor an explicit boundary if available from the architecture or options
and fall back to a robust detection: update NeuralNetworkArchitecture<T> to
include an optional VisionLayerEnd (or similar) and in InitializeLayers use
Architecture.VisionLayerEnd when present, otherwise compute the boundary by
scanning Architecture.Layers for the first non-vision/decoder transition
(inspect layer type/metadata) and set _visionLayerEnd accordingly; keep
ComputeCausalBoundary for the default native-mode construction path.
- Around line 41-55: GenerateFromImage currently tokenizes prompt into
promptTokens but discards it; integrate promptTokens into the decoder input by
converting promptTokens to token embeddings (using the model's text/token
embedding routine—e.g., call the existing TokenEmbedding or Embedding method)
and concatenating or interleaving those embeddings with the vision output
(visionOut) before running the causal decoder layers (Layers index >=
_visionLayerEnd). Replace the unused local promptTokens with a variable holding
the embedded token tensors, ensure any grounding/location tokens are inserted
according to the model contract, and then run the for-loop over Layers starting
at _visionLayerEnd on the combined sequence so the prompt influences the output;
if no text embedding function exists, add a helper to map tokens -> embeddings
and use that helper here.
- Around line 29-37: The file currently crams multiple field declarations, two
KOSMOS2 constructors, and property declarations onto single lines; refactor by
splitting into one declaration/statement per line: declare each private readonly
field (_options, _optimizer, _tokenizer), each private field (_useNativeMode,
_disposed, _visionLayerEnd) and each public property (EmbeddingDimension,
MaxGenerationLength, DecoderEmbeddingDim) on its own line, and expand both
KOSMOS2 constructors so parameter list, base call, null-coalescing option
creation, assignment of _useNativeMode/_optimizer/_tokenizer,
base.ImageSize/ImageChannels/EmbeddingDim assignments, modelPath
validation/OnnxModel creation, _options.ModelPath assignment, and
InitializeLayers() are each on separate lines while preserving the exact logic
and method calls (InitializeLayers, OnnxModel,
ClipTokenizerFactory.CreateSimple, AdamWOptimizer).
In `@src/VisionLanguage/Generative/OpenFlamingo.cs`:
- Around line 41-58: GenerateFromImage currently tokenizes the prompt into
promptTokens but never uses them, so the decoder (layers from _perceiverLayerEnd
onward) is unconditioned; fix by taking the TokenizeText(prompt) result and
integrating it into the decoder input before running the decoder layers (for
example build a decoderInput that combines perceiverOut and the prompt token
embeddings — e.g., obtain token embeddings from TokenizeText output or an
embedding method, concatenate or append them to perceiverOut or set them as
cross-attention memory for the decoder layers), then replace the decoder loop to
run Layers[i].Forward(decoderInput) (or pass both decoder state and
cross-attention memory if your Layer.Forward supports it); ensure you handle
prompt == null by using perceiverOut alone and preserve shapes/types expected by
Layers and downstream methods.
- Around line 29-37: The file packs multiple field declarations, constructors
and property implementations into single compressed lines which hurts
readability; split each field declaration (e.g., private readonly
OpenFlamingoOptions _options; private readonly IGradientBasedOptimizer<T,
Tensor<T>, Tensor<T>>? _optimizer; private readonly ITokenizer? _tokenizer;
private bool _useNativeMode; private bool _disposed; private int
_visionLayerEnd; private int _perceiverLayerEnd;) onto their own lines and
reformat the two constructors (public OpenFlamingo(...)) so each statement
(assignments, checks, OnnxModel instantiation, tokenizer creation,
InitializeLayers call) is on its own line with normal indentation; likewise
split property implementations (GetOptions, EmbeddingDimension,
IVisualEncoder.ImageSize, ImageChannels, MaxGenerationLength,
DecoderEmbeddingDim) onto separate lines for clarity while preserving existing
names and behavior.
---
Duplicate comments:
In `@src/Helpers/LayerHelper.cs`:
- Around line 22664-22672: The loop in LayerHelper.cs that builds encoder layers
currently returns identical MultiHeadAttentionLayer<T> instances for every
iteration, but must implement DaViT’s alternating attention types; modify the
encoder construction inside the for-loop (around the MultiHeadAttentionLayer<T>,
LayerNormalizationLayer<T>, DenseLayer<T> returns) to alternate between a
SpatialWindowAttentionLayer<T> for odd layers and a
ChannelGroupAttentionLayer<T> (or a DaViTChannelGroupAttentionLayer<T>
implementation) for even layers (or replace the two returns with a single
DaViTBlock<T> that encapsulates the alternating attention, normalization, FFN
and optional Dropout), keeping the existing LayerNormalizationLayer<T>,
DenseLayer<T> (encoderFfnDim/encoderEmbeddingDim) and dropout logic intact and
ensuring numEncoderHeads, encoderEmbeddingDim and dropoutRate are passed through
to the chosen attention block constructors.
- Around line 22571-22580: The CNN-stage loop currently yields
DenseLayer/LayerNormalizationLayer placeholders; replace those with actual
CoAtNet/MBConv building blocks or delegate to a dedicated helper. Specifically,
in the loop over cnnStages (which uses numVisionLayers, visionEmbeddingDim,
visionFfnDim, geluActivation, identityActivation, dropoutRate), remove the
DenseLayer/LayerNormalizationLayer/DropoutLayer sequence and instead yield
proper MBConv/CoAtNet blocks (e.g., MBConvBlock/CoAtNetConvStage instances) that
implement depthwise conv, expansion/projection, SE/attention as required, or
call a new helper method CreateMbconvStage(visionEmbeddingDim, visionFfnDim,
geluActivation, dropoutRate) that returns the correct IEnumerable<Layer<T>> for
each stage; ensure the new block types replace DenseLayer,
LayerNormalizationLayer and maintain dropout behavior controlled by dropoutRate.
- Around line 22516-22527: The current vision encoder loop in LayerHelper.cs
yields DenseLayer/TLayerNormalization placeholders (using visionEmbeddingDim,
visionFfnDim, numVisionLayers, dropoutRate) which must be replaced with a real
EfficientNet‑B7 / MBConv stack (or delegated to an existing implementation)
before release: remove the DenseLayer/LayerNormalizationLayer/DropoutLayer
sequence and instead instantiate the proper EfficientNetB7 feature extractor (or
call a helper such as
CreateEfficientNetB7Layers/CreateMbconvStack/GetEfficientNetLayers) that
implements MBConv blocks, se/activation params, strides, and pre/post
batchnorm/layernorm semantics, ensuring the produced layers match the expected
embedding dimension and dropoutRate handling and integrate with the surrounding
pipeline (use visionEmbeddingDim/visionFfnDim to configure expansion/project
widths).
In `@src/VisionLanguage/Generative/IDEFICS.cs`:
- Around line 52-53: The promptTokens produced by TokenizeText(prompt) are
created but never used, so the LLaMA-based decoder never receives prompt
conditioning; update the decoder invocation in this class (where the LLaMA-based
decoder is run) to accept and include promptTokens (e.g., prepend or provide
them as the decoder's initial context input) whenever prompt is not null,
ensuring TokenizeText(prompt) output (promptTokens) is passed into the decoder
call instead of being discarded.
In `@src/VisionLanguage/Generative/IDEFICS2.cs`:
- Around line 52-53: The code tokenizes the prompt into promptTokens via
TokenizeText but never uses or forwards promptTokens to the Mistral-7B decoder,
so the decoder receives no conditioning; update the decoder invocation (where
the Mistral-7B decoder layer/function is called) to accept and pass promptTokens
(or its encoded representation) instead of discarding it, ensuring
TokenizeText(prompt) result is stored in a variable in scope and threaded into
the decoder input pipeline (e.g., supply promptTokens to the decoder's input
parameter or concatenate it with existing input token buffer before calling the
decoder routines).
In `@src/VisionLanguage/Generative/IDEFICS3.cs`:
- Around line 52-53: The prompt tokens are computed but never used, so compute
promptTokens via TokenizeText(prompt) and concatenate or merge them into the
existing perceiverOut token stream before the decoder call; specifically, after
TokenizeText(prompt) assign or append the result into perceiverOut (or the input
buffer fed to the Llama 3.1 decoder) so the decoder receives text conditioning
(use the same token shape/format as perceiverOut), then pass that augmented
perceiverOut into the decoder invocation (the method that runs the decoder in
IDEFICS3.cs) instead of the original perceiverOut.
In `@src/VisionLanguage/Generative/PaLI.cs`:
- Around line 48-49: The prompt tokenization result is being declared inside the
if block and discarded; declare a promptTokens variable in the surrounding scope
(e.g., initially null or an empty token list), assign it from
TokenizeText(prompt) when prompt is not null, and then pass that promptTokens
into the mT5 decoder conditioning step (the method that consumes projected
visual features and text conditioning). Replace the inner "var promptTokens =
TokenizeText(prompt);" with an assignment to the outer promptTokens so the
decoder receives the tokenized prompt via the existing decoder call.
In `@src/VisionLanguage/Generative/PaLI3.cs`:
- Around line 48-49: The promptTokens are being declared inside the if block and
immediately discarded so the decoder receives no conditioning; change the code
in PaLI3.cs (and analogous locations) to declare a variable (e.g., promptTokens)
in the surrounding scope, assign it from TokenizeText(prompt) when prompt is not
null, and pass that stored promptTokens into the decoder conditioning call
(where the model performs decoding). Ensure you remove the inline-scoped
declaration "var promptTokens" and instead reference the outer-scope
promptTokens variable (or null/empty token list) in the decoder invocation so
the prompt actually conditions generation.
In `@src/VisionLanguage/Generative/PaLIX.cs`:
- Around line 48-49: The promptTokens computed by calling TokenizeText(prompt)
are currently scoped and discarded, so the UL2 decoder receives only visual
encoder output; change this by declaring a promptTokens variable outside the if
(e.g., var promptTokens = null), assign it inside the if (prompt is not null) {
promptTokens = TokenizeText(prompt); }, and then pass that promptTokens into the
UL2 decoder call that consumes visual encoder output (the UL2 decoder/DecodeUL2
invocation) so the decoder uses both visual encoder output and tokenized prompt
for conditioning.
…re types Implements LLaVA-1.5, LLaVA-NeXT, LLaVA-OneVision, LLaVA-OneVision-1.5, Phi-3-Vision, InternVL/2/2.5/3, DeepSeek-VL/2 (MLP projection), MiniGPT-4/v2 (Q-Former), Qwen-VL/2VL/2.5VL/3VL (cross-attention resampler), CogVLM/2 (visual expert). Adds IInstructionTunedVLM interface, InstructionTunedVLMOptions base class, and CreateDefaultVisualExpertVLMLayers LayerHelper method. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 27 VLM models across 3 architecture types: - MLP Projection (23): Phi4Multimodal, Llama32Vision, Gemma3, Pixtral, PixtralLarge, Cambrian1, NVLM, Eagle, Eagle25, Molmo, Ovis, MiniCPMV, MiniCPMo, SmolVLM, Moondream, Monkey, Dragonfly, Mantis, VILA, VILAU, Aria, AquilaVL, Maya - Visual Abstractor (3): MPLUGOwl, MPLUGOwl2, MPLUGOwl3 - Direct Patch (1): Fuyu Each model has dual ONNX/native mode, proper serialization, and model-specific options. Adds VisualAbstractor and DirectPatch to InstructionTunedArchitectureType enum. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…els 104-133) Category 6 - Visual Reasoning (6 models): QVQ72B, KimiVL, KimiVLThinking, SkyworkR1V, SkyworkR1V2, LLaVACoT Category 7 - Visual Grounding (11 models): GroundingDINO, GroundingDINO15, DINOX, GroundedSAM2, GLaMM, Ferret, FerretV2, Groma, Shikra, OWLViT, OWLv2 Category 8 - Document Understanding (12 models): LayoutLMv3, Donut, Pix2Struct, Nougat, MPLUGDocOwl, MPLUGDocOwl15, MPLUGDocOwl2, TextMonkey, GOTOCR2, Surya, DocPedia, UReader Includes 3 new interfaces (IReasoningVLM, IVisualGroundingModel, IDocumentUnderstandingModel) and 3 base option classes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-153) Add 3 new interfaces (IVideoLanguageModel, IUnifiedVisionModel, IImageEditingVLM), 3 base options classes, and 20 model implementations across categories: - 9 video-language models (VideoLLaVA, LLaVA-NeXT-Video, LLaVA-Video, etc.) - 8 unified understanding+generation models (Chameleon, Show-o, Janus, etc.) - 3 image editing models (Emu Edit, MGIE, SmartEdit) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… (models 154-166) Add 2 new interfaces (IVisionLanguageAction, IThreeDVisionLanguageModel), 2 base options classes, and 13 model implementations: - 7 VLA/robotics models (RT-2, PaLM-E, Octo, pi-zero, Helix, GR00T-N1, 3D-VLA) - 6 3D vision-language models (PointLLM, 3D-LLM, LEO-VL, 3DGraphLLM, GPT4Point, Scene-LLM) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ry (models 167-177) Add 3 new interfaces (IMedicalVLM, IRemoteSensingVLM, IProprietaryVLM), 3 base options classes, and 11 model implementations: - 5 medical VLMs (LLaVA-Med, RadFM, Med-Flamingo, PathVLM, Dragonfly-Med) - 3 remote sensing VLMs (GeoChat, RSGPT, SkyEyeGPT) - 3 proprietary reference VLMs (Gemini Vision, Claude Vision, Grok Vision) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 200 out of 374 changed files in this pull request and generated 10 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 150 out of 376 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ates and unused vars Replace fake cosine/sine weighted attention loops in GenerateFromImage with proper ConcatenateTensors for visual-text fusion across all Generative VLMs: GIT, CoCa, KOSMOS1/2, Emu/2/3, InstructBLIP, BLIP3, OpenFlamingo, PaLI, PaLI-X, PaLI-3, IDEFICS/2/3. Remove unused NumDecoderHeads from PaLI/PaLI-X/PaLI-3 options (models use NumHeads). Clean up unused dim, visLen, promptLen variables. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…es and unused vars Replace fake fusion loops with proper ConcatenateTensors in 8 Document models: MPLUGDocOwl, MPLUGDocOwl15, MPLUGDocOwl2, GOTOCR2, TextMonkey, DocPedia, Surya, UReader. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 157 out of 376 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…d unused vars Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…d gating Replace fake text-visual fusion in 6 grounding models with proper ConcatenateTensors + decoder layers pattern. Groma, OWLViT, FerretV2, GroundedSAM2, Ferret, Shikra all now fuse text with visual features via concatenation rather than sin/cos embeddings or sigmoid gating. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 159 out of 376 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…s fusion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… fusion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… fusion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s, ThreeD, Medical, Proprietary, Unified, RemoteSensing, VideoLanguage) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add copy constructors to 186 Options classes - Add parameterless constructors to 17 Options classes - Add "For Beginners" XML docs to 168 Options files - Add "For Beginners" XML docs to 166 model files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Feb 23, 2026
6 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Batches 1 and 2 of issue #273 (Vision-Language Stack), adding 25 new vision-language models with full infrastructure.
Batch 1: Contrastive Vision-Language Encoders (16 models)
Batch 2: Vision Encoders / Foundation Models (9 models)
Infrastructure
VisionLanguageModelBase<T>- base class with dual ONNX inference + native training modeIContrastiveVisionLanguageModel<T>,IVisualEncoder<T>,ITextEncoder<T>interfacesContrastiveEncoderOptions/VisionEncoderOptionsbase option classesViTVariant,PositionalEmbeddingType,PoolingStrategy,Florence2ModelSize, etc.All models support:
Test plan
Closes #273 (partial - Batches 1-2 of 8)
🤖 Generated with Claude Code
Summary by CodeRabbit