test: add DistributedTraining integration tests - #764
Merged
Merged
Conversation
Add 78 comprehensive integration tests for the DistributedTraining module covering: - Enums: ReductionOperation - Communication backends: InMemoryCommunicationBackend (constructor, initialization, single-process operations) - Configuration: ShardingConfiguration (default, high-bandwidth, low-bandwidth presets) - ParameterAnalyzer: ParameterGroup, AnalyzeParameters, AnalyzeForDistribution, ValidateGrouping, CalculateDistributionStats - CommunicationManager: static API, initialization, shutdown, collective operations Also fixes stale CLBlast project reference in test project. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings. WalkthroughThis PR adds a comprehensive integration test suite for the DistributedTraining module, covering communication backends (InMemoryCommunicationBackend), sharding configurations, parameter analysis, distributed training operations, and model wrappers (DDPModel, FSDPModel, ZeRO variants, and others). The test suite validates nominal and edge-case scenarios across multiple components. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
…ining-integration-tests
Add comprehensive tests for distributed training models: - DDPModel tests (construction, metadata, serialization, clone) - FSDPModel tests (construction, sharding, metadata) - ZeRO1/2/3Model tests (construction, metadata, sharding) - PipelineParallelModel tests - TensorParallelModel tests - HybridShardedModel tests - Edge case tests (null handling, environment isolation) - MockDistributedModel helper for testing Total: 107 distributed training tests (78 original + 29 new) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
3 tasks done
This was referenced Feb 10, 2026
ooples
added a commit
that referenced
this pull request
Jul 10, 2026
…elper, facade, resident parity Addresses the blocking review comments on the tape-converted time-series models: - TemporalFusionTransformer: restore a REAL quantile head. The tape conversion had collapsed the quantile/pinball head to a single MSE point forecast, so PredictQuantiles returned identical (degenerate) bands. Now the head projects to [H*Q] (quantile-major) and trains with the summed pinball loss over every QuantileLevels level, so PredictQuantiles returns a genuine ordered spread (verified: q0.1 < q0.5 < q0.9 across the horizon); the point forecast is the median-level head. Also validates each quantile level is in (0,1) at construction. - new Random(42) -> RandomHelper.CreateSeededRandom(42) in Informer/NBEATS/NHiTS (deterministic + the project's crypto-secure helper), matching DLinear/Autoformer. - DeepARModel.TrainingLossHistory: public IReadOnlyList (leaked the mutable backing List) -> internal + .AsReadOnly(), matching NBEATSModel.LastRunEpochLosses and the facade-only rule. - NBEATS GPU-resident path: on AiDotNet.Tensors 0.112.0 (#759 fixed the TensorPermute/TensorBroadcastAdd multi-consumer grad-buffer blowup, #764 the resident-parameter mistrain) the resident run now trains faithfully — verified on GPU that it lowers the eager-forward training MSE and improves held-out MSE, so the correctness gate accepts it. Updated the stale "does not reliably reproduce eager gradients" docs; the gate is retained as a generalization safety net. - testconsole/AutoformerVerify: seeded RandomHelper + Environment.Exit(1) on verification failure so a regression fails the process. - Directory.Packages.props: AiDotNet.Tensors + Native 0.111.1 -> 0.112.0 (brings the resident-param + N-BEATS grad-buffer fixes). Real TFT tests (unit + TS model-family) + NBEATSModelTests: 46/46 green. The Generated.* generic-NN tests feed TS models non-time-series data (1-point series) and fail pre-existing regardless of these changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ooples
added a commit
that referenced
this pull request
Jul 10, 2026
…elper, facade, resident parity Addresses the blocking review comments on the tape-converted time-series models: - TemporalFusionTransformer: restore a REAL quantile head. The tape conversion had collapsed the quantile/pinball head to a single MSE point forecast, so PredictQuantiles returned identical (degenerate) bands. Now the head projects to [H*Q] (quantile-major) and trains with the summed pinball loss over every QuantileLevels level, so PredictQuantiles returns a genuine ordered spread (verified: q0.1 < q0.5 < q0.9 across the horizon); the point forecast is the median-level head. Also validates each quantile level is in (0,1) at construction. - new Random(42) -> RandomHelper.CreateSeededRandom(42) in Informer/NBEATS/NHiTS (deterministic + the project's crypto-secure helper), matching DLinear/Autoformer. - DeepARModel.TrainingLossHistory: public IReadOnlyList (leaked the mutable backing List) -> internal + .AsReadOnly(), matching NBEATSModel.LastRunEpochLosses and the facade-only rule. - NBEATS GPU-resident path: on AiDotNet.Tensors 0.112.0 (#759 fixed the TensorPermute/TensorBroadcastAdd multi-consumer grad-buffer blowup, #764 the resident-parameter mistrain) the resident run now trains faithfully — verified on GPU that it lowers the eager-forward training MSE and improves held-out MSE, so the correctness gate accepts it. Updated the stale "does not reliably reproduce eager gradients" docs; the gate is retained as a generalization safety net. - testconsole/AutoformerVerify: seeded RandomHelper + Environment.Exit(1) on verification failure so a regression fails the process. - Directory.Packages.props: AiDotNet.Tensors + Native 0.111.1 -> 0.112.0 (brings the resident-param + N-BEATS grad-buffer fixes). Real TFT tests (unit + TS model-family) + NBEATSModelTests: 46/46 green. The Generated.* generic-NN tests feed TS models non-time-series data (1-point series) and fail pre-existing regardless of these changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ooples
added a commit
that referenced
this pull request
Jul 10, 2026
…ntTape + IEngine) + GPU-residency seam (#1841) * fix: fix NHiTS backward pass: train via GradientTape autodiff so the model actually learns NHiTSModel.TrainCore looped over ForwardWithGradients/ApplyGradients, but ForwardWithGradients built an EMPTY gradient dictionary ("// Backward removed") so ApplyGradients updated nothing — training was a no-op that returned an untrained model. Fix (mirrors the working NBEATSModel path): - Re-express the stack MLP forward under GradientTape<T> using IEngine tensor ops (NHiTSStackTensor.ForwardTape: TensorPermute / TensorMatMul / TensorBroadcastAdd / ReLU). Autodiff now produces gradients for every weight/bias and AdamOptimizer.Step applies them. The forward is fully GPU-dispatchable (the point of the residency campaign). - Register every weight/bias via RegisterTrainableParameter so TapeTrainingStep.CollectParameters picks them up. - TrainCore: z-normalize the series, build [B,L]->[B,H] windowed batches, run tape forward over all stacks (multi-rate avg-pooled inputs), MSE loss, tape.ComputeGradients + optimizer.Step. Denormalize at inference. - Fix a train/inference mismatch: inference ForwardInternal used a hand-rolled scalar 2D-indexer matmul that disagreed with the tape path (~3500x off), producing garbage predictions from correctly-trained weights. It now uses the same Engine.Tensor* ops as ForwardTape, so a trained model's inference output matches what training optimized. - Add best-checkpoint / early-stopping restore: snapshot params at the best epoch and restore at the end, so late-epoch mini-batch Adam noise cannot degrade the returned model. - Remove the dead hand-derived Backward/UpdateParameter/GetParameterNames and the unused activation caches. Public API and NHiTSOptions are unchanged. Verification on a noisy sinusoid+trend series (default hyperparameters, lookback 20 / horizon 8 / 3 stacks): windowed forecast MSE 287.08 -> 18.44 (ratio 0.064, well under 0.5), beats the repeat-last-value baseline (21.85), and probe forecasts track actuals closely (e.g. 18.13 vs 18.16, 16.87 vs 17.07). Before this change the model returned untrained output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: batched GradientTape + IEngine forward (autodiff backward, GPU-dispatchable) Re-express InformerModel<T> training with automatic differentiation and batched Engine.Tensor* ops, mirroring the NBEATSModel / NHiTSModel tape pattern, so autodiff produces the whole backward pass (no hand-derived gradients) and the model runs through the GPU engine. Model (src/TimeSeries/InformerModel.cs): - TrainCore: z-normalizes the series, stacks windows into a [B, L] batch, runs a batched tape forward (ForwardBatch), MeanSquaredErrorLoss.ComputeTapeLoss, tape.ComputeGradients, AdamOptimizer.Step. Supervises the full H-step horizon per sample; snapshots/restores best-epoch parameters. - ForwardBatch: batched [B,L,1] -> [B,H] forward built entirely from Engine.* ops (embedding, multi-head scaled-dot-product attention via ScaledDotProductAttention with the [B,H,S,headDim] layout, LayerNorm, ReLU FFN, batched distilling conv+maxpool, generative decoder, per-position output projection). Token-wise ops run on a flattened [B*S, d] matrix. - ATTENTION SIMPLIFICATION: replaced ProbSparse self-attention with full multi-head scaled-dot-product attention. ProbSparse's top-u query selection is a data-dependent host gather that neither batches nor differentiates cleanly, and for these sequence lengths it already reduces to full attention. Self-attention distilling is retained (now trained: its conv weights were added to CollectTrainableParameters). - Inference (ForwardEngine/PredictSingle/Predict/ForecastHorizon) uses the SAME batched Engine ops (B=1), normalizing the input and denormalizing the forecast — no scalar/tape divergence. Normalization stats are serialized for round-trip. - Removed the dead hand-derived gradient/scalar code: ComputeGradients/ForwardWithCache/ EmbedInput/CreateDecoderInput/ComputeOutput/AccumulateGradients/ApplyGradients and the scalar Forward/Backward/ProbSparse/FeedForward/attention in the encoder/decoder/distilling layer classes (~630 lines). Public API and InformerOptions unchanged. testconsole: informer-verify (double CPU correctness) and informer-gpu (float32 GPU util) harnesses + Program dispatch entries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(timeseries): gPU-resident training seam for NBEATS via compiled fused plan Adds a reusable GPU-residency path for the tensorized TimeSeries forecasters and wires NBEATSModel<float> to it, following NeuralNetworkBase's compiled fused-optimizer template. TimeSeriesModelBase<T>: - CanTrainOnGpu guard (float + DirectGpuTensorEngine + compilation enabled), mirroring NeuralNetworkBase.CanTrainOnGpu. - TryFusedResidentStep(...): thin reusable wrapper over CompiledTapeTrainingStep<T>.TryStepWithFusedOptimizer that compiles forward+backward+Adam into a single on-device graph (weights, activations and Adam moments stay resident; no per-op host<->device round-trip). Applies gradient clipping (maxGradNorm=1.0) to match the eager Adam path. NBEATSModel<T>: - TryTrainGpuResident: runs the doubly-residual stack through the fused compiled plan with a constant batch shape (required for capture/replay). - Correctness-first validation gate: the resident run is kept only if it improves validation MSE over the untrained baseline; on divergence or no-improvement it re-initializes the blocks and returns false so TrainCore falls back to the eager tape path. Gated to epoch-bounded mode so a rejected attempt can't consume a wall-clock training budget. - LastRunEpochLosses / LastRunUsedGpuResidentPath instrumentation (both paths). Behavior: - The double/CPU training path is unchanged (CanTrainOnGpu is float+GPU only); verified NBEATS still trains cleanly on CPU (normalized loss 0.44 -> 0.0004). - Float+GPU: the eager tape already dispatches every op to the CUDA engine; the resident attempt is validated and safely falls back to eager when the compiled plan doesn't reproduce eager gradients. Known limitation (precise blocker): on the currently-linked Tensors build the compiled fused plan does not reliably reproduce eager gradients for NBEATS's per-layer TensorPermute + TensorBroadcastAdd op graph, so the resident attempt typically validation-rejects for NBEATS and hands off to eager. NBEATS is also host-bound (small per-op tensors), which caps GPU occupancy. The seam is retained for TimeSeries models whose op graphs the compiler handles faithfully. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(timeseries): gPU-resident training seam for NHiTS via compiled fused plan Mirrors NBEATSModel.TryTrainGpuResident so N-HiTS routes forward + backward + Adam through a single on-device compiled plan (weights / activations / Adam moments resident across every step) when float + DirectGpuTensorEngine + compilation are all live. * PoolBatchedTape: batched, tape-recordable average pooling via Reshape + ReduceMean. [B, L] -> [B, L/k]. Requires L % k == 0; returns null (fused path unsupported) for configs where the pool size doesn't divide the lookback cleanly, so callers cleanly fall back to the eager scalar pool. * RunForwardBatched: full multi-rate stack on-tape via per-stack PoolBatchedTape + ForwardTape + sum. Single [B, L] input suits the fused step's constant-shape replay contract. * TryTrainGpuResident: NBEATS-parallel structure — pre-MSE baseline, batch loop through TryFusedResidentStep, divergence guard, validation gate. On reject the stacks are re-initialized so the eager fallback starts clean. * LastRunUsedGpuResidentPath: telemetry flag on the model instance. * TrainCore now attempts the resident path first (epoch-bounded only, per the same wall-clock hazard NBEATS docs) and falls through to the existing eager tape loop on failure. Builds green net471 / net8.0 / net10.0. Correctness gate rejects the run if it doesn't improve validation MSE by 2%, guaranteeing the resident path can never ship worse weights than eager. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(timeseries): gPU-resident training seam for Informer via compiled fused plan Same seam NBEATSModel + NHiTSModel use. Informer's forward is SDPA + LayerNorm + FFN + distilling — a very different op graph from NBEATS's Permute + BroadcastAdd chain that trips the divergence guard on the linked Tensors build, so the compiled fused plan should engage cleanly here even while NBEATS keeps falling back. * CollectTrainableLayers: flat ITrainableLayer<T> list over encoder + distilling + decoder layers (all LayerBase-derived). Needed by TryFusedResidentStep to ZeroGrad per step. * ValidationMseGpu: 256-window baseline for the accept/reject gate. * TryTrainGpuResident: NBEATS-parallel structure — pre-MSE, batch loop through TryFusedResidentStep, divergence guard, validation gate. On reject calls InitializeModel() so the eager fallback starts clean. * LastRunUsedGpuResidentPath telemetry flag. * TrainCore attempts the resident path first (epoch-bounded only, same rationale as NBEATS/NHiTS) and falls through to the existing eager tape+optimizer loop on failure. Builds green net471 / net8.0 / net10.0. This completes wiring for the three tape-based deep forecasters currently on the PR branch (NBEATS + NHiTS + Informer). Autoformer / DeepAR / TFT branches referenced in the PR body haven't landed on the remote yet — will be wired the same way when they do. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(tft): tape-train Temporal Fusion Transformer with batched IEngine ops Re-express the TFT forward pass entirely with GradientTape + batched Engine.Tensor* ops so autodiff produces the backward pass (no hand-derived gradients) and every op is GPU-dispatchable — matching the NHiTS/Informer tape conversions. - ForwardBatch: [B,L]->[B,H] point forecast via embedding + positional encoding, softmax-gated variable selection (GRN), static-enrichment GRN, interpretable multi-head ScaledDotProductAttention, post-attention gated skip (GRN), mean-pool + forecast head. - GatedResidualNetwork rewritten batched/tape-safe (Engine ELU/Sigmoid/ LayerNorm with learned gamma/beta); removes scalar NumOps activation loops. - TrainCore: batched tape forward -> MSE -> ComputeGradients -> Adam.Step, z-normalized windows, best-epoch snapshot/restore. Inference (ForwardEngine) uses the identical ops. - Simplifications (documented): quantile/pinball head -> single point (mean) MSE forecast; sequential LSTM -> positional encodings + attention; univariate variable selection -> learned per-channel softmax gate. GRN + variable selection + interpretable attention retained. - Remove dead scalar VariableSelectionNetwork.cs. Public API and TemporalFusionTransformerOptions unchanged. Verify (standalone harness, double, sinusoid+trend series, 1100 pts, 40 ep): MSE before 205.45 -> after 0.472 (0.23% of init, <0.5x); beats naive repeat-last baseline 39.09 by ~83x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(autoformer): tape-based automatic-differentiation training (no hand-derived gradients) Convert AutoformerModel<T> training to a GradientTape + IEngine tensor-op forward so autodiff produces the entire backward pass and every op is GPU-dispatchable (matching the NHiTS/Informer conversions). - ForwardCore: whole forward (embedding, series-decomposition moving-avg trend/seasonal split, encoder/decoder, output projection) is now built from Engine.Tensor* ops. The DEFINING series decomposition is kept, and a tape-differentiable time-delay auto-correlation (R(lag) spectrum -> top-k softmax -> rolled-value aggregation) replaces the old scalar FFT-style attention; only the top-k index SELECTION is non-differentiable (as in the official implementation). - TrainCore rewritten: z-normalize the series, build [L]->[H] windows, mini-batch gradient by accumulating each sample's gradient across separate (immediately-disposed) tapes, then one averaged Adam step per batch; best-epoch params snapshotted and restored. Removed the output-bias level seeding (normalization handles level). - Inference (Predict/PredictSingle/PredictMultiple) uses the SAME ForwardCore ops (normalize in, denormalize out) so there is no train/predict divergence. - Removed the dead scalar/hand-derived code: ComputeGradientsMultiStep, ComputeMovingAverageNode, TopologicalSort, Accumulate/ApplyGradients, ForwardWithCache, AutoformerCache, the gradient-accumulator infrastructure, the ComputeGradients override, and all per-layer scalar forward/gradient methods (~1400 net lines removed). Layers now hold parameters + (de)serialization only. - Normalization stats persisted in Serialize/DeserializeCore. Public API and AutoformerOptions unchanged. Adds a testconsole `autoformer-verify` verb (trains on a learnable noisy sinusoid+trend, reports before/after horizon MSE vs a repeat-last baseline). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(deepar): tape-based BPTT training + GPU-dispatchable forward Re-express DeepARModel<T> training with GradientTape<T> + Engine.Tensor* ops so autodiff produces the full backward pass (no hand-derived gradients) and the forward path is GPU-dispatchable, mirroring the NHiTS/Informer/NBEATS conversions. - Unroll the LSTM recurrence over the L-step lookback as tape-tracked engine ops (gate matmuls via TensorMatMul/TensorBroadcastAdd, gates split with TensorNarrow, Sigmoid/Tanh/TensorMultiply), carrying [H,B] column-major hidden/cell state across steps under the tape. Two internal LayerBase subclasses (DeepARLstmCellTape, DeepARGaussianHead) register every weight/bias via RegisterTrainableParameter so TapeTrainingStep.CollectParameters picks them up. - TrainCore: batched teacher-forced forward -> loss -> tape.ComputeGradients -> AdamOptimizer.Step, with per-epoch best-checkpoint snapshot/restore. Inference (PredictDistribution) runs the identical Step ops (B=1) so there is no scalar/tape divergence. Removed all hand-derived scalar gradient code (ComputeGradients/ApplyGradients/BackwardInternal). - Gaussian likelihood retained: mean trained on MSE (undivided gradient) and the scale trained on the Gaussian NLL term over a StopGradient-ed mean, so sigma cannot inflate to collapse the mean (verified failure mode). Mean is emitted as a residual on the current observation (mu_t = x_t + delta(h_t)), which keeps the model at the persistence prior at init and lets training learn the one-step change. - Public API + DeepAROptions unchanged; added read-only TrainingLossHistory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(timeseries): fix review comments — visibility, serialization, input validation, checkpoint Addresses CodeRabbit review comments on PR#1841 (the clear, non-design ones; the GPU-resident gradient root cause is handled separately): - API visibility (keep the public/external-subclass surface unchanged, per the facade objective): TimeSeriesModelBase.CanTrainOnGpu and TryFusedResidentStep are now `private protected` (still usable by in-assembly models, hidden from external subclasses); NBEATS/NHiTS/Informer LastRunUsedGpuResidentPath and NBEATS LastRunEpochLosses are now `internal` (visible to tests/serving via InternalsVisibleTo). LastRunEpochLosses is exposed as an immutable snapshot (AsReadOnly) so callers cannot mutate the backing list. - InformerModel.InitializeModel is now idempotent — it clears _encoderLayers, _distillingLayers and _decoderLayers before rebuilding. TryTrainGpuResident re-invokes it on a rejected resident run; previously it APPENDED a second full layer set, doubling the model and corrupting ParameterCount/ForwardBatch/the forecast. Safe on the constructor path (lists already empty). - NHiTSModel now validates external inputs: every PoolingKernelSize must be positive (a zero divided by zero in the downsampled-length calc; a negative gave an invalid length), and a training series shorter than LookbackWindow + ForecastHorizon is rejected up front (an empty series divided by zero in the mean pass; a short series trained silently on zero windows). - NHiTSModel.SerializeCore/DeserializeCore now persist _normMean/_normStd. A reloaded model previously kept the defaults (0/1) and denormalized forecasts incorrectly, so it no longer matched the original trained model. - NHiTS best-checkpoint selection now scores each epoch on its FROZEN end-of-epoch weights (via the existing ValidationMse forward) instead of the mean of pre-update batch losses. bestLoss now measures exactly the weights bestSnapshot captures, so a late-diverging epoch can no longer be selected as best. The added pass is forward-only (no backprop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(timeseries): resident-path review fixes — holdout gate, dead-code, failed-step fallback Addresses the remaining CodeRabbit comments on the N-BEATS / N-HiTS GPU-resident path (the compiled-plan gradient root cause is fixed separately in AiDotNet.Tensors): - [3] Accept/reject gate now validates on a time-ordered HOLDOUT. Both models reserved their windows for BOTH resident optimization AND the pre/post MSE gate, so a 2% training-loss improvement proved nothing about generalization. Reserve the latest ~20% of windows as a holdout the optimizer never trains on; train on the earlier windows; score preMse/postMse on the holdout alone. - [4] Removed the unreachable time-bounded logic from NBEATSModel.TryTrainGpuResident. TrainCore only calls it when MaxTrainingTimeSeconds <= 0, so timeBounded, the stopwatch, and both timeout branches were always dead. The loop now uses _options.Epochs directly and keeps the cancellation checks. (N-HiTS already had no such dead code.) - [5] A failed fused step after the plan engaged now diverges + breaks (→ the gate reinitializes and hands off to eager) instead of silently `continue`-ing the batch, which could leave a partially-executed resident run to be accepted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(nbeats): stop eager training divergence blowing up forecasts On noisy / fat-tailed real series the eager tape-Adam training loop diverges: the normalized training loss climbs instead of falling (measured 1.20 -> 1.55 over 15 epochs on SPY daily log-returns) and the final weights produce forecasts blown up to ~hundreds of times the target scale (a walk-forward eval measured N-BEATS prediction variance ~692x the target vs ~0.07-4x for Informer/DLinear). Two coupled guards, both on vectorized Tensor/IEngine ops (no jagged arrays, no per-element scalar loops): 1. Global-norm gradient clipping (Oreshkin et al. 2020). New NBEATSModelOptions.GradientClipNorm (default 1.0). The global L2 norm is computed with Engine.TensorSumOfSquares per gradient and the whole set is scaled in place with Engine.TensorMultiplyScalarInPlace when it exceeds the threshold. 2. Best-epoch-weights checkpointing. Adam's adaptive step makes clipping alone insufficient, so the parameters from the lowest-loss epoch are kept (as Tensor clones, refreshed via Engine.TensorMultiplyScalarInto) and restored after training — inference never runs on a diverged tail. After the fix a 6-fold walk-forward over SPY daily returns keeps every fold's prediction/target std ratio in 0.18x-0.74x (worst max|pred| 0.049 vs a target std of 0.011) — the ~692x blow-up is gone. Existing NBEATSModelTests: 15/15 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(timeseries): resolve review comments — TFT quantile head, RandomHelper, facade, resident parity Addresses the blocking review comments on the tape-converted time-series models: - TemporalFusionTransformer: restore a REAL quantile head. The tape conversion had collapsed the quantile/pinball head to a single MSE point forecast, so PredictQuantiles returned identical (degenerate) bands. Now the head projects to [H*Q] (quantile-major) and trains with the summed pinball loss over every QuantileLevels level, so PredictQuantiles returns a genuine ordered spread (verified: q0.1 < q0.5 < q0.9 across the horizon); the point forecast is the median-level head. Also validates each quantile level is in (0,1) at construction. - new Random(42) -> RandomHelper.CreateSeededRandom(42) in Informer/NBEATS/NHiTS (deterministic + the project's crypto-secure helper), matching DLinear/Autoformer. - DeepARModel.TrainingLossHistory: public IReadOnlyList (leaked the mutable backing List) -> internal + .AsReadOnly(), matching NBEATSModel.LastRunEpochLosses and the facade-only rule. - NBEATS GPU-resident path: on AiDotNet.Tensors 0.112.0 (#759 fixed the TensorPermute/TensorBroadcastAdd multi-consumer grad-buffer blowup, #764 the resident-parameter mistrain) the resident run now trains faithfully — verified on GPU that it lowers the eager-forward training MSE and improves held-out MSE, so the correctness gate accepts it. Updated the stale "does not reliably reproduce eager gradients" docs; the gate is retained as a generalization safety net. - testconsole/AutoformerVerify: seeded RandomHelper + Environment.Exit(1) on verification failure so a regression fails the process. - Directory.Packages.props: AiDotNet.Tensors + Native 0.111.1 -> 0.112.0 (brings the resident-param + N-BEATS grad-buffer fixes). Real TFT tests (unit + TS model-family) + NBEATSModelTests: 46/46 green. The Generated.* generic-NN tests feed TS models non-time-series data (1-point series) and fail pre-existing regardless of these changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: franklinic <franklin@ivorycloud.com>
ooples
added a commit
that referenced
this pull request
Jul 10, 2026
…mirrors 0.113.0 publishes Tensors PR #763 — this branch's gating dependency. #763 makes the compiled/persistent backward honor createGraph=true (GradientTape.Compute- Gradients previously gated the compiled path on !createGraph), so the WGAN-GP gradient penalty's inner backward now differentiates into the disc weights through the fused compiled plan instead of silently returning zeros (issue #1844). Against 0.111.2 the fused GPU-resident WGAN-GP path would have silently degraded to plain WGAN. Also carries #765 (compiled ReduceMax axis fill) and #764 (resident-param fused-Adam fix). Now that 0.113.0 is published to NuGet, retire the local mirror classes that stood in until it shipped (they were 1:1 copies with identical public APIs), and point every consumer at the Engines.Training.* versions in the package: - Delete src/Training/{DpSgdFusedStep,WganGpFusedStep,MultiSlotFusedStep}.cs. - Repoint all call sites (added by #1847's fused-primitive centralization) to AiDotNet.Tensors.Engines.Training.*: * WganGpFusedStep — CausalGAN, CTGAN, CopulaGAN, TableGAN * MultiSlotFusedStep — DiffusionModelBase, CSDI, TabDDPM, TabSyn * DpSgdFusedStep — DPCTGAN, MedSynth Signatures are identical, so this is a pure namespace swap. GpuResidentFusedStep stays consumer-side (not part of #763; it composes the loss — including the createGraph=true gradient penalty — and drives the generic fused plan). Bumps AiDotNet.Native.OneDNN / OpenBLAS to 0.113.0 in lockstep (both published). Builds green on net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ooples
added a commit
that referenced
this pull request
Jul 11, 2026
…for fused paths) 0.113.0 publishes Tensors PR #763 — this branch's gating dependency. #763's key fix (4C): the compiled/persistent backward now honors createGraph=true (GradientTape. ComputeGradients previously gated the compiled path on !createGraph), so the WGAN-GP gradient penalty's inner backward differentiates into the disc weights through the fused compiled plan instead of silently returning zeros (issue #1844). Against 0.111.2 the fused GPU-resident WGAN-GP path would have silently degraded to plain WGAN. The src/Training fused-primitive mirrors (WganGpFusedStep, MultiSlotFusedStep, DpSgdFusedStep) stay as the consumer-side primitives that #1847/#1848 centralize every consumer through; they call the public engine API, so the bump alone routes them onto #763's fixed compiled backward. Also picks up #765 (compiled ReduceMax axis fill) and #764 (resident-param fused-Adam fix). Native OneDNN/OpenBLAS/CLBlast bumped in lockstep (all published). Builds green on net8.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ooples
added a commit
that referenced
this pull request
Jul 11, 2026
…1843) * feat(training): gPU-resident fused step for non-TS single-net models Kicks off the non-time-series GPU-residency sweep. Adds a shared helper for classes that don't inherit from NeuralNetworkBase or TimeSeriesModelBase but still want their Train() to route forward + backward + optimizer through the compiled fused plan (weights / activations / Adam moments resident on-device across the whole step). * src/Training/GpuResidentFusedStep.cs — shared helper. TryResolveOptimizerConfig maps a runtime IGradientBasedOptimizer to the fused-plan OptimizerType + hyperparameters (Adam / AdamW / SGD supported via case-insensitive class-name match + reflection over Options.InitialLearningRate / Beta1 / Beta2 / Epsilon / WeightDecay). TryStep is the one-shot entry. Wired the following single-net models through the helper (mirrors NeuralNetworkBase.TrainWithFusedStep): * GraphClassificationModel (NN base + GNN + pooling + cross-entropy) — routes tape training through the fused plan when float + DirectGpu + compilation are live; falls through to the existing eager tape+optimizer loop on any failure. * LinkPredictionModel (NN base + GNN + node embeddings + BCE) — same wire-up as GraphClassificationModel. * FourierNeuralOperator (lift → FourierLayers → project, hardcoded SGD + learning rate 0.001) — captures the whole spectral-conv chain in a fused SGD plan; falls through to the in-place SGD loop below. GAN-family generators (CTGAN, DPCTGAN, PATEGAN, ...), CLAPModel (learned scalar outside layer hierarchy), AutoDiffTabGenerator (per-sample variable shape defeats plan replay) and other non-conforming shapes are queued for follow-up commits. The shared helper is the common seam so they all wire the same way once their per-step structure fits the fused-plan contract (constant shape + Adam/AdamW/SGD optimizer + ITrainableLayer-carried params). Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): gPU-resident fused step for Finance foundation forecasters Wires TFC, TOTEM, CSDI, MQCNN through the compiled fused SGD plan (all four use in-place SGD with lr=0.001 as their eager path). Extends the shared helper with an inline IsGpuResidentAvailable check so callers don't need to duplicate the availability gate. * GpuResidentFusedStep — adds IsGpuResidentAvailable static property (mirrors TimeSeriesModelBase.CanTrainOnGpu / NeuralNetworkBase.CanTrainOnGpu). TryStep short-circuits when unavailable so callers stay on their eager path. * TFC — supervised + contrastive branches fused into one closure; the fused plan captures both losses in a single backward. * TOTEM — reconstruction + VQ commitment terms fused; the recompute-loss closure re-derives the commitment loss on each replay for graph parity. * CSDI — denoising-score-matching: the forward closure re-samples timestep + noise each step (via ComputeDenoisingPairTape) so replay produces fresh training data even though the plan's captured graph shape is fixed. * MQCNN — multi-quantile pinball loss captured directly via ComputeMultiQuantilePinballLossTape. All four fall through to the existing in-place SGD path on any fused-plan failure (unsupported op, non-compilable graph, etc). Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): gPU-resident fused generator step for GAN-family synthetic generators + TVAE Wires the generator side of the dual-network training loop through the fused compiled plan for 9 SyntheticData generators. Each covers a distinct pattern: * PATEGAN — per-sample loop; fused plan compiles on first sample, replays across the batch with fresh noise per iteration. * MedSynth — batched noise -> decoder -> discriminator-frozen -> non-saturating log-sigmoid loss. * CausalGAN — batched noise -> generator -> optional causal-structure -> output activations -> disc-frozen -> -avgFake. * OCTGAN — per-sample loop; noise -> gen -> disc-frozen embedding -> SVDD dist². * TableGAN — noise + real batch -> gen -> composite loss (fake-scores + information-loss + optional classification). * CTGAN — the tricky one: pack (noise, cond, mask) into one persistent input tensor so the closure can slice cond and mask back out on replay; captures both -avgFake AND conditional cross-entropy on the fused plan. * DPCTGAN — same pattern as CTGAN but simpler (no mask, no CE). * CopulaGAN — same pattern as DPCTGAN. * TVAE — encoder + reparam + decoder + composite ELBO (recon + KL) all fused; Reparameterize re-samples inside the closure so training stays stochastic. Discriminator/critic/student layers are NOT passed to the fused step (their weights stay frozen on the gen step, matching the eager path semantics). Falls through to the eager tape+optimizer path on any failure. The discriminator STEPS of these GANs are not yet wired — they need a separate pass because their loss involves gradient penalty (WGAN-GP) or per-example DP clipping (DPCTGAN) which don't compose cleanly with the single-closure fused step yet. Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): gPU-resident fused step for TabSyn VAE + MisGAN mask/data gens Two more SyntheticData generators covered — each covers a distinct pattern: * TabSyn.TrainVAEBatch — per-row VAE ELBO training. Fused plan compiles on first row, replays across the batch with refreshed input per iteration. Reparameterize re-samples inside the closure so training stays stochastic. * MisGAN.TrainDataGeneratorStep — dual-noise generator (data noise + mask noise) packed into a single persistent input tensor; closure slices them back out on replay. Loss = -E[D_x(fakeRow ⊙ fakeMask)]. * MisGAN.TrainMaskGeneratorStep — simpler single-noise version; per-sample loop with fused plan replay across the batch. The DataDiscriminator step (WGAN-style critic + weight clipping) and mask discriminator step aren't wired — they need separate plans and the ClipWeights post-step interacts poorly with the fused optimizer's in-place update. Follow-up. Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): gPU-resident fused step for TimeGAN P1/P2 + MedSynth non-DP disc * TimeGAN Phase 1 (TrainReconstructionStepBatched) — embedder + recovery reconstruction MSE. Straight single-network fused-step conversion. * TimeGAN Phase 2 (TrainSupervisedStepBatched) — supervisor's next-step MSE in the embedded space, with the embedder frozen (its layers not in the trainable set for this step). Two-tensor closure (ht, htNext). * MedSynth non-DP disc step — pack (real, fake) into a single input tensor along axis 0; slice back in the loss for BCE-real + BCE-fake. Generator runs OUTSIDE the fused plan so its weights stay frozen on the critic step. TimeGAN Phase 3 (adversarial + supervised joint) and MedSynth DP-SGD paths still use their eager tape — the WGAN-GP gradient penalty and per-example DP clipping don't compose with the single-closure fused optimizer step. Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): extra-tensors parameter on fused-step API Extends the fused compiled-plan training step to accept raw trainable tensors that aren't naturally carried by an ITrainableLayer (e.g. GOGGLE's soft adjacency A; CLAP's learned _logTemperature scalar). Solves the SOLID concerns raised in review of the ITrainableLayer-wrapper alternative: * ISP — no forcing raw tensors to implement the full ILayer surface with no-op Forward / SetTrainingMode / GetParameterGradients members * LSP — no risk of "layer" wrappers with identity Forward diverging from the layer contract's behavioral expectations elsewhere in the codebase Wire-up: * CompiledTapeTrainingStep.TryStepWithFusedOptimizer — new optional extraTensors param. Threaded into the dedup-aware parameter collection (CollectDeduplicatedParametersWithExtras) so extras get moment buffers, gradient accumulation, and GPU-residency in exactly the same code path that layer-carried params do. * GpuResidentFusedStep.TryStep — plumbs extras through. A callsite with only extras (no layers) is a valid config for models whose whole trainable surface is raw tensors. Dedup is by Tensor<T> reference across BOTH sources, so an extra tensor that also happens to be layer-carried is registered exactly once — the same shared/tied-weight protection the layer-only collector provides. Enables Phase 4E (GOGGLE + CLAP wire-ups). Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): wire GOGGLE + CLAP through fused-step extra-tensors API Consumes Phase 4A's extraTensors parameter to route models with raw trainable state (outside the ITrainableLayer contract) through the fused compiled plan. * GOGGLE — soft adjacency A (initialised in InitializeModel, projected after each optimizer step via ProjectAdjacencyConstraints) is passed through extraTensors. The fused optimizer allocates its moment buffer, accumulates gradients, and applies the update in-place. Reparameterize re-samples inside the forward closure so training stays stochastic; ProjectAdjacencyConstraints runs after the fused step to keep the adjacency on the valid soft-adjacency manifold. * CLAP — the learned _logTemperature scalar (Radford 2021 / Wu 2023 contrastive-alignment temperature) goes through extraTensors. Both audio and text encoders are in the layers list; the fused step handles all three parameter classes uniformly. Both fall through to the existing eager tape+optimizer path on any failure (unsupported optimizer, non-compilable graph, etc). Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(diffusion): batched per-element timesteps for DDPM-canonical training Adds industry-standard batched-per-element timestep API to the diffusion stack (Ho et al. 2020 §3, HuggingFace diffusers reference). Previously DiffusionModelBase.Train sampled ONE timestep for the whole batch — reducing the signal-to-noise diversity of the training gradient. Canonical DDPM training samples a distinct timestep per batch element. * INoiseScheduler.AddNoiseBatched(cleanBatch, noiseBatch, timesteps) — default interface implementation delegates to the scalar AddNoise per element for backward compatibility. Concrete schedulers can override with a fused batched implementation. * NoisePredictorBase.PredictNoiseBatched(noisyBatch, timesteps, conditioning) — virtual with default slice-then-call-scalar implementation. Subclasses that want a fused batched forward override this to keep the training loop on-device. * DiffusionModelBase.PredictNoiseBatched — model-level counterpart with the same default slice-then-call-scalar behavior. * DiffusionModelBase.Train detects batched vs rank-1 input and routes through the batched noise scheduler + batched predictor when input.Rank >= 2. Rank-1 unbatched inputs stay on the scalar path for backward compatibility. This is the foundation for the industry-exceeding "batched-per-element + fused-resident" diffusion training path — see follow-up commits that wire DiffusionModelBase.Train through the fused compiled plan. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): wGAN-GP correctness fix + fused disc + DP-SGD wire-ups (4F/4G/4H) Consumes Tensors PR ooples/AiDotNet.Tensors#763 (compiled backward with createGraph=true, DP-SGD helper, multi-slot persistent inputs). ## 4G — WGAN-GP correctness fix + fused disc steps (5 files) Root cause fixed for AiDotNet issue #1844: every ComputeGradientPenalty implementation used the inner GradientTape's ComputeGradients WITHOUT createGraph=true. The inner backward's ops didn't record on the outer tape, so inputGradients had no GradFn chain back to the discriminator weights. Effect: the gradient penalty was included in the loss VALUE but NOT in the disc's gradient — WGAN-GP silently degraded to plain WGAN with no 1-Lipschitz enforcement. Fix (5 files: CTGAN, DPCTGAN, CopulaGAN, CausalGAN, TableGAN): add createGraph: true to the inner ComputeGradients so the outer tape can differentiate the penalty through the disc weights. Also wired 4 disc steps through the fused compiled plan (CausalGAN, CTGAN, CopulaGAN, TableGAN — DPCTGAN's disc is DP so goes via 4H). Packs (real, fake) into one persistent input along axis 0; loss closure splits scores back out and computes wasserstein + λ·GP. The fused path activates once Tensors PR #763 lands (its 4C change removes the !createGraph gate at GradientTape.cs:727); until then these wire-ups fall through to the (now-corrected) eager tape path. ## 4H — DP-SGD wire-ups (2 files) Local AiDotNet.Training.DpSgdStep<T> — drop-in mirror of the same helper in Tensors PR #763 (Engines.Training.DpSgdStep<T>). Enables the wire-ups to land in this PR without waiting on the Tensors NuGet publish. Both implementations enforce the Abadi 2016 §3 Algorithm 1 clip-BEFORE-aggregate order via their structure. * DPCTGAN.TrainDiscriminatorStepBatchedDP — routes the per-example WGAN-GP + DP-SGD critic through DpSgdStep<T>. Objective: Wasserstein + λ·GP per example, clipped per-example, aggregated, noised, averaged. * MedSynth.TrainDiscriminatorStepPerExampleDPSGD — routes the per- example non-saturating BCE + DP-SGD critic through the same helper. Once Tensors PR #763 merges + NuGet publishes, the local mirror can be swapped for the Tensors version by changing the `using` — the API surface is identical by design. ## 4F — Batched-per-element diffusion foundation DiffusionModelBase.Train samples per-batch-element timesteps for rank ≥ 2 inputs (Ho et al. 2020 canonical pattern; HuggingFace diffusers reference), routing through NoiseSchedulerBase.AddNoiseBatched + DiffusionModelBase.PredictNoiseBatched. Rank-1 unbatched inputs keep the scalar path for backward compat. net471 doesn't support default interface implementations, so AddNoiseBatched lives on NoiseSchedulerBase<T> (as virtual with a default per-element delegate) rather than on the INoiseScheduler<T> interface. DiffusionModelBase's fused-training wire-up (via Tensors PR #763's PersistentInputRegistry) is queued for the NuGet-bump follow-up commit — the API foundation is already in place. Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(review): resolve 12 CodeRabbit findings on PR #1843 Addresses every unresolved review comment on the non-TS GPU-residency PR. ## Correctness fixes * DiffusionModelBase.Train — RecomputeForward closure now dispatches isBatched ? PredictNoiseBatched(inp, timesteps) : PredictNoise(inp, timestep), so optimizer re-evaluation matches the recorded batched forward objective. * NoisePredictorBase.PredictNoiseBatched — detect batch-aligned conditioning (leading dim == batchSize) and slice per element; preserve shared conditioning when the leading dim doesn't match. Fixes shape-mismatch / conditioning-shared bugs in classifier-free-guidance-style predictors. * CSDI.Train — remove the fused-plan branch. ComputeDenoisingPairTape samples fresh (timestep, noise) each call, and calling it in both Fwd and Loss produces independent samples that don't match. The compiled plan can't refresh the RNG per replay either. Path stays on the eager tape until Tensors' PersistentInputRegistry (PR ooples/AiDotNet.Tensors#763) lands. * TFC.Train — ComputeContrastiveLossTape now runs INSIDE ForwardCombined so it consumes the current-step persistent input (`inp`), not the outer `input` which would freeze at compile time. Closure-captured local; Loss reads it with a null-guard covering the Fwd-then-Loss ordering invariant. * TOTEM.Train — capture the commitment tensor from ForwardNativeForTrainingWithCommitment's first call; reuse it in Loss instead of running the quantizer a second time. Without this, the compiled path performs an EMA SetCodebookValue update TWICE per step and diverges from the eager path. * NoiseSchedulerBase.AddNoiseBatched — validate full shape parity (rank + every dim), not only the leading batch dim. Prevents indexing beyond noiseBatch's span when a caller passes [B, smaller...] shapes. ## Correctness cleanup * CTGAN.TrainGeneratorStepBatched — capture (act, condFromInput, maskFromInput) from Fwd's single generator pass; reuse in Loss so the conditional-CE term doesn't re-run GeneratorForwardWithResidualBatched + ApplyOutputActivationsBatched (was doubling the per-step generator forward cost). * TabSynGenerator.TrainVAEBatch — capture (mean, logVar) from Fwd's single encoder pass; reuse in Loss so the encoder doesn't run twice per row. Also: if the fused step fails mid-batch after fusedEngaged=true, drop to the eager path for the remaining rows (no row is silently skipped). * FourierNeuralOperator.TapeTrainStep — remove the duplicate _fourierLayers loop in both the fused-path allTrainable collection AND the eager paramList. Fourier layers are already registered into Layers at construction, so iterating them again double-registered each Fourier parameter and drove the eager SGD to apply the update twice per step. ## API cleanup * GpuResidentFusedStep<T> — public → internal. Aligns with CompiledTapeTrainingStep<T> which is already internal; keeps in-assembly training plumbing out of the public API surface. Also fixed an inadvertent null-forgiving operator (`!`) usage that slipped into the review-fix edits. All new nullable annotations use explicit null-guards with descriptive InvalidOperationException on invariant violation (per CLAUDE.md's "never use null-forgiving operators" rule). INoiseScheduler default-interface issue (net471 incompatibility) was already fixed in an earlier commit (AddNoiseBatched moved to NoiseSchedulerBase). Builds green net471 / net8.0 / net10.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): vectorized fused DP-SGD/WGAN-GP/multi-slot primitives + consumer wire-ups Adds three fused-plan training primitives (mirrors of AiDotNet.Tensors PR #763) and wires the DP-SGD consumers through them: * DpSgdFusedStep<T> — Abadi 2016 §3 Algorithm 1 per-example clip-before-aggregate. Runs each per-example forward+backward through a compiled plan (LR=0 so weights don't drift between replays), then computes the global L2 norm, clips, noises, and aggregates via vectorized IEngine ops (TensorMultiply/ReduceSum for the norm, TensorMultiplyScalar/TensorAdd for the accumulator, TensorRandomNormalInto for on-device Gaussian noise). Returns aggregated gradients so the caller's configured optimizer (Adam/AdamW/SGD/...) applies the update — no hardcoded SGD step inside. * WganGpFusedStep<T> — Gulrajani 2017 WGAN-GP critic step. Composes E[D(fake)] − E[D(real)] + λ·(‖∇_x̃ D(x̃)‖₂ − 1)² inside one compiled plan with the inner ∇_x̃ D(x̃) recorded via createGraph=true so it differentiates into disc weights (issue #1844 fix). OnesLike uses vectorized Engine.TensorFill. * MultiSlotFusedStep<T> — N-slot persistent input mechanism with plan cache keyed by composite shape + parameter identity. Slots refreshed via AsSpan().CopyTo(AsWritableSpan()) — no per-element loops on the hot path. Consumer wire-ups (this PR): * DPCTGANGenerator.TrainDiscriminatorStepBatchedDP — primary path routes through DpSgdFusedStep.TryStep, falls back to the existing ComputePerExampleNoisedGradients when the fused path can't engage (non-GPU host / compilation disabled). * MedSynthGenerator.TrainDiscriminatorStepPerExampleDPSGD — same primary/fallback layering. * Both legacy fallback loops (ComputePerExampleNoisedGradients + MedSynth eager) are now themselves vectorized: global-L2-norm via ReduceSum(g·g), clipped accumulation via TensorMultiplyScalar+TensorAdd, on-device Gaussian noise via TensorRandomNormalInto+TensorAdd — no scalar per-element loops anywhere on the DP-SGD path. Codebase convention adopted: * Class-scope `private static IEngine Engine => AiDotNetEngine.Current;` and `private static readonly INumericOperations<T> Ops = MathHelper.GetNumericOperations<T>();` on each fused-step class (matches the ~20 activation/optimizer/etc. bases). No IEngine or INumericOperations<T> threaded through method signatures. Skipped (follow-ups filed): * WGAN-GP consumer rewire → #1845 (blocked on IGradientBasedOptimizer<T> config accessor extension; consumers already correct via GpuResidentFusedStep + local createGraph=true ComputeGradientPenalty). * Diffusion consumer rewire → #1846 (blocked on per-generator forward-refactor to move _timestepProjection inside compiled plan while treating raw timestep as a persistent slot). Verification: * net8.0 + net471 + net10.0 all build clean on both AiDotNet and AiDotNet.Tensors. * Old DpSgdStep.cs deleted (replaced by DpSgdFusedStep). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(review): tFC + TOTEM preprocessing rewritten with traceable engine ops Resolves the two BLOCKING CodeRabbit findings on PR #1843: both fused-plan training paths were freezing preprocessing into the compiled plan because their inner ops used host-side .Data.Span loops, so later replays reused the first batch's normalized values / spectrum / argmin decisions instead of recomputing per batch. TFC — ApplyInstanceNormalization (RevIN) + ComputeFrequencyRepresentation: * ApplyInstanceNormalization now delegates to a new stateless NormalizeWithStats that returns (normalized, mean, std) as tensors. Under the hood: ReduceMean + ReduceVariance + TensorSqrt + TensorBroadcastSubtract + TensorBroadcastDivide. All ops record on the tape and re-execute per replay under the compiled plan. * DenormalizeForecast likewise delegates to DenormalizeForecastWithStats, which takes mean/std as explicit tensor parameters. ForwardNative threads them through as locals so the compile-mode replay uses CURRENT-step stats instead of frozen trace-time values. * _revinMean/_revinStd (Vector<T> scalars) → _revinMeanTensor/_revinStdTensor (Tensor<T>? nullable) — kept for the abstract override's external callers, but the fused path never reads them. * ComputeFrequencyRepresentation now uses Engine.RFFT for batched real FFT → reshape to [B, halfN, 2] pairs → TensorMultiply + ReduceSum(axis=2) for magSquared → TensorSqrt + TensorMultiplyScalar(1/n) for the one-sided magnitude spectrum → TensorSlice + TensorFlip + TensorConcatenate to mirror bins [1..n-halfN] into the tail. Handles even and odd n identically to the old scalar impl. * Fused-plan fast path in TFC.Train restored — now safe because both preprocessing methods trace correctly. TOTEM — VectorQuantize (VQ-VAE) traceable rewrite + post-Step EMA: * New VectorQuantizeTraceable returns (quantized, commitmentLoss, argmin, head) using Engine ops end-to-end: TensorBroadcastSubtract + TensorMultiply + ReduceSum for distances, TensorArgMin along the codebookSize axis, per-c TensorSliceAxis + TensorIndexSelectDiff + TensorStack for the gather, Engine.StopGradient for the straight-through estimator, ReduceSum-based commitment loss weighted by β/totalLen. * EMA moved OUT of the compiled forward into a new UpdateCodebookEMA(head, argmin). Called POST-Step by the fused path with the trace-time graph-node references — their .Data reflects the LAST replay so the update lands exactly once per batch (matches CodeRabbit's "EMA must execute exactly once per batch" contract). Under the compiled plan, argmin/head are refreshed by every _plan.Step() so post-Step reads see the current batch's values. * UpdateCodebookEMA expresses the per-codebook scatter as: current codebook slice + TensorScatterAdd((1-decay)·(head - gathered), argmin) → new slice, then TensorConcatenate across codebook axis into the full [numCodebooks, codebookSize, codebookDim] tensor, then Engine.TensorCopy back into the _codebooks tensor object to preserve identity (future reads via the same reference see the update). * Legacy VectorQuantize is now a thin adapter around VectorQuantizeTraceable for callers that don't need the extras. * ForwardNativeForTrainingWithCommitment delegates to a new ForwardNativeForTrainingWithVQExtras that exposes the argmin/head; the original (forecast, commitmentLoss) contract is preserved for callers that don't need EMA state. * Fused-plan fast path in TOTEM.Train restored — now safe because VQ is fully traceable AND the EMA runs exactly once per batch in post-Step eager code. No new Tensors primitives were needed — the engine already has RFFT, ReduceMean/Variance, TensorSqrt, TensorBroadcastSubtract/Divide, TensorFlip, TensorConcatenate, TensorArgMin, TensorIndexSelectDiff, TensorSliceAxis, TensorStack, StopGradient, TensorScatterAdd, and TensorCopy, all in the autodiff/compile registry per OpRegistry.cs. Verification: * net8.0 + net471 + net10.0 all build clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): centralize WGAN-GP + diffusion consumers through fused primitives (#1847) Closes #1845 and #1846. Routes all four WGAN-GP critics through WganGpFusedStep (the fused-plan primitive from PR #1843) and adds MultiSlotFusedStep wire-ups to the three tabular diffusion consumers plus a base-class opt-in hook for DiffusionModelBase subclasses. ## Optimizer config plumbing Zero new API surface: IFusedOptimizerSpec.TryGetFusedOptimizerConfig already exposes (OptimizerType, LR, Beta1, Beta2, Epsilon, WeightDecay, Schedule, UseBf16Moments). Widened NeuralNetworkBase<T>.TryMapToFusedOptimizerConfig from private to internal so the sibling generators in the same assembly can reuse the existing helper. ## WGAN-GP consumers (#1845) * CTGANGenerator, CopulaGANGenerator, TableGANGenerator, CausalGANGenerator — each critic training method now attempts WganGpFusedStep.TryStep FIRST with the discriminator's optimizer hyperparameters extracted via TryMapToFusedOptimizerConfig, falls back to the existing GpuResidentFusedStep path (secondary fused), then the eager tape (final fallback). The ε ∈ [0, 1]^B epsilon sampler uses Engine.TensorRandomUniformRange to match each critic's local ComputeGradientPenalty behavior. * Non-Adam optimizers (Lion, LBFGS) that don't implement IFusedOptimizerSpec cleanly fall through — TryMapToFusedOptimizerConfig returns false and the code path skips to GpuResidentFusedStep as before. ## Diffusion consumers (#1846) * TabDDPMGenerator — refactored to expose a slot-based forward (BuildTabDDPMSlots + DenoiserForwardFromTensors + ComputeDiffusionLossTapeFromTensors). Per-row TrainBatch loop now attempts MultiSlotFusedStep with (numNoisy, actualNoise, catNoisy, catClean, rawSinusoidalTimeEmbed) as persistent slots. The learnable _timestepProjection stays INSIDE the compiled forward closure so its weights participate in the backward pass. Plan is compiled once on the first row and replayed via slot-data refresh for subsequent rows. * TabSynGenerator — TrainDiffusionBatch's per-row loop wired with MultiSlotFusedStep on (noisyLatent, actualNoise, projectedTimeEmbed). Matches the existing eager path's semantic that _timestepProjection is NOT in _diffMLPLayers (kept detached in the eager path too), so the projected embedding is precomputed host-side per row and passed as slot data. * Finance/Forecasting/Foundation/CSDI — - ApplyInstanceNormalization rewritten with traceable engine ops (ReduceMean + ReduceVariance + TensorSqrt + broadcast subtract/divide) — same pattern as the TFC RevIN fix. The previous `.Data.Span` per-batch loop froze at trace time. - New BuildCsdiSlots + DenoiserForwardFromSlots express the DDPM x_t formation and packed denoising input via TensorConcatenate + engine scalar multiplies. Replaces the `.Data.Span[i] = xt[0, i]` fill that baked the trace batch's x_t into the compiled plan. - Train() attempts MultiSlotFusedStep first, falls back to the existing eager ComputeDenoisingPairTape path when the fused path can't engage. * Diffusion/DiffusionModelBase — - New opt-in `protected virtual bool SupportsFusedDenoising => false;` property. Base default is false so no existing subclass changes behavior. - Train() attempts MultiSlotFusedStep when SupportsFusedDenoising is true AND the training optimizer maps cleanly to a fused config. Slots: (noisySample, noise). Loss = MSE(pred, noise). QAT shadow restoration is preserved on the fused-success path. - Subclasses with fully-traceable PredictNoise / PredictNoiseBatched (e.g. after auditing to remove `.Data.Span` host loops) can opt in via a single-line override; no infrastructure changes needed elsewhere. ## Verification * net8.0, net471, net10.0 all build clean. * No API surface changes on IGradientBasedOptimizer<T> — the existing IFusedOptimizerSpec interface (already implemented by all fuse-able optimizers) provided everything needed. * All consumers preserve eager fallback path for non-fuse-able optimizers and non-GPU hosts. Co-authored-by: franklinic <franklin@ivorycloud.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * feat(training): wire additional WGAN-GP + diffusion consumers through fused primitives (#1848) Extends PR #1847 with three additional consumer wire-ups discovered during a comprehensive audit against the primitives added in PR #1843. Each consumer attempts the fused path first via the existing IFusedOptimizerSpec / TryMapToFusedOptimizerConfig plumbing and falls back to eager tape training when the fused path can't engage. ## WGAN-GP consumers (additional beyond #1845) * NeuralNetworks/WGANGP.cs — standalone WGAN-GP class. Previously used the legacy flat-vector round-trip (three Critic.Predict calls per step, GetParameterGradients + host-side vector combine + UpdateCriticWithOptimizer). TrainCriticBatchWithGP now attempts WganGpFusedStep.TryStep first, using Critic.Layers as the parameter source and a per-batch ε ~ U(0, 1) sampler matching each critic's local ComputeGradientPenalty behavior. Falls back to the legacy path when the critic's optimizer has no fused-kernel mapping. ## Diffusion consumers (additional beyond #1846) * NeuralNetworks/SyntheticData/FinDiffGenerator.cs — per-row DDPM training (Sattarov et al. 2023). TrainBatch caches a MultiSlotFusedStep across rows so the compiled plan is built once and replayed via slot-data refresh for subsequent rows. Slots: (packedDenoiserInput, targetNoise). Falls back to the existing Train(input, targetNoise) call on miss. * NeuralNetworks/SyntheticData/AutoDiffTabGenerator.cs — per-row DDPM training with a custom TapeStepOver optimizer step. Same MultiSlotFusedStep caching pattern as FinDiff. Slots: (denoiserInput, targetNoise). Falls back to the existing tape-based TapeStepOver path on miss. ## Explicitly not wired in this PR (documented) * NeuralNetworks/GenerativeAdversarialNetwork.cs — uses BCE loss with optional GP regularization (a separate auxiliary optimizer step), NOT Wasserstein + GP. Wiring WganGpFusedStep would change training semantics from BCE to Wasserstein. Requires a separate design decision. * Finance/Forecasting/Foundation/{CCDM,MGTSD,TSDiff,TimeDiff,TimeGrad}.cs and Finance/Probabilistic/DiffusionTS.cs — each uses .Data.Span host-side packs for the denoising input (same trace-freeze issue TFC/CSDI hit in PR #1843). Each needs a TFC/CSDI-scale traceable rewrite BEFORE fused wiring is safe. Deferred to individual per-consumer PRs. * MetaLearning/Algorithms/{MetaDDPMAlgorithm,MetaDMAlgorithm,MetaDiffAlgorithm}.cs — hand-rolled Vector<T> params with index arithmetic, NOT the ILayer/Engine/tape infrastructure. Would need full rewrite to fit MultiSlotFusedStep. * DiffusionModelBase subclasses (DDPMModel, DiffWaveModel, LatentDiffusionModelBase) — the SupportsFusedDenoising opt-in hook (added in PR #1847) is available, but flipping it safely requires per-class audit of PredictNoise → UNet traceability. Left as follow-up work; the mechanism is in place. ## Verification * net8.0, net471, net10.0 all build clean. * No API surface changes. Co-authored-by: franklinic <franklin@ivorycloud.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * chore(deps): bump AiDotNet.Tensors 0.111.2 -> 0.113.0 (activates #763 for fused paths) 0.113.0 publishes Tensors PR #763 — this branch's gating dependency. #763's key fix (4C): the compiled/persistent backward now honors createGraph=true (GradientTape. ComputeGradients previously gated the compiled path on !createGraph), so the WGAN-GP gradient penalty's inner backward differentiates into the disc weights through the fused compiled plan instead of silently returning zeros (issue #1844). Against 0.111.2 the fused GPU-resident WGAN-GP path would have silently degraded to plain WGAN. The src/Training fused-primitive mirrors (WganGpFusedStep, MultiSlotFusedStep, DpSgdFusedStep) stay as the consumer-side primitives that #1847/#1848 centralize every consumer through; they call the public engine API, so the bump alone routes them onto #763's fixed compiled backward. Also picks up #765 (compiled ReduceMax axis fill) and #764 (resident-param fused-Adam fix). Native OneDNN/OpenBLAS/CLBlast bumped in lockstep (all published). Builds green on net8.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: franklinic <franklin@ivorycloud.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test Coverage
Bug Fix
AiDotNet.Native.CLBlast.csprojproject reference from test project (now a NuGet package)Linked Issue
Closes #654
Test plan
🤖 Generated with Claude Code