fix(#1405): moe default optimizer overshoots — use amsgrad adam(1e-4)

[ooples]

Summary

MixtureOfExpertsNeuralNetworkTests.MoreData_ShouldNotDegrade was failing because the network's built-in default optimizer (vanilla AdamOptimizer<T,Tensor,Tensor>(this)) diverges in the long-training regime: short training fits, but loss climbs again as training proceeds.

This is the same signature as the DenseNet divergence fixed in #1393, and the same recipe applies.

Root cause

Two reinforcing problems with the previous default:

1. LR too aggressive for MoE gradient paths. In MoE, the gating network plus each expert all touch shared parameters on every step, so the effective per-parameter update is larger than for a typical feed-forward net. Vanilla Adam's default LR makes step sizes grow late in training.
2. Vanilla Adam's v̂ denominator drifts late in training (Reddi et al. 2018, On the Convergence of Adam and Beyond). AMSGrad fixes this by keeping the running max of v instead of the EMA, which prevents the denominator from shrinking and the step from blowing up.

Fix

Switch the built-in default in MixtureOfExpertsNeuralNetwork to AMSGrad-mode Adam at lr = 1e-4:

_optimizer = optimizer ?? new AdamOptimizer<T, Tensor<T>, Tensor<T>>(
    this,
    new AdamOptimizerOptions<T, Tensor<T>, Tensor<T>>
    {
        InitialLearningRate = 1e-4,
        UseAMSGrad = true
    });

Users who pass an explicit optimizer are unaffected.

Test plan

• MixtureOfExpertsNeuralNetworkTests.MoreData_ShouldNotDegrade passes on net10.0
• Release build clean (0 errors)

Closes #1405

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• Bug Fixes
  • Enhanced default optimizer configuration for Mixture of Experts Neural Network with optimized learning rate and AMSGrad mode to improve training stability and convergence.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
aidotnet_website	Ready	Preview, Comment	May 20, 2026 6:48pm
aidotnet-playground-api	Ready	Preview, Comment	May 20, 2026 6:48pm

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d58217a1-1db5-413a-bae3-8db84caaafe9

📥 Commits

Reviewing files that changed from the base of the PR and between 330380e and 6bdcd34.

📒 Files selected for processing (1)

• src/NeuralNetworks/MixtureOfExpertsNeuralNetwork.cs

---

Walkthrough

MixtureOfExpertsNeuralNetwork<T> adds using AiDotNet.Models.Options; and updates the default optimizer initialization in its parameterized constructor to instantiate AdamOptimizer with explicit AdamOptimizerOptions configuration: InitialLearningRate = 1e-4 and UseAMSGrad = true. Existing optimizer overrides remain unchanged.

Changes

Default AMSGrad optimizer configuration

Layer / File(s)	Summary
Default AMSGrad optimizer configuration src/NeuralNetworks/MixtureOfExpertsNeuralNetwork.cs	Adds using AiDotNet.Models.Options; import and replaces parameterless AdamOptimizer default with explicit AdamOptimizerOptions configuration (learning rate 1e-4, AMSGrad enabled) in the constructor's _optimizer initialization.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• ooples/AiDotNet#1403: Applies identical AMSGrad optimizer default configuration (learning rate 1e-4, UseAMSGrad = true) to DenseNetNetwork constructor, following the same pattern as this MixtureOfExpertsNeuralNetwork change.

Poem

🧠 The mixture finds its balance true,
With AMSGrad's steadying hand to guide,
A learning rate of 1e-4 sets the view—
No more divergence, just adaptive stride. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: switching MoE default optimizer to AMSGrad Adam with 1e-4 learning rate to fix overshooting behavior.
Linked Issues check	✅ Passed	The PR fully addresses issue #1405 by implementing the exact fix specified: changing default optimizer to AMSGrad Adam (lr=1e-4) to prevent loss divergence in late training, directly resolving the failing MoreData_ShouldNotDegrade test.
Out of Scope Changes check	✅ Passed	All changes are strictly scoped to the MixtureOfExpertsNeuralNetwork default optimizer configuration; no unrelated modifications or ancillary changes are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-1405-moe-overshoot

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.