Overview
This issue tracks the implementation of comprehensive integration tests for the Optimizers module. This is a P0 (Critical Priority) module for training neural networks and optimization problems.
Parent Issue: #615
Current Status
- Coverage: 7% (only scheduler integration tested)
- Source Files: 42 files (40 optimizers + 2 base classes)
- Test Files: OptimizerSchedulerIntegrationTests.cs (tests only Adam, AdamW, Lion with schedulers)
Module Location
src/Optimizers/
Missing Optimizer Tests (37 optimizers)
Gradient-Based Optimizers (Need Tests)
First-Order Methods:
- GradientDescentOptimizer
- StochasticGradientDescentOptimizer (SGD)
- MiniBatchGradientDescentOptimizer
- ModifiedGradientDescentOptimizer
- MomentumOptimizer
- NesterovAcceleratedGradientOptimizer (NAG)
Adaptive Learning Rate:
- AdagradOptimizer
- AdaDeltaOptimizer
- RootMeanSquarePropagationOptimizer (RMSProp)
- AdaMaxOptimizer
- AMSGradOptimizer
- NadamOptimizer
- LAMBOptimizer
- LARSOptimizer
- FTRLOptimizer
Second-Order Methods:
- NewtonMethodOptimizer
- BFGSOptimizer
- LBFGSOptimizer
- DFPOptimizer
- ConjugateGradientOptimizer
- LevenbergMarquardtOptimizer
- TrustRegionOptimizer
Proximal Methods:
- ProximalGradientDescentOptimizer
- CoordinateDescentOptimizer
- ADMMOptimizer
Already Tested (3 optimizers)
- AdamOptimizer (scheduler integration only)
- AdamWOptimizer (scheduler integration only)
- LionOptimizer (scheduler integration only)
Metaheuristic Optimizers (Need Tests)
- GeneticAlgorithmOptimizer
- ParticleSwarmOptimizer
- DifferentialEvolutionOptimizer
- SimulatedAnnealingOptimizer
- AntColonyOptimizer
- TabuSearchOptimizer
- CMAESOptimizer (Covariance Matrix Adaptation)
- BayesianOptimizer
- NelderMeadOptimizer
- PowellOptimizer
Test Categories Required
1. Convergence Tests
2. Gradient Computation Tests
3. Learning Rate Tests
4. Algorithm-Specific Tests
SGD/Momentum:
Adam/AdamW:
RMSProp:
Adagrad:
Second-Order Methods:
Metaheuristic:
5. Edge Cases
6. Serialization Tests
7. Performance Tests
Mathematical Correctness Verification
SGD with Momentum
v_t = momentum * v_{t-1} + gradient
param = param - learning_rate * v_t
Adam
m_t = beta1 * m_{t-1} + (1 - beta1) * g_t
v_t = beta2 * v_{t-1} + (1 - beta2) * g_t^2
m_hat = m_t / (1 - beta1^t)
v_hat = v_t / (1 - beta2^t)
param = param - lr * m_hat / (sqrt(v_hat) + eps)
RMSProp
v_t = decay * v_{t-1} + (1 - decay) * g_t^2
param = param - lr * g_t / sqrt(v_t + eps)
Test Functions (Optimization Benchmarks)
- Sphere: f(x) = sum(x_i^2) - Simple convex
- Rosenbrock: f(x,y) = (1-x)^2 + 100*(y-x^2)^2 - Non-convex valley
- Rastrigin: f(x) = 10n + sum(x_i^2 - 10cos(2pi*x_i)) - Multimodal
- Ackley: Multimodal with many local minima
- Beale: Non-convex with steep walls
Priority Order
-
Critical (Test First):
- SGD (base optimizer)
- Momentum
- Adam (expand beyond scheduler tests)
- AdamW (expand beyond scheduler tests)
- RMSProp
-
High:
- Adagrad
- NAG (Nesterov)
- L-BFGS
- AdaMax
-
Medium:
- All other gradient-based optimizers
- Metaheuristic optimizers
Acceptance Criteria
References
Overview
This issue tracks the implementation of comprehensive integration tests for the Optimizers module. This is a P0 (Critical Priority) module for training neural networks and optimization problems.
Parent Issue: #615
Current Status
Module Location
src/Optimizers/Missing Optimizer Tests (37 optimizers)
Gradient-Based Optimizers (Need Tests)
First-Order Methods:
Adaptive Learning Rate:
Second-Order Methods:
Proximal Methods:
Already Tested (3 optimizers)
Metaheuristic Optimizers (Need Tests)
Test Categories Required
1. Convergence Tests
2. Gradient Computation Tests
3. Learning Rate Tests
4. Algorithm-Specific Tests
SGD/Momentum:
Adam/AdamW:
RMSProp:
Adagrad:
Second-Order Methods:
Metaheuristic:
5. Edge Cases
6. Serialization Tests
7. Performance Tests
Mathematical Correctness Verification
SGD with Momentum
Adam
RMSProp
Test Functions (Optimization Benchmarks)
Priority Order
Critical (Test First):
High:
Medium:
Acceptance Criteria
References