GPU performance gap vs TorchSharp/TensorFlow (Phase B)

[ooples]

Summary

Benchmarks show AiDotNet GPU ops are significantly slower than TorchSharp CPU and TensorFlow.NET CPU for micro-ops (10x-235x slower on elementwise and reductions, ~9x-20x on MatMul/Conv2D). This suggests launch + transfer overheads dominate and kernel implementations are not yet competitive.

Evidence

• Benchmark docs: docs/GPU_OPTIMIZATION_REPORT.md
• TorchSharp run (CPU only): AiDotNetBenchmarkTests/BenchmarkDotNet.Artifacts/results/AiDotNetBenchmarkTests.TorchSharpComparisonBenchmarks-report-github.md
• TensorFlow.NET run (CPU oneDNN): BENCHMARKS.md

Hypotheses / Bottlenecks

• Per-op allocations and CPU<->GPU transfers (Allocate1D, CopyFromCPU, CopyToCPU) dominate.
• Frequent _accelerator.Synchronize() and _gpuLock serialize work and prevent overlap.
• Kernel dispatch overhead and lack of fusion for common op chains.
• Naive GEMM/Conv2D kernels lacking tiling/shared memory.
• High managed allocations in hot paths (Gen0/Gen1 pressure in BDN).

Proposed work (prioritized)

P0

• Add GPU-resident tensor paths and minimize host/device transfers across chained ops.
• Use pooled GPU buffers in all hot paths and reuse outputs where safe.
• Reduce or defer synchronization and shrink lock scope to allow async execution.

P1

• Ensure kernel caching covers all hot paths (remove runtime LoadAutoGroupedStreamKernel).
• Add kernel fusion for common op sequences (bias+activation, add+activation, etc.).
• Optimize GEMM/Conv2D with tiling/shared memory and vectorized loads.

P2

• Optional vendor lib interop (cuBLAS/cuDNN or rocBLAS/MIOpen) for GEMM/Conv2D.
• Recompute adaptive thresholds from size-sweep benchmarks.
• Add profiling to split kernel vs transfer time in benchmarks.

Notes

TorchSharp CUDA is installed, but torch.cuda.is_available() is false on the current AMD GPU (gfx902). GPU-vs-GPU comparisons should be re-run on NVIDIA once available.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• ISSUE-412

Related Issues

• [Inference Optimization] Implement Kernel Optimization and Custom Operators #412
• [Inference Optimization] Implement Graph Optimization and Operator Fusion #409
• [Deployment] Implement TensorRT Integration and Mobile Optimization #414
• Adopt Industry-Standard Meta-Learning Data Abstractions from PR #307 and PR #299 #442
• [Gap Analysis] Implement Smart Distributed Training (FSDP-Inspired) #309

👤 Suggested Assignees

• ooples

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[ooples]

Applied quick-win fixes in GpuEngine: removed duplicate kernel launches and explicit accelerator synchronizations (now rely on blocking CopyToCPU). Re-ran TorchSharpComparisonBenchmarks; results were within noise of the prior run, indicating host-device transfers still dominate. Next focus: GPU-resident tensors, kernel fusion, and tiled GEMM/Conv2D kernels.