GPU CI fails on main with CUDNN_STATUS_EXECUTION_FAILED_CUDART

[ChrisRackauckas-Claude]

GPU CI is failing on main with a cuDNN execution error. I hit the same failure while checking PR #221, then verified it also appears on the unmodified base branch.

Main run: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940793272
Main GPU job: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940793272/job/85862242539

PR run with same failure: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940670571/job/85861840439

CI reproduction context from the job:

import Pkg
Pkg.test(;
    coverage = true,
    julia_args = ["--check-bounds=yes", "--compiled-modules=yes", "--depwarn=yes"],
    force_latest_compatible_version = false,
    allow_reresolve = true,
)

with GROUP=GPU on Julia 1.12.6.

Representative failure output:

Warning: No valid algorithm found, probably bad params for convolution.
x_size: (3, 3, 1, 3): Error During Test at test/layers_tests.jl:41
CUDNNError: CUDNN_STATUS_EXECUTION_FAILED_CUDART (code 5003)
...
Layers Tests | 734 pass, 36 error, 770 total
DEQ/cuda | 540 pass, 36 error, 576 total

The same CI log also shows a CUDA runtime artifact download data error earlier in the job:

Downloading artifact: CUDA_Runtime
ERROR: Data Error : jl_dgluVCHUZS-download
Failure artifact: CUDA_Runtime

I did not bisect locally because this checkout is shallow/grafted and I do not have a local GPU reproduction environment available here. This appears independent of PR #221's documentation-only changes because it reproduces on main.

[ChrisRackauckas]

This is a hardware bug according to the response.

State: blocked
Now: diagnosis done; waiting on decisions before opening any PRs.
Done so far:

Root cause of #222 (cuDNN error): the GPU lane runs on V100 (sm_70), and cuDNN ≥ 9.11 has no kernels for it. It's hardware, not a DEQ bug.
Every V100 run fails all 36 conv tests. The only green run since July (2026-07-10) got a T4 runner, with identical package versions (CUDA 6.2.1, CUDNN_jll 9.24.0+0) to the failing runs around it.
9ee48db (2026-07-12) pinned the lane to the gpu-v100 runner label, so every run since has been on V100.
NVIDIA's current cuDNN support matrix lists CC 7.5 as the minimum. cuobjdump on the CUDNN_jll CUDA 12 artifacts: libcudnn_cnn/libcudnn_ops ship sm_70 SASS in 9.10.0 and none in 9.12, 9.13, 9.17.1, 9.20 or 9.24 (sm_75 minimum). The only PTX is sm_12x, which can't JIT down to sm_70.
Already reported upstream as JuliaGPU/CUDA.jl#2890 (open, no fix PR). The CUDA/cuDNN Julia wrappers don't take the device's compute capability into account when choosing cuDNN.
Second, independent regression that the cuDNN failure was hiding: on an RTX 4080 SUPER (sm_89) there are no cuDNN errors, but 23 Scalar indexing is disallowed errors remain, all in the NewtonRaphson(autodiff = AutoForwardDiff()) cases.
Bisected with everything else fixed: ForwardDiff 1.4.3 passes, 1.4.4 fails. CI shows the same pattern: runs on 1.4.1/1.4.3 have 0 scalar errors, runs on 1.4.5/1.4.6 have them.
Cause: GPU ForwardDiff Jacobians only work here because Lux overloads ForwardDiff.seed! for GPU arrays (src/helpers/forwarddiff_training.jl). ForwardDiff#821 (released in 1.4.4) moved partial-zeroing to a new seed_zero_partials!, which Lux doesn't cover, so GPU arrays fall back to the scalar loop.
ForwardDiff#816 (open, awaiting review) fixes this inside ForwardDiff. A Lux-side fix is two extra overloads, as PolyesterForwardDiff#34 did for its own code; on ForwardDiff 1.4.6 a minimal DEQ repro fails without it and passes with it.
Decisions: none taken yet. What I'd propose:

DEQ: switch test/test_groups.toml GPU runner from gpu-v100 to gpu-t4 (NeuralOperators.jl and DiffEqGPU.jl use it). One-line CI change; no DEQ code change is needed for #222.
Lux: PR adding ForwardDiff.seed_zero_partials! GPU overloads next to the existing seed! ones, with a GPU test. Alternatively, wait for ForwardDiff#816.
Add the per-version architecture table and a minimal reproducer to JuliaGPU/CUDA.jl#2890.
Before opening PRs I'd run the full DEQ GPU suite with both fixes applied; so far only the single-case repro has been checked with the Lux patch.

Blocked on / questions: @ChrisRackauckas

Was gpu-v100 chosen for a reason (memory, queue time) that rules out gpu-t4?
For the ForwardDiff regression, should I open the Lux PR, or will ForwardDiff#816 land soon enough?
OK to comment on JuliaGPU/CUDA.jl#2890 with the evidence?
Links: local branch ci/gpu-t4-runner (not pushed). CI logs used: main runs 34685669460 (V100, fail), 29114758096 (T4, pass).

from @SebastianM-C