RuntimeError: CUDA error: an illegal memory access was encountered

[Northern-mountains]

File "E:\ProgramData\anaconda3\envs\fast_gaussian_splatting\lib\site-packages\diff_gaussian_rasterization_fastgs\__init__.py", line 103, in forward num_rendered, num_buckets, color, radii, geomBuffer, binningBuffer, imgBuffer, sampleBuffer, accum_metric_counts = _C.rasterize_gaussians(*args) RuntimeError: CUDA error: an illegal memory access was encountered

Hello, author. I also often encounter this error randomly when running the code. May I ask how you solved it? Thank you! My device is an RTX 5080 with CUDA version 12.9. I also randomly get this error when using an RTX 3090 with CUDA 11.6 on another device.
I also looked up some common solutions and tried them one by one, but none of them worked. May I ask if you've encountered this problem before? Do you have any solutions?

[Tianci-Wen]

@Shiwei-Ren

[Shiwei-Ren]

@Northern-mountains Hello, Northern-mountains,

We’re very sorry for the trouble this issue has caused you! I encountered this problem myself when I first wrote the code. I believe it’s not an issue with the internal code of diff_gaussian_rasterization_fastgs, but rather with the variables passed into render.py. After finishing the code, I ran it many times on my own machine and never ran into this problem.

I recommend using a lower CUDA version, for example, 11.8. I also have a few questions to confirm:

1.For your RTX 3090 with CUDA 11.6, is your local CUDA version really 11.6? Could you please provide the output of 'nvcc -V' to confirm your CUDA version?

2.Did you encounter this issue on datasets like mipnerf360, db, or tanks? Or was it on other datasets?

[Fregettia]

I'd like to add my own test results to this issue, as I was able to reproduce the illegal memory access error. My findings suggest this might be a hardware-specific issue related to the newer RTX 50-series GPUs, rather than just the CUDA version.

I ran a controlled test using the same Conda environment (miniforge3 and the environment.yml) but on different GPUs.

Here are my results:

Test 1:NVIDIA RTX 5090   CUDA Version: 12.8
Result: Reproduced the error: RuntimeError: CUDA error: an illegal memory access was encountered.

Test 2:NVIDIA RTX 4090   CUDA Version: 12.8         
Result: Works perfectly. No errors.

Test 3:NVIDIA RTX 4090   CUDA Version: 11.8
Result: Works perfectly. No errors.

The key takeaway is that with the exact same software stack (Miniforge, CUDA 12.8), the code fails on the 5090 but succeeds on the 4090.

This seems to support @Shiwei-Ren's theory that the issue might not be in the kernel logic itself, but perhaps in how memory is being handled or passed to the CUDA kernel on this newer GPU architecture. There might be a subtle change in the driver, compute capability, or memory management for the 50-series that is exposing a latent issue.

[BruceLz]

Same problem...
NVIDIA RTX 3090 CUDA Version: 12.8 python3.12 pytorch2.9

[Shiwei-Ren]

@BruceLz Hello! Please try to create a conda environment identical to the one in environment.yml.

[BruceLz]

@Shiwei-Ren I tried the same environment, but still encountered problems. Is it because I'm trying to input multiple perspectives at every moment in the fast-d3dgs?

[IamShubhamGupto]

Can confirm the same with 5070
python -m torch.utils.collect_env

Collecting environment information...
PyTorch version: 2.9.1+cu130
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)
Python platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 13.1.80
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5070
Nvidia driver version: 591.44
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.17.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.17.0
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               16
On-line CPU(s) list:                  0-15
Vendor ID:                            AuthenticAMD
Model name:                           AMD Ryzen 7 9700X 8-Core Processor
CPU family:                           26
Model:                                68
Thread(s) per core:                   2
Core(s) per socket:                   8
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             7585.55
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm avx512_vp2intersect
Virtualization:                       AMD-V
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            384 KiB (8 instances)
L1i cache:                            256 KiB (8 instances)
L2 cache:                             8 MiB (8 instances)
L3 cache:                             32 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-15
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.0.0.19
[pip3] nvidia-cuda-cupti==13.0.48
[pip3] nvidia-cuda-nvrtc==13.0.48
[pip3] nvidia-cuda-runtime==13.0.48
[pip3] nvidia-cudnn-cu13==9.13.0.50
[pip3] nvidia-cufft==12.0.0.15
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.3.29
[pip3] nvidia-cusparse==12.6.2.49
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-nccl-cu13==2.27.7
[pip3] nvidia-nvjitlink==13.0.39
[pip3] nvidia-nvtx==13.0.39
[pip3] torch==2.9.1+cu130
[pip3] torchvision==0.24.1+cu130
[pip3] triton==3.5.1
[conda] numpy                                 2.3.5            py311h2e04523_0       conda-forge
[conda] nvidia-cublas                         13.0.0.19        pypi_0                pypi
[conda] nvidia-cuda-cupti                     13.0.48          pypi_0                pypi
[conda] nvidia-cuda-nvrtc                     13.0.48          pypi_0                pypi
[conda] nvidia-cuda-runtime                   13.0.48          pypi_0                pypi
[conda] nvidia-cudnn-cu13                     9.13.0.50        pypi_0                pypi
[conda] nvidia-cufft                          12.0.0.15        pypi_0                pypi
[conda] nvidia-curand                         10.4.0.35        pypi_0                pypi
[conda] nvidia-cusolver                       12.0.3.29        pypi_0                pypi
[conda] nvidia-cusparse                       12.6.2.49        pypi_0                pypi
[conda] nvidia-cusparselt-cu13                0.8.0            pypi_0                pypi
[conda] nvidia-nccl-cu13                      2.27.7           pypi_0                pypi
[conda] nvidia-nvjitlink                      13.0.39          pypi_0                pypi
[conda] nvidia-nvtx                           13.0.39          pypi_0                pypi
[conda] torch                                 2.9.1+cu130      pypi_0                pypi
[conda] torchvision                           0.24.1+cu130     pypi_0                pypi
[conda] triton                                3.5.1            pypi_0                pypi

[Shiwei-Ren]

@Shiwei-Ren I tried the same environment, but still encountered problems. Is it because I'm trying to input multiple perspectives at every moment in the fast-d3dgs?

@BruceLz Hi! Sorry about the issue. I haven’t tested it on a 3090 GPU, so I’m not sure whether it’s related to the GPU you’re using.
And I’m also not quite sure what you mean by ‘‘I’m trying to input multiple perspectives at every moment in the fast-d3dgs.’’

[Shiwei-Ren]

@IamShubhamGupto Hi! Sorry about the issue. As Fregettia mentioned in an earlier comment, ‘this might be a hardware-specific issue related to the newer RTX 50-series GPUs.’ Could you test it on an RTX 4090 GPU to see if the problem persists?

[BruceLz]

@Shiwei-Ren I would like to confirm whether the submodules of FastGS, Fast-D3DGS and other branches are the same

[Shiwei-Ren]

@BruceLz In the current version, they are not the same. The difference between the submodules of FastGS and Fast-D3DGS lies in diff_gaussian_rasterization_fastgs: Fast-D3DGS returns a depth map, while FastGS does not. FastGS and Fast-DropGaussian share the same submodules.

[Demifiend217]

I also encountered similar issues while training on the 5090, possibly related to the new architecture of the 50-series GPUs. After trying to modify some code without success and consulting several LLMs, I came up with the following solution. Experts can refer to it if needed:

# Uninstall the old diff-g-r-fastgs package  
pip uninstall diff-gaussian-rasterization-fastgs -y  

cd submodules/diff-gaussian-rasterization_fastgs  
# Clean the cache
rm -rf build dist *.egg-info  

# Set the architecture list  
export TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0+PTX"  

# Recompile and install  
pip install .  

# Return to the root directory  and train
cd ../..

I've also written a custom dataset script:

#!/bin/bash
# FastGS Training Script (RTX 5090 Optimized)

# ===== 1. Basic Configuration =====
DATASET_NAME="yourDatasetName"
DATASET_PATH="yourDatasetPath"
GPU_ID=0

# Output path
OUTPUT_PATH="./output/$DATASET_NAME"

# ===== 2. FastGS Key Parameters =====
ITERATIONS=30000
DENSIFY_INTERVAL=1000
TEST_ITERATIONS="7000 30000"

# ===== 3. Environment & Performance Optimization =====
echo "======================================"
echo "Starting FastGS Training (RTX 5090 Mode)"
echo "======================================"

# [Key Fix] Compatibility settings for RTX 5090 (Arch 12.0)
# Force PyTorch to recognize PTX code for newer architectures
export TORCH_CUDA_ARCH_LIST="8.0;8.6;8.9;9.0+PTX"

# Dynamically set library path
export LD_LIBRARY_PATH=$(python -c "import torch; import os; print(os.path.dirname(torch.__file__))")/lib:$LD_LIBRARY_PATH

# ===== 4. Dataset Check. Note the sparse path - this is where my dataset is located.=====
if [ ! -d "$DATASET_PATH/sparse/0" ] && [ -d "$DATASET_PATH/colmap/sparse/0" ]; then
    echo " Creating symbolic link: sparse -> colmap/sparse"
    ln -sf "$DATASET_PATH/colmap/sparse" "$DATASET_PATH/sparse"
fi

# ===== 5. Start Training Command =====
echo "Training on GPU $GPU_ID..."

CUDA_VISIBLE_DEVICES=$GPU_ID python train.py \
    -s "$DATASET_PATH" \
    -m "$OUTPUT_PATH" \
    -r 1 \
    --eval \
    --position_lr_init 0.000016 \
    --scaling_lr 0.002

# ===== 6. Result Feedback =====
if [ $? -eq 0 ]; then
    echo "✅ Training completed successfully!"
    echo "Results saved to: $OUTPUT_PATH"
else
    echo "❌ Training failed. Please check the error messages."
    exit 1
fi

[Shiwei-Ren]

@Demifiend217 Hello, thank you very much for your enthusiastic help!

[Demifiend217]

@Demifiend217 Hello, thank you very much for your enthusiastic help!

(∠・ω< )⌒☆

[turandai]

same on RTX 5090 and CUDA 12.8, tried reinstalling but still got the error. Do you guys know what is the problem? Thx!