🐛 Describe the bug
Environment
- macOS 26.5 (Apple Silicon), executorch
main @ d614f9d, FVPs via FVPs-on-Mac Docker wrappers
Observed
Two distinct launch failures both produced a success report:
Case 1 — wrong-architecture binary exec'd:
.../Linux64_armv8l_GCC-9.3/FVP_Corstone_SSE-300_Ethos-U55: cannot execute binary file
[backends/arm/scripts/run_fvp.sh] Simulation complete, 0
Checking for a etdump in log
Checking for problems in log:
No problems found!
Case 2 — docker unavailable inside the wrapper:
docker: command not found
[backends/arm/scripts/run_fvp.sh] Simulation complete, 0
...
No problems found!
In both cases the simulator never started and the ELF never ran, but the script exits 0 with an all-clear. In CI or any scripted use this is a silent false green.
Root cause
The FVP invocation ends with ... | sed ... | tee ${log_file} || true, so the launch failure status is discarded; the subsequent echo "Simulation complete, $?" reports the status of that pipeline (forced 0). The "problems" check then greps the log for known error patterns — an FVP that never launched produces none of them, so an empty/near-empty log passes.
Suggested fix
- Capture the FVP's real exit status (e.g.
PIPESTATUS[0] / set -o pipefail around the launch) and fail on nonzero, and
- Treat the absence of positive evidence as failure: require the executor runner banner (or an explicit success marker like
Model run: / Program complete) in the log before declaring success, rather than only grepping for known-bad patterns.
Versions
PyTorch version: 2.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 26.5.2 (arm64)
GCC version: Could not collect
Clang version: 21.0.0 (clang-2100.1.1.101)
CMake version: version 3.31.10
Libc version: N/A
Python version: 3.12.14 (main, Aug 12 2026, 13:57:54) [Clang 21.0.0 (clang-2100.1.1.101)] (64-bit runtime)
Python platform: macOS-26.5.2-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A
CPU:
Apple M4 Pro
Versions of relevant libraries:
[pip3] executorch==1.5.0+d614f9d
[pip3] numpy==2.5.2
[pip3] pytorch_tokenizers==1.3.0
[pip3] torch==2.13.0
[pip3] torchao==0.18.0.dev20260729+cpu
[pip3] torchaudio==2.11.0
[pip3] torchdata==0.11.0+cpu
[pip3] torchsr==1.0.4
[pip3] torchtune==0.0.0
[pip3] torchvision==0.28.0
[conda] Could not collect
🐛 Describe the bug
Environment
main@ d614f9d, FVPs via FVPs-on-Mac Docker wrappersObserved
Two distinct launch failures both produced a success report:
Case 1 — wrong-architecture binary exec'd:
Case 2 — docker unavailable inside the wrapper:
In both cases the simulator never started and the ELF never ran, but the script exits 0 with an all-clear. In CI or any scripted use this is a silent false green.
Root cause
The FVP invocation ends with
... | sed ... | tee ${log_file} || true, so the launch failure status is discarded; the subsequentecho "Simulation complete, $?"reports the status of that pipeline (forced 0). The "problems" check then greps the log for known error patterns — an FVP that never launched produces none of them, so an empty/near-empty log passes.Suggested fix
PIPESTATUS[0]/set -o pipefailaround the launch) and fail on nonzero, andModel run:/Program complete) in the log before declaring success, rather than only grepping for known-bad patterns.Versions
PyTorch version: 2.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 26.5.2 (arm64)
GCC version: Could not collect
Clang version: 21.0.0 (clang-2100.1.1.101)
CMake version: version 3.31.10
Libc version: N/A
Python version: 3.12.14 (main, Aug 12 2026, 13:57:54) [Clang 21.0.0 (clang-2100.1.1.101)] (64-bit runtime)
Python platform: macOS-26.5.2-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A
CPU:
Apple M4 Pro
Versions of relevant libraries:
[pip3] executorch==1.5.0+d614f9d
[pip3] numpy==2.5.2
[pip3] pytorch_tokenizers==1.3.0
[pip3] torch==2.13.0
[pip3] torchao==0.18.0.dev20260729+cpu
[pip3] torchaudio==2.11.0
[pip3] torchdata==0.11.0+cpu
[pip3] torchsr==1.0.4
[pip3] torchtune==0.0.0
[pip3] torchvision==0.28.0
[conda] Could not collect