Repository navigation
[release/2.13] Use fp32 tolerance for fp32 grads in test_batchnorm_nhwc_cpu (#202210) - #3932
Open
amd-maradosa wants to merge 1 commit into
Open
amd-maradosa wants to merge 1 commit into
amd-maradosa wants to merge 1 commit into
Conversation
…#202210) Supported by @jeffdaily `test_batchnorm_nhwc_cpu` fails with 1–2 OpenMP threads: ``` BatchNorm3d (4, 8, 2, 10, 10), float16 input, float32 params (mixed_dtype=True) self.assertEqual(bn.weight.grad, ref_bn.weight.grad, atol=precision, rtol=precision) Mismatched elements: 1 / 8 (12.5%) Greatest absolute difference: 2.3365020751953125e-05 at index (0,) (up to 1e-05 allowed) ``` The test compares the channels_last and contiguous CPU batch norm backward kernels. Both reduce `weight`/`bias` grads over N×spatial elements in fp32, but in different orders: [channels_last](https://github.com/pytorch/pytorch/blob/0564ffec41075bbb70b8113eb79e27fc8668decd/aten/src/ATen/native/cpu/batch_norm_kernel.cpp#L1204) accumulates each channel serially per thread chunk, and [contiguous](https://github.com/pytorch/pytorch/blob/0564ffec41075bbb70b8113eb79e27fc8668decd/aten/src/ATen/native/cpu/batch_norm_kernel.cpp#L1055) accumulates across vector lanes. Against a float64 reference, both are well within fp32 summation error; they just disagree with each other by more than the fp32 default `atol=1e-5`. The 3D fp32 cases already pass `precision=1e-4`, and pytorch#189180 applied it to `bias.grad` as well. But the 2D loop passes no precision, and the 3D float16 entry is `None`. So with `mixed_dtype=True`, the fp32 params' grads are held to the fp32 default. This change applies `1e-4` whenever the params are fp32 and no precision was given. It surfaced after pytorch#175988, which changed CPU fp16/bf16 `randn`/`uniform_`, so the test's fixed seed draws different inputs. On the failing inputs, the kernels give bit-identical grads in 2.5.1 and on current main. Upstream CI doesn't hit it because [`.ci/pytorch/test.sh` floors `OMP_NUM_THREADS` at 4](https://github.com/pytorch/pytorch/blob/0564ffec41075bbb70b8113eb79e27fc8668decd/.ci/pytorch/test.sh#L213) on ARC runners, naming this test. Test plan: ``` OMP_NUM_THREADS=1 python test/test_nn.py -k test_batchnorm_nhwc_cpu # fails before, passes after python test/test_nn.py -k test_batchnorm_nhwc_cpu ``` Pull Request resolved: pytorch#202210 Approved by: https://github.com/jeffdaily (cherry picked from commit 64ef93a)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of upstream pytorch#202210 (commit 64ef93a) into release/2.13.
Why:
test_nn.py::TestNN::test_batchnorm_nhwc_cpuexists on this branch and fails on some GPUs (e.g. gfx1103) because channels_last and contiguous kernels reduce fp32 weight/bias grads in different orders, exceeding the default tolerance.Applied cleanly; changed lines are identical to upstream.