Closed
Description
Describe the bug
======================================================================
FAIL: test_train_quick_5 (tests.test_gaussian_filter.TestGaussianFilterBackprop)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/parameterized/parameterized.py", line 533, in standalone_func
return func(*(a + p.args), **p.kwargs)
File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_gaussian_filter.py", line 89, in test_train_quick
self.code_to_run(input_args)
File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_gaussian_filter.py", line 84, in code_to_run
s.cpu().item(), gt.cpu() if len(gt.shape) == 0 else gt[idx].cpu().item(), rtol=1e-2
File "/opt/conda/lib/python3.6/site-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
verbose=verbose, header=header, equal_nan=equal_nan)
File "/opt/conda/lib/python3.6/site-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.01, atol=0
Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.18121517
Max relative difference: 0.36243033
x: array(0.681215)
y: array(0.5, dtype=float32)
----------------------------------------------------------------------
To Reproduce
this error happens once in ~50 runs
saw this error with the following env and the latest codebase c39f6fa
Numpy version: 1.18.5
Pytorch version: 1.6.0a0+9907a3e
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: None
Optional dependencies:
Pytorch Ignite version: 0.4.5
Nibabel version: 3.2.1
scikit-image version: 0.15.0
Pillow version: 8.3.1
Tensorboard version: 1.15.0+nv
gdown version: 3.13.0
TorchVision version: 0.7.0a0
tqdm version: 4.62.2
lmdb version: 0.98
psutil version: 5.7.0
pandas version: 0.24.2
einops version: 0.3.0
PT16+CUDA110
Expected behavior
no error