Repository navigation
Undeterministic segmentation on with different instances of model #91
Description
I try to create cellpose plugin to my tool https://github.com/4DNucleome/PartSeg_cellpose I found that In some scenario different instances of cellpose model returns different predictions. I try to prepare code which deterministic reproduce this, but not always next instance of model fail. I need to recreate it again in third cell to got this result.
But in my plug-in this happens always.
It is possible to fix it?
img02.png is from your example notebook.
Activity
marius10p commentedon Jul 3, 2020
Sorry, I'm not sure I understand. There should not be any randomness in that function call (right @carsen-stringer ?).
Czaki commentedon Jul 3, 2020
As I expect there should be no randomness. But I found that I got different result in notebook and in gui (my one). Then in test I found that I can reproduce this in notebook.
Czaki commentedon Jul 13, 2020
Any information?
carsen-stringer commentedon Aug 30, 2020
In the notebook you are running with diameter=None so it is estimating the diameter for you. In the GUI are you also using the "calibration" button to get the diameter?
I don't understand how it could be different in the same notebook with the same code, can you provide more information if you are still having this issue?
Czaki commentedon Sep 1, 2020
I also do not understand this. But I notice that problematic runs take much longer time (9 seconds against 90-160 seconds).
Now it happens time to time when using multiple instances of model from notebook. It does not depend if I set
diametertoNoneor0or set positive value30.I do not have any idea how debug this. But maybe You have idea why wrong calculation in current version takes longer time?
Czaki commentedon Sep 2, 2020
I check this on second computer (both ubuntu) and same problem (different segmentation form different instance of model for same file) shows.
chrisroat commentedon Dec 1, 2020
I have seen similar non-determinism in some unit tests I have, which makes them flaky. It's often on just small, isolated groups of pixels near segment boundaries. From apache/mxnet#13829 (comment), it seems that mxnet autotuning can be non-deterministic.
Setting the following during my tests has fixed my problem:
MXNET_CUDNN_AUTOTUNE_DEFAULT=0
carsen-stringer commentedon Apr 28, 2021
thanks @chrisroat , closing this for now, perhaps the torch version doesn't have this problem