Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Mxnet Training deterministic #13829

Closed
Closed
Mxnet Training deterministic#13829
Labels
BackendIssues related to the backend of MXNetBug

Description

@jiangzhengkai
jiangzhengkai
opened on Jan 10, 2019

Hi, I'm using mxnet for research experiments. However, the experiments results is not deterministic.

I'm using np, random, mxnet.nd, so i fix the seed for all of them. besides, i add os.environ['MXNET_ENFORCE_DETERMINISM'] = '1' and disable cudnn option in convolution option.

here is the log for different two times
image
image

It starts different when 10 iters.

here is the debugging process.
I first confirm that the load data is totally same. when disables backward process, the forward loss is totally same. So the problem is in the backward process.

there is any random process for bachward?

anyone helps me?

Thanks

Activity

  1. zachgk commented on Jan 10, 2019

    @zachgk
    zachgk
    on Jan 10, 2019
    Contributor

    Thank you for submitting the issue! I'm labeling it so the MXNet community members can help resolve it.
    @mxnet-label-bot add [Backend, Bug]

  2. marcoabreu
    added
    BackendIssues related to the backend of MXNet
    Bug
    on Jan 10, 2019

  3. jiangzhengkai commented on Jan 11, 2019

    @jiangzhengkai
    jiangzhengkai
    on Jan 11, 2019
    Author

    The reason may be the atomic_add, it seems that it can not be deterministic. Any other reason?

  4. zachgk commented on Jan 11, 2019

    @zachgk
    zachgk
    on Jan 11, 2019
    Contributor

    Hi @jiangzhengkai, can you provide some information about your environment and the code that is not behaving deterministically (or a minimal example?

    Environment info (Required)

    What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

    Package used (Python/R/Scala/Julia):
    (I'm using ...)

    For Scala user, please provide:

    1. Java version: (java -version)
    2. Maven version: (mvn -version)
    3. Scala runtime if applicable: (scala -version)

    For R user, please provide R sessionInfo():

    Build info (Required if built from source)

    Compiler (gcc/clang/mingw/visual studio):

    MXNet commit hash:
    (Paste the output of git rev-parse HEAD here.)

    Build config:
    (Paste the content of config.mk, or the build command.)

    Minimum reproducible example

    (If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

  5. apeforest commented on Feb 24, 2019

    @apeforest
    apeforest
    on Feb 24, 2019
    Contributor

    @jiangzhengkai There is a default cuDNN autotune switch controlled by an environment variable: MXNET_CUDNN_AUTOTUNE_DEFAULT. You will need to turn it off to disable the cuDNN algorithm exploration. Otherwise, the conv layer may still generate non-deterministic result due to different cuDNN algorithm used.

    Please set MXNET_CUDNN_AUTOTUNE_DEFAULT=0.

  6. jiangzhengkai
    closed this as completedon Mar 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BackendIssues related to the backend of MXNetBug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions