restart training from saved model

[ciphercharly]

Error while trying to restart the training of a model from a previously saved checkpoint

tried to restart a training run from a saved 10000 epochs checkpoint, configuration otherwise identical

got the error in the attached log, not sure if I did something wrong or there's an issue with the code

restart-error_stdout_train_30000epochs_gmi_unet_lw1rw1_learnrate1e-5.log

[mathpluscode]

@ciphercharly it might be related to the gpu strategy. How many GPUs are you using?

I propose to try some simple fixes:

1. in https://github.com/DeepRegNet/DeepReg/blob/main/deepreg/train.py#L162 add indent to model.compile(optimizer=optimizer) so that it's inside the scope of the strategy

2. If this does not work, try using at most 1 GPU, and remove the strategy.

We will investigate it further later. Also be aware that, in saved checkpoint, we DO NOT have the status of the optimizer, and we are planning to fix it, as mentioned in #469. Which means that you might observe some performance drop after loading the checkpoint.

[ciphercharly]

thanks, yes I was using 2 GPUs, will try what you suggest

[mathpluscode]

Hi @ciphercharly, are you still facing problems on this?

Regarding the error you got, it might be linked to the inconsistency of the training strategy, normally we should use mirror strategy in case of multiple GPUs. We've updated it recently, check https://github.com/DeepRegNet/DeepReg/blob/main/deepreg/train.py#L153

Regarding the warnings, it's normal, as the variables linked to the optimizer will be loaded later, please check more details & discussions here:

tensorflow/tensorflow#33150 (comment)

[ciphercharly]

I did not try again, but yes, I think that this is related to using more than 1 GPU for the training, and perhaps gpu_allow_growth = Truetoo? anyway you can close the issue I think, this is not something that can be fixed upstream...

[mathpluscode]

I did not try again, but yes, I think that this is related to using more than 1 GPU for the training, and perhaps gpu_allow_growth = Truetoo? anyway you can close the issue I think, this is not something that can be fixed upstream...

Sure. We thought there was a bug linked to model loading, but recently we think the loading is working as expected (means the optimizer status is in checkpoints and will be loaded).

I'm closing this issue then. Thx!