You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been having this problem and I decreased the learning rate as per various discussions on SO and that seemed to work. After a while I tried increasing it by 0.01 and started getting NaN's again. I've tried restoring the checkpoint and re-running with the lower learning rate, but I'm still seeing NaN. Does this mean my checkpoint is useless?
@hate5six I'm still thinking about an appropriate solution. Each dist is a tensor shape (batch_size, extended_vsize), so I am not sure if dist != 0 will work. Also, I want the log_dists length to be the same as final_dists
The below change worked for me. Add the below code to def _calc_final_dist(self, vocab_dists, attn_dists). Info is in comments.
# OOV part of vocab is max_art_oov long. Not all the sequences in a batch will have max_art_oov tokens.
# That will cause some entries to be 0 in the distribution, which will result in NaN when calulating log_dists
# Add a very small number to prevent that.
def add_epsilon(dist, epsilon=sys.float_info.epsilon):
epsilon_mask = tf.ones_like(dist) * epsilon
return dist + epsilon_mask
final_dists = [add_epsilon(dist) for dist in final_dists]
return final_dists
Reacted by Mike Tian-Jian Jiang a.k.a. Chiang, Tien Chien, sunny hate5six, Little_Savior, Pedro Paulo Balage, Foleosy, Nitin Kishore Sai Samala, Leon Zhang, Daniel Lindsäth, Han Guo, Shujian2015 and 7 more
@lizaigaoge550 did this work for you? final_dists = [tf.clip_by_value(dist,1e-10,1.) for dist in final_dists]
Could you let me know which line you put this at?
Many thanks
I've looked further into this and still don't understand where the NaNs are coming from. I changed the code to detect when a NaN occurs, then dump the attention distribution, vocabulary distribution, final distribution and some other stuff to file.
Looking at the dump file, I find that attn_dists and vocab_dists are both all NaN, on every decoder step, for every example in the batch and across the encoder timesteps (for attn_dists) and across the vocabulary (for vocab_dists). Consequently final_dists contains NaNs, therefore log_dists does and the final loss is NaN too.
This is different than what I was expecting. I was expecting to find zero values in final_dists and therefore NaNs in log_dists, but it seems that the problem occurs earlier, somehow causing NaNs everywhere in attn_dists and vocab_dists.
Given this information, I don't see why adding epsilon to final_dists works as a solution, because if attn_dists and vocab_dists contain NaNs, it shouldn't work.
Thanks for your update! The strange thing is, it does work @abisee. By adding epsilon I have not encountered NaNs again. But it does affect the training convergence a bit. By how much, I dunno.
@bluemonk482 I tried adding epsilon as mentioned in the previous discussions above. I still get the NaN loss after one day of training. Can you tell me what was the learning rate you used for your experiments? I tried using 0.1.
@shahbazsyed Well I used as big a learning rate as 0.15, and had no NaN error after adding epsilon. How did you add the epsilon? Like this (as @Rahul-Iisc has suggested):
def add_epsilon(dist, epsilon=sys.float_info.epsilon):
epsilon_mask = tf.ones_like(dist) * epsilon
return dist + epsilon_mask
final_dists = [add_epsilon(dist) for dist in final_dists]
@bluemonk482 Yes, I added epsilon just as @Rahul-Iisc suggested. I tried with a higher learning rate of 0.2 , but am still getting the same error. Did you change any other parameters ? Mine are the following :
@shahbazsyed I see you have changed the default parameter setting. I used 128 for emb_dim. And why did you increase your learning rate to 0.2 when you had NaN error at 0.1? You can try a smaller learning rate.
@bluemonk482 Still got the NaN with lower learning rate after 42000 steps. I tried decoding with the model trained so far, it was just a bunch of UNKs.
I encountered the same problem and found that it is because of the zeros examples in the batch. If the training corpus contains $k * batch_size - a$ where $a<k$ examples/samples there would be $k - a$ zero examples. (single pass)
There might be another cause for NANs. There are some stories in the dataset which only contain highlights and not the article itself. That causes a sequence of [PAD] tokens with attention distribution of all zeros which causes the probability of [PAD] to be zero in final distribution (because of scatter function). And since the only target for that sequence is [PAD] with zero probability it will generate NAN. I removed those articles and it seems to be working.
Reacted by Bryan McCann, Leon Zhang, Shujian2015, Ruhong, Ruhong, Sabyasachi18, AngelaSunny and NhanToThanh
I also encounter NaNs when pgen is 1 or 0. This means that either vocab_dists or attn_dists will be 0-filled and final_dists will have 0s in it (for words only in vocabulary or only in input). This will cause the loss to be inf (and the backprop will put NaNs in layers) if a word in reference has a 0 probability.
I will add espilon if pgen is 0 and remove epsilon if pgen is 1.
But as @tianjianjiang, i am concerned that this will bias the model.
when I add this model to transformer, I get nan, when I train model, I try many ways to solve this problem, but it still nan, if anyone interested in it, we can discuss.
when I add this model to transformer, I get nan, when I train model, I try many ways to solve this problem, but it still nan, if anyone interested in it, we can discuss.
Please I'm trying your code of pointer generator built on LCSTS dataset. I changed the dataset with amazon reviews for my academic work purpose; I'm getting all UNKs in output..Please help if you have solved it
Activity
abisee commentedon May 5, 2017
@StevenLOL I see this happen sometimes too -- seems to be a very common problem with Tensorflow training in general.
hate5six commentedon May 7, 2017
I've been having this problem and I decreased the learning rate as per various discussions on SO and that seemed to work. After a while I tried increasing it by 0.01 and started getting NaN's again. I've tried restoring the checkpoint and re-running with the lower learning rate, but I'm still seeing NaN. Does this mean my checkpoint is useless?
aar-gee commentedon May 8, 2017
I am also getting NaN. Found out the culprit to be
line #227:
log_dists = [tf.log(dist) for dist in final_dists]in model.py
hate5six commentedon May 8, 2017
@Rahul-Iisc Is your workaround to filter out cases where
dist == 0, for example:log_dists = [tf.log(dist) for dist in final_dists if dist != 0]?aar-gee commentedon May 8, 2017
@hate5six I'm still thinking about an appropriate solution. Each dist is a tensor shape (batch_size, extended_vsize), so I am not sure if
dist != 0will work. Also, I want thelog_distslength to be the same asfinal_distsaar-gee commentedon May 8, 2017
Trying to convert NaNs to 0 for now. Need to further look for the cause of 0s coming up in the distribution. @abisee @StevenLOL reopen the issue?
UPDATE: This didn't work. Loss ended up 0 instead of NaN.
aar-gee commentedon May 9, 2017
The below change worked for me. Add the below code to
def _calc_final_dist(self, vocab_dists, attn_dists). Info is in comments.lizaigaoge550 commentedon May 9, 2017
final_dists = [tf.clip_by_value(dist,1e-10,1.) for dist in final_dists]
jamesposhtiger commentedon May 25, 2017
@lizaigaoge550 did this work for you?
final_dists = [tf.clip_by_value(dist,1e-10,1.) for dist in final_dists]Could you let me know which line you put this at?
Many thanks
lizaigaoge550 commentedon May 27, 2017
@jamesposhtiger
after finishing the final_dists
Fixes issue abisee#4 for NaN loss
Fixes issue abisee#4 for NaN loss
apoorv001 commentedon Jun 1, 2017
can we restore the already trained model after it starts getting NaN ?
7 remaining items
abisee commentedon Aug 5, 2017
I've looked further into this and still don't understand where the NaNs are coming from. I changed the code to detect when a NaN occurs, then dump the attention distribution, vocabulary distribution, final distribution and some other stuff to file.
Looking at the dump file, I find that attn_dists and vocab_dists are both all NaN, on every decoder step, for every example in the batch and across the encoder timesteps (for
attn_dists) and across the vocabulary (forvocab_dists). Consequentlyfinal_distscontains NaNs, thereforelog_distsdoes and the final loss is NaN too.This is different than what I was expecting. I was expecting to find zero values in
final_distsand therefore NaNs inlog_dists, but it seems that the problem occurs earlier, somehow causing NaNs everywhere inattn_distsandvocab_dists.Given this information, I don't see why adding epsilon to
final_distsworks as a solution, because ifattn_distsandvocab_distscontain NaNs, it shouldn't work.bwang482 commentedon Aug 7, 2017
Thanks for your update! The strange thing is, it does work @abisee. By adding epsilon I have not encountered NaNs again. But it does affect the training convergence a bit. By how much, I dunno.
shahbazsyed commentedon Aug 10, 2017
@bluemonk482 I tried adding epsilon as mentioned in the previous discussions above. I still get the NaN loss after one day of training. Can you tell me what was the learning rate you used for your experiments? I tried using 0.1.
bwang482 commentedon Aug 10, 2017
@shahbazsyed Well I used as big a learning rate as 0.15, and had no NaN error after adding epsilon. How did you add the epsilon? Like this (as @Rahul-Iisc has suggested):
shahbazsyed commentedon Aug 12, 2017
@bluemonk482 Yes, I added epsilon just as @Rahul-Iisc suggested. I tried with a higher learning rate of 0.2 , but am still getting the same error. Did you change any other parameters ? Mine are the following :
bwang482 commentedon Aug 12, 2017
@shahbazsyed I see you have changed the default parameter setting. I used 128 for
emb_dim. And why did you increase your learning rate to 0.2 when you had NaN error at 0.1? You can try a smaller learning rate.shahbazsyed commentedon Aug 14, 2017
@bluemonk482 Still got the NaN with lower learning rate after 42000 steps. I tried decoding with the model trained so far, it was just a bunch of UNKs.
aar-gee commentedon Aug 14, 2017
Use tfdbg.
abisee commentedon Aug 16, 2017
Hello everyone, and thanks for your patience. We've made a few changes that help with the NaN issue.
The README also contains a section about NaNs.
Edit: We've also provided a pretrained model, which we trained using this new version of the code. See the README.
eduOS commentedon Jan 11, 2018
I encountered the same problem and found that it is because of the zeros examples in the batch. If the training corpus contains$k * batch_size - a$ where $a<k$ examples/samples there would be $k - a$ zero examples. (single pass)
I just add a line next to line 329 in batcher:
May this help a little.
mahnazkoupaee commentedon Feb 1, 2018
There might be another cause for NANs. There are some stories in the dataset which only contain highlights and not the article itself. That causes a sequence of [PAD] tokens with attention distribution of all zeros which causes the probability of [PAD] to be zero in final distribution (because of scatter function). And since the only target for that sequence is [PAD] with zero probability it will generate NAN. I removed those articles and it seems to be working.
ygorg commentedon Feb 6, 2019
I also encounter NaNs when pgen is 1 or 0. This means that either
vocab_distsorattn_distswill be 0-filled andfinal_distswill have 0s in it (for words only in vocabulary or only in input). This will cause the loss to beinf(and the backprop will put NaNs in layers) if a word in reference has a 0 probability.I will add espilon if pgen is 0 and remove epsilon if pgen is 1.
But as @tianjianjiang, i am concerned that this will bias the model.
xiongma commentedon May 8, 2019
when I add this model to transformer, I get nan, when I train model, I try many ways to solve this problem, but it still nan, if anyone interested in it, we can discuss.
GaneshDoosa commentedon Mar 29, 2020
Please I'm trying your code of pointer generator built on LCSTS dataset. I changed the dataset with amazon reviews for my academic work purpose; I'm getting all UNKs in output..Please help if you have solved it
tohidarehman1988 commentedon Dec 12, 2020
can you please help me how you resolved this? how you print this loss graph?