Repository navigation
Problems with CoT, Coconut, SIM-CoT training #6
Description
Hi, thank you for an interesting and insightful work.
I tried to train GPT2 with your code for SIM-CoT reasoning but during training I got strange model behavior.
Firstly, as I understood from README and training configs in order to train a model for SIM-CoT you need 3 stages:
- Train in CoT regime using SFT
- Train in Coconut regime
- Train in SIM-CoT regime
I tried to change as little as possible in your code and configs when I ran my experiments. I just changed precision to bf16 and added tensorboard support. So when I trained model in CoT regime the validation loss wouldn't go below 0.6 and final validation accuracy was just 8 %. Previously, I had already conducted training of GPT2 for CoT using my own code and the model reached validation loss of 0.1 and accuracy of 40 %. I didn't manage to find the problem with your code and the reason why I couldn't reach similar metrics with it. Maybe you had similar issues?
Secondly, when I continued training in Coconut regime the loss behaviour was also strange. With each latent schedule iteration the validation loss would only increase. It reached minimal value early in the training process and after that increased to the absolute value of 1.2 by the end of training. The accuracy bahave correspondingly. It reached the value of 23 % and after that started to decrease up to 16 %. Maybe you could also clarify this beahavior?
I also launched SIM-CoT training (from standard GPT2 checkpoint) and got the same validation loss behaviour as in the case of Coconut training. Only the accuracy was just 4 %.
Thank you in advance for response!
Activity
Wiselnn570 commentedon Dec 2, 2025
Have you solved this issue?
What is the model’s performance on the test set after training? You can try running it with the default experimental settings first to check whether the issue is caused by the bf16 precision.
Oleg20502 commentedon Jan 21, 2026
It seems the problem was indeed with bf16 precision. I switched to fp32 precision and managed to train GPT2 for CoT using SFT just fine reaching accuracy 0.44. But I have issues training using Coconut and SIM-CoT methods.
I want to train GPT2 using SIM-CoT method. I have very similar questions as in the threads #4 and #7. However I didn't quite understand your explanations. From those discussions it seemed to me that to obtain the results as in the paper some hyperparameters should be changed in the configs. If that is true then what hyperparameters should be changed in the repository's configs?
Also those discussions put doubt on my understanding how you trained the model. I thought that first I need to train GPT2 for CoT in SFT method. Then start training with Coconut method using curriculum learning. And only after that start training GPT2 with SIM-CoT method also using curriculum learning. Is that true? If not or if I forgot some details could you please clarify the exact training procedure to obtain the results for SIM-CoT + Coconut approach?
Wiselnn570 commentedon Jan 27, 2026
The YAML settings in the repository are correct. Only when training on LLaMA 1B based on Coconut did we adjust the training strategy to eliminate formatting effects. The specific changes can be found in Issue #7, where the corresponding training results are also discussed.
Oleg20502 commentedon Feb 1, 2026
I train GPT2 model. I used your scripts and your configs with all the main settings preserved. The only difference can be in the batch size. It was 96 for Coconut (6 gpus with bs 8 on each with gradient accumulation 2) and 48 for SIM-CoT (6 gpus with bs 8 on each with gradient accumulation 1).
I tried to replicate Figure 3 from your paper. Specifically SIM-CoT accuracy for different number of latents on GSM8k-Aug. For this I used your eval scripts. However, I didn't manage to get something similar to your results. Below is what I get
To obtain this curves, I used different checkpoints (saved at the end of different curriculum learning steps) as indicated in the legend. For each checkpoint I used various number of latents controled by
resumekeyword. You can see that the max accuracy I got is 0.41 but when only 1 latent is used. Your curve has a pick at 5 latents with accuracy 0.448. In my experiments the best accuracy I got for 5 latents is about 0.36.In your paper I couldn't find important detailes on how you obtain these results for SIM-CoT. What chekpoint number did you use, from what curriculum learning step? And did you use different checkpoints for different number of latents? Maybe these are the details that allow to replicate your results?
Also, in #4 here you write
What do you mean by failed and normal checkpoints?
Wiselnn570 commentedon Feb 2, 2026
Can you reproduce your Coconut baseline results? Please verify them first using a total batch size of 128, as specified in the paper.
krkawzq commentedon Feb 7, 2026
Hi @Oleg20502
I encountered the same issue regarding low accuracy and strange loss behavior, and I believe I have found the root cause.
It seems the problem comes from the training dataset size. If we use the standard GSM8k dataset, it only contains about 7.5k training examples, which is insufficient for reproducing the paper's results.
I switched the training dataset to zen-E/GSM8k-Aug (which contains roughly 386k training samples), and after doing so, the model's accuracy returned to normal levels. I highly recommend checking if your training config is pointing to the Augmented dataset rather than the original one.
Hope this helps!