Activity

1. Wiselnn570 commented on Dec 29, 2025

   

   Wiselnn570

   on Dec 29, 2025

   Last edited by Wiselnn570

   Collaborator

   More actions

   Hello, thank you for your interest in SIM-CoT.

   Regarding the experiments on Llama-1B reported in Table 2, due to

   1. the catastrophic forgetting issue when the coconut framework is generalized to larger models (https://openreview.net/forum?id=tG4SgayTtk), and
   2. our experimental observation that the training format has a stronger impact on larger models,

   we therefore divided the training into three stages.

   Experimental Settings

   • c_thought: 6
   • epochs_per_stage: 1
   • max_latent_stage: 1

   Training Procedure

   1. Stage 1: llm-only Training
      In this stage, all defined implicit tokens participate in training from the beginning, and the explicit reasoning steps are gradually removed. By the end of this stage, all explicit steps are completely removed, and the model performance is approximately ~39.

   2. Stage 2: Decoder-only Training
      In this stage, only the decoder is trained. Since llm is frozen, the model performance remains unchanged.

   3. Stage 3: Joint Training of LLM and Decoder
      In this stage, llm and the decoder are trained jointly. After this stage, the model performance reaches approximately ~42.

   Additional Recommendation

   For larger models (1B, 3B, and 8B), we recommend using SIM-CoT based on CODI, which achieves better performance.
2. neerajgangwar commented on Dec 29, 2025

   

   neerajgangwar

   on Dec 29, 2025

   Author

   More actions

   Thank you for your response. I have follow-up questions based on your response.

   You mentioned max_latent_stage to be 1. Does this mean there is only one latent stage, as opposed to 3 stages used in the original Coconut paper or 5 in your paper with GPT-2?

   Additionally, Stage 1 under the above-mentioned training procedure is the Coconut baseline training. If it achieves 39%, shouldn't Table 2 mention 39% instead of 33.2% for the Coconut baseline (without SIM-CoT)?
3. Wiselnn570 commented on Dec 29, 2025

   

   Wiselnn570

   on Dec 29, 2025

   Collaborator

   More actions

   You mentioned max_latent_stage to be 1. Does this mean there is only one latent stage, as opposed to 3 stages used in the original Coconut paper or 5 in your paper with GPT-2?

   Yes.

   Additionally, Stage 1 under the above-mentioned training procedure is the Coconut baseline training. If it achieves 39%, shouldn't Table 2 mention 39% instead of 33.2% for the Coconut baseline (without SIM-CoT)?

   Our first stage is not Coconut. Under Coconut’s training strategy (with c_thought = 2 and max_latent_stage = 3), the number of implicit tokens is gradually increased via curriculum training, reaching 2 × 3 = 6 total implicit tokens. In contrast, we use six implicit tokens throughout the entire training process from the very beginning.

   Moreover, Coconut replaces one explicit reasoning step each time a new group of implicit tokens is appended. Our method does not perform this simultaneous add-and-remove operation. Instead, we only gradually remove explicit reasoning steps, while keeping the number of implicit reasoning tokens fixed.
4. zaiquanyang commented on Jan 23, 2026

   

   zaiquanyang

   on Jan 23, 2026

   More actions

   Hello, thank you for your interest in SIM-CoT.

   Regarding the experiments on Llama-1B reported in Table 2, due to

   1. the catastrophic forgetting issue when the coconut framework is generalized to larger models (https://openreview.net/forum?id=tG4SgayTtk), and
   2. our experimental observation that the training format has a stronger impact on larger models,

   we therefore divided the training into three stages.

   Experimental Settings

   • c_thought: 6
   • epochs_per_stage: 1
   • max_latent_stage: 1

   Training Procedure

   1. Stage 1: llm-only Training
      In this stage, all defined implicit tokens participate in training from the beginning, and the explicit reasoning steps are gradually removed. By the end of this stage, all explicit steps are completely removed, and the model performance is approximately ~39.
   2. Stage 2: Decoder-only Training
      In this stage, only the decoder is trained. Since llm is frozen, the model performance remains unchanged.
   3. Stage 3: Joint Training of LLM and Decoder
      In this stage, llm and the decoder are trained jointly. After this stage, the model performance reaches approximately ~42.

   Additional Recommendation

   For larger models (1B, 3B, and 8B), we recommend using SIM-CoT based on CODI, which achieves better performance.

   hi, you mentioned that three stages are needed for reproducing Table 2. Could you give the references about the epochs for different stages.
5. Wiselnn570 commented on Jan 27, 2026

   

   Wiselnn570

   on Jan 27, 2026

   Collaborator

   More actions

   @zaiquanyang Because of a cluster migration, the logs are no longer available. From what I remember, each stage was trained for around 20 epochs, consistent with the epoch settings specified in the YAML files. You can do a quick check to verify this.
6. neerajgangwar commented on Feb 7, 2026

   

   neerajgangwar

   on Feb 7, 2026

   Author

   More actions

   Experimental Settings

   • c_thought: 6
   • epochs_per_stage: 1
   • max_latent_stage: 1

   Training Procedure

   1. Stage 1: llm-only Training
      In this stage, all defined implicit tokens participate in training from the beginning, and the explicit reasoning steps are gradually removed. By the end of this stage, all explicit steps are completely removed, and the model performance is approximately ~39.

   @Wiselnn570 How do you remove the explicit reasoning steps gradually with the above configuration in the current version of the code? From what I understand, the current code removes all explicit reasoning steps if scheduled_stage $$>$$ max_latent_stage. Assuming explicit CoT to be $$0^{th}$$ scheduled_stage, the first scheduled_stage will have 6 latent tokens, followed by all but the first explicit reasoning step. And then the next scheduled_stage will remove all explicit reasoning steps. Could you please clarify this and provide the configuration accordingly?

   Additionally, in #4, you mentioned that the SIM-CoT is continued from a Coconut checkpoint. Does this mean that "Stage 1: llm-only Training" starts from that checkpoint?
7. Wiselnn570 commented on Feb 24, 2026

   

   Wiselnn570

   on Feb 24, 2026

   Collaborator

   More actions

   My previous wording may have been a bit unclear. What I intended to convey is that, after SFT, we append all the implicit tokens at once to the model and begin training. Once the training stabilizes, we gradually remove the explicit reasoning steps, while keeping the implicit tokens fixed throughout.

   Previously, I implemented this through a quick patch in the data loader, which was removed when preparing the open-source release. I will take time to properly refine and clean up the code accordingly.
8. neerajgangwar commented on Mar 5, 2026

   

   neerajgangwar

   on Mar 5, 2026

   Last edited by neerajgangwar

   Author

   More actions

   Hi @Wiselnn570,

   Are you referring to the progressive_train flag in the get_cot_with_explainable_latent_dataset method of dataset.py shown below?
   

   SIM-CoT/Coconut/dataset.py

   Line 502 in f3ae967

   if hasattr(configs, 'progressive_train') and configs.progressive_train:

   Also, I am confused about the total number of latent tokens used for Llama3-1b. In this thread, you have mentioned the following as the configuration:

   c_thought: 6
   max_latent_stage: 1
   

   This would result in 6 latent tokens.

   However, on #11, the user asked if c_thought: 2 epochs_per_stage: 3 max_latent_stage: 5 was the correct configuration for Coconut, and you did not disagree. This would mean Coconut uses 10 latent tokens. Also, the paper mentions 13.2 (I assume 10 latent tokens + 2 tokens for start and end latent + 1.2 tokens for the final answer from No-CoT) as the number of tokens. I am not sure which configuration is correct. Could you please confirm this?

   Thank you!