Repository navigation
Error in method and attribute conflicts #29
Description
After running installing required packages, and tried the quick start code, I have the following error message:
[1572](file:///C:/Users/quang/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/tokenization_utils_base.py:1572) for key in kwargs:
[1573](file:///C:/Users/quang/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/tokenization_utils_base.py:1573) if hasattr(self, key) and callable(getattr(self, key)):
-> [1574](file:///C:/Users/quang/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/tokenization_utils_base.py:1574) raise AttributeError(f"{key} conflicts with the method {key} in {self.__class__.__name__}")
[1576](file:///C:/Users/quang/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/tokenization_utils_base.py:1576) self.init_kwargs = copy.deepcopy(kwargs)
[1577](file:///C:/Users/quang/AppData/Local/Programs/Python/Python311/Lib/site-packages/transformers/tokenization_utils_base.py:1577) self.name_or_path = kwargs.pop("name_or_path", "")
AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizer
Which happens to be a problem with the RobertaTokenizer having conflicts in method name and attributes. Please help me fix this problem, or verify if it is an issue with the transformers package.
Thanks a lot.
Phat.
Activity
Frdoe commentedon Sep 2, 2024
What worked for me was to comment out add_special_tokens in resource_management/data_lookups.py, line 82, and in utilities/general_utilities.py line 130 and line 137.
Hope that helps.
thaiquangphat commentedon Sep 3, 2024
Thank you, @Frdoe, for the solution! I really appreciate your help.
dhgottesman commentedon Apr 18, 2025
@Frdoe did you manage to train the model?
Frdoe commentedon Apr 20, 2025
I’m sorry, but I never really tried.
I don’t use refined anymore, as it seems quite dead, and there was new bugs everytime i updated th dependencies.
dhgottesman commentedon Apr 20, 2025
@Frdoe no worries, do you have a recommendation for a different model?
tileh commentedon Sep 21, 2026
In case someone is still looking for a fix: You have to use python 3.11 when installing your venv and change the requirements of the transformers version to
transformers>=4.2.2,<4.37.0in your requirements.txt to make it work