Repository navigation
fix: from_dict must not deserialize into the caller's dictionary - #13616
duoduohappy wants to merge 1 commit into
Conversation
…s serialized `default_from_dict` copies `init_parameters` before replacing serialized sub-objects with their instances, so that a second deserialization does not receive already-parsed objects. Several components did their own deserialization into `data["init_parameters"]` before calling it, which defeats that copy: the caller's mapping ended up holding live callables, enums or components, so it could no longer be serialized with `json.dumps` and a second `from_dict` raised. Deserialize into a copy in those components, and hand `default_from_dict` a fresh envelope that carries it. The same defect was previously fixed one component at a time (MetadataRouter deepset-ai#13421, MockChatGenerator deepset-ai#13471, ListJoiner deepset-ai#13563, Tool deepset-ai#13606). This covers the remaining instances found by scanning every `from_dict` implementation in the repository for writes through `data.get("init_parameters", {})` and `data["data"]`.
|
@duoduohappy is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @duoduohappy, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
|
Thanks for the PR! Our recommendation is to copy the dict yourself (e.g. |
Related Issues
Tool.from_dictmutates the caller's dictionary, so it can't be re-serialized or deserialized twice #13606 (Tool.from_dictmutates the caller's dictionary).Toolis deliberately not touched here — fix: Tool.from_dict must not deserialize into the caller's dictionary #13607 already covers it.Proposed Changes:
default_from_dictcopiesinit_parametersbefore replacing serialized sub-objects with their instances. Its own comment says why:Sixteen components do their own deserialization into
data["init_parameters"]before calling it, so that copy protects nothing. They all follow the same shape:Grouped by what ends up in the caller's mapping:
DocumentSplitter,MockTextEmbedder,MockDocumentEmbedder,OutputAdapter,FunctionHook,LLM,Agent(streaming_callback),ChatPromptBuilder,FallbackChatGeneratorDocumentWriter(DuplicatePolicy),InMemoryBM25RetrieverandInMemoryEmbeddingRetriever(FilterPolicy)Agent(chat_generator,tools),LLM,FallbackChatGenerator,ToolResultOffloadHook(store),ConfirmationHook(confirmation_strategies),ComponentTool,PipelineToolThree consequences, all covered by the new test:
json.dumpson it now raisesTypeError, so the payload can no longer be persisted,KeyError/TypeError/AttributeError).Fix: deserialize into a copy, and hand
default_from_dicta fresh envelope that carries it.For the two tool classes the change is smaller, because they build the instance directly from the mapping (
cls(**inner_data)) rather than throughdefault_from_dict, so copying the mapping is enough.Scope of the sweep. I scanned every
from_dictimplementation in the repository (AST, tracking one level of aliasing throughdata.get(...)/data[...]) for writes into the caller's mapping before any rebinding to a fresh dict. It reported 17 sites; the one left out isTool, which #13607 handles. The same defect has already been fixed one component at a time — MetadataRouter (#13421), MockChatGenerator (#13471), ListJoiner (#13563), all merged — so this closes the remaining instances rather than adding a fourth one-off.Toolis the last one left and is already claimed by #13607.How did you test it?
New file
test/core/test_from_dict_does_not_mutate.pyparameterises four components over the three invariants above (mapping unchanged, still JSON-serializable, secondfrom_dictsucceeds). Four cases cover the enum and nested-dataclass shapes:DocumentWriter,InMemoryBM25Retriever,InMemoryEmbeddingRetriever,ChatPromptBuilder.I checked the test actually fails without the fix — reverting only
haystack/and re-running the equivalent probe gives:and after the fix:
Existing suites for the touched components: 297 passed, 36 failed, and the same 36 fail on the unmodified tree (
git stashbaseline, identical set) — they need optional test dependencies (transformers,sentence-transformers) that are not installed here. No regressions.ruff checkandruff format --checkpass on the changed files.Notes for the reviewer
dict(...)alone.ChatPromptBuilder.from_dictreadsdata["init_parameters"]rather than.get(...), so a payload without that key still raisesKeyErroras before — behaviour unchanged.default_from_dict-adjacent helper that takes the mapping and returns the envelope would remove the footgun; I did not add one because it would touch every component in this PR and make the diff harder to review.