[MEDI] Establish pattern for handling LLM failures

[adamsitnik]

Currently, a failure in one of the requests send to the LLM would tear down entire pipeline.

We should figure out a pattern for handling such failures and expose it to the end user.

The user should be allowed to:

• retry
• ignore the failure
• provide default value

Since all our enrichers accept an IChatClient client and ChatOptions? chatOptions = null, we should consider creating an option bag that would take mandatory client and let the users specify chat options and error handling strategy.

Context: #6957 (comment)

[stephentoub]

This has to be addressed, and not just "failure" in terms of HTTP responses, but even minor hallucinations. With how the enrichers are currently written, if for example a classification enricher had classes "stephen" and "adam" and it sent back "steven" for just a single piece of data, the entire pipeline will fail.