diff --git a/providers/common/ai/docs/approval_gates.rst b/providers/common/ai/docs/approval_gates.rst index 411a13bd02235..386822f969af8 100644 --- a/providers/common/ai/docs/approval_gates.rst +++ b/providers/common/ai/docs/approval_gates.rst @@ -85,6 +85,11 @@ list does not take effect. Reviewing uncertain output -------------------------- +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A classifier model such as TypeSafe's reports a confidence for every field of a structured output, in ``provider_details`` on the model response. It is a summary of how concentrated the model's probability distribution was, not the diff --git a/providers/common/ai/docs/classifier_models.rst b/providers/common/ai/docs/classifier_models.rst index 2bb35f5671d3f..2f003806e56c7 100644 --- a/providers/common/ai/docs/classifier_models.rst +++ b/providers/common/ai/docs/classifier_models.rst @@ -18,6 +18,11 @@ Classifier models ================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Every other model in this provider writes text. A classifier model does not: you give it some text and a typed question, and it answers with a value from a set you named in advance, plus a confidence. Ask it for a string and the request is refused before it diff --git a/providers/common/ai/docs/code_mode.rst b/providers/common/ai/docs/code_mode.rst index 32256d88fb14a..c32fa15700ccc 100644 --- a/providers/common/ai/docs/code_mode.rst +++ b/providers/common/ai/docs/code_mode.rst @@ -20,6 +20,11 @@ Code mode ========= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Set ``code_mode=True`` to collapse the agent's tools into a single ``run_code`` tool powered by the `Monty `__ sandbox (via pydantic-ai-harness). Instead of one model round-trip per tool call, the model diff --git a/providers/common/ai/docs/connections/langchain.rst b/providers/common/ai/docs/connections/langchain.rst index 1949d99dea810..f5b4c54bcc706 100644 --- a/providers/common/ai/docs/connections/langchain.rst +++ b/providers/common/ai/docs/connections/langchain.rst @@ -20,6 +20,11 @@ LangChain connection ==================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The ``langchain`` connection type configures access to LLM providers via `LangChain `__'s universal ``init_chat_model`` / ``init_embeddings`` entry points. It backs diff --git a/providers/common/ai/docs/connections/llamaindex.rst b/providers/common/ai/docs/connections/llamaindex.rst index e74a688f48d29..2b89a4689fb07 100644 --- a/providers/common/ai/docs/connections/llamaindex.rst +++ b/providers/common/ai/docs/connections/llamaindex.rst @@ -20,6 +20,11 @@ LlamaIndex connection ===================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The ``llamaindex`` connection type configures access to LLM and embedding providers for `LlamaIndex `__. It backs :class:`~airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook` (see diff --git a/providers/common/ai/docs/durable_execution.rst b/providers/common/ai/docs/durable_execution.rst index b34daa591b8a3..262f48f6aa5f0 100644 --- a/providers/common/ai/docs/durable_execution.rst +++ b/providers/common/ai/docs/durable_execution.rst @@ -20,6 +20,11 @@ Durable execution ================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Agent tasks can involve multiple LLM calls and tool invocations. If a task fails mid-run (network error, timeout, transient API failure), a plain retry re-executes every LLM call and tool call from scratch -- repeating work that diff --git a/providers/common/ai/docs/guardrails.rst b/providers/common/ai/docs/guardrails.rst index a15918aa46c38..dff3cf6093ea1 100644 --- a/providers/common/ai/docs/guardrails.rst +++ b/providers/common/ai/docs/guardrails.rst @@ -48,6 +48,12 @@ Guardrail capabilities use the same passthrough pattern. This example uses ``InputGuard`` from ``pydantic-ai-shields`` to reject a prompt before the agent run starts. +.. note:: + + Experimental: the ``shields`` extra can change or be removed in a minor release of this + provider. + See :ref:`howto/stability`. + .. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_agent_capabilities.py :language: python :start-after: [START howto_operator_agent_capabilities_input_guard] diff --git a/providers/common/ai/docs/hooks/langchain.rst b/providers/common/ai/docs/hooks/langchain.rst index 579829bdb425d..b141b36e881c7 100644 --- a/providers/common/ai/docs/hooks/langchain.rst +++ b/providers/common/ai/docs/hooks/langchain.rst @@ -20,6 +20,11 @@ LangChain models: ``LangChainHook`` =================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + .. toctree:: :titlesonly: :hidden: diff --git a/providers/common/ai/docs/hooks/llamaindex.rst b/providers/common/ai/docs/hooks/llamaindex.rst index c388f4f921c04..44dd4e2335c35 100644 --- a/providers/common/ai/docs/hooks/llamaindex.rst +++ b/providers/common/ai/docs/hooks/llamaindex.rst @@ -20,6 +20,11 @@ Using LlamaIndex directly: ``LlamaIndexHook`` ============================================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook` to bridge an Airflow connection to `LlamaIndex `__ chat and embedding models. The hook reads credentials (API key, optional diff --git a/providers/common/ai/docs/index.rst b/providers/common/ai/docs/index.rst index ba37e88a6466c..a245d5e83db60 100644 --- a/providers/common/ai/docs/index.rst +++ b/providers/common/ai/docs/index.rst @@ -119,6 +119,7 @@ waits for the result, use that vendor's provider. Example Dags Configuration + Stable and experimental features Python API <_api/airflow/providers/common/ai/index> .. toctree:: diff --git a/providers/common/ai/docs/observability.rst b/providers/common/ai/docs/observability.rst index 1282df14f0fba..fc641a375df94 100644 --- a/providers/common/ai/docs/observability.rst +++ b/providers/common/ai/docs/observability.rst @@ -18,6 +18,12 @@ Observability (OpenTelemetry tracing) ===================================== +.. note:: + + Experimental: the spans and their attributes can change in a minor release of this + provider. + See :ref:`howto/stability`. + pydantic-ai ships native OpenTelemetry instrumentation that emits GenAI spans for each agent run, model call, and tool call, following the `OpenTelemetry GenAI semantic conventions `__. diff --git a/providers/common/ai/docs/operators/document_loader.rst b/providers/common/ai/docs/operators/document_loader.rst index d425fe30f4875..b86f1f0d7ab79 100644 --- a/providers/common/ai/docs/operators/document_loader.rst +++ b/providers/common/ai/docs/operators/document_loader.rst @@ -20,6 +20,11 @@ Load documents: ``DocumentLoaderOperator`` ========================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.operators.document_loader.DocumentLoaderOperator` to parse files into ``list[dict(text, metadata)]`` for downstream embedding pipelines. The operator bridges Airflow's connectivity layer (hooks that diff --git a/providers/common/ai/docs/operators/llamaindex_embedding.rst b/providers/common/ai/docs/operators/llamaindex_embedding.rst index fff4ea461b0cb..d3a101c3f811c 100644 --- a/providers/common/ai/docs/operators/llamaindex_embedding.rst +++ b/providers/common/ai/docs/operators/llamaindex_embedding.rst @@ -20,6 +20,11 @@ Embed documents: ``LlamaIndexEmbeddingOperator`` ================================================ +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Chunk a ``list[dict]`` of documents and produce embedding vectors using LlamaIndex. Designed to feed the output of :class:`~airflow.providers.common.ai.operators.document_loader.DocumentLoaderOperator` diff --git a/providers/common/ai/docs/operators/llamaindex_retrieval.rst b/providers/common/ai/docs/operators/llamaindex_retrieval.rst index c346cd9dd28e1..1a0b254023b87 100644 --- a/providers/common/ai/docs/operators/llamaindex_retrieval.rst +++ b/providers/common/ai/docs/operators/llamaindex_retrieval.rst @@ -20,6 +20,11 @@ Retrieve context: ``LlamaIndexRetrievalOperator`` ================================================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Load a persisted LlamaIndex index and run similarity search. Designed to sit between :class:`~airflow.providers.common.ai.operators.llamaindex_embedding.LlamaIndexEmbeddingOperator` diff --git a/providers/common/ai/docs/operators/llm.rst b/providers/common/ai/docs/operators/llm.rst index 83cf366261001..6f026710f6a39 100644 --- a/providers/common/ai/docs/operators/llm.rst +++ b/providers/common/ai/docs/operators/llm.rst @@ -212,6 +212,11 @@ reviewer. The full guide, including timeouts, notifiers and assigned reviewers, Reviewing uncertain output -------------------------- +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A ``decision_policy`` with a confidence bar sends output the model is unsure about to the same review flow. See :doc:`../approval_gates`. diff --git a/providers/common/ai/docs/operators/llm_batch.rst b/providers/common/ai/docs/operators/llm_batch.rst index f0050f2b70852..806517c4051ad 100644 --- a/providers/common/ai/docs/operators/llm_batch.rst +++ b/providers/common/ai/docs/operators/llm_batch.rst @@ -20,6 +20,11 @@ Batch processing: ``LLMBatchOperator`` ====================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.operators.llm_batch.LLMBatchOperator` to run many prompts through a provider's **batch API** instead of one synchronous call per prompt: roughly half the per-token cost of :class:`~airflow.providers.common.ai.operators.llm.LLMOperator`, diff --git a/providers/common/ai/docs/operators/llm_branch.rst b/providers/common/ai/docs/operators/llm_branch.rst index 2d066baa2a0e8..9e572d21e0e84 100644 --- a/providers/common/ai/docs/operators/llm_branch.rst +++ b/providers/common/ai/docs/operators/llm_branch.rst @@ -171,6 +171,11 @@ behaviour are inherited from :ref:`LLMOperator `. Reviewing Uncertain Picks ------------------------- +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A classifier model such as TypeSafe's returns a confidence with every pick, a number from 0 to 1 that summarizes how concentrated its probability distribution was: near 1 when one branch stood out, low when two or more diff --git a/providers/common/ai/docs/operators/llm_file_analysis.rst b/providers/common/ai/docs/operators/llm_file_analysis.rst index 220af100c23e4..4fb0931410d77 100644 --- a/providers/common/ai/docs/operators/llm_file_analysis.rst +++ b/providers/common/ai/docs/operators/llm_file_analysis.rst @@ -20,6 +20,11 @@ Analyze files and images: ``LLMFileAnalysisOperator`` ===================================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.operators.llm_file_analysis.LLMFileAnalysisOperator` or the ``@task.llm_file_analysis`` decorator to analyze files from object storage or local storage with a single prompt. diff --git a/providers/common/ai/docs/operators/llm_schema_compare.rst b/providers/common/ai/docs/operators/llm_schema_compare.rst index 6fea1f2ca5589..e825c873181c8 100644 --- a/providers/common/ai/docs/operators/llm_schema_compare.rst +++ b/providers/common/ai/docs/operators/llm_schema_compare.rst @@ -20,6 +20,11 @@ Detect schema drift: ``LLMSchemaCompareOperator`` ================================================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.operators.llm_schema_compare.LLMSchemaCompareOperator` to compare schemas across different database systems and detect drift using LLM reasoning. diff --git a/providers/common/ai/docs/operators/llm_sql.rst b/providers/common/ai/docs/operators/llm_sql.rst index ccaaaf02a39ae..6f3b74b5aab57 100644 --- a/providers/common/ai/docs/operators/llm_sql.rst +++ b/providers/common/ai/docs/operators/llm_sql.rst @@ -20,6 +20,11 @@ Natural language to SQL: ``LLMSQLQueryOperator`` ================================================ +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Use :class:`~airflow.providers.common.ai.operators.llm_sql.LLMSQLQueryOperator` to generate SQL queries from natural language using an LLM. diff --git a/providers/common/ai/docs/rag_pipelines.rst b/providers/common/ai/docs/rag_pipelines.rst index 0825eafd6508c..96592e0e04867 100644 --- a/providers/common/ai/docs/rag_pipelines.rst +++ b/providers/common/ai/docs/rag_pipelines.rst @@ -20,6 +20,11 @@ Document and RAG pipelines ========================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A retrieval pipeline in this provider is three ordinary tasks. :doc:`operators/document_loader` parses files (text, CSV, JSON, PDF, DOCX) into a list of ``{"text", "metadata"}`` dicts with no AI framework involved. :doc:`operators/llamaindex_embedding` chunks those documents and diff --git a/providers/common/ai/docs/retry_policies.rst b/providers/common/ai/docs/retry_policies.rst index cd88962473f03..52f0977902b46 100644 --- a/providers/common/ai/docs/retry_policies.rst +++ b/providers/common/ai/docs/retry_policies.rst @@ -159,6 +159,11 @@ an attempt is a separate mechanism on the connection; see ClassifierRetryPolicy --------------------- +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + ``ClassifierRetryPolicy`` takes the retry decision away from the model. Its ``categories`` maps a category name to an :class:`~airflow.providers.common.ai.policies.retry.ErrorCategory`: what diff --git a/providers/common/ai/docs/sandbox/backends.rst b/providers/common/ai/docs/sandbox/backends.rst index 39fc416adde5a..1934ecd742aa6 100644 --- a/providers/common/ai/docs/sandbox/backends.rst +++ b/providers/common/ai/docs/sandbox/backends.rst @@ -18,6 +18,11 @@ Sandbox backends ================ +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + .. _sandbox-backend-modal: Modal (hosted) diff --git a/providers/common/ai/docs/sandbox/configuration.rst b/providers/common/ai/docs/sandbox/configuration.rst index 7f5713d7565cf..d4e347f27cbbc 100644 --- a/providers/common/ai/docs/sandbox/configuration.rst +++ b/providers/common/ai/docs/sandbox/configuration.rst @@ -18,6 +18,11 @@ Sandbox configuration and lifecycle =================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + .. _sandbox-configuring: Configuring a sandbox diff --git a/providers/common/ai/docs/sandbox/index.rst b/providers/common/ai/docs/sandbox/index.rst index 7a8cfb26934da..6d201ee09fcfb 100644 --- a/providers/common/ai/docs/sandbox/index.rst +++ b/providers/common/ai/docs/sandbox/index.rst @@ -20,6 +20,11 @@ Sandboxed execution for agents ============================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + An agent that is asked to do open-ended work writes code, and then something has to run that code. By default that something is the Airflow worker: a skill script, a hand-written tool that shells out, or generated glue in diff --git a/providers/common/ai/docs/stability.rst b/providers/common/ai/docs/stability.rst new file mode 100644 index 0000000000000..93e47ed417a6a --- /dev/null +++ b/providers/common/ai/docs/stability.rst @@ -0,0 +1,168 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _howto/stability: + +Stable and experimental features +================================ + +A stable feature keeps its behaviour across minor releases of this provider, from version +1.0.0 on. An experimental feature is documented and maintained, but can change or be removed +in a minor release, as Airflow's :ref:`experimental feature policy ` +allows. A breaking change to an experimental feature is announced in the changelog. + +Stable features +--------------- + +A stable feature keeps its public parameters, their defaults and the behaviour described +below until the next major release. A minor release can add an optional parameter. +Changing a default, renaming a parameter or narrowing what a feature does needs a major +release. + +Stability covers behaviour, not wording. Log lines, error messages, tool descriptions and +the prompt text this provider sends to a model can change in any release. The toolsets stay +Pydantic AI toolsets, but the Pydantic AI class they inherit from can change. + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Feature + - What stays the same + * - ``@task.llm`` and :class:`~airflow.providers.common.ai.operators.llm.LLMOperator` + - Sends the prompt to the model from ``llm_conn_id`` and returns the output as the + task's return value: a string by default, or an instance of ``output_type``, + dumped to a dict when ``serialize_output=True``. Before Airflow 3.3 an instance is + always dumped to a dict. ``decision_policy`` is experimental; see below. + * - ``@task.agent`` and :class:`~airflow.providers.common.ai.operators.agent.AgentOperator` + - Runs an agent with the model from ``llm_conn_id`` and the given toolsets, and + returns its output under the same rules as ``@task.llm``. ``durable``, + ``code_mode`` and per-tool approval are experimental; see below. + * - ``message_history`` on ``AgentOperator`` + - Seeds the run with the given conversation, as a list of messages or its JSON form, + and publishes the finished conversation under the ``message_history`` XCom key so a + later run can continue it. Cannot be combined with ``enable_hitl_review``. + * - ``@task.llm_branch`` and + :class:`~airflow.providers.common.ai.operators.llm_branch.LLMBranchOperator` + - The model picks among the task's direct downstream tasks, one by default or several + with ``allow_multiple_branches=True``, and only the picked tasks run. The + descriptions in ``branches``, as strings or as ``BranchOption(description=...)``, + are sent to the model with the options. ``decision_policy`` and + ``BranchOption.min_confidence`` are experimental; see below. + * - :class:`~airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook` and its + connection types + - The connection fields documented for each type in :doc:`connections/pydantic_ai`, + :doc:`connections/pydantic_ai_bedrock`, :doc:`connections/pydantic_ai_vertex` and + :doc:`connections/pydantic_ai_azure` keep their meaning, so an existing connection + keeps producing the same model. + * - :class:`~airflow.providers.common.ai.toolsets.sql.SQLToolset` + - Exposes ``list_tables``, ``get_schema``, ``query`` and ``check_query``. Read-only + unless ``allow_writes=True``. When ``allowed_tables`` is set, other tables are + refused. ``query`` returns at most ``max_rows`` rows, and keeps the rows and column + names within ``max_result_bytes`` bytes. When ``query`` fails or refuses a + statement, the model gets the error and can correct it, up to the tool's retry + limit, after which the task fails. + * - :class:`~airflow.providers.common.ai.toolsets.hook.HookToolset` + - Exposes exactly the hook methods in ``allowed_methods``, each named after its method + with ``tool_name_prefix`` in front, and raises an error when the toolset is created + if a listed method does not exist on the hook. + * - :class:`~airflow.providers.common.ai.toolsets.mcp.MCPToolset` and + :class:`~airflow.providers.common.ai.hooks.mcp.MCPHook` + - Exposes the tools of the MCP server configured by ``mcp_conn_id``, each named + ``_`` when ``tool_prefix`` is set. The connection fields for each + transport keep their meaning. + * - :class:`~airflow.providers.common.ai.policies.retry.LLMRetryPolicy` + - Returns a retry decision from a model's reading of the exception. Values registered + as secrets are masked in the exception text before it reaches the model, unless + ``redact_exception=False`` or a custom ``redactor`` replaces the masking. If the + model call fails, it falls back to ``fallback_rules`` and then to the task's own + retry settings; it never fails the task itself. + * - Output review: ``require_approval``, ``enable_hitl_review`` and the review plugin + - With ``require_approval=True``, the task defers after generating output and returns + it only once a person approves it on the Required Actions page. A rejection, or a + timeout under the default ``on_approval_timeout="fail"``, fails the task, except on + ``@task.llm_branch``, where it skips the downstream tasks unless + ``fail_on_reject=True``. + +Experimental features +--------------------- + +Everything this provider ships that is not in the table above is experimental. + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - Feature + - Why it is experimental + * - :class:`~airflow.providers.common.ai.policies.retry.ClassifierRetryPolicy` + (:doc:`classifier_models`) + - The confidence threshold, the fallback order and the behaviour when the classifier + is unavailable are still settling. + * - :class:`~airflow.providers.common.ai.policies.decision.DecisionPolicy`, and + ``min_confidence`` on + :class:`~airflow.providers.common.ai.policies.decision.BranchOption` + (:doc:`approval_gates`) + - New. The threshold semantics and the recorded decision may change as they are + used. + * - ``@task.llm_batch`` and batch adapters (:doc:`operators/llm_batch`) + - Submission, re-attachment on retry, cancellation and handling of partial results + are still settling, and ``BatchAdapter`` has no implementation outside this + provider yet. + * - ``durable=True`` on ``AgentOperator`` (:doc:`durable_execution`) + - Which tool results are replayed on retry will change, so that a tool can declare + whether its result may be reused. + * - Per-tool approval on ``AgentOperator`` (``tool_approval_timeout``, + ``on_tool_approval_timeout`` and ``tool_approval_assigned_users``; + :doc:`tool_approval`) + - New, and needs Airflow 3.3. How a paused run resumes may change. + * - ``code_mode`` (:doc:`code_mode`), the Agent Skills toolset + (:doc:`toolsets/skills`) and the ``shields`` extra (used in :doc:`guardrails`) + - Thin integrations of packages outside this provider whose APIs are still + changing: ``pydantic-ai-harness``, ``pydantic-ai-skills`` and + ``pydantic-ai-shields``. + * - :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset` and its + backends (:doc:`sandbox/index`) + - The sandbox runtime belongs to the backend; ownership and cleanup across worker + failures are still being designed. + * - LangChain and LlamaIndex hooks and the LangChain tool bridge + (:doc:`hooks/langchain`, :doc:`toolsets/langchain`, :doc:`hooks/llamaindex`) + - Each follows the API of a framework that changes often. + * - The retrieval operators: ``DocumentLoaderOperator``, + ``LlamaIndexEmbeddingOperator`` and ``LlamaIndexRetrievalOperator`` + (:doc:`rag_pipelines`) + - New. The shape of the pipeline, from loading documents through embedding to + retrieval, has not settled. + * - :class:`~airflow.providers.common.ai.toolsets.datafusion.DataFusionToolset` + (:doc:`toolsets/datafusion`) + - Runs on the DataFusion engine of ``common.sql``, which pins ``datafusion`` below 52 + and follows its API. + * - Managed agent toolsets (:doc:`toolsets/managed_agent`) + - A contract for vendor providers that no vendor implements yet. + * - OpenTelemetry spans (``[common.ai] otel_export_enabled`` and + ``capture_content``; :doc:`observability`) + - Span names and attributes come from Pydantic AI's instrumentation and the + OpenTelemetry GenAI conventions, which are still in development. + * - :class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` used directly + (:doc:`toolsets/logging`) + - The wrapper ``AgentOperator`` applies for ``enable_tool_logging``; its constructor + may change. + * - ``@task.llm_schema_compare``, ``@task.llm_file_analysis`` and ``@task.llm_sql`` + (:doc:`operators/llm_schema_compare`, :doc:`operators/llm_file_analysis`, + :doc:`operators/llm_sql`) + - Each adds its own input handling to ``@task.llm``: schema introspection, file + sampling, or SQL validation. Their options are still settling. diff --git a/providers/common/ai/docs/tool_approval.rst b/providers/common/ai/docs/tool_approval.rst index e3ca1aacfaba0..a5ede5270e0d4 100644 --- a/providers/common/ai/docs/tool_approval.rst +++ b/providers/common/ai/docs/tool_approval.rst @@ -20,6 +20,11 @@ Approve an agent's tool calls ============================= +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + .. seealso:: To approve, edit or reject an LLM operator's output instead, see :doc:`approval_gates`; to review an agent's final answer over several rounds, see :doc:`hitl_review`. diff --git a/providers/common/ai/docs/toolsets/datafusion.rst b/providers/common/ai/docs/toolsets/datafusion.rst index fc83a90f8b9db..27db2d8a109d0 100644 --- a/providers/common/ai/docs/toolsets/datafusion.rst +++ b/providers/common/ai/docs/toolsets/datafusion.rst @@ -18,6 +18,11 @@ Files with DataFusion: ``DataFusionToolset`` ============================================ +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Curated toolset wrapping :class:`~airflow.providers.common.sql.datafusion.engine.DataFusionEngine` with three tools (``list_tables``, ``get_schema``, and ``query``) for diff --git a/providers/common/ai/docs/toolsets/langchain.rst b/providers/common/ai/docs/toolsets/langchain.rst index 2901d4a836dc9..d1c59881ddbf2 100644 --- a/providers/common/ai/docs/toolsets/langchain.rst +++ b/providers/common/ai/docs/toolsets/langchain.rst @@ -18,6 +18,11 @@ LangChain tools in both directions ================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Tools bridge in both directions between common.ai's toolsets and LangChain. **LangChain tools → ``AgentOperator``.** No Airflow code is needed. pydantic-ai diff --git a/providers/common/ai/docs/toolsets/logging.rst b/providers/common/ai/docs/toolsets/logging.rst index 4e40a75329cb6..816df3a217ad4 100644 --- a/providers/common/ai/docs/toolsets/logging.rst +++ b/providers/common/ai/docs/toolsets/logging.rst @@ -18,6 +18,11 @@ Tool call logging: ``LoggingToolset`` ===================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` is a ``WrapperToolset`` that intercepts ``call_tool()`` to log each tool invocation in real time. ``AgentOperator`` applies it automatically (see diff --git a/providers/common/ai/docs/toolsets/managed_agent.rst b/providers/common/ai/docs/toolsets/managed_agent.rst index fdab5f3ddb704..4ab923fa5f471 100644 --- a/providers/common/ai/docs/toolsets/managed_agent.rst +++ b/providers/common/ai/docs/toolsets/managed_agent.rst @@ -20,6 +20,11 @@ Vendor-managed agents: ``BaseManagedAgentToolset`` ================================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Cloud vendors now run agents on your behalf: Snowflake Cortex Agents, Amazon Bedrock AgentCore runtimes, Azure AI Foundry hosted agents, Vertex AI Agent Engine. Their reasoning loops execute on the vendor's infrastructure, so they @@ -27,8 +32,8 @@ are not something ``AgentOperator`` runs; they are something an Airflow task *consults*. :class:`~airflow.providers.common.ai.toolsets.managed_agent.BaseManagedAgentToolset` -is the contract for exposing one of those as a tool. Each provider package -ships its own subclass, so credentials keep flowing through that provider's +is the contract for exposing one of those as a tool. A provider package would +ship its own subclass, so credentials keep flowing through that provider's existing hook and no new connection types are needed. A subclass implements two members: diff --git a/providers/common/ai/docs/toolsets/skills.rst b/providers/common/ai/docs/toolsets/skills.rst index 9fb2adf576759..ed32a2fb2c711 100644 --- a/providers/common/ai/docs/toolsets/skills.rst +++ b/providers/common/ai/docs/toolsets/skills.rst @@ -20,6 +20,11 @@ Agent Skills: ``AgentSkillsToolset`` ==================================== +.. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :class:`~airflow.providers.common.ai.toolsets.skills.AgentSkillsToolset` loads `Agent Skills `__ -- ``SKILL.md`` bundles (instructions, and optionally scripts and resources) that the model discovers and loads *on diff --git a/providers/common/ai/src/airflow/providers/common/ai/batch/__init__.py b/providers/common/ai/src/airflow/providers/common/ai/batch/__init__.py index 1b0cc20ea7df5..d79f408010f45 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/batch/__init__.py +++ b/providers/common/ai/src/airflow/providers/common/ai/batch/__init__.py @@ -17,7 +17,7 @@ """ Batch execution for ``@task.llm_batch``. -The stable surface for other packages is :class:`BatchAdapter` (the contract a +The extension points for other packages, all experimental, are :class:`BatchAdapter` (the contract a provider batch engine implements), :class:`BatchRequest` (one input item) and :func:`register_adapter` (how another provider package plugs its adapter in). """ diff --git a/providers/common/ai/src/airflow/providers/common/ai/batch/anthropic.py b/providers/common/ai/src/airflow/providers/common/ai/batch/anthropic.py index b69aa91a4ad44..1fc58b45aeb50 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/batch/anthropic.py +++ b/providers/common/ai/src/airflow/providers/common/ai/batch/anthropic.py @@ -94,6 +94,11 @@ class AnthropicBatchAdapter(BatchAdapter): """ Batch adapter for Anthropic's Message Batches API. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :param api_key: Passed straight to the ``anthropic.Anthropic`` client. ``None`` falls back to the SDK's own env-var resolution (``ANTHROPIC_API_KEY``). diff --git a/providers/common/ai/src/airflow/providers/common/ai/batch/base.py b/providers/common/ai/src/airflow/providers/common/ai/batch/base.py index 3fcf189e67af8..e0d435fcbf77e 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/batch/base.py +++ b/providers/common/ai/src/airflow/providers/common/ai/batch/base.py @@ -42,6 +42,11 @@ class BatchRequest(TypedDict, total=False): """ One input item for ``@task.llm_batch``. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A bare string ``"foo"`` is shorthand for ``{"prompt": "foo"}``; the operator normalizes to this shape before anything else sees the input. """ @@ -138,6 +143,11 @@ class BatchAdapter(ABC): """ Adapter contract between the common.ai batch surface and one provider's batch API. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + ``batch/dispatch.py`` selects a concrete subclass by the ``model_id`` prefix (request shape) and checks the connection type against :attr:`conn_types` (auth). Operator, trigger, and the state/results layers diff --git a/providers/common/ai/src/airflow/providers/common/ai/batch/dispatch.py b/providers/common/ai/src/airflow/providers/common/ai/batch/dispatch.py index a8d0a29cf0aea..22615e2f46f20 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/batch/dispatch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/batch/dispatch.py @@ -67,6 +67,11 @@ def register_adapter(adapter_cls: type[BatchAdapter]) -> None: """ Register an adapter class for its own :attr:`~airflow.providers.common.ai.batch.base.BatchAdapter.name` prefix. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Intended for other provider packages that ship a batch engine (Bedrock, Vertex, Azure OpenAI). A registration overrides a built-in or entry-point adapter with the same prefix. diff --git a/providers/common/ai/src/airflow/providers/common/ai/batch/openai.py b/providers/common/ai/src/airflow/providers/common/ai/batch/openai.py index 2a8ff497ae904..11273789c3a6d 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/batch/openai.py +++ b/providers/common/ai/src/airflow/providers/common/ai/batch/openai.py @@ -104,6 +104,11 @@ class OpenAIBatchAdapter(BatchAdapter): """ Batch adapter for OpenAI's Batch API (``/v1/chat/completions``). + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :param api_key: Passed straight to the ``openai.OpenAI`` client. ``None`` falls back to the SDK's own env-var resolution (``OPENAI_API_KEY``). :param base_url: Passed straight to the ``openai.OpenAI`` client. Pointing diff --git a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_batch.py b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_batch.py index a9f3f7ebf2d36..d7f351b3728ef 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_batch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_batch.py @@ -110,6 +110,11 @@ def llm_batch_task( """ Wrap a function that returns a batch's inputs into an ``@task.llm_batch`` task. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The function body constructs the list of inputs (it can use Airflow context, XCom, etc.). Results are written to ``result_path`` as JSONL; the XCom value is a manifest describing where to find them (see diff --git a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_file_analysis.py b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_file_analysis.py index e6ad0fcf2bd19..305184fb0d102 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_file_analysis.py +++ b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_file_analysis.py @@ -101,6 +101,11 @@ def llm_file_analysis_task( """ Wrap a callable that returns a prompt into an LLM-backed file-analysis task. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Any file-analysis keyword arguments accepted by :class:`~airflow.providers.common.ai.operators.llm_file_analysis.LLMFileAnalysisOperator`, including ``sample_rows``, can be passed through this decorator. diff --git a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_schema_compare.py b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_schema_compare.py index d159da63ca93d..2e106509215e8 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_schema_compare.py +++ b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_schema_compare.py @@ -110,6 +110,11 @@ def llm_schema_compare_task( """ Wrap a function that returns a prompt into an LLM schema comparison task. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The function body constructs the prompt (can use Airflow context, XCom, etc.). The decorator handles: schema introspection from multiple data sources, LLM-powered cross-system type comparison, and structured mismatch output. diff --git a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_sql.py b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_sql.py index 5dca4e3d92fb4..7ea9765ddcdbb 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_sql.py +++ b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_sql.py @@ -112,6 +112,11 @@ def llm_sql_task( """ Wrap a function that returns a natural language prompt into an LLM SQL task. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The function body constructs the prompt (can use Airflow context, XCom, etc.). The decorator handles: LLM connection, schema introspection, SQL generation, and safety validation. diff --git a/providers/common/ai/src/airflow/providers/common/ai/hooks/langchain.py b/providers/common/ai/src/airflow/providers/common/ai/hooks/langchain.py index 87a0f44025062..55b779bba0ffa 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/hooks/langchain.py +++ b/providers/common/ai/src/airflow/providers/common/ai/hooks/langchain.py @@ -31,6 +31,11 @@ class LangChainHook(BaseHook): """ Bridge an Airflow connection to LangChain chat and embedding models. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The hook resolves credentials (API key, optional base URL) from the Airflow connection and returns LangChain model objects via two universal entry-point functions: diff --git a/providers/common/ai/src/airflow/providers/common/ai/hooks/llamaindex.py b/providers/common/ai/src/airflow/providers/common/ai/hooks/llamaindex.py index f5c6ee5ce0d54..f0b45370b5ba9 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/hooks/llamaindex.py +++ b/providers/common/ai/src/airflow/providers/common/ai/hooks/llamaindex.py @@ -34,6 +34,11 @@ class LlamaIndexHook(BaseHook): """ Bridge an Airflow connection to LlamaIndex chat and embedding models. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The hook resolves credentials (API key, optional API base URL) from the Airflow connection and returns native LlamaIndex objects ready to pass to ``VectorStoreIndex(..., embed_model=...)``, diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py b/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py index f261b6a5d47e3..1b6f5d5f0b7ca 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py @@ -242,7 +242,7 @@ class AgentOperator(CancellableAgentRunMixin, BaseOperator, HITLReviewMixin): request cap. See :ref:`howto/operator:llm` for the full set of caveats, and :ref:`howto/operator:agent` for the ``durable=True`` replay double-counting warning. - :param durable: When ``True``, enables step-level caching of model + :param durable: Experimental. When ``True``, enables step-level caching of model responses and tool results for durable execution. On retry, cached steps are replayed instead of re-executing. Each cached step is verified against the current request before replay: if the prompt, @@ -264,7 +264,7 @@ class AgentOperator(CancellableAgentRunMixin, BaseOperator, HITLReviewMixin): not: a replayed tool result describes a workspace state the replay did not reproduce, and the first call that misses the cache runs against whatever the sandbox holds now. - :param code_mode: When ``True``, wraps the agent's tools in a single + :param code_mode: Experimental. When ``True``, wraps the agent's tools in a single ``run_code`` tool powered by the Monty sandbox (pydantic-ai-harness ``CodeMode``). Instead of one model round-trip per tool call, the model writes Python that calls the tools as functions, with loops and @@ -318,7 +318,7 @@ class AgentOperator(CancellableAgentRunMixin, BaseOperator, HITLReviewMixin): :param hitl_poll_interval: Seconds between XCom polls while waiting for a human response. Default ``10``. - **Per-tool approval** (Airflow 3.3+): + **Per-tool approval** (Airflow 3.3+, experimental): Mark the tools a human must approve with pydantic-ai's own API -- ``toolset.approval_required(...)``, or ``requires_approval=True`` on a function @@ -334,13 +334,13 @@ class AgentOperator(CancellableAgentRunMixin, BaseOperator, HITLReviewMixin): requires approval fails the task as before. A ``SandboxToolset`` attached to a sandbox another task owns is fine: the sandbox outlives the pause. - :param tool_approval_timeout: How long the pause waits for a decision. + :param tool_approval_timeout: Experimental. How long the pause waits for a decision. ``None`` (default) waits indefinitely. Must be positive. - :param on_tool_approval_timeout: What a timed-out pause does: ``"fail"`` + :param on_tool_approval_timeout: Experimental. What a timed-out pause does: ``"fail"`` (default) fails the task, ``"deny"`` rejects the pending calls so the agent carries on without them, and needs a ``tool_approval_timeout``. There is no approve-on-timeout. - :param tool_approval_assigned_users: Users allowed to decide. ``None`` (default) + :param tool_approval_assigned_users: Experimental. Users allowed to decide. ``None`` (default) leaves it to anyone who can act on the task's Required Actions. :param serialize_output: If ``True`` and ``output_type`` is a Pydantic diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/document_loader.py b/providers/common/ai/src/airflow/providers/common/ai/operators/document_loader.py index 42f630ffcf591..a226b89ef76ab 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/document_loader.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/document_loader.py @@ -45,6 +45,11 @@ class DocumentLoaderOperator(BaseOperator): """ Parse files into ``list[dict(text, metadata)]`` for downstream embedding. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Bridges Airflow's connectivity layer (hooks that produce bytes or local files) and the AI embedding layer (operators that need structured text with metadata). Framework-agnostic: no LlamaIndex, LangChain, or other diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_embedding.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_embedding.py index 34cd441ef0565..bfc8001091f3d 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_embedding.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_embedding.py @@ -38,6 +38,11 @@ class LlamaIndexEmbeddingOperator(BaseOperator): """ Chunk documents and produce embedding vectors using LlamaIndex. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Bridges document loading (e.g. :class:`~airflow.providers.common.ai.operators.document_loader.DocumentLoaderOperator` output) and vector storage (pgvector, Pinecone, Weaviate, ...). Input is diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_retrieval.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_retrieval.py index 9725cdace48d1..67f8fa9bfa3b6 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_retrieval.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llamaindex_retrieval.py @@ -37,6 +37,11 @@ class LlamaIndexRetrievalOperator(BaseOperator): """ Retrieve relevant document chunks from a persisted LlamaIndex index. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Loads a previously persisted vector store index (from ``LlamaIndexEmbeddingOperator(persist_dir=...)``) and performs similarity search against the provided query. Output is a list of chunks with text, diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py index a06f27f04a128..d54a089acc7cb 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py @@ -154,7 +154,8 @@ class LLMOperator(CancellableAgentRunMixin, BaseOperator, LLMApprovalMixin): ``{"id": ..., "name": ...}`` dicts where ``id`` is the auth manager's user id. ``None`` (default) lets any user with the permission respond. The list is fixed when the review is first created. Needs Airflow 3.1+. - :param decision_policy: A :class:`~airflow.providers.common.ai.utils.decision.DecisionPolicy` + :param decision_policy: Experimental. A + :class:`~airflow.providers.common.ai.policies.decision.DecisionPolicy` saying how confident the model has to be for the operator to return its answer by itself (``min_confidence``) and what happens otherwise (``on_uncertain``: ``"review"`` or ``"fail"``). Confidence comes from models that report one per output field, such as diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_batch.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_batch.py index bc342077ee792..6f05a216a8750 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_batch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_batch.py @@ -61,6 +61,11 @@ class LLMBatchOperator(BaseOperator): """ Submit prompts as a provider batch job and land results on object storage. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Routes to the OpenAI or Anthropic batch API based on ``model_id``'s ``":"`` prefix (see :mod:`~airflow.providers.common.ai.batch.dispatch`). Unlike diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py index 939e3fe5b4e91..02f603ba3afab 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py @@ -97,13 +97,15 @@ class LLMBranchOperator(LLMOperator, BranchMixIn): to a string as shorthand for its description. The description travels in the output schema next to the option, so the model reads "here is an option, here is what it means" rather than guessing from the task ID; ``min_confidence`` on an - option is a bar for that branch alone. A downstream task without an entry is + option, which is experimental, is a bar for that branch alone. A downstream task + without an entry is presented by its ID alone and takes the policy's bar. A key that is not a downstream task ID fails the task before the model is called. Descriptions support Jinja templating. :param allow_multiple_branches: When ``False`` (default) the LLM returns a single task ID. When ``True`` the LLM may return one or more task IDs. - :param decision_policy: A :class:`~airflow.providers.common.ai.utils.decision.DecisionPolicy`: + :param decision_policy: Experimental. A + :class:`~airflow.providers.common.ai.policies.decision.DecisionPolicy`: the confidence a pick needs for the operator to branch on it without a person (``min_confidence``) and what happens under it (``on_uncertain``: ``"review"`` or ``"fail"``). Confidence comes from models that report one, such as a classifier diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py index fd78907049044..4bb91f454d51f 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_file_analysis.py @@ -38,6 +38,11 @@ class LLMFileAnalysisOperator(LLMOperator): """ Analyze files from object storage or local storage using a single LLM call. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The operator resolves ``file_path`` via :class:`~airflow.providers.common.compat.sdk.ObjectStoragePath`, normalizes supported formats into text context, and optionally attaches images/PDFs as diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_schema_compare.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_schema_compare.py index d82b3df2da011..cc2b07c853b95 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_schema_compare.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_schema_compare.py @@ -81,6 +81,11 @@ class LLMSchemaCompareOperator(LLMOperator): """ Compare schemas across different database systems and detect drift using LLM reasoning. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The LLM handles complex cross-system type mapping that simple equality checks miss (e.g., ``varchar(255)`` vs ``string``, ``timestamp`` vs ``timestamptz``). diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py index 4a5964e6e091c..847e3d1ee2608 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py @@ -50,6 +50,11 @@ class LLMSQLQueryOperator(LLMOperator): """ Generate SQL queries from natural language using an LLM. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Inherits from :class:`~airflow.providers.common.ai.operators.llm.LLMOperator` for LLM access and optionally uses a :class:`~airflow.providers.common.sql.hooks.sql.DbApiHook` diff --git a/providers/common/ai/src/airflow/providers/common/ai/policies/decision.py b/providers/common/ai/src/airflow/providers/common/ai/policies/decision.py index 83997f9b925e2..4e60b2bc2ade0 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/policies/decision.py +++ b/providers/common/ai/src/airflow/providers/common/ai/policies/decision.py @@ -49,6 +49,11 @@ class DecisionPolicy: """ When an LLM operator may act on the model's answer by itself, and what happens when it may not. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :param min_confidence: The confidence, from 0 to 1, the answer needs for the operator to act without a person. ``None`` (default) is no gate: the operator behaves as it always has. Confidence comes from models that report one, such as a classifier model (TypeSafe's), @@ -87,6 +92,12 @@ class BranchOption: """ One branch the model may pick: what choosing it means, and the confidence it needs. + .. note:: + + Experimental: ``min_confidence`` can change or be removed in a minor release of this + provider. ``description`` is stable. + See :ref:`howto/stability`. + The value of :class:`~airflow.providers.common.ai.operators.llm_branch.LLMBranchOperator`'s ``branches`` mapping, keyed by downstream task ID. A bare string in that mapping is shorthand for ``BranchOption(description=...)``. diff --git a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py index e2c0a0059ed56..f3bd80c9fd8d7 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py +++ b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py @@ -129,6 +129,11 @@ class ErrorCategory: """ One kind of failure a :class:`ClassifierRetryPolicy` may name, and what it does when it does. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The value of the policy's ``categories`` mapping, keyed by the category name the model answers with. @@ -429,6 +434,11 @@ class ClassifierRetryPolicy(_ModelRetryPolicy): """ Retry policy where the model names the kind of failure and the author's table decides. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The model's only job is to pick one of ``categories``; it reads each one's description from the output schema. Whether that category is retried, after how long, and how sure the model has to be all come from the :class:`ErrorCategory` in the worker process. diff --git a/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py b/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py index 235428b77c63a..0a69c54c48c8d 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py +++ b/providers/common/ai/src/airflow/providers/common/ai/sandbox/base.py @@ -115,6 +115,11 @@ class SandboxSpec: """ What a single sandbox should be provisioned with. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Passed to :meth:`SandboxBackend.create`. Every field is optional and a backend may not be able to honor all of them; a backend that cannot enforce a field it was given must raise rather than silently ignore it, so a DAG @@ -259,6 +264,11 @@ class SandboxBackend(ABC): """ Contract for running commands and file operations in an isolated sandbox. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + The lifecycle is create -> (any number of operations) -> destroy, driven by :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`. The four operation methods are named after the four tools the toolset @@ -439,6 +449,11 @@ class AttachableSandboxBackend(SandboxBackend): """ A backend whose sandboxes outlive the process that created them and can be found again. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + This is what lets one task provision a sandbox and a later agent task use it: the provisioning task stamps the sandbox with :attr:`SandboxSpec.owner`, the :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset` attaches diff --git a/providers/common/ai/src/airflow/providers/common/ai/sandbox/modal.py b/providers/common/ai/src/airflow/providers/common/ai/sandbox/modal.py index 3fae83d692633..c67ae08085715 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/sandbox/modal.py +++ b/providers/common/ai/src/airflow/providers/common/ai/sandbox/modal.py @@ -171,6 +171,11 @@ class ModalSandboxBackend(AttachableSandboxBackend): """ Sandbox backend that runs agent commands in a `Modal `__ sandbox. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Each sandbox is a gVisor-isolated container in Modal's infrastructure, provisioned over the API. Nothing has to be installed on the Airflow worker and model-written code never executes on the worker host, which makes this the backend to reach for diff --git a/providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py b/providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py index 4ce1290609639..07aaaed61d703 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py +++ b/providers/common/ai/src/airflow/providers/common/ai/sandbox/opensandbox.py @@ -159,6 +159,11 @@ class OpenSandboxBackend(SandboxBackend): """ Run sandbox tools through an OpenSandbox server. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + OpenSandbox supports Docker and Kubernetes runtimes behind the same API. Airflow workers need only network access to that API; the OpenSandbox deployment owns container provisioning and isolation. diff --git a/providers/common/ai/src/airflow/providers/common/ai/sandbox/sbx.py b/providers/common/ai/src/airflow/providers/common/ai/sandbox/sbx.py index 7292262c94d42..02e39430a7f7e 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/sandbox/sbx.py +++ b/providers/common/ai/src/airflow/providers/common/ai/sandbox/sbx.py @@ -65,6 +65,11 @@ class SbxSandboxBackend(SandboxBackend): """ Sandbox backend that runs agent commands in a Docker Sandboxes (``sbx``) microVM. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Drives the ``sbx`` CLI: ``create`` provisions a per-session microVM, ``exec`` runs commands in it, and ``rm`` tears it down. Each sandbox is a microVM with its own kernel, so agent code is isolated by a hardware boundary rather than a diff --git a/providers/common/ai/src/airflow/providers/common/ai/skills.py b/providers/common/ai/src/airflow/providers/common/ai/skills.py index 20238ec85115a..13c053313bc96 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/skills.py +++ b/providers/common/ai/src/airflow/providers/common/ai/skills.py @@ -59,6 +59,11 @@ class GitSkills: """ Agent Skills cloned from a Git repository when resolved. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + :param repo_url: HTTPS or SSH URL of the repository to clone. :param conn_id: Airflow ``git`` connection used for credentials, resolved through the Git provider's ``GitHook`` (HTTPS token in the connection diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py index cc771bda05a64..299d61fb9b7db 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py @@ -85,6 +85,11 @@ class DataFusionToolset(AbstractToolset[Any]): """ Curated toolset that gives an LLM agent SQL access to object-storage data via Apache DataFusion. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Provides three tools — ``list_tables``, ``get_schema``, and ``query`` — backed by :class:`~airflow.providers.common.sql.datafusion.engine.DataFusionEngine`. diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/langchain_bridge.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/langchain_bridge.py index 41ed44c281892..90120cf2ab07f 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/langchain_bridge.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/langchain_bridge.py @@ -54,6 +54,11 @@ def airflow_toolset_to_langchain_tools( """ Convert a pydantic-ai toolset into a list of LangChain ``StructuredTool`` objects. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Each returned tool is backed by ``toolset.call_tool`` and carries the ``args_schema`` derived from the tool's JSON schema, so a LangChain agent or chain can call it the same way it calls any native LangChain tool. diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/logging.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/logging.py index 3c325e4d66ee8..805f7cf32e2b1 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/logging.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/logging.py @@ -35,7 +35,14 @@ @dataclass class LoggingToolset(WrapperToolset[Any]): - """Wrap a toolset to log each tool call with timing.""" + """ + Wrap a toolset to log each tool call with timing. + + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + """ logger: Logger | logging.Logger = field(default_factory=lambda: logging.getLogger(__name__)) diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/managed_agent.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/managed_agent.py index e2e35c22b4de9..aaddb2b0fa9cc 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/managed_agent.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/managed_agent.py @@ -55,6 +55,11 @@ class BaseManagedAgentToolset(AbstractToolset[Any]): """ Base class exposing a vendor-managed agent as a single pydantic-ai tool. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + A managed agent runs its own reasoning loop on the vendor's infrastructure (Snowflake Cortex Agents, Amazon Bedrock AgentCore, Azure AI Foundry hosted agents, Vertex AI Agent Engine). Airflow submits one request and reads one @@ -242,6 +247,11 @@ class FailoverManagedAgentToolset(BaseManagedAgentToolset): """ Present several interchangeable managed agents to the model as one tool. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Active/passive failover for a managed agent: members are tried in order and the first answer wins. Because this is itself a :class:`BaseManagedAgentToolset`, the calling model sees a single tool and diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/sandbox.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/sandbox.py index d069f0f2c85e5..d0a92c20e1779 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/sandbox.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/sandbox.py @@ -144,6 +144,11 @@ class SandboxToolset(AbstractToolset[Any]): """ Give an agent shell and file access inside a disposable sandbox, off the Airflow worker. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Exposes four tools -- ``run_command``, ``read_file``, ``write_file`` and ``list_directory`` -- against a sandbox provisioned by the given :class:`~airflow.providers.common.ai.sandbox.SandboxBackend`. The same four diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/skills.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/skills.py index fdbf5933c8401..8dc1fb68ba955 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/skills.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/skills.py @@ -51,6 +51,11 @@ class AgentSkillsToolset(AbstractToolset): """ A pydantic-ai toolset that loads Agent Skills, with Git credentials from Airflow connections. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Sources are local directory paths and/or :class:`~airflow.providers.common.ai.skills.GitSkills`. diff --git a/providers/common/ai/src/airflow/providers/common/ai/triggers/llm_batch.py b/providers/common/ai/src/airflow/providers/common/ai/triggers/llm_batch.py index 8e019686355db..ee989cfbaa56b 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/triggers/llm_batch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/triggers/llm_batch.py @@ -34,6 +34,11 @@ class LLMBatchTrigger(BaseTrigger): """ Poll a batch adapter until the batch reaches a terminal state. + .. note:: + + Experimental: this can change or be removed in a minor release of this provider. + See :ref:`howto/stability`. + Deliberately thin: this trigger only polls and, on kill or timeout, cancels. It never downloads or validates results; that is :meth:`~airflow.providers.common.ai.operators.llm_batch.LLMBatchOperator.execute_complete`'s