Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 49 additions & 32 deletions airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,31 +22,40 @@ Non-Python Task SDKs

|experimental|

Airflow Dags are always defined in Python, but individual task *implementations* can be written in other
languages. When a task runs, Airflow's worker calls out the target language to execute the task logic, which
communicates the result back. The Dag author uses a lightweight Python *stub* to declare where the task lives;
everything else, including the actual business logic, Airflow API calls, and any library dependencies, lives
in a non-Python implementation.
Airflow Dags and tasks can be written in languages other than Python. A language SDK is used in one of two
ways:

* **Dag definition.** The whole Dag, with its schedule, tasks and dependencies, is defined in the SDK's language.
See :ref:`language-sdks/dag-definition`.
* **Python Dag with stub TaskHandler.** A Python Dag declares the tasks and their dependencies, and a
TaskHandler in the SDK's language implements each stub task. See :ref:`language-sdks/stub-taskhandler`.

Either way, a task runs on a *coordinator*, which starts the language's runtime for the task and relays
messages between it and Airflow. See :ref:`language-sdks/coordinator-config`.

.. list-table:: Available language SDKs
:header-rows: 1
:widths: 15 40 15 30
:widths: 15 35 15 15 20

* - Language
- Coordinator class
- Min. runtime
- Dag definition
- Guide
* - JVM languages (e.g. Java)
- :class:`task-sdk:airflow.sdk.coordinators.java.JavaCoordinator`
- JRE 17
- Yes
- :doc:`java`
* - Go
- :class:`task-sdk:airflow.sdk.coordinators.executable.ExecutableCoordinator`
- None (native binary)
- Yes
- :doc:`go`
* - TypeScript
- :class:`task-sdk:airflow.sdk.coordinators.node.NodeCoordinator`
- Node.js 22
- Yes
- :doc:`typescript`

.. toctree::
Expand All @@ -56,36 +65,38 @@ in a non-Python implementation.
go
typescript

How it works
------------
.. _language-sdks/dag-definition:

Dag definition
--------------

The whole Dag is declared in the SDK's language: its schedule, its tasks, their options and the dependencies
between them. No Python file is involved.

The execution model has three moving parts.
The SDK builds the Dag into an artifact, and the artifact goes into a Dag bundle like any other Dag file. The
Dag processor runs the artifact with the language's runtime to read its Dags, so the Dag processor needs that
runtime and the ``[sdk]`` coordinator configuration. From there, Airflow treats the Dag like any Python Dag:
the scheduler plans its runs, it shows up in the UI, and each task runs on the coordinator its queue maps to,
from the same Dag bundle.

**Stub tasks in the Dag**
The Dag file declares tasks using :func:`@task.stub <airflow.sdk.task.stub>`. A stub is a normal Airflow
task from the scheduler's perspective. It participates in dependencies, retries, pools, and all other
task-level features exactly like other ``@task``-decorated Python functions. The only difference is that
the worker does not execute the Python code inside the function definition; instead it delegates execution
to a *coordinator*.
The **Code** view in the Airflow UI shows the source the SDK embeds in the artifact.

**Coordinators**
A coordinator is a Python object registered in the ``[sdk] coordinators`` configuration. This is considered
a part of an Airflow worker. When the worker picks up a stub task, it looks up the coordinator mapped to
that task's specified ``queue``, and uses the coordinator to execute the task. The coordinator is
responsible for managing the target language's runtime, forwarding messages from, and relaying results back
to Airflow. All coordinators extend
:class:`task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator`.
For TypeScript, see :ref:`typescript-sdk/dag-definition`.

**Language runtime**
The coordinator calls out one short-lived runtime per task instance. In most cases, this would be a
subprocess of an executable implemented in a non-Python language. The runtime receives messages from the
worker to identify the workload, executes the task, and communicates through the coordinator as a proxy
back to the worker process.
.. _language-sdks/stub-taskhandler:

Python Dag with stub TaskHandler
--------------------------------

A Python Dag declares a task that another language implements as a *stub task*. The scheduler sees a normal
task: it takes part in dependencies, retries, pools and every other task-level feature like any other
``@task`` function. When a worker picks it up, the worker does not run the Python function body; it hands the
task to the coordinator its queue maps to, which runs the implementation in the target language.

.. _language-sdks/stub-tasks:

Stub tasks
----------
~~~~~~~~~~

A stub task is declared with the :func:`@task.stub <airflow.sdk.task.stub>` decorator. Since it is still a
Python task declaration, every parameter available on a normal Dag or task applies. Task dependencies are also
Expand Down Expand Up @@ -129,16 +140,22 @@ documentation on how to do this correctly.

.. note::

For a Dag containing stub tasks, the **Code** view in the Airflow UI shows only the Python Dag
file — including the stub declarations — as the Dag's source. The non-Python implementation source
is not displayed anywhere in the UI; consult your project repository or the build artifact shipped
in the bundle to inspect it. This is an intentional architecture decision, not a bug.
For a Dag containing stub tasks, the **Code** view in the Airflow UI shows only the Python Dag file,
including the stub declarations, as the Dag's source. The non-Python implementation source is not
displayed anywhere in the UI; consult your project repository or the build artifact shipped in the bundle
to inspect it. This is an intentional architecture decision, not a bug.

.. _language-sdks/coordinator-config:

Coordinator configuration
-------------------------

A coordinator is a Python object registered in the ``[sdk] coordinators`` configuration, and runs as part of an
Airflow worker. When the worker picks up a task, it looks up the coordinator mapped to the task's ``queue``,
and uses it to run the task. The coordinator starts one short-lived runtime per task instance, usually a
subprocess of an executable written in the target language, and relays messages between that runtime and the
worker. All coordinators extend :class:`task-sdk:airflow.sdk.execution_time.coordinator.BaseCoordinator`.

Coordinators are registered in ``airflow.cfg`` (or via environment variables) under ``[sdk]``.

``coordinators``
Expand Down
Loading