Skip to content

Re-organize repo into multiple projects using uv's workspaces "core" (scheduler + web/api servers), "providers" (a single project for all providers for now), "task-sdk" (new dist for this project) #42857

Description

@kaxil

List discussion https://lists.apache.org/thread/dyv5jhvt65xs6l5o2byc2b67f4wlwf6r

We will try out uv's workspace feature for this.

The projects we want are:

  • "core", containing the scheduler, web/api servers, dag parser etc.
  • "providers", a single project containing all the provivders
  • "task-sdk", the new dist that is key to this AIP.

There will be other top level folders such as the helm chart etc.

Tests should probably be split to live alongside the projects.

Example code layout (picking a few representative files in each group):

core/pyproject.toml/
core/src/airflow/__init__.py
core/src/airflow/models/…
core/src/airflow/api_fastapi/…
core/src/airflow/ui/package.json # React UI etc.
core/tests/jobs/test_scheduler_job.py
providers/src/airflow/providers/celery/provider.yaml
providers/src/airflow/providers/celery/__init__.py
providers/tests/airflow/providers/celery/executors/test_celery_executor.py
providers/src/airflow/providers/cncf/kubernetes/provuder.yaml
providers/src/airflow/providers/cncf/kubernetes/__init__.py
task-sdk/pyproject.toml
task-sdk/src/airflow/sdk/__init__.py
task-sdk/src/airflow/sdk/defintions/dag.py
task-sdk/src/airflow/sdk/defintions/task_group.py
task-sdk/tests/airflow/sdk/defintions/test_task_group.py

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions