Skip to content

Handling “Airflow confs” in Task SDK #44352

Open
Open
Handling “Airflow confs” in Task SDK#44352
Assignees
amoghrajesh
Labels
area:corearea:task-execution-interface-aip72AIP-72: Task Execution Interface (TEI) aka Task SDKpriority:mediumBug that should be fixed before next release but would not block a release
Milestone
Airflow 3.5.0

Description

@kaxil
kaxil
opened on Nov 25, 2024

Figure out how to retrieve Airflow configurations (from airflow.cfg) in the Task SDK without importing Airflow core.

Investigate different approaches for configuration architecture in separated server and SDK components, analyze trade-offs, and recommend the best approach.

Configuration Split Options to Investigate

Option 1: Complete Independence

  • Approach: Separate configuration files for server and SDK
  • Pros: Clean separation, no network dependencies
  • Cons: Configuration duplication, sync issues
  • Use case: When server and workers are in different environments

Option 2: API-Based Inheritance

  • Approach: SDK queries server API for configuration values
  • Pros: Centralized configuration management
  • Cons: Network dependency, potential latency
  • Use case: When workers need dynamic configuration updates

Option 3: Bootstrap Injection

  • Approach: Server injects essential config at task startup
  • Pros: No runtime API calls, centralized control
  • Cons: Static configuration, larger task payloads
  • Use case: When configuration is stable and performance is critical

Option 4: Hybrid Local-First + API Fallback

  • Approach: Local config first, API fallback for inherited values, caching
  • Pros: Performance + flexibility, graceful degradation
  • Cons: Complexity, cache invalidation challenges
  • Use case: Best of both worlds approach

## Open Questions

  • What about ProvidersManager class that is used to fetch configs from Providers

Activity

  1. kaxil
    added this to AIP-72 - Task Execution Interface and SDKon Nov 25, 2024
  2. kaxil
    converted this from a draft issue on Nov 25, 2024
  3. kaxil
    added
    area:task-execution-interface-aip72AIP-72: Task Execution Interface (TEI) aka Task SDK
    priority:mediumBug that should be fixed before next release but would not block a release
    on Nov 25, 2024
  4. dosubot
    added
    area:core
    on Nov 25, 2024
  5. kaxil
    self-assigned this
    on Nov 29, 2024

  6. kaxil commented on Dec 10, 2024

    @kaxil
    kaxil
    on Dec 10, 2024
    MemberAuthor

    @potiuk 's options here: https://lists.apache.org/thread/8jtgr1m072lgsy4tl3km9ptb216vqjds

    we have three
    possible options to tell our users:.

    1. either copy to workers only the configuration that is needed (but which
      ones are needed and how to separate those config entries in the docs?)
    2. or NO configuration should be present in the worker - and all
      configuration that is needed should be passed by Task SDK to the worker
    3. or we assess that task and dag file processors do not need any
      configuration at all

    I think if we assess that 3) is the case - there is no other option but to
    remove conf. That would also be really nice as we would not have to have
    "conf" reading code in TaskSDK at all.
    But if we think that tasks and dag parsing will actually need some
    configuration, then we can go either 1) or 2) route

    I think option 2) is much better from less confusion, security and "single
    source of truth" point of view. And if we go for 2) then namespace = conf.get("kubernetes", "NAMESPACE") will not work because configuration
    will not be present there and in this case what would make more sense is to
    leave "conf" as part of the context but only pass the "needed"
    configuration there.
    If we go for option 1) then likely yes we could remove it from context.

  7. potiuk commented on Dec 10, 2024

    @potiuk
    potiuk
    on Dec 10, 2024
    Member

    Thanks @kaxil - yeah, that's the right place to discuss it :)

  8. kaxil
    added 2 commits that reference this issue on Dec 12, 2024

    AIP-72: Add some basic Task Context keys

    4b759e4

    AIP-72: Add some basic Task Context keys

    0de709d
  9. kaxil
    removed their assignment
    on Jan 6, 2025
  10. kaxil
    moved this from Next 2 Weeks to Todo in AIP-72 - Task Execution Interface and SDKon Jan 6, 2025
  11. kaxil
    added this to the Airflow 3.0.0 milestone on Jan 17, 2025
  12. kaxil
    modified the milestones: Airflow 3.0.0, Airflow 3.1+ on Apr 11, 2025

  13. 7 remaining items

  14. amoghrajesh commented on Aug 25, 2025

    @amoghrajesh
    amoghrajesh
    on Aug 25, 2025
    Contributor

    @potiuk @kaxil

    Alright, nice discussions above. I would be in favour of Option 1: Complete Independence as I think it is the most practical and architecturally sound approach with Airflow3 architecture. Let me explain the fundamental problems that eliminate Options 2, 3, and 4 from consideration:

    • Option 2: API-Based Inheritance
      It creates network dependencies between the workers and the API server and now makes the workers running task sdk dependent on API server even for the most basic dag authoring experience, I do not know how we would be able to deal with operator and dag defaults here, not to mention it might turn into really huge payloads across the networks making a mark on performance of Airflow by having too many API calls and also by making "context" as a dumping groung.

    • Option 3: Bootstrap Injection
      Could potentially lead into slower bootstrapping for configs that need to be calculated and it still requires API calls and doesn't solve the DAG processor/triggerer problem. Another issue I see is static nature of injected configuration means no dynamic updates are possible, leading to inconsistency when configuration changes on the server side while tasks are running (could be an issue for things like execution url changing)

    • Option 4: Hybrid Local-First + API Fallback
      This is a decent option and my proposal would be more like a hybrid of option 4 and 1, but 4 in itself could be overly complex for very little benefit?

    I propose that we move the AirflowConfigParser (not configs) to a shared module, while keeping the actual configuration files and values separate and independent for each component.

    Each component now gets exactly what it needs - the shared parsing logic / abstraction to read and interpret configuration files, but with complete autonomy over what configuration values they actually contain.

  15. amoghrajesh commented on Aug 25, 2025

    @amoghrajesh
    amoghrajesh
    on Aug 25, 2025
    Contributor

    Performed a small analysis on the pattern of conf imports used by the providers, there are 400+ usages which are mostly either provider configs, or operator related (task sdk), the only ones (non tests) using core are:

    Config Key File Location Usage Purpose
    core.security providers/apache/hdfs/src/airflow/providers/apache/hdfs/hooks/webhdfs.py _kerberos_security_mode = conf.get("core", "security") == "kerberos" Kerberos authentication detection
    core.security providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py if conf.get("core", "security") == "kerberos": (4 occurrences) Kerberos authentication checks
    core.execution_api_server_url providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py server=conf.get("core", "execution_api_server_url", fallback=default_execution_api_server) Celery worker API communication
    core.execution_api_server_url providers/edge3/src/airflow/providers/edge3/cli/worker.py execution_api_server_url = conf.get("core", "execution_api_server_url", fallback="") Edge worker API endpoint
    core.internal_api_clock_grace providers/edge3/src/airflow/providers/edge3/worker_api/auth.py clock_grace = conf.getint("core", "internal_api_clock_grace", fallback=30) API authentication timing tolerance
    core.internal_api_secret_key providers/edge3/src/airflow/providers/edge3/worker_api/auth.py secret_key=conf.get("core", "internal_api_secret_key") API authentication secret
    core.PARALLELISM providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py PARALLELISM: int = conf.getint("core", "PARALLELISM") System-wide task parallelism limit
    core.dbapihook_resolve_target_fields providers/common/sql/src/airflow/providers/common/sql/hooks/sql.py _resolve_target_fields = conf.getboolean("core", "dbapihook_resolve_target_fields", fallback=False) SQL hook field resolution behavior
    core.enable_xcom_pickling providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/adx.py if conf.getboolean("core", "enable_xcom_pickling", fallback=False): XCom serialization method
    core.enable_xcom_pickling providers/microsoft/winrm/src/airflow/providers/microsoft/winrm/operators/winrm.py enable_pickling = conf.getboolean("core", "enable_xcom_pickling", fallback=False) XCom serialization method
    core.enable_xcom_pickling providers/ssh/src/airflow/providers/ssh/operators/ssh.py enable_pickling = conf.getboolean("core", "enable_xcom_pickling", fallback=False) XCom serialization method
    core.default_timezone providers/fab/src/airflow/providers/fab/www/extensions/init_jinja_globals.py server_timezone = conf.get("core", "default_timezone") Web UI timezone display

    Impact for Worker Configuration:

    All these core settings must be included in worker configuration files since providers running on workers depend on them for proper functionality.

  16. kaxil
    moved this from Next 2 Weeks to In Progress in AIP-72 - Task Execution Interface and SDKon Sep 16, 2025

  17. amoghrajesh commented on Nov 25, 2025

    @amoghrajesh
    amoghrajesh
    on Nov 25, 2025
    Contributor

    Update on this one:

    1. Move Airflow Config Parser to shared library #57744 has been merged. This PR achieves the client server separation with respect to conf for airflow 3.1.2.
    2. Further changes about providers consuming conf from core etc can come after 3.1.2.
  18. amoghrajesh
    removed a parent issue on Feb 17, 2026
  19. amoghrajesh
    added a parent issue on Feb 17, 2026
  20. amoghrajesh
    removed this from AIP-72 - Task Execution Interface and SDKon Feb 17, 2026
  21. amoghrajesh
    modified the milestones: Airflow 3.2.0, Airflow 3.3.0 on Mar 25, 2026

  22. amoghrajesh commented on Mar 25, 2026

    @amoghrajesh
    amoghrajesh
    on Mar 25, 2026
    Contributor

    This issue can be worked on 3.3+ when we try to decouple providers from airflow-core. For now its fine.

  23. amoghrajesh
    modified the milestones: Airflow 3.3.0, Airflow 3.4.0 on Jun 15, 2026

  24. amoghrajesh commented on Jun 15, 2026

    @amoghrajesh
    amoghrajesh
    on Jun 15, 2026
    Contributor

    Wasn't committed for Airflow 3.3, moving to Airflow 3.4 as best effort work.

  25. vatsrahul1001
    modified the milestones: Airflow 3.4.0, Airflow 3.5.0 on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:corearea:task-execution-interface-aip72AIP-72: Task Execution Interface (TEI) aka Task SDKpriority:mediumBug that should be fixed before next release but would not block a release

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions