SDK conf does not resolve provider metadata config defaults (has_option returns False) #64201

Apache Airflow version

3.1.8

If "Other Airflow 3 version" selected, which one?

No response

What happened?

airflow.sdk.configuration.conf does not resolve config options defined in provider metadata (provider.yaml). For example, conf.has_option("celery", "pool") returns False on the SDK conf, while the core conf correctly returns True.

This was discovered during the #60000 conf import migration. celery_command.py uses conf.has_option("celery", "pool") to decide whether to pass --pool to the Celery worker. After switching to the compat SDK conf, this returns False and the --pool prefork argument is silently dropped.

From reading the code, this may be caused by a circular dependency in the lookup sequence. When has_option() triggers _get_option_from_provider_metadata_config_fallbacks (parser.py:318), the cached property _provider_metadata_config_fallback_default_values (parser.py:342) calls ProvidersManagerTaskRuntime().provider_configs, which in turn calls conf.load_providers_configuration(). load_providers_configuration() uses already_initialized_provider_configs (parser.py:1202) which may return an empty dict if initialization hasn't completed yet.

What you think should happen instead?

sdk_conf.has_option("celery", "pool") should return True, consistent with core_conf.has_option("celery", "pool").

How to reproduce

from airflow.configuration import conf as core_conf
from airflow.sdk.configuration import conf as sdk_conf

print("core:", core_conf.has_option("celery", "pool"))  # True
print("sdk:", sdk_conf.has_option("celery", "pool"))    # False — expected True

celery.pool is defined in providers/celery/provider.yaml:287 with default prefork.

Operating System

Breeze (Debian 12)

Versions of Apache Airflow Providers

Apache Airflow main branch.

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct