Skip to content

Adding UIAlert in local settings fails with Sentry enabled #31442

Closed as not planned
Closed as not planned
Adding UIAlert in local settings fails with Sentry enabled#31442
Labels
Stale Bug Report_eol_affected_version:2.6Issues Reported for 2.6area:corekind:bugThis is a clearly a bug

Description

@potiuk
potiuk
opened on May 21, 2023

Discussed in #31408

Originally posted by gil-tober May 18, 2023

Apache Airflow version

2.6.1

What happened

When upgrading Airflow from 2.5.3 to 2.6.1 the airflow-run-migration job fails.

I have deployed the same setup to our DEV environment (where Sentry is disabled) and the upgrade succeeded without an issue.

After that I tried deploying to our PROD environment where Sentry is enabled and the job failed with the error below.

I tried enabling Sentry on DEV and the upgrade failed.

Took a look at airflow.executors.executor_loader.py at lines 162-185 when the exception is raised from. Setting the environment variable _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1 fixed the error getting in the upgrade job. (Our metadb is Postgres)

Traceback (most recent call last):
  File “/home/airflow/.local/bin/airflow”, line 5, in <module>
    from airflow.__main__ import main
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/__init__.py”, line 66, in <module>
    settings.initialize()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 522, in initialize
    import_local_settings()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py”, line 467, in import_local_settings
    import airflow_local_settings
  File “/opt/airflow/dags/repo/airflow_local_settings.py”, line 5, in <module>
    from airflow.www.utils import UIAlert
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/www/utils.py”, line 44, in <module>
    from airflow.models.dagrun import DagRun
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/dagrun.py”, line 57, in <module>
    from airflow.models.taskinstance import TaskInstance as TI
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py”, line 100, in <module>
    from airflow.sentry import Sentry
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 195, in <module>
    Sentry = ConfiguredSentry()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/sentry.py”, line 92, in __init__
    executor_class, _ = ExecutorLoader.import_default_executor_cls()
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 158, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 134, in import_executor_cls
    return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 130, in _import_and_validate
    cls.validate_database_executor_compatibility(executor)
  File “/home/airflow/.local/lib/python3.10/site-packages/airflow/executors/executor_loader.py”, line 181, in validate_database_executor_compatibility
    from airflow.settings import engine
ImportError: cannot import name ‘engine’ from ‘airflow.settings’ (/home/airflow/.local/lib/python3.10/site-packages/airflow/settings.py)

What you think should happen instead

migrateDatabaseJob should not fail when Sentry is enabled

How to reproduce

Upgrade from 2.5.3 to 2.6.1 with CeleryKubernetesExecutor and Sentry enabled

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-docker==3.6.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-jenkins==3.2.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-postgres==5.3.1
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-salesforce==5.3.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0
apache-airflow-providers-tableau==4.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow deployed on an EKS cluster using the official Airflow Helm Chart (V1.9.0)
MetaBD is a Postgres RDS
CeleryKubernetesExecutor

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. potiuk
    added
    kind:bugThis is a clearly a bug
    pending-response
    area:core
    _eol_affected_version:2.6Issues Reported for 2.6
    on May 21, 2023

  2. potiuk commented on May 21, 2023

    @potiuk
    potiuk
    on May 21, 2023
    MemberAuthor

    comment: Seems that indeed we have a circular import happening when UIAlert is created in local_settins.

  3. potiuk
    added
    good first issue
    and removed
    pending-response
    on May 21, 2023
  4. potiuk
    added this to the Airflow 2.6.2 milestone on May 21, 2023

  5. potiuk commented on May 21, 2023

    @potiuk
    potiuk
    on May 21, 2023
    MemberAuthor

    Very interesting bug :) .

    The root cause is AIP-51 related change #28375 which added executor initialization in Sentry's ConfiguredSentry() __init__ class. This is done only in order to see if the executor supports sentry.

                if executor_class.supports_sentry:
                from sentry_sdk.integrations.celery import CeleryIntegration

                sentry_celery = CeleryIntegration()
                integrations.append(sentry_celery)

    It's an interesting sequence of events:

    • local settings are being imported
    • UIAlertView is created in the local settings
    • while creating it, taskinstace class is imported
    • which in turn initializes Sentry
    • which instantiates the executor class
    • which needs settings (but settings are not fully initialized yet so we raise the exception).

    @o-nikolas @rkarish you might want to take a look and do some other way of checking if the executor we have supports sentry - without necessarily initializing it. This might be tricky.

  6. potiuk commented on May 21, 2023

    @potiuk
    potiuk
    on May 21, 2023
    MemberAuthor

    BTW. Initializing UI Alert view in local settings is perfectly fine and recommended by us in case anyone asked.

  7. potiuk
    removed
    good first issue
    on May 21, 2023

  8. potiuk commented on May 21, 2023

    @potiuk
    potiuk
    on May 21, 2023
    MemberAuthor

    cc: @gil-tober who created the original issue.

    As discussed in #31408 (reply in thread) - maybe a good solution could be utilizing listeners on_starting to create the UIAlertView after the initialization is complete. I don't think we want to make it the "only" way though - local_settings is a more "natural" way of doing it, so possibly fixing this circular import might be a good idea anyway.

  9. gilt-cl commented on May 21, 2023

    @gilt-cl
    gilt-cl
    on May 21, 2023

    This is mentioned in #31408
    Adding it here for anyone else having this issue

    I will try the workaround you suggested and will post my findings.

    For now, I found that setting _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1 also works as workaround.

    airflow/airflow/executors/executor_loader.py

    Lines 160 to 186 in 9137740

    @classmethod
    @functools.lru_cache(maxsize=None)
    def validate_database_executor_compatibility(cls, executor: type[BaseExecutor]) -> None:
    """Validate database and executor compatibility.
    Most of the databases work universally, but SQLite can only work with
    single-threaded executors (e.g. Sequential).
    This is NOT done in ``airflow.configuration`` (when configuration is
    initialized) because loading the executor class is heavy work we want to
    avoid unless needed.
    """
    # Single threaded executors can run with any backend.
    if executor.is_single_threaded:
    return
    # This is set in tests when we want to be able to use SQLite.
    if os.environ.get("_AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK") == "1":
    return
    from airflow.settings import engine
    # SQLite only works with single threaded executors
    if engine.dialect.name == "sqlite":
    raise AirflowConfigException(f"error: cannot use SQLite with the {executor.__name__}")

    Our metadb is Postgres, so the check for sqlite is not relevant for us

  10. eladkal
    removed this from the Airflow 2.6.2 milestone on Jun 11, 2023

  11. tanvn commented on Apr 8, 2024

    @tanvn
    tanvn
    on Apr 8, 2024
    Contributor

    I faced a similar issue today with Airflow 2.8.4 (Db migration worked successfully but later, the scheduler and webserver can not start and keep crashing due to the error below)

    Our environment:

    • Airflow 2.8.4
    • KubernetesExecutor
    • MySQL 8
    • Sentry enabled
    • airflowLocalSettings in our helm-chart values.yaml contains from airflow.www.utils import UIAlert

    Error log from the scheduler

    ERROR! Maximum number of retries (20) reached.

Last check result:
$ airflow db check
Failed to import airflow_local_settings.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 500, in import_local_settings
    import airflow_local_settings
  File "/opt/airflow/config/airflow_local_settings.py", line 2, in <module>
    from airflow.www.utils import UIAlert
  File "/usr/local/lib/python3.10/site-packages/airflow/www/utils.py", line 44, in <module>
    from airflow.models.dagrun import DagRun
  File "/usr/local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 59, in <module>
    from airflow.models.taskinstance import TaskInstance as TI
  File "/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 99, in <module>
    from airflow.sentry import Sentry
  File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 198, in <module>
    Sentry = ConfiguredSentry()
  File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 90, in __init__
    executor_class, _ = ExecutorLoader.import_default_executor_cls(validate=False)
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 167, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name, validate=validate)
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 141, in import_executor_cls
    return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 135, in _import_and_validate
    executor = import_string(path)
  File "/usr/local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 40, in <module>
    from airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException, PodReconciliationError
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator.py", line 50, in <module>
    from airflow.providers.cncf.kubernetes.pod_generator_deprecated import (
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator_deprecated.py", line 33, in <module>
    from airflow.utils.hashlib_wrapper import md5
  File "/usr/local/lib/python3.10/site-packages/airflow/utils/hashlib_wrapper.py", line 26, in <module>
    from airflow import PY39
ImportError: cannot import name 'PY39' from partially initialized module 'airflow' (most likely due to a circular import) (/usr/local/lib/python3.10/site-packages/airflow/__init__.py)
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.10/site-packages/airflow/__init__.py", line 68, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 552, in initialize
    import_local_settings()
  File "/usr/local/lib/python3.10/site-packages/airflow/settings.py", line 500, in import_local_settings
    import airflow_local_settings
  File "/opt/airflow/config/airflow_local_settings.py", line 2, in <module>
    from airflow.www.utils import UIAlert
  File "/usr/local/lib/python3.10/site-packages/airflow/www/utils.py", line 44, in <module>
    from airflow.models.dagrun import DagRun
  File "/usr/local/lib/python3.10/site-packages/airflow/models/dagrun.py", line 59, in <module>
    from airflow.models.taskinstance import TaskInstance as TI
  File "/usr/local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 99, in <module>
    from airflow.sentry import Sentry
  File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 198, in <module>
    Sentry = ConfiguredSentry()
  File "/usr/local/lib/python3.10/site-packages/airflow/sentry.py", line 90, in __init__
    executor_class, _ = ExecutorLoader.import_default_executor_cls(validate=False)
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 167, in import_default_executor_cls
    executor, source = cls.import_executor_cls(executor_name, validate=validate)
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 141, in import_executor_cls
    return _import_and_validate(cls.executors[executor_name]), ConnectorSource.CORE
  File "/usr/local/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 135, in _import_and_validate
    executor = import_string(path)
  File "/usr/local/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 39, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py", line 40, in <module>
    from airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException, PodReconciliationError
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator.py", line 50, in <module>
    from airflow.providers.cncf.kubernetes.pod_generator_deprecated import (
  File "/usr/local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/pod_generator_deprecated.py", line 33, in <module>
    from airflow.utils.hashlib_wrapper import md5
  File "/usr/local/lib/python3.10/site-packages/airflow/utils/hashlib_wrapper.py", line 26, in <module>
    from airflow import PY39
ImportError: cannot import name 'PY39' from partially initialized module 'airflow' (most likely due to a circular import) (/usr/local/lib/python3.10/site-packages/airflow/__init__.py)

    Setting _AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK=1 does not work.

  12. potiuk
    added a commit that references this issue on Apr 16, 2024

    Moves airflow import in deprecated pod_generator to local

    ef711b2

  13. potiuk commented on Apr 16, 2024

    @potiuk
    potiuk
    on Apr 16, 2024
    MemberAuthor

    I faced a similar issue today with Airflow 2.8.4 (Db migration worked successfully but later, the scheduler and webserver can not start and keep crashing due to the error below)

    This is a different issue and likely to be addressed by #39062 - which should be available in the next cncf.kubernetes after merging.

  14. potiuk
    added a commit that references this issue on Apr 16, 2024

    Moves airflow import in deprecated pod_generator to local (#39062)

    b59cef1
  15. grrolland
    added a commit that references this issue on Apr 19, 2024

    Moves airflow import in deprecated pod_generator to local (apache#39062)

    d8b4f23
  16. Taragolis
    added a commit that references this issue on May 6, 2024

    Fix SFTPSensor.newer_than not working with jinja logical ds/ts expres…

    8965f2e
  17. pateash
    added a commit that references this issue on May 13, 2024

    Fix SFTPSensor.newer_than not working with jinja logical ds/ts expres…

    105979e

  18. github-actions commented on Apr 17, 2025

    @github-actions
    github-actionsbot
    on Apr 17, 2025 – with GitHub Actions
    Contributor

    This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.

  19. github-actions
    added
    Stale Bug Report
    on Apr 17, 2025

  20. github-actions commented on Jun 2, 2025

    @github-actions
    github-actionsbot
    on Jun 2, 2025 – with GitHub Actions
    Contributor

    This issue has been closed because it has not received response from the issue author.

  21. github-actions
    closed this as not plannedon Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stale Bug Report_eol_affected_version:2.6Issues Reported for 2.6area:corekind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions