airflow db clean crashes with ForeignKeyViolation when deleting dag_version in Airflow 3.1.8 #63703

You have requested that we purge all data prior to 2026-01-01 00:00:00+00:00 for tables ['_xcom_archive', 'asset_event', 'callback_request', 'celery_taskmeta', 'celery_tasksetmeta', 'dag', 'dag_run', 'dag_version', 'deadline', 'import_error', 'job', 'log', 'sla_miss', 'task_instance', 'task_instance_history', 'task_reschedule', 'trigger', 'xcom'].
This is irreversible.  Consider backing up the tables first and / or doing a dry run with option --dry-run.
Enter 'delete rows' (without quotes) to proceed.
delete rows

Checking table import_error
Found 0 rows meeting deletion criteria.

Checking table task_instance_history
Found 0 rows meeting deletion criteria.

Checking table xcom
Found 98 rows meeting deletion criteria.
Performing Delete...
Moving data to table _airflow_deleted__xcom__20260316091330
Finished Performing Delete

Checking table task_instance
Found 43572 rows meeting deletion criteria.
Performing Delete...
Moving data to table _airflow_deleted__task_instance__20260316091330
Finished Performing Delete

Checking table deadline
Found 0 rows meeting deletion criteria.

Checking table dag_run
Found 38877 rows meeting deletion criteria.
Performing Delete...
Moving data to table _airflow_deleted__dag_run__20260316091343
Finished Performing Delete

Checking table dag_version
Found 8 rows meeting deletion criteria.
Performing Delete...
Moving data to table _airflow_deleted__dag_version__20260316091351
Traceback (most recent call last):
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.ForeignKeyViolation: update or delete on table "dag_version" violates foreign key constraint "task_instance_dag_version_id_fkey" on table "task_instance"
DETAIL:  Key (id)=(019a828c-a2ec-7700-bad6-725dd65b53a1) is still referenced from table "task_instance".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/sbin/airflow", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/local/lib/python3.13/site-packages/airflow/__main__.py", line 55, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.13/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/cli.py", line 114, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/airflow/cli/commands/db_command.py", line 296, in cleanup_tables
    run_cleanup(
    ~~~~~~~~~~~^
        table_names=args.tables,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        batch_size=args.batch_size,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/db_cleanup.py", line 541, in run_cleanup
    _cleanup_table(
    ~~~~~~~~~~~~~~^
        clean_before_timestamp=clean_before_timestamp,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        batch_size=batch_size,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/db_cleanup.py", line 372, in _cleanup_table
    _do_delete(
    ~~~~~~~~~~^
        query=query,
        ^^^^^^^^^^^^
    ...<3 lines>...
        batch_size=batch_size,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/db_cleanup.py", line 262, in _do_delete
    raise e
  File "/usr/local/lib/python3.13/site-packages/airflow/utils/db_cleanup.py", line 258, in _do_delete
    session.execute(delete)
    ~~~~~~~~~~~~~~~^^^^^^^^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
           ~~~~~~~~~~~~~~~~~~~~~~^
        statement,
        ^^^^^^^^^^
    ...<4 lines>...
        _add_event=_add_event,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/orm/session.py", line 2258, in _execute_internal
    result = conn.execute(
        statement, params or {}, execution_options=execution_options
    )
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
        self,
        distilled_parameters,
        execution_options or NO_OPTIONS,
    )
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self, distilled_params, execution_options
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
        dialect,
    ...<8 lines>...
        cache_hit=cache_hit,
    )
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        dialect, context, statement, parameters
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        e, str_statement, effective_parameters, cursor, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib64/python3.13/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table "dag_version" violates foreign key constraint "task_instance_dag_version_id_fkey" on table "task_instance"
DETAIL:  Key (id)=(019a828c-a2ec-7700-bad6-725dd65b53a1) is still referenced from table "task_instance".

[SQL: DELETE FROM dag_version USING _airflow_deleted__dag_version__20260316091351 WHERE dag_version.id = _airflow_deleted__dag_version__20260316091351.id]
(Background on this error at: https://sqlalche.me/e/20/gkpj)