SQLAlchemy / Pandas incompatibility

Apache Airflow version

2.8.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After upgrading to Airflow 2.8.2 one of our DAGs started failing. It seems to come from updating Pandas to 2.2.1 while SQLAlchemy is still on 1.4.X. According to Pandas docs v2.2.1 needs at least 2.0.

https://pandas.pydata.org/docs/whatsnew/v2.2.0.html#increased-minimum-versions-for-dependencies

The issue we get is from within pandas

File "./virtualenvs/airflow/lib/python3.11/site-packages/pandas/io/sql.py", line 2670, in execute
cur = self.con.cursor()
^^^^^^^^^^^^^^^
AttributeError: 'Connection' object has no attribute 'cursor'

What you think should happen instead?

No response

How to reproduce

Upgrade to 2.8.2 using the Constraint File for Python 3.11

Operating System

Ubuntu 22.04.4 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

pandas-dev/pandas#57049

@aa-matthias

We updated constraints, and re-built the images. When users will be using reference image or install airflow 2.8.2 following our recommendation (with constraints https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html ) - they should get Pandas 2.1.4 when installing Airflow 2.8.2. Thansk @aa-matthias for notifying us! 1

@potiuk

@potiuk - looks like the issue reappeared in Airflow 2.8.3 ?

@potiuk

@potiuk - looks like the issue reappeared in Airflow 2.8.3 ?

Can you please elaborate how it reappeared? What makes you think so ?

Can you please elaborate how it reappeared? What makes you think so ?

We exclude #37748 from 2.8.3 and originally I did not add this tests #37779 as 2.8.3 milestone ¯\_(ツ)_/¯.
The things happen, just need to think how we might prevent this into the future.

So for Python > 3.8 we generate constraints, build docker image with pandas greater or equal than 2.2

Let me see - I can check and regenerate the images again. I think the way to do it in the future is to "remember" :) . There is no other way

@potiuk

@potiuk

The constraint is no more and it's breaking pandas x SQLAlchemy in Airflow 3 right now.

Now

df.to_sql(
    "table",
    con=engine,
)

... gives me:

AttributeError: 'Engine' object has no attribute 'cursor'

$ docker run --rm --pull=always --entrypoint='' 'apache/airflow:3.0.3-python3.12' bash -c 'pip list --format=freeze' | grep -Ei '^(pandas|sqlalchemy)=='
3.0.3-python3.12: Pulling from apache/airflow
Digest: sha256:66d35503f78e8c87de2db43f0a3311efd26a5d34584599509a8bd90fb308adc7
Status: Image is up to date for apache/airflow:3.0.3-python3.12
pandas==2.2.3
SQLAlchemy==1.4.54

PS: When SQLAlchemy<2, pandas<2.2 is a must.

Refs:

BUG: Pandas 2.2 breaks SQLAlchemy 1.4 compatibility pandas-dev/pandas#57049

The constraint is no more

Not sure what it means

Breaking pandas x SQLAlchemy in Airflow 3 right now.

You can upgrade things independently from constraints. But this has been fixed in #53535 and new constraints are coming in 3.0.4 - right after we release current wave of providers that is being voted.

The constraint is no more

Not sure what it means

I meant these

BTW. The new providers are released - if you - for example try to install latest common.sql provider - it will not allow to upgrade pandas beyond what is supported.

The constraints-3-0 generated today (and those will be used for 3.0.4) contain this:

As you can see - pandas is limited in constraints to 2.1.4 now.

@potiuk

@potiuk, what pandas version should be used with Airflow 3.1.7 (latest) and Python 3.12? I tried the one from constraints (https://raw.githubusercontent.com/apache/airflow/constraints-3.1.7/constraints-3.12.txt), i.d. pandas=2.3.3, but had the same error (AttributeError: 'Connection' object has no attribute 'cursor').
Everything works with pandas=2.1.4 🤔, although I'm afraid that this version is old.

Did you install everything with constraints ? Including providers and other reps or you installed things independently ? If independently then sometimes it might be trial and error because some dependencies might not work with others. I recommend to install airflow following our installation guide (with everything you need with constraints) and then downgrade or upgrade any other dependencies as needed and test. This is exactly what our recommendations at installation page are. If you do it differently, there are too many variables to help you and give you concrete answers for potential incompatibilities between other dependencies

Likely you have some sqlalchemy version that conflicts

@potiuk

@potiuk, thank you for the fast answer!

Did you install everything with constraints ?

Yes, I did, but I used uv add instead of uv pip install (as the installation guide recommends) since the latter doesn't update pyproject.toml which I need to maintain for running tests. I noticed that the constraints state SQLAlchemy==2.0.46, whereas I have sqlalchemy==1.4.54 in my test env. So, uv add doesn't work correctly for some reason 😢.

UPD:
Fixed the versions: uv add "sqlalchemy==2.0.46" "pandas==2.3.3", everything works now.

The constraints are only active in the command they are used. This is how constraints work. If you do uv add or whatever without constraints they can feely upgrade or downgrade other deps. And even If you specify constraints and install only one or two packages - those constraints only apply to things being explicitly installed, if uv decides to downgrade packages that are not specified in the installation command, they do not follow constraints. If you notice that uv downgrades some package with add and you want to keep it, you need to add this package explicitly in your add command I guess

Navigation Menu

Repository navigation

SQLAlchemy / Pandas incompatibility #37742

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SQLAlchemy / Pandas incompatibility #37742

Description

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

aa-matthias commented on Feb 27, 2024

potiuk commented on Feb 27, 2024

aa-matthias commented on Mar 12, 2024

potiuk commented on Mar 12, 2024

Taragolis commented on Mar 12, 2024

Taragolis commented on Mar 12, 2024

potiuk commented on Mar 12, 2024

lucasfcnunes commented on Aug 1, 2025

potiuk commented on Aug 1, 2025

lucasfcnunes commented on Aug 2, 2025

potiuk commented on Aug 3, 2025

TopCoder2K commented on Feb 25, 2026

potiuk commented on Feb 25, 2026

potiuk commented on Feb 25, 2026

TopCoder2K commented on Feb 25, 2026

potiuk commented on Feb 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions