Repository navigation
TypeError when importing operators from airflow.providers.google.cloud.operators.dataproc #46357
Description
Apache Airflow Provider(s)
Versions of Apache Airflow Providers
12.0.0
Apache Airflow version
2.10.4
Operating System
Ubuntu 24.04.1 LTS
Deployment
Virtualenv installation
Deployment details
No response
What happened
When importing any of the existing operators in airflow.providers.google.cloud.operators.dataproc (for example DataprocCreateClusterOperator, DataprocDeleteClusterOperator or DataprocSubmitJobOperator) at the beginning of a dag, a dag import error occurs when starting the airflow scheduler.
What you think should happen instead
When I start the airflow scheduler the following error occurs:
[2025-02-02T19:12:46.029+0100] {logging_mixin.py:190} INFO - [2025-02-02T19:12:46.028+0100] {dagbag.py:387} ERROR - Failed to import: /home/iscipar/Projects/airflow-tutorial/dags/6_dataproc_airflow.py
Traceback (most recent call last):
File "/home/iscipar/Proyectos/airflow-tutorial/airflow_env/lib/python3.12/site-packages/airflow/models/dagbag.py", line 383, in parse
loader.exec_module(new_module)
File "", line 995, in exec_module
File "", line 488, in _call_with_frames_removed
File "/home/iscipar/Proyectos/airflow-tutorial/dags/6_dataproc_airflow.py", line 5, in
from airflow.providers.google.cloud.operators.dataproc import (
File "/home/iscipar/Proyectos/airflow-tutorial/airflow_env/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 57, in
from airflow.providers.google.cloud.openlineage.utils import (
File "/home/iscipar/Proyectos/airflow-tutorial/airflow_env/lib/python3.12/site-packages/airflow/providers/google/cloud/openlineage/utils.py", line 204, in
class BigQueryJobRunFacet(RunFacet):
TypeError: function() argument 'code' must be code, not str
How to reproduce
The error is reproduced simply by adding the following import inside the code of a dag without importing the source code of the tasks of said dag:
from airflow.providers.google.cloud.operators.dataproc import (
DataprocCreateClusterOperator,
DataprocDeleteClusterOperator,
DataprocSubmitJobOperator,
)
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Feb 2, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
mohamedmeqlad99 commentedon Feb 2, 2025
I think that Airflow's Google provider package is using code that is incompatible with Python 3.12. This is likely due to changes in Python 3.12 affecting the exec() function or types.CodeType
I suggest Downgrade to Python 3.10 or 3.11
Fix TypeError when importing operators from airflow.providers.google.…
iscipar commentedon Feb 3, 2025
Hi.
This is my first time opening an issue in this community.
I see that there are two solutions:
As I prefer the second option, would a new version of the package with the fix be available now or do I have to wait? What npm install command would I have to run?
Thanks,