Repository navigation
airflow standalone deployment is unable to run DockerOperator tasks on MacOS #28487
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Airflow version 2.4.3 running on MacOS, running a DAG using DockerOperators fails in combination with a local standalone deployment:
*** Reading local file: /Users/schustmi/airflow/logs/dag_id=test/run_id=manual__2022-12-20T09:42:32.767926+00:00/task_id=docker_task/attempt=1.log
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1383} INFO - Executing <Task(DockerOperator): docker_task> on 2022-12-20 09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:55} INFO - Started process 98208 to run task
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test', 'docker_task', 'manual__2022-12-20T09:42:32.767926+00:00', '--job-id', '17', '--raw', '--subdir', 'DAGS_FOLDER/dag.py', '--cfg-path', '/var/folders/45/1tkl8h1d3tvf2q72t8p6bw5r0000gn/T/tmpe6h4cv9f']
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:83} INFO - Job 17: Subtask o
[2022-12-20, 09:42:35 UTC] {task_command.py:376} INFO - Running <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [running]> on host schustmi-mac-work.fritz.box
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test
AIRFLOW_CTX_TASK_ID=docker_task
AIRFLOW_CTX_EXECUTION_DATE=2022-12-20T09:42:32.767926+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-12-20T09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-12-20, 09:42:35 UTC] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
What you think should happen instead
On other operating systems, the same combination of standalone deployment and DockerOperators works.
How to reproduce
pip install apache-airflow==2.4.3 apache-airflow-providers-docker==3.3.0
airflow standaloneThen run the following DAG:
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime
with DAG(dag_id="test", schedule="@once", start_date=datetime.utcnow()) as dag:
docker_task = DockerOperator(image="alpine:latest", command="echo test", task_id="docker_task")Operating System
MacOS Ventura 13.0 (22A380)
Versions of Apache Airflow Providers
apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.3.0
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-sqlite 3.3.1
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Dec 20, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
potiuk commentedon Dec 20, 2022
This is problem with MacOS M1 not Airflow, not even Docker. There are various workarounds for this. We've implemented a better message for this case in #27381 but due to the nature of this error, we cannot do more - because it is really problem of some of the libraries that do not work on M1 (ARM) Macbook. There are various workarouds you can apply (see the message in #27381) and you can also look for some other cases (search for SIGSEGV our issues or in SIGESEGV MacOS M1 via google).
schustmi commentedon Dec 20, 2022
The same error happens on non-M1 macs, but the
no_proxy=*workaround in the PR you linked still worked, thanks!Is there a general option to run tasks without forking? For example setting
AIRFLOW__CORE__DEFAULT_IMPERSONATION=...also resolves the issue for me, asStandardTaskRunner.start()then runs without forking.potiuk commentedon Dec 20, 2022
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter
potiuk commentedon Dec 20, 2022
And yes we think that on MacOS we should enable it by default
kajen3 commentedon Jan 16, 2023
Could you share a link to the solution? I couldn't find a workaround in the linked PR?
potiuk commentedon Jan 16, 2023
Because there is not one, generally accepted - SIGSEGV might be generated by various causes and libraries and if you look through linked issues (and maybe search for just SIGSEGV) and read what worked for various people - maybe you will find a solution that will work for you.
One generall thing you can attempt is to set this config to True: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter
This might help in a number of similar cases but there is no guarantee - because there might be various reasons and libraries raising SIGSEGV. The linked PR might help you to see how you can try to narrow it down and possibly find a root cause.
That thing is - This is not an Airflow Problem - this is a problem that some libraries are not working on ARM and some of them might require some workarounds - but you have to find those workarounds on your own - first attempt various workarouds people found for their libraries, if it does not work, finding which library causes it and maybe finding or figuring out a new workaround you might need.
Sorry we can't help more, but other than helping you to investigate by better information and showing how you can investigate it is the best we can do.