Repository navigation
External python operator logging #27012
Description
Apache Airflow version
2.4.1
What happened
When using the external python operator for running tasks inside a different environment, logs do not appear for the task instance.
The log files are always empty if the task is successful.
If the task fails, one gets very rudimentary logs about the process command failing.
Both environments use the same Python version (3.9.10) and Airflow uses Celery executor
What you think should happen instead
The External Operator is great for cases where you might have multiple environments that require other packages and versions, so they don't mix up with the airflow environment and so other teams could potentially use their own environments inside DAGs. Unfortunately, the External Python Operator does not provide any meaningful logs when the task fails or succeeds.
An example of a successful task without logs:
In case a task fails, the following log will appear (regardless of the actual error):
[2022-10-12, 11:23:05 CEST] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/decorators/base.py", line 188, in execute
return_value = super().execute(context)
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 370, in execute
return super().execute(context=serializable_context)
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 678, in execute_callable
return self._execute_python_callable_in_subprocess(python_path, tmp_path)
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 426, in _execute_python_callable_in_subprocess
execute_in_subprocess(
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/utils/process_utils.py", line 168, in execute_in_subprocess
execute_in_subprocess_with_kwargs(cmd, cwd=cwd)
File "/usr/local/airflow_env/lib/python3.9/site-packages/airflow/utils/process_utils.py", line 191, in execute_in_subprocess_with_kwargs
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/usr/local/aiops_env/bin/python', '/tmp/tmdclodljc4/script.py', '/tmp/tmdclodljc4/script.in', '/tmp/tmdclodljc4/script.out', '/tmp/tmdclodljc4/string_args.txt']' returned non-zero exit status 1.
[2022-10-12, 11:23:05 CEST] {standard_task_runner.py:102} ERROR - Failed to execute job 1976894 for task external_python_pipeline (Command '['/usr/local/aiops_env/bin/python', '/tmp/tmdclodljc4/script.py', '/tmp/tmdclodljc4/script.in', '/tmp/tmdclodljc4/script.out', '/tmp/tmdclodljc4/string_args.txt']' returned non-zero exit status 1.; 23728)
How to reproduce
Using celery executor, so the task is ran on another machine. Simply run a DAG that uses the External Python Operator. A simple hello world will reproduce the issue
Operating System
CentOS 7
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.2
Deployment
Virtualenv installation
Deployment details
No response
Anything else
I have not yet gone into the details, but am willing to contribute!
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Oct 12, 2022
Thanks for opening your first issue here! Be sure to follow the issue template!
pierrejeambrun commentedon Oct 15, 2022
@juraa14 Thank you for being willing to help on this one.
I just assigned you :)
potiuk commentedon Oct 24, 2022
And that should also fix the same case for Python Virtualenv Operator (which should have the same problem).
juraa14 commentedon Dec 21, 2022
Been a while
I've looked into the issue a little bit to find out what is happening.
The task is run using python's subprocess module. It uses Popen which executes a child program in a new process. Now, we can capture STDOUT and STDERR from such a subprocess and output that once the program is finished. So I'll explain what is happening first inside the operator:
The external task is run using subprocess.Popen and all the output is logged at INFO level. If your Airflow logging level is set to WARNING or higher, you will not see the output of the subprocess. So using print() and return statements will output to Airflow logging at INFO level.
You can also output other logs by using
handler = logging.StreamHandler(sys.stdout)and then all logs are forwarded to stdout.Here is a sample DAG that output using print, logging and return:
This DAG outputs the following:
The logs are pretty ugly and there are some encoding issues using logger, but it seems to be sort of a hack around this. Changing this would require changes in the subprocess module, which I guess wouldn't be possible
Here's a prettier example of logs where I printed a spark dataframe using show():
One thing I nearly forgot to mention, getting the traceback from python is possible again if you put logging level to INFO :)
potiuk commentedon Jan 16, 2023
Are there any actions form that?
juraa14 commentedon Jan 17, 2023
No, we can only capture the STDOUT and STDERR output from a subprocess and the comment above shows that, but we cannot control the way subprocesses work in Python.
potiuk commentedon Jan 17, 2023
So should we close it ? Is that the suggestion?
potiuk commentedon Jan 17, 2023
(asking because you are the author - I would simply close it).
Docdez commentedon Jul 18, 2025
I'm surprised that there is no common handler to pretty parse the logs of the subprocess output at this time.
potiuk commentedon Jul 19, 2025
You can always contribute one. This is how things work here - there are more than 3300 contributors here - many of them contributed something they needed. I am actually quite surprised you have not done it yet @Docdez