Repository navigation
Scripts/systemd weren't updated to Airflow 3 #51117
Description
Activity
It looks like the script is out-dated. I also recently observed some changes between the 2.x docker-compose file and 3.x docker-compose. Also, in the script folder, haven’t found anything related with database initialization. At high level, it can be helpful to see how these scripts are used to deploy airflow. How it is different from the docker-compose file. 🤔 and wondering if it is a deprecated process since the code is not updated for a while.
Yes - if you want to update the scripts either of you - feel free - either of you. We are not actively checking or testting systemd setup but if someone does and would like to update the scripts based on their setup, it's a good idea.
Reacted by Niko OliveiraQuestion: It seems like the initial systemd files assume that airflow is installed at the system level, instead of in a virtual environment.
For example:
whereas in our service file we have:
ExecStart= bash -c 'source /home/airflow/airflow_venv/bin/activate ; /home/airflow/airflow_venv/bin/airflow scheduler'Which is activating the virtual environment (and loading any variables configured there) and then starting the scheduler.
Any preferences?
I could update the documentation to be more explicit about the airflow installation location and what to modify if the install is within a virtual environment.
Also, we don't run flower, kerberos or worker, so I can't touch those with authority if there's something to change there.
I want to put my two cents that might be worth to consider. From my understanding, your case may be to deploy and run Airflow as a standalone instance. flower and worker more relate to if you want to use CeleryExecutor and want to have a service to monitor the workers. From the service files, looks it assumes a running instance of Postgres, MySQL, or Redis, on the same machine. I think it can also be helpful to manage Python dependencies in the Airflow environment through the virtual environment, but if the machine only runs the single Airflow instance, probably it’s not necessary. Feel free to correct me if I am wrong on any aspect.
Reacted by Raphael DumasI'm in agreement with that @sjyangkevin. From your knowledge, do the Flower & Worker files look like they would need updating to Airflow 3?
@potiuk, does the
scriptsfolder in GitHub get included in the installation? I'm wonder if I should also update the documentation to point people more explicitly at where they find that folder, since the references don't include links back to GitHub.airflow/airflow-core/docs/howto/run-with-systemd.rst
Lines 26 to 28 in 241144a
Apache Airflow version
3.0.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
The documentation (https://airflow.apache.org/docs/apache-airflow/stable/howto/run-with-systemd.html) suggests using the scripts in the below folder for setting up with SystemD
https://github.com/apache/airflow/tree/main/scripts/systemd
However, they have not been updated to Airflow 3 and they do not start the
dag-processorortriggerer, and start thewebserverinstead of theapi-serverWhat you think should happen instead?
No response
How to reproduce
Follow the step in the documentation
Operating System
RHEL
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
We can try to submit a PR to correct this, but honestly at the moment our in-place Airflow 3 upgrade on RHEL hasn't been working super well, so I'm not sure we're the best placed to be making this update (though I think this is mostly issues with configs and not systemd unit files).
Are you willing to submit PR?
Code of Conduct