Repository navigation
Support network tags in airflow.providers.google.cloud.operators.dataproc.DataprocCreateBatchOperator() #40232
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
I want to take a stab at this, can you please assign this to me?
Assigned.
Thanks for taking a stab at this. By the way, can we also add support for tags for DataprocCreateClusterOperator as well?
Thanks for taking a stab at this. By the way, can we also add support for tags for DataprocCreateClusterOperator as well?
@githubwua Just to confirm, your ask here is to add the tags as mentioned here right?
--tags=[TAGS,…]
Network tags for traffic control.The reason I ask is - It seems you should be already able to pass in tags in this particular operator as I see that it is assigned to the _build_gce_cluster_config(). This
_build_gce_cluster_config()is in turn being used to form thecluster_dataattribute within the_build_cluster_data()here, which is invoked from themake()function here. Thismake()function is what's being used inside theDataprocCreateClusterOperatorto create the final cluster config, so my understanding is,tagsshould be able to be passed as part of the**kwargsattributes.Would have @potiuk confirm if I am making sense here?
One more thing @githubwua. Regarding the original requirement here to add
tagstoDataprocCreateBatchOperator, can you pass your network tags in the below syntax to the batch config and try once?BATCH_CONFIG = { "environment_config": { "execution_config": { "subnetwork_uri": "<your-subnet>", "network_tags": ["dev", "user0124"], # This is where you can place your tags, I used these dummy values to test } }, "spark_batch": { "jar_file_uris": ["file:///usr/lib/spark/examples/jars/spark-examples.jar"], "main_class": "org.apache.spark.examples.SparkPi", }, }
and then pass this
BATCH_CONFIGwhile instantiating the operator?create_batch = DataprocCreateBatchOperator( task_id="create_batch", project_id=PROJECT_ID, region=REGION, batch=BATCH_CONFIG, batch_id=BATCH_ID, )I dug deep into the GCP docs for the
BatchControllerClient()and found this out, better to try if this works before making a code change on airflow side. Seems to be working for me if I understand your requirement correctly here -
@githubwua FYI in case this fell under your radar :)
Thank you for finding the right place to specify network tags. I didn't realize it is available deep inside the cluster config field. I will try and see if I can update the documentation to point this out.
Reacted by Kunal Bhattacharya
Description
When submitting a batch job to Dataproc, it is possible to specify network tags, as shown here
But airflow.providers.google.cloud.operators.dataproc.DataprocCreateBatchOperator() does not have an argument to accept network tags. So it is impossible to specify network tags when submitting Dataproc batch jobs from Airflow.
This is a feature request to add support for network tags in DataprocCreateBatchOperator()
Use case/motivation
When a network has firewall rules that allow/deny network traffic, we need to specify network tags to apply specific firewall rules to worker and master nodes. Without network tags, nodes cannot communicate with each other.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct