Repository navigation
DockerSwarmOperator can not mount a volume #9047
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
@gundoganalperen Good investigation. Can you create a PR?
Anyone willing to fix this. I see there is an attached pull request, but that's stale now. I have run into this issue also.
Won't be able to take this immediately. But here is a diff (of an untested commit) from our internal Airflow which was trying to fix this issue. Hopefully, this helps give some idea of what needs to be done.
Anyone can please feel free to use this. Unfortunately, the commit doesn't have test cases.diff --git a/airflow/providers/docker/operators/docker_swarm.py b/airflow/providers/docker/operators/docker_swarm.py index f35644de..312b3809 100644 --- a/airflow/providers/docker/operators/docker_swarm.py +++ b/airflow/providers/docker/operators/docker_swarm.py @@ -142,12 +142,22 @@ def __init__( def _execute(self): self.log.info('Starting docker service from image %s', self.image) + self.mounts = [] + for volume in self.volumes: + path = volume.split(":") + self.mounts.append(types.Mount( + source = path[0], + target = path[1], + type = "bind" + )) + self.service = self.cli.create_service( types.TaskTemplate( container_spec=types.ContainerSpec( image=self.image, command=self.get_command(), env=self.environment, + mounts = self.mounts, user=self.user, tty=self.tty, ),Also, here is the complete specification of Docker's
ContainerSpecif they'd like to implement it completely (and not just themountsargument) - https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec .
Apache Airflow version: 1.10.10
Kubernetes version (if you are using kubernetes) (use
kubectl version):Environment:
uname -a): Linux 4.4.0-179-genericWhat happened:
with dag as dag: t1 = DockerSwarmOperator( api_version='auto', command='***', image='***', auto_remove=False, task_id='airflow_test', volumes=['/tmp/.X11-unix:/tmp/.X11-unix'], environment = {'QT_X11_NO_MITSHM':1, 'DISPLAY':0.0} )Mounting of a volume is not succesfull with Docker Swarm Operator.
What you expected to happen:
I am able to create a service with Docker swarm operator but I can not mount a volume to the container. I understand the problem when I run "docker service inspect <service_name>", there is no volume mounted to the host PC.
Solution proposal:
Inside _run_image() function in docker_swarm_operator.py file we create a service but it has a missing argument for the volumes.
I simply fixed the problem when I add
mounts=self.volumesas an argument to aContainerSpecfunction.