Distributed training is slow compared to single machine #69

Activity

1. saikishor

   changed the title [-]Distributed learning is slow comapred to single machine[/-] [+]Distributed training is slow comapred to single machine[/+] on Nov 22, 2017
2. saikishor

   changed the title [-]Distributed training is slow comapred to single machine[/-] [+]Distributed training is slow compared to single machine[/+] on Nov 22, 2017
3. lingyuhaunti commented on Nov 3, 2018

   

   lingyuhaunti

   on Nov 3, 2018

   More actions

   I am trying to perform a distributed training on models from Tensorflow Object Detection models. I am able to perform distributed learning, but to my notice the distributed training is too slow compared to single machine training. As the step duration is high in distributed training than that of single machine. I am using Hadoop HDFS to feed data to all hosts .

   From Distributed Training (All machines have more or less same step duration):

   INFO:tensorflow:global step 1247: loss = 3.1943 (1.972 sec/step)
   
   INFO:tensorflow:global step 1251: loss = 2.5009 (1.218 sec/step)
   
   INFO:tensorflow:global step 1254: loss = 2.2030 (1.746 sec/step)
   
   INFO:tensorflow:global step 1257: loss = 1.9776 (2.222 sec/step)
   
   INFO:tensorflow:global step 1262: loss = 2.6990 (1.161 sec/step)
   
   INFO:tensorflow:global step 1264: loss = 1.9501 (1.682 sec/step)
   
   INFO:tensorflow:global step 1268: loss = 1.6125 (1.288 sec/step)
   
   INFO:tensorflow:global step 1271: loss = 2.0434 (1.223 sec/step)
   

   From single machine training:

   INFO:tensorflow:global step 236: loss = 4.1127 (0.319 sec/step)
   
   INFO:tensorflow:global step 237: loss = 3.0208 (0.316 sec/step)
   
   INFO:tensorflow:global step 238: loss = 3.2838 (0.367 sec/step)
   
   INFO:tensorflow:global step 239: loss = 2.9822 (0.324 sec/step)
   
   INFO:tensorflow:global step 240: loss = 2.9753 (0.322 sec/step)
   
   INFO:tensorflow:global step 249: loss = 2.8071 (0.318 sec/step)
   
   INFO:tensorflow:global step 250: loss = 3.4335 (0.328 sec/step)
   
   INFO:tensorflow:global step 251: loss = 4.3550 (0.322 sec/step)
   

   Why do we encounter such difference between distributed learning as well as Single machine training?. Is there a way to resolve this?

   Can you share your codes with us?
4. DanyOrtu97 commented on Dec 19, 2020

   

   DanyOrtu97

   on Dec 19, 2020

   More actions

   I have the same problem as you. When I use a spark cluster with two or more nodes, the algorithm needs the same time as if I have only one node. I don't understand why. Please, someone answer this doubt