This repository was archived by the owner on Nov 28, 2025. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 28, 2025. It is now read-only.
TensorFlow on Hadoop YARN #33
Description
Hadoop YARN is a commonly deployed cluster manager. Having the ability to run TensorFlow on YARN would be very useful in such environment.
Our team is currently working on a YARN application for this purpose, and would like to contribute our work here. We will provide more details of our contribution soon.
-Jason
Activity
jhseu commentedon Feb 16, 2017
Thanks! That'd be really useful.
zhe-thoughts commentedon Feb 17, 2017
@tslam75 Have you looked at https://issues.apache.org/jira/browse/YARN-6043?
tslam75 commentedon Feb 22, 2017
@zhe-thoughts Thanks for the reference! Looked over YARN-6043, and both uses a native application master for TensorFlow.
Attaching a design document here now. We also have an implementation based on this design, and will publish the code soon.
TensorFlow_on_YARN.pdf
tslam75 commentedon Mar 21, 2017
Sorry for the delay.
Created pull request #39 while waiting for the CLA to be signed.
IDerr commentedon Apr 11, 2017
Awesome job :O
leftnoteasy commentedon Apr 17, 2017
In Hadoop 3.0, YARN native services can support running Tensorflow services on YARN without adding any dependencies or implement a new YARN application master.
Please see our blogpost: https://hortonworks.com/blog/distributed-tensorflow-assembly-hadoop-yarn/ and let me know if you have any questions. Thanks!
tbchj commentedon Apr 21, 2017
focus ...
butterluo commentedon May 14, 2017
@tslam75 Does your 'TensorFlow on Yarn' support fault tolerance ? If yes, how?
zhanglistar commentedon Dec 25, 2017
mark
oliverhu commentedon Feb 1, 2018
+1
zhe-thoughts commentedon Sep 27, 2018
We (LinkedIn Hadoop team) just open sourced TonY:
Repo: https://github.com/linkedin/TonY
Blog post: https://engineering.linkedin.com/blog/2018/09/open-sourcing-tony--native-support-of-tensorflow-on-hadoop
Comments / discussions very welcome!