Repository navigation
Build new Spark-Tensorflow-Connector for scala 2.12 #194
Description
Activity
Spark has released 3.x which built with scala 2.12, but th newest version of Spark-Tensorflow-Connector is built with scala 2.11 which means not compatiable with spark 3.x. Maven repository is still 2.11.
You can build 2.12 by yourself!!!
Spark has released 3.x which built with scala 2.12, but th newest version of Spark-Tensorflow-Connector is built with scala 2.11 which means not compatiable with spark 3.x. Maven repository is still 2.11.
Also, you use spark-tfrecord https://github.com/linkedin/spark-tfrecord, that support scala 2.12.
It would be great to have an example of how to build this for Spark 3.x and Tensorflow 2.x
I tried a number of combinations with no success. I can build the jar, but it writes an empty TFRecord file.
If others have successfully built the jar for Spark 3.x and Tensorflow 2.x I'd love to hear the details.
@dennisobrien Hi, try this: https://github.com/wuxianxingkong/spark-tfrecord (Built with spark 3.1.1 and scala 2.12)
It would be super helpful if TF team could kindly help releasing spark-tensorflow-connector built with scala 2.12 to maven artifact 🙌 Right now the best workaround seems to be
spark-tfrecordbut there'll be migration effort across products using it ..Reacted by Ratul Ray, Fernando Raganhan Barbosa and Wojtek MalinowskiIs this still not available? It seems it is not in Maven but Databricks is working fine, seems they compiled it for scala 2.12.
Spark has released 3.x which built with scala 2.12, but th newest version of Spark-Tensorflow-Connector is built with scala 2.11 which means not compatiable with spark 3.x. Maven repository is still 2.11.