No module named Tensorframes on EMR installation #113

Activity

1. taklwu commented on Jul 17, 2017

   

   taklwu

   on Jul 17, 2017

   More actions

   I had the same issue when just use pyspark with automatic package installation.

   pyspark --packages databricks:tensorframes:0.2.8-s_2.11
   

   So, at the end I decided to build the jars from source and use the jar as extra library, also, you might need to install pandas before running pyspark

   PYTHONPATH=tensorframes-assembly-0.2.8.jar \
   pyspark --jars tensorframes-assembly-0.2.8.jar
   
2. thunterdb commented on Jul 27, 2017

   

   thunterdb

   on Jul 27, 2017

   Contributor

   More actions

   @nikparmar can you tell me more about your current spark setup?
3. nikparmar commented on Jul 31, 2017

   

   nikparmar

   on Jul 31, 2017

   Author

   More actions

   @thunterdb So it's an Amazon EMR setup where I was installing this.

   @taklwu I will try and update you
4. thunterdb commented on Aug 11, 2017

   

   thunterdb

   on Aug 11, 2017

   Contributor

   More actions

   @nikparmar I am not very familiar with EMR's setup. Did the suggestion above work for you?
6. tanjialiang commented on Nov 17, 2017

   

   tanjialiang

   on Nov 17, 2017

   Last edited by tanjialiang

   More actions

   Hi @thunterdb ,

   I have similar issues. I have Spark running in yarn-cluster mode (nodes running CentOS7), version 2.0.1. I added databricks:tensorframes:0.2.9-s_2.11 to spark.jars.packages and it seemed to load the jar according to Spark UI. But I still get no module named tensorframes error when I try to import tensorframes. I have python version 2.7.5 installed across all nodes. I cannot go the other method that sets PYTHONPATH to every node because as I am running Spark in cluster mode, and in this case I will have to maintain it in every single node along with the assemly jars. Is this a bug? If not where do you see the issues are?

   Thanks,
   Jialiang

   👍React with 👍1Reacted by UlionTse