Repository navigation
NoClassDefFoundError: org/apache/spark/Logging #56
Description
I am trying to run sample code with $pyspark --packages databricks:tensorframes:0.2.3-s_2.10
I am getting this error (below attached) with spark-1.6, spark-2.0 and custom build on spark with branch-2.0.
I tried to find org.apache.spark.logging jar, which I am unable to find anywhere.
I am using Mac OS ElCapitain.
Please help.
Py4JJavaError Traceback (most recent call last)
in ()
8 # The TensorFlow placeholder that corresponds to column 'x'.
9 # The shape of the placeholder is automatically inferred from the DataFrame.
---> 10 x = tfs.block(df, "x")
11 # The output that adds 3 to x
12 z = tf.add(x, 3, name='z')
/private/var/folders/_f/fzxfs9dj33j0381wvy5bdsdm0000gn/T/spark-1831a94f-7cdd-4691-a32a-4cbc3d9bc3a5/userFiles-66cd46aa-619d-41cf-806d-3dc6e509e13d/databricks_tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py in block(df, col_name, tf_name)
313 :return: a TensorFlow placeholder.
314 """
--> 315 return _auto_placeholder(df, col_name, tf_name, block = True)
316
317 def row(df, col_name, tf_name = None):
/private/var/folders/_f/fzxfs9dj33j0381wvy5bdsdm0000gn/T/spark-1831a94f-7cdd-4691-a32a-4cbc3d9bc3a5/userFiles-66cd46aa-619d-41cf-806d-3dc6e509e13d/databricks_tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py in _auto_placeholder(df, col_name, tf_name, block)
331
332 def _auto_placeholder(df, col_name, tf_name, block):
--> 333 info = _java_api().extra_schema_info(df._jdf)
334 col_shape = [x.shape() for x in info if x.fieldName() == col_name]
335 if len(col_shape) == 0:
/private/var/folders/_f/fzxfs9dj33j0381wvy5bdsdm0000gn/T/spark-1831a94f-7cdd-4691-a32a-4cbc3d9bc3a5/userFiles-66cd46aa-619d-41cf-806d-3dc6e509e13d/databricks_tensorframes-0.2.3-s_2.10.jar/tensorframes/core.py in _java_api()
28 # You cannot simply call the creation of the the class on the _jvm due to classloader issues
29 # with Py4J.
---> 30 return _jvm.Thread.currentThread().getContextClassLoader().loadClass(javaClassName)
31 .newInstance()
32
/usr/local/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py in call(self, *args)
931 answer = self.gateway_client.send_command(command)
932 return_value = get_return_value(
--> 933 answer, self.gateway_client, self.target_id, self.name)
934
935 for temp_arg in temp_args:
/usr/local/spark/python/pyspark/sql/utils.pyc in deco(_a, *_kw)
61 def deco(_a, *_kw):
62 try:
---> 63 return f(_a, *_kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
/usr/local/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
310 raise Py4JJavaError(
311 "An error occurred while calling {0}{1}{2}.\n".
--> 312 format(target_id, ".", name), value)
313 else:
314 raise Py4JError(
Py4JJavaError: An error occurred while calling o41.loadClass.
: java.lang.NoClassDefFoundError: org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 22 more
Activity
ushnish commentedon Sep 1, 2016
I have the same issue and using Mac OS El Capitan as well.
dk67604 commentedon Sep 28, 2016
Is someone have the solution for this issue ?
Kai-Chen commentedon Nov 21, 2016
To make it work on Mac, clone this repository, and then change the file
build.sbtby replacing the line"tensorflow-0.8.0-1.2-macosx-x86_64.jar" // This is not the main target, excludingwith
tensorflow-0.8.0-1.2-linux-x86_64.jarIt bundles the tensorflow native libraries for mac (instead of linux).
Then follow the developer's instruction in README, with the update from this PR https://github.com/databricks/tensorframes/pull/61/files.
That should do it.
yanji84 commentedon Dec 20, 2016
Any update on this issue?
It seems that this exception is due to the fact that org/apache/spark/Logging class has been removed from Spark since 1.5.2. Is there any reason why tensorframe is taking a dependency on this library?
Include the stackoverflow link that explains the issue:
http://stackoverflow.com/questions/40287289/java-lang-noclassdeffounderror-org-apache-spark-logging
thunterdb commentedon Mar 30, 2017
@yanji84 this issue should be corrected with the latest published release (
0.2.5-rc1). Can you give it a try? Note that the support for Spark 1.6 has been removed (there has been too many changes in the developer API to be able to maintain compatibility between the two releases).thunterdb commentedon Mar 30, 2017
I just published a new version (
0.2.5-rc2) that should work on some macos configurations. It seems there is no clear pattern, it is an issue upstream with javacpp-tensorflow.oldmonk101 commentedon May 11, 2017
Ok. I am using Spark 2.1. I built tensorframes from source (latest version as of today) on my mac an the assembly jar seems to contain the lazylogging class.
jar tf tensorframes-assembly-0.2.8.jar |grep log
com/typesafe/scalalogging/
com/typesafe/scalalogging/slf4j/
com/typesafe/scalalogging/BaseLogger.class
com/typesafe/scalalogging/Logger.class
com/typesafe/scalalogging/Logging.class
com/typesafe/scalalogging/slf4j/LazyLogging$class.class
com/typesafe/scalalogging/slf4j/LazyLogging.class
com/typesafe/scalalogging/slf4j/Logger$.class
com/typesafe/scalalogging/slf4j/Logger.class
Not sure why I am still getting this error -
Py4JJavaError Traceback (most recent call last)
in ()
4 # The TensorFlow placeholder that corresponds to column 'x'.
5 # The shape of the placeholder is automatically inferred from the DataFrame.
----> 6 x = tfs.block(df, "x")
7 # The output that adds 3 to x
8 z = tf.add(x, 3, name='z')
........
.............
..................
Py4JJavaError: An error occurred while calling o46.loadClass.
: java.lang.NoClassDefFoundError: com/typesafe/scalalogging/slf4j/LazyLogging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)