Skip to content
This repository was archived by the owner on Nov 28, 2025. It is now read-only.
This repository was archived by the owner on Nov 28, 2025. It is now read-only.

BytesList with length 0 or 1 is inferred to have StringType instead of ArrayType #159

Open

Description

@jukujala

If BytesList in TFRecords has always length of 0 or 1, then the feature is inferred to have StringType instead of ArrayType. Is there a reason for this behavior? With this behavior you can write a DataFrame as TFRecords, but you can't read those TFRecords back to a DataFrame. Zero length BytesList is valid in Tensorflow.

Below is the implementation of the parseBytesList from
https://github.com/tensorflow/ecosystem/blob/master/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/TensorFlowInferSchema.scala#L144:

  private def parseBytesList(feature: Feature): DataType = {
    val length = feature.getBytesList.getValueCount

    if (length == 0) {
      null
    }
    else if (length > 1) {
      ArrayType(StringType)
    }
    else {
      StringType
    }
  }

Activity

  1. liusulizzu commented on Jun 11, 2022

    @liusulizzu

    i also hit this problem , do you have any solutions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions