support human shape annotation in meta column info

data = [Row(x=[float(x), float(2 * x)], key=str(x % 2)) for x in range(1, 6)]
df_analyzed = tfs.analyze(sqlContext.createDataFrame(data))
df = sqlContext.createDataFrame(data)

# this would work
grouped_df_analyzed = df_analyzed.groupby('key')

In [79]: with tf.Graph().as_default() as g:
    ...:     x = tf.placeholder(tf.double, [None, 2], name='x_input')
    ...:     y = tf.reduce_mean(x, 0, name='x')
    ...:     df_1 = tfs.aggregate(y, grouped_df_analyzed)

#while this would fail..

grouped_df  = df.groupby('key')

In [80]: with tf.Graph().as_default() as g:
    ...:     x = tf.placeholder(tf.double, [None, 2], name='x_input')
    ...:     y = tf.reduce_mean(x, 0, name='x')
    ...:     df_1 = tfs.aggregate(y, grouped_df)

# with reason

java.lang.Exception: The data column 'x' has shape [?,?], not compatible with shape [?,2] requested by the TF graph

which would be useful if there is a function the take human input into the meta data...

@tjhunter

@tjhunter would builder or experiment operator a good place to append human annotation?

Hello Peng, yes you need to analyze the dataframe first, unless it contains some simple (scalar) types. This may be relaxed in the future.

On January 23, 2018 at 6:18:34 PM, Peng Yu ***@***.***) wrote: turns out it can be done, but i need to `tfs.analyze(df)` first...

wondering if it is possible

to give it to df instead of analysis it -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

#137 (comment)

Hello Peng, what do you mean by human annotation?

On January 23, 2018 at 7:18:05 PM, Peng Yu ***@***.***) wrote: @tjhunter would builder or experiment operator a good place to append

human annotation?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

#137 (comment)

Hi, I was referring to whether we should provide a interface let user pass in the shape of the dataframe columns. In my example, if I can add shape=(None,2) to column x's meta data, I don't need to analyze the whole dataframe, which have expensive cost

@yupbank

@yupbank sorry I missed your response. There is a workaround documented there that should do the trick for your use case:
https://groups.google.com/forum/#!topic/tensorframes/g3Dm97oFvVw
It would be great to add this functionality into tensorframes. If it works for you, you should feel free to open a pull request.

Navigation Menu

Repository navigation

support human shape annotation in meta column info #137

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

support human shape annotation in meta column info #137

Description

Activity

yupbank commented on Jan 23, 2018

thunterdb commented on Jan 26, 2018

thunterdb commented on Jan 26, 2018

yupbank commented on Jan 26, 2018

thunterdb commented on Feb 8, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions