VectorType attribute with dynamic dimension while dealing with csv files

[rishi-git]

I am working on training and evaluation in ML.NET. The data comes from a .csv file. According to the requirements, there may be variable number of columns in the file. Until now, I have achieved this by providing a fixed vectorType dimension. But I am not aware of making this work dynamically.

Here is the a block of code working using the fixed dimension,

public class InputClass
{
public bool Lavel;

[VectorType(5)] // I want this to be dynamic
public string[] SpecVec;

public string PredictedTarget;

}

public static void Main(string[] args)
{
MLContext mlContext = new MLContext(seed:0);
TextLoader.Columns[] columns = new TextLoader.Columns[3];
columns[0] = new TextLoader.Column("Label", DataKind.Boolean, 0);
columns[2] = new TextLoader.Column("PredictedTarget", DataKind.String, 1);
columns[1] = new TextLoader.Column("SpecVec", DataKind.String, 2,6);

IDataView dataview = mlContext.Data.LoadFromTextFile(filepath, columns,
separatorChar:',', hasHeader:true, allowQuoting:false, trimWhitespace: true,
allowSparse:true);

//-- creating pipeline and training a model further

}
This code works without any issue. But I just need the VectorType to accept the dimension at runtime. Now it accepts only constant values. I am looking if there is any workaround to achieve this. I am a beginner into ML.Net, please do understand if I missed any steps or anything to be explained. I really appreciate any help to get this worked.

[michaelgsharp]

@rishi-git when you say dynamically what exactly are you meaning? If you do this:

[VectorType()]
public string[] SpecVec;

It should mark SpecVec as a vector of unknown size. (I think you can even leave off the VectorType and it will just do a vector of unknown size)

For the text loader, you can use the constructor that takes a Range to set it to load from a certain start point until the end of the datafile.

new TextLoader.Column("Features",
                                          DataKind.Single,
                                          new[] { new TextLoader.Range(0, null) }) // This will load everything from column 0 till the end of the file. You can change the start column to whatever you want.

Does something like this work in your situation?

[rishi-git]

@michaelgsharp Thank you very much for your reply.
When I say dynamically, I mean the dimensions for VectorType attribute must be initialized at the runtime.
I tried using your code within the program.
But it throws an error as "System.ArgumentOutOfRangeException : Schema mismatch for score column 'Score': expected Single, got Vector<Single, 5>".
This error occurs at the time of evaluation at the following line of code,

mlContext.BinaryClassification.Evaluate(predictions, labelColumnName : "Label");

I appreciate any further help, thanks.

[michaelgsharp]

Hmm.. If thats the case I wonder if we have a bug because just doing [VectorType()] with no parameters should declare it as a vector type, but its clearly not working. Let me take a closer look and see what I can find.

[mitsuomiyazato]

Hey, did anybody resolve that problem? I am also dealing with this...