Repository navigation
AutoML experiments in non declarative style not working #6446
Description
System Information (please complete the following information):
- OS & Version: Windows 11
- ML.NET Version: 2.0
- .NET Version: .NET6
Describe the bug
Running the "old" AutoML experiments does not work for all trainers. Using CreateMulticlassClassificationExperiment or CreateRegressionExperiment instead of the new declarative style will result in an exception (possibly due to custom schemas).
An exception will be raised in the new SweepablePipeline:
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.ML.AutoML.SweepablePipeline..ctor(Dictionary`2 estimators, Entity schema, String currentSchema)
at Microsoft.ML.AutoML.SweepablePipeline.AppendEntity(Boolean allowSkip, Entity entity)
at Microsoft.ML.AutoML.AutoCatalog.MultiClassification(String labelColumnName, String featureColumnName, String exampleWeightColumnName, Boolean useFastForest, Boolean useLgbm, Boolean useFastTree, Boolean useLbfgs, Boolean useSdca, FastTreeOption fastTreeOption, LgbmOption lgbmOption, FastForestOption fastForestOption, LbfgsOption lbfgsOption, SdcaOption sdcaOption, SearchSpace`1 fastTreeSearchSpace, SearchSpace`1 lgbmSearchSpace, SearchSpace`1 fastForestSearchSpace, SearchSpace`1 lbfgsSearchSpace, SearchSpace`1 sdcaSearchSpace)
at Microsoft.ML.AutoML.MulticlassClassificationExperiment.CreateMulticlassClassificationPipeline(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer)
at Microsoft.ML.AutoML.MulticlassClassificationExperiment.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
at Microsoft.ML.AutoML.MulticlassClassificationExperiment.Execute(IDataView trainData, String labelColumnName, String samplingKeyColumn, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
To Reproduce
var experimentSettings = new MulticlassExperimentSettings
{
MaxExperimentTimeInSeconds = 30,
OptimizingMetric = MulticlassClassificationMetric.MacroAccuracy
};
experimentSettings.Trainers.Clear();
experimentSettings.Trainers.Add(MulticlassClassificationTrainer.LbfgsMaximumEntropy);
ctx.Auto().CreateMulticlassClassificationExperiment(experimentSettings).Execute(trainDv);Where the schema is of a custom type:
var schemaDef = SchemaDefinition.Create(typeof(ModelInput));
schemaDef["Features"].ColumnType = new VectorDataViewType(NumberDataViewType.Single, numberOfFeatures);
schemaDef.Remove(schemaDef["LabelFeaturized"]); // removed when not applicablepublic class ModelInput
{
public uint Label;
public float LabelFeaturized;
public float[] Features;
}Expected behavior
No regression expected for CreateMulticlassClassificationExperiment and CreateRegressionExperiment.
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]AutoML experiments not working[/-][+]AutoML experiments in non declarative style not working[/+]LittleLittleCloud commentedon Nov 10, 2022
Thanks for reporting this bug. It's because
LbfgsMaximumEntropyis not used when constructing sweepable pipeline for multiclass classification (machinelearning/src/Microsoft.ML.AutoML/API/MulticlassClassificationExperiment.cs
Line 314 in 17c061a
We'll push a fix for this bug, in the meanwhile, you can add other trainer (like lightGbm) as a work-around
thoron commentedon Nov 11, 2022
Thank you for your answer! We will downgrade until the fix is deployed.
zeroskyx commentedon Dec 3, 2022
Any chance of getting this fix rolled out before ML.NET 3.0 as an update to ML.NET 2.0? Projects depending on this would otherwise need to stay at
0.7.1/0.19.1:(michaelgsharp commentedon Dec 5, 2022
@luisquintanilla how does the suggestion by @zeroskyx sound for a servicing release?
thoron commentedon Dec 6, 2022
I believe it should be in a servicing release as it will produce a runtime exception when simply upgrading and using a previously functional workflow (without any syntax or compilation warnings / errors).
zeroskyx commentedon Dec 23, 2022
Possibly related to #6529
fix #6446 (#6447)