Repository navigation
RegressionEvaluator return 0 for MSE && RMSE if all scores are NaN #6361
Description
It seems that RegressionEvaluator filters out all NaN scores when calculating metrics. So if a predictor produces NaN only for the entire evaluation dataset, the RMSE and MSE from RegressionEvaluator would be 0, which is ironically the best result it can get.
While it may need further discussion, I wonder if it's necessary to FILTER out the NaN value when calculating metrics, as NaN in prediction result usually indicates a model is not well-trained, and thus should be reflected in evaluation metrics.
Activity
michaelgsharp commentedon Oct 10, 2022
What do you think it should do? If we did math with nan values it would cause the final result to be nan, even if there was only 1 nan result. Maybe we just should just change it so that if all results are NAN that the final result is NAN instead of 0?
@luisquintanilla
LittleLittleCloud commentedon Oct 10, 2022
I would vote for just return NaN even if just one result is NaN because a NaN in score usually means the model is not well-trained.
michaelgsharp commentedon Feb 6, 2023
@luisquintanilla are you ok with returning NaN if one result is a NaN?
rosebyte commentedon Oct 23, 2026
Triage: we will take a look when we will have time.