Conversation
At a long bucket length a trend sub-model sees few values per unit of decay time, so the prediction errors recorded before its regression is identified never age out of the model selection MSEs. Both the constant and the linear MSE are then dominated by the same startup transient, their ratio tends to one, and the F-test in selectModelOrdersForForecasting can never select a higher order model. A clean linear ramp at bucket_span 1d is consequently forecast with a constant, and the blend across sub-models makes the forecast drift the wrong way. Skip the MSE update until the component has seen at least as many values as the regression has parameters.
|
Pinging @elastic/ml-core (Team:ML) |
|
Hi @valeriy42, I've updated the changelog YAML for you. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
At a long bucket span a trend sub-model sees few values per unit of decay time, so the prediction errors recorded before its regression is identified never age out of the model selection MSEs. The constant and the linear MSE then stay dominated by the same startup transient, their ratio tends to one, and the F-test in
selectModelOrdersForForecastingnever selects a model which extrapolates the trend. A clean linear ramp atbucket_span: 1dis forecast with a constant, and blending sub-models which hold different levels pulls the prediction back towards the mean. That is the decreasing forecast reported in #2740.Measured on a 120 day ramp of +17/day sampled once per day: the selected orders were all 1 and the forecast slope was -2.5/day. A single untrained prediction error accounted for 99.6% of the accumulated MSE of the slowest sub-model. The same ramp at a 10 minute bucket span selects order 2 for every sub-model, because 144 times more values per day dilute that one error.
This skips the MSE update until the component has seen at least as many values as the regression has parameters. The forecast slope for the case above becomes +17.9/day.
Existing models keep the errors already recorded, so a job whose snapshot predates this change still needs a model reset to benefit. Repairing that state in place was explored and rejected: doing it on restore breaks persist idempotency, and doing it during forecasting recovers only until the stale errors decay below the detection bar but not yet far enough for the F-test, which makes the behaviour non-monotonic over time.