Skip to content

[ML] Learn a linear trend at long bucket spans - #3210

Open
valeriy42 wants to merge 3 commits into
elastic:mainfrom
valeriy42:investigation/2740-linear-trend-1d
Open

valeriy42 wants to merge 3 commits into
elastic:mainfrom
valeriy42:investigation/2740-linear-trend-1d

Conversation

@valeriy42

Copy link
Copy Markdown
Contributor

Summary

At a long bucket span a trend sub-model sees few values per unit of decay time, so the prediction errors recorded before its regression is identified never age out of the model selection MSEs. The constant and the linear MSE then stay dominated by the same startup transient, their ratio tends to one, and the F-test in selectModelOrdersForForecasting never selects a model which extrapolates the trend. A clean linear ramp at bucket_span: 1d is forecast with a constant, and blending sub-models which hold different levels pulls the prediction back towards the mean. That is the decreasing forecast reported in #2740.

Measured on a 120 day ramp of +17/day sampled once per day: the selected orders were all 1 and the forecast slope was -2.5/day. A single untrained prediction error accounted for 99.6% of the accumulated MSE of the slowest sub-model. The same ramp at a 10 minute bucket span selects order 2 for every sub-model, because 144 times more values per day dilute that one error.

This skips the MSE update until the component has seen at least as many values as the regression has parameters. The forecast slope for the case above becomes +17.9/day.

Existing models keep the errors already recorded, so a job whose snapshot predates this change still needs a model reset to benefit. Repairing that state in place was explored and rejected: doing it on restore breaks persist idempotency, and doing it during forecasting recovers only until the stale errors decay below the detection bar but not yet far enough for the F-test, which makes the behaviour non-monotonic over time.

At a long bucket length a trend sub-model sees few values per unit of decay
time, so the prediction errors recorded before its regression is identified
never age out of the model selection MSEs. Both the constant and the linear
MSE are then dominated by the same startup transient, their ratio tends to
one, and the F-test in selectModelOrdersForForecasting can never select a
higher order model. A clean linear ramp at bucket_span 1d is consequently
forecast with a constant, and the blend across sub-models makes the forecast
drift the wrong way.

Skip the MSE update until the component has seen at least as many values as
the regression has parameters.
@elasticsearchmachine

Copy link
Copy Markdown

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine

Copy link
Copy Markdown

Hi @valeriy42, I've updated the changelog YAML for you.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants