Skip to content

feat(ai): add data classes for time series forecasting and imputation - #2873

Merged
miladh merged 18 commits into
masterfrom
ai-ts-forecasting/01-data-classes
Nov 9, 2026
Merged

miladh merged 18 commits into
masterfrom
ai-ts-forecasting/01-data-classes

Conversation

@miladh

@miladh miladh commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

First PR in a stack that adds SDK support for the ai/timeseries/forecast and ai/timeseries/impute endpoints. The API is in beta.

The stack builds the feature in two layers, both under client.ai.time_series:

  • forecast / impute: a low-level client that mirrors the API. PRs 1–4.
  • forecast_with_retrieval / impute_with_retrieval: convenience methods for CDF time series by instance id. PRs 5–8.

This PR only adds data classes, shaped like the API's request and response, with no API calls yet:

  • ForecastInputDatapoint and ForecastInputTimeSeries: the request objects, used by both endpoints. They're dataclasses, like DatapointsQuery. ForecastInputDatapoint.load creates a datapoint from a tuple, a list or an API-style dict.
  • QuantileDatapoint: one predicted point, quantile level -> value.
  • ForecastResult(List) and ImputeResult(List): one result per input series. The lists hold the response's metadata (quantile levels and cohorts) as ForecastMetadata, following SDK precedent: _load_raw_api_response and dump_raw like NodeList, and a non-optional field with an empty default for lists built from results alone, like SyncRecordList.
  • to_pandas() gives one column per series with the median; include_all_quantiles=True gives (label, quantile level) columns for every level.

Usage (this PR on its own)

from cognite.client.data_classes.ai import ForecastInputDatapoint, ForecastInputTimeSeries, ForecastResultList

series = ForecastInputTimeSeries(
    "23-PT-1101",
    [
        ForecastInputDatapoint(1790812800000, 42.1),
        ForecastInputDatapoint(1790812860000, None),                 # no reading: hidden from the model
        ForecastInputDatapoint(1790812920000, 42.9, missing=True),   # hidden; impute returns a prediction for it
        ForecastInputDatapoint.load((1790812980000, 43.1)),          # from a tuple
    ],
    cohort="compression-train-a",
)
series.dump()   # the API request shape: {"label": ..., "cohort": ..., "datapoints": [...]}

res = ForecastResultList._load_raw_api_response([api_response])   # done by the API in PR 3
res.metadata.quantile_levels                   # as the API returns them
res.metadata.cohorts                           # ["compression-train-a"]
res.get("23-PT-1101").forecast[0].quantiles    # {0.05: 41.0, ..., 0.95: 43.0}
res.to_pandas()                                # DatetimeIndex, one median column per series
res.to_pandas(include_all_quantiles=True)      # (label, quantile level) columns
res.dump_raw()                                 # {"items": [...], "metadata": {...}}, like the API response

Checklist:

  • Tests added/updated: test_data_classes/test_ai.py. The pandas tests are marked dsl.
  • Documentation updated: docstrings. Rendered docs come in PR 3.
  • The PR title follows the Conventional Commit spec.

🤖 Generated with Claude Code

@niekbeckers niekbeckers left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my own understanding, we cannot reuse the Datapoint and Timeseries data classes, or have composites that rely on a combination of primitives (e.g., Sequence) with those datamodels?

Request and result types for the upcoming ai/timeseries/{forecast,impute}
endpoints:

- InputDatapoint / InputTimeSeries: request objects (dataclasses, like
  DatapointsQuery). Datapoints can be InputDatapoint, (timestamp, value)
  tuples or API-shaped dicts; timestamps accept ms, datetime or time-shift
  strings.
- QuantileDatapoint: one step with quantile level -> value. Non-numeric keys
  are ignored on load.
- ForecastResult(List) / ImputeResult(List): one result per input label, with
  quantile_levels and cohorts from the response metadata (like NodeList's
  typing/debug), lookup by .get(label=...), and to_pandas() with a
  DatetimeIndex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@miladh
miladh force-pushed the ai-ts-forecasting/01-data-classes branch from ea1105b to 12e6a69 Compare November 2, 2026 11:15
@miladh
miladh marked this pull request as ready for review November 2, 2026 11:20
@miladh
miladh requested review from a team as code owners November 2, 2026 11:20

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces new data classes and helper functions in cognite/client/data_classes/ai.py to support forecasting and imputation features, including InputDatapoint, InputTimeSeries, QuantileDatapoint, ForecastResult, ForecastResultList, ImputeResult, and ImputeResultList, along with corresponding unit tests. Feedback on the changes includes handling empty inputs in _concat_columns to prevent pandas errors, supporting lists in addition to tuples for input datapoints, and safely handling potential null metadata values in API responses to avoid attribute errors.

Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
pd.concat raised "No objects to concatenate" for a ForecastResultList or
ImputeResultList with no results (e.g. an empty slice). Other SDK lists
return an empty DataFrame, so do the same.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@miladh

miladh commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Re @niekbeckers' question on reusing Datapoint and TimeSeries: the existing classes don't fit this API. The input needs a nullable value and a missing flag, results are quantiles per step rather than a single value or aggregate, and series are identified by a caller-chosen label, not a CDF id. The CDF-identified variants (TimeSeriesForecast, TimeSeriesImpute, keyed by instance_id) come later in the stack, in PR 06.

— Written by Claude

Direct pd.to_datetime calls are only allowed in _pandas_helpers.py (enforced
by test_pandas_timestamp_constructors_only_used_in_pandas_helpers), so build
the empty DatetimeIndex with to_pandas_datetime_index instead.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.87640% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.72%. Comparing base (1ab3df0) to head (d38ce4c).
⚠️ Report is 34 commits behind head on master.

Files with missing lines Patch % Lines
cognite/client/data_classes/ai.py 97.82% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2873      +/-   ##
==========================================
+ Coverage   93.49%   93.72%   +0.22%     
==========================================
  Files         527      541      +14     
  Lines       55489    57533    +2044     
==========================================
+ Hits        51879    53920    +2041     
- Misses       3610     3613       +3     
Files with missing lines Coverage Δ
tests/tests_unit/test_base.py 98.94% <100.00%> (+<0.01%) ⬆️
tests/tests_unit/test_data_classes/test_ai.py 100.00% <100.00%> (ø)
cognite/client/data_classes/ai.py 97.65% <97.82%> (+0.28%) ⬆️

... and 109 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
miladh and others added 9 commits November 4, 2026 19:12
The forecast and impute endpoints return their series under "items", like
other CDF list endpoints, rather than "timeSeries".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ForecastInputTimeSeries

InputDatapoint and InputTimeSeries were too generic for data_classes.ai, which
also holds the document AI classes. The forecast and impute endpoints take the
same request body, so both use these classes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a load method

ForecastInputTimeSeries.datapoints accepted datapoints, tuples and dicts,
which widened its type for little gain. It now takes ForecastInputDatapoint
only, and ForecastInputDatapoint.load creates one from a tuple, a list or an
API-style dict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API keys quantiles by their level only ("0.50", or more decimals when
needed, like "0.025"), so there are no other keys to skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
metadata, with quantileLevels and cohorts, is always in the response, and
the levels are numbers already, so there is nothing to default or convert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results have no id, external id or instance id, so get takes just the
label. Labels are unique, since duplicate labels in a request are rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_handle_unknown_arguments_when_loading adds a made-up key to every dict
in a resource. Every key in a quantiles dict must be a quantile level, like
"0.50", so loading fails on the made-up key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@miladh

miladh commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @haakonvt for the useful feedback! It's addressed in 8772cc0e..9ab0e18f, with a reply on each thread. One question is still open, on the _is_number thread.

The range also includes a recent API change: the array of series is now called items instead of timeSeries (a3ec603). This PR only covers the response side; the request side follows in #2875.

It also adds a quantile_levels argument to to_pandas, which now includes only the median (0.5) by default, so DataFrames and the notebook view stay narrow (9ab0e18).

Results hold every quantile level the API returns, which makes the DataFrame
and the notebook view wide, and the levels can change over time. to_pandas
now takes quantile_levels, the levels to include: the median (0.5) by
default, a list of levels, or None for all of them. The notebook view uses
the default.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread cognite/client/data_classes/ai.py Resolved

@haakonvt haakonvt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments 🥳

Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
miladh and others added 4 commits November 6, 2026 14:31
Co-authored-by: Håkon V. Treider <haakonvt@gmail.com>
The supported formats are matched directly: a datapoint, a (timestamp,
value) or (timestamp, value, missing) sequence, or a dict with a timestamp.
A dict without a timestamp, or a sequence of another length, raises the
same TypeError as any other unsupported input. The docstring of
ForecastInputTimeSeries.datapoints says to load datapoints with it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The result lists took the API response in a separate _load_response.
CogniteResourceList already has _load_raw_api_response for lists that need
more than the items of a response, like NodeList with typing, so use it,
with its signature. _load keeps taking the items only, as load passes them.
The responses of all requests of a call are merged: their items, and
their cohorts in the order the API lists them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o_pandas

Taking quantile levels as floats is more than to_pandas needs. It now takes
include_all_quantiles: by default the result holds the median in one column
per time series, with no quantile level in the columns, and with
include_all_quantiles every level is a separate column, as
(label, quantile level) columns for a list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
haakonvt
haakonvt previously approved these changes Nov 6, 2026

@haakonvt haakonvt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a few more nits, but nothing blocking as we are "a long stack away" from making these endpoints public 😅

Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
Comment thread cognite/client/data_classes/ai.py Resolved
…ists

The API now returns quantileLevels and cohorts in a metadata object in
every forecast and impute response. Keep that shape, following the SDK's
precedents for lists with data beyond the items of a response:

- ForecastMetadata holds both fields, required as in the API, and is a
  CogniteResource like TypeInformation, which NodeList holds for typing.
- ForecastResultList and ImputeResultList have it as metadata, the key the
  API uses, instead of two separate attributes.
- Like SyncRecordList, whose nextCursor and hasNext are always in the
  response, metadata is not optional, and a list built from results alone,
  like a slice or load(dump()), gets an empty default.
- Like NodeList, dump_raw returns the items and the metadata as the API
  does, and _load_raw_api_response loads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@haakonvt haakonvt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🦄 one nit you can put in a later pr

# order the API lists them.
metadata = ForecastMetadata(
quantile_levels=responses[0]["metadata"]["quantileLevels"],
cohorts=list(dict.fromkeys(c for response in responses for c in response["metadata"]["cohorts"])),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use from cognite.client.utils._auxiliary import remove_duplicates_keep_order

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added it to #2875 in d55bb65.

@haakonvt haakonvt self-assigned this Nov 9, 2026
@haakonvt haakonvt added risk-review-ongoing Risk review is in progress waiting-for-team Waiting for the submitter or reviewer of the PR to take an action labels Nov 9, 2026
@miladh
miladh added this pull request to the merge queue Nov 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
@miladh
miladh added this pull request to the merge queue Nov 9, 2026
Merged via the queue into master with commit e1ca668 Nov 9, 2026
24 checks passed
@miladh
miladh deleted the ai-ts-forecasting/01-data-classes branch November 9, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk-review-ongoing Risk review is in progress waiting-for-team Waiting for the submitter or reviewer of the PR to take an action

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants