Type cursor.description columns from the returned values

[aminghadersohi]

Problem

cursor.description does not describe what a SELECT returns:

• With an explicit column list, every column is STRING, whatever the attributes hold. SELECT id, amount FROM t describes two numbers as STRING.
• A SELECT evaluated by the superset connector's query DB (subqueries, aggregates) reports no type (None).
• DynamoDB has no date type, so dates and timestamps are stored as ISO-8601 strings and are never described as DATE/DATETIME unless the query wraps them in DATETIME(...).

Tools that build a dataset from cursor.description see no numeric or temporal column. For example, a Superset virtual dataset over SELECT id, amount, ts, label FROM t gets four STRING columns, so it cannot use SUM metrics or time grains.

Fix

The executor records the type of every non-NULL value it returns, per column. The description uses that type:

• numbers → NUMBER
• booleans → BOOL
• ISO-8601 dates → DATE
• ISO-8601 timestamps (T or space separator, optional fraction and Z/±HH:MM) → DATETIME; a mix of dates and timestamps is also DATETIME
• anything else, or mixed types → STRING

A column with a type conversion function (NUMBER(x), DATETIME(x)) keeps that function's type. A column with no non-NULL value keeps its previous type. Rows produced by the superset connector's query DB are typed the same way. The query DB's own column types are not changed.

Tests

• New tests/test_value_types.py covers the type merge rules: 15 tests, no database needed.
• Live assertions against DynamoDB Local:
  • test_reserved_word: size[0] returns 100 and 500 and is now described as NUMBER. This test previously asserted the old STRING.
  • test_function_in_columns: a plain column list over number, date, timestamp and text attributes.
  • test_execute_nested_select: the superset connector's query-DB result.
• Full suite on DynamoDB Local 3.3.1 (-sharedDb -inMemory), Python 3.11.15, SQLAlchemy 2.0.52:
  • main: 156 passed, 1 skipped.
  • New tests on main's code: 3 failed, and the new unit module cannot import.
  • This branch: 171 passed, 1 skipped.
• black --check passes.

[passren]

@aminghadersohi Thank you for creating this PR. Actually, I did consider adding a feature to return the actual data type. The reason I didn’t implement it is that DynamoDB, as a NoSQL database, does not enforce a fixed data type for each column (or field). Determining the data type implicitly could require scanning the entire dataset for every query, which could introduce performance concerns.

That’s why I added the data type annotation function, which allows users to explicitly specify the expected data type when needed.

[aminghadersohi]

Thanks for the context — and agreed, inferring types by scanning a table up front would be a real problem for a NoSQL store like this.

This isn't doing that: observe_value_types (pydynamodb/executor.py) only inspects the rows a query has already returned via ExecuteStatement, after they're paged in — the exact rows the caller is fetching anyway, not a separate read. The superset connector's query-DB path works the same way: it types the rows that query DB already produced (pydynamodb/superset_dynamodb/pydynamodb.py), again with no extra call to DynamoDB. So there's no additional scan or performance cost beyond what the query already does.

The type-annotation function stays authoritative: for a column using NUMBER(x)/DATETIME(x), that function's type is used as-is and the observed value type is ignored for it (pydynamodb/result_set.py, the column_info.function is None check). This change only fills in a default for columns with no annotation, based on the values actually returned — so the annotation function remains the way to force a type when the returned rows don't reliably indicate one (e.g., a column that's genuinely mixed, or all-NULL on a given page).

If you'd rather this be opt-in instead of the new default, I'm happy to gate it behind a cursor/connect-string flag — let me know and I'll add it.

[passren]

@aminghadersohi A flag in the connection string would be a great idea. It would allow users to decide whether they want to enable data type conversion by default. Thank you!