[aminghadersohi]

Problem

Every statement parser runs Base.root_parse_results, which calls parseString() without an end anchor. pyparsing therefore stops at the first token the grammar does not recognize and silently discards the rest of the statement. The driver then executes something other than what was written:

• SELECT with a table alias drops the WHERE clause and returns wrong rows. SELECT id FROM t AS a WHERE id = ? runs as SELECT id FROM "t" and returns every row, although DynamoDB itself rejects aliasing (ValidationException: Aliasing is not supported). Bare aliases (FROM t a), GROUP BY, ORDER BY without a direction and a second statement after ; are dropped the same way.
• Other statement types can act on the wrong target. DROP TABLE Issues Topics drops Issues, and DESC Issues Topics describes Issues. A trailing token after an UPDATE/DELETE WHERE clause is dropped and the statement still runs.
• INSERT truncates literal nested maps. The item grammar stops at the first }, so VALUE {'a': {'k': '1'}} is sent without its final brace.

A related case: with connector=superset, a single-table SELECT that PartiQL cannot express fails to parse. This covers aggregates and GROUP BY on a table, which is what a chart on a physical table issues, e.g. SELECT txt, COUNT(*) FROM t WHERE ... GROUP BY txt. With a table alias it runs without its WHERE clause. Only the nested form (... FROM (SELECT ...) AS virtual_table ...) was evaluated by the query DB.

Fix

• Full match for all parsers. Every parser (CREATE / CREATE GLOBAL, ALTER, DROP / DROP GLOBAL, INSERT, UPDATE, DELETE, SELECT, LIST / LIST GLOBAL, DESC / DESC GLOBAL, and the superset SELECT) now requires the whole statement to match. At most one trailing semicolon is allowed; anything else raises ParseException.
• Balanced INSERT items. The item is matched with balanced braces, ignoring braces inside quoted strings, and passed through unchanged.
• Single-table aliases (commit a45e259). FROM t [AS] a with a.col references is rewritten to plain PartiQL on both cursors, since DynamoDB rejects table aliases and reads a.col as a document path. Joins and subqueries are not rewritten.
• SQL comments (commit a45e259). Comments are stripped before parsing (quoted text is kept), so a trailing -- tag line appended by query logging is not a parse error under the full match.
• ORDER BY (commit a45e259) accepts a key list with optional ASC/DESC; the grammar required exactly one key with a direction, so ORDER BY id failed.
• Flat SELECTs under connector=superset. A base SELECT is used only when it matches the whole statement (after alias resolution). Otherwise a single-table SELECT is read with a PartiQL scan (SELECT * FROM "t") and evaluated as a whole by the query DB, exactly like the outer part of a nested SELECT. Bind parameters are rejected for this form with a clear NotSupportedError, because only the scan is sent to DynamoDB.
• Query DB loader. It now reads all pages before creating its table and pads rows read before an attribute first appeared (items are schemaless, so a later page can add attributes). Map/list/set values are stored as JSON text. Previously a SELECT * load on heterogeneous items failed with a binding error.

Changed existing tests

Three tests asserted the old lenient behaviour. Silently ignoring trailing tokens can make the driver act on the wrong target, so these now expect an error or the complete statement:

• test_ddl_drop.py::test_parse_drop: DROP TABLE Issues Topics now raises instead of dropping Issues.
• test_util_sql.py::test_desc_table: desc Issues Topics now raises instead of describing Issues.
• test_dml_insert.py::test_parse_simple_case_2: the nested-map INSERT now keeps its closing brace. The old expected statement was unbalanced.

New tests

• tests/test_parse_trailing_tokens.py:
  • each parser accepts its complete statement (with and without ;) and rejects Unexpected, Unexpected Tokens, ; DROP TABLE Other and ;;;
  • nested and quoted-brace INSERT items are kept intact, and unbalanced items are rejected;
  • the superset SELECT rejects a second statement.
• SELECT parser tests for aliases, GROUP BY, ORDER BY and the trailing semicolon.
• Superset parser tests:
  • an aggregate / GROUP BY flat SELECT becomes scan + query DB evaluation with the WHERE intact;
  • a plain base SELECT is unchanged;
  • a nested virtual_table query is unchanged.
• Live tests:
  • SQLAlchemy alias: both cursors return exactly the filtered rows, with literal or bound values; a subquery (default cursor) or a join (connector=superset) still raises;
  • a flat COUNT/SUM ... GROUP BY returns the same result as the nested equivalent;
  • a parameterized flat SELECT raises a clear error.

Against DynamoDB Local 3.3.1 (Python 3.11, SQLAlchemy 2.0.52, pyparsing 3.3.2): this branch gives 244 passed, 1 skipped. main source with the new tests gives 167 passed / 77 failed (every new or changed expectation). black --check pydynamodb and flake8 pydynamodb are clean.

Follow-up: series-limit JOINs (commit 74f9ad1)

A Superset series-limit chart JOINs the table against its own top-N subquery: (SELECT col AS col__, ...) AS series_limit ON col = col__. Two things made it fail with connector=superset:

• Dropped labels. The statement compiler dropped every label, so the subquery lost AS col__. Labels inside subqueries are now rendered when they rename their element. Top-level labels are still omitted, because PartiQL has no column aliases.
• Unscanned table name. The flat form evaluated FROM <cache> while the subquery still named the DynamoDB table. The query DB now evaluates the statement with a CTE named after the scanned table, so every reference reads the scanned rows.

Without connector=superset, such a SELECT cannot be expressed in PartiQL. The default cursor now raises NotSupportedError saying so, instead of a raw ParseException.

New tests:

• compiled labels (top level vs subquery);
• a live series-limit JOIN returns the expected top-N aggregate;
• the default cursor raises the explanatory error.

This head gives 247 passed / 1 skipped. The previous head with these tests gives 4 failed.

Follow-up: comments, ORDER BY lists and single-table aliases (commit a45e259)

The full-match parse made trailing SQL comments a parse error, and ORDER BY id without a direction failed. Comments are now stripped before parsing, ORDER BY takes a key list with optional directions, and a single-table alias is resolved to plain PartiQL on both cursors instead of being rejected (with connector=superset, an aliased SELECT therefore runs as PartiQL, with bind parameters, rather than as a scan evaluated in the query DB). Joins and subqueries are unchanged.

New tests in tests/test_comments_order_by_aliases.py (comment stripping incl. markers inside quotes, ORDER BY key lists, alias resolution, statements that must not be rewritten, both cursors running an aliased SELECT). One existing SELECT parser case now uses ORDER BY id NULLS FIRST as its unsupported trailing clause, since ORDER BY id is now valid.

DynamoDB Local 3.3.1: full suite 273 passed, 1 skipped. With the source changes reverted and the new tests kept: 16 failed.

Follow-up: pin pyparsing below 3.3.3 (commit c64b482)

pyparsing 3.3.3 (released after the last update above) rewrote infix_notation and nested_expr, which changed how much of a statement the WHERE-clause grammar consumes. Under 3.3.3 the full-match parse this PR added starts rejecting valid statements (SELECT id FROM t WHERE id = 'x' ORDER BY id fails with "Expected end of text, found 'WHERE'") and a contains(...)/begins_with(...) call in a WHERE clause loses the comma between its arguments. Verified: with pyparsing 3.3.2 the full suite is green; with 3.3.3, eight tests fail. setup.py, requirements.txt, and the test workflow's install step now cap pyparsing<3.3.3 until the grammar is adapted to the new implementation.

Also fixed test_reflected_default_schema_select (pulled in by the merge from main/#83), which seeded two rows into the table shared with test_sqlalchemy_dynamodb.py::test_reflect_table and never removed them, making that test's exact row-count assertion depend on execution order. It now cleans up the rows it seeds.

This head (merged with current main) gives 310 passed against DynamoDB Local, verified on Python 3.9-3.14 with both SQLAlchemy 1.x and 2.x via the fork's own Actions run.

[aminghadersohi]

Added a45e259: the full-match parse made trailing SQL comments a parse error (for example a -- tag line that query logging appends), and ORDER BY id without a direction failed because the grammar required exactly one key with ASC/DESC. Comments are now stripped before parsing (quoted text kept), and ORDER BY takes a key list with optional directions. A single-table alias (FROM t [AS] a, a.col) is now resolved to plain PartiQL on both cursors instead of being rejected. Joins and subqueries are unchanged.

DynamoDB Local 3.3.1: full suite 273 passed, 1 skipped. With the source changes reverted and the new tests kept: 16 failed.

[aminghadersohi]

Pushed c64b482: pins pyparsing below 3.3.3 (a newly released version that rewrote infix_notation/nested_expr and broke WHERE-clause full-statement matching — 8 tests failed under 3.3.3, all pass under 3.3.2), and fixes a test-isolation bug in test_reflected_default_schema_select (pulled in from the merge with main/#83) that leaked two rows into a table shared with test_sqlalchemy_dynamodb.py::test_reflect_table.

Also merged current main into this branch (clean, no conflicts).

Verified locally and via the fork's own Actions run (https://github.com/aminghadersohi/PyDynamoDB/actions/runs/36667961587): all pytest/black/flake8 steps pass across Python 3.9-3.14 for both SQLAlchemy 1.x and 2.x (310 passed each). The only failing jobs there are SonarCloud and Codecov upload, which fail solely because their tokens are repo secrets not available to a fork-triggered run — unrelated to this PR's code.

Note for visibility: this repo's run-test.yaml triggers only on push, not pull_request, so it never runs against the base repo for a fork-sourced PR and won't show up under this PR's Checks tab. That's a pre-existing repo-wide condition (affects #83, #85, #86 too), not something specific to this branch, so I left the workflow trigger as-is.

[passren]

@aminghadersohi Thank you. Acknowledged and Merged!