wheels test ignores --filter and --db arguments #2489

Activity

1. bpamiri commented on May 8, 2026

   

   bpamiri

   on May 8, 2026 – with Claude

   Collaborator

   More actions

   Research findings + decisions

   Researched how Rails, Laravel, Symfony, Django, Phoenix, PHPUnit, and pytest handle the two parts of this issue, then mapped the result back to Wheels.

   --filter — already works

   The filter side was fixed by PR #2394 (commit 4c5233b on develop). $normalizeTestFilter maps short names like models to the dotted path tests.specs.models that the app runner's regex accepts. If you saw wheels test --filter=models ignored, you were probably on a snapshot built before that landed — current develop correctly scopes to tests/specs/models/.

   The convention this matches:

   Framework	Mechanism
   Rails	Positional paths (bin/rails test test/models), -n /regex/, file.rb:27
   Laravel	php artisan test --filter=Pattern, --testsuite=Unit
   PHPUnit	phpunit tests/Unit, --filter, --group
   pytest	Positional, pytest tests/foo.py::test_method, -k expression
   Phoenix	Positional, :line suffix, --only tag

   Wheels' --filter, --directory alias, and bare positional all work consistently with the ecosystem.

   --db — design correction, no spec change

   Surveyed how every comparable framework selects the test database:

   Framework	Mechanism
   Rails	DATABASE_URL=postgres://... bin/rails test env var
   Laravel	phpunit.xml <env name="DB_CONNECTION" value="testing"/>
   PHPUnit	<env> blocks in phpunit.xml
   Symfony	APP_ENV=test, DATABASE_URL=... env
   Django	DATABASES settings dict, DJANGO_SETTINGS_MODULE env
   Phoenix / Ecto	MIX_ENV=test, sandbox; matrix runs via env vars

   No framework exposes per-invocation CLI database switching for app tests. The reason is structural: the framework's transactional test isolation needs to know the connection at boot, and switching mid-run breaks that contract. DB selection is a process-level decision; filtering is per-invocation.

   So wheels test --db=mysql for app tests isn't a feature this issue should add. What was a real bug, though, is that the CLI accepted the flag, attached it to the test URL, and printed Running app tests (mysql)... while the framework's vendor/wheels/tests/app-runner.cfm silently ignored url.db — so the user saw a misleading echo of their own flag. The CLI was lying about which engine it ran against.

   Wheels' --core matrix is the legitimate place for --db: the framework's self-test wires up wheelstestdb_<engine> per supported engine. App tests don't have that need.

   Decisions shipped in #2517

   The fixes target the lying-CLI bug and document the convention, without adding the per-invocation switch:

   1. --db warns when passed to an app run. Yellow warning block: "--db only applies to --core tests; ignoring for the app suite." Points at the testing guide for the env-var pattern instead. No silent drop.

   2. The preamble shows the real datasource for app tests. New $resolveAppTestDataSource helper reads .env DATASOURCE_NAME and falls back to config/settings.cfm's set(dataSourceName=...), applying _test when --useTestDB is on. So users see Running app tests (myapp_test)... instead of the misleading (sqlite).

   3. Docs explain the env-var pattern.

      • command-line-tools/wheels-commands/testing.mdx — flag table now annotates --db as --core only; new "Testing against different engines" section walks through the env-var pattern and names the Rails / Laravel / Symfony / Django / Phoenix parity.
      • testing/running-tests-locally.mdx — Aside calling out the same boundary near the example block.

   The research that informed the decisions is on the issue thread above. Closing as completed by #2517.

   ---

   Generated by Claude Code
2. bpamiri

   closed this as completedin #2517on May 8, 2026
3. zainforbjs

   added a commit that references this issue on May 11, 2026

   fix(cli): wheels test honours --db only for --core; surfaces real dat…

   ...

   ab1b099
4. zainforbjs commented on May 11, 2026

   

   zainforbjs

   on May 11, 2026

   Last edited by zainforbjs

   ContributorAuthor

   More actions

   Thanks for the detailed investigation and ecosystem comparison.

   That clears things up — especially the distinction between app-test datasource configuration vs framework/core test matrix execution.

   The clarification around --db only applying to --core tests makes sense, and the updated warning/output behavior should help avoid confusion going forward.

   However, after testing again on the latest snapshot, it appears the --filter behavior may still not be working correctly for app tests.
   Example:

   wheels test --filter=models
   

   still executes all test suites instead of only model tests.

   Current test structure:

   models.PostSpec
   controllers.PostsControllerSpec
   functional.ExampleSpec
   

   but the runner still reports:

   Running app tests (testappdb_test_test)...
   3 passed (0.02s)
   

   I also tested:

   wheels test --filter=model
   wheels test --filter=controller
   

   with the same result — all tests continue running.

   Screenshots

   

   Additionally, there appears to be a small datasource display issue where set(coreTestDataSourceName="testappdb_test"); is being displayed as testappdb_test_test, which looks like an extra _test suffix is being appended in the output.

   Appreciate the fixes and documentation updates overall — just wanted to report these remaining behaviors after retesting on the newer snapshot.
5. zain4bjs

   reopened this on May 11, 2026
6. wheels-dev

   deleted a comment from wheels-bot on May 12, 2026
7. bpamiri

   closed this as completedon May 12, 2026
8. bpamiri

   reopened this on May 12, 2026
9. wheels-bot commented on May 12, 2026

   

   wheels-botBot

   on May 12, 2026

   Contributor

   More actions

   Wheels Bot — Triage

   Classified as bug.

   Suspected layer

   cli — the Wheels CLI's test command. The relevant code lives in cli/lucli/Module.cfc; see also the in-repo CLI guide at cli/CLAUDE.md and the framework-side runner at vendor/wheels/tests/app-runner.cfm.

   Fix sketch

   Two sub-bugs in cli/lucli/Module.cfc:

   1. --filter still ignored after the snapshot built post-fix(cli): wheels test --db is --core only; surface real datasource for app suite #2517. The reporter retested on a current build and short filters (models, model, controller) still run every spec. $normalizeTestFilter (lines ~491–511) maps models → tests.specs.models, and runTests appends &directory=… to the test URL (line ~3752), so the CLI side looks correct on paper. The fix needs to verify three things end-to-end: (a) that the normalized directory= value actually arrives at vendor/wheels/tests/app-runner.cfm, (b) that the runner's directory regex matches the dotted form CLI emits, and (c) that a freshly-reloaded server picks up the change (Lucee inspectTemplate=once can serve a stale runner across reloads). Likely fix surface: app-runner.cfm plus a regression spec exercising the full wheels test --filter=models flow.

   2. testappdb_test_test datasource doubling. $resolveAppTestDataSource (lines 5057–5085) unconditionally appends _test when useTestDB=true, even if the resolved base already ends in _test (the reporter's set(coreTestDataSourceName=\"testappdb_test\") shows up as testappdb_test_test). Mechanical fix: skip the suffix when base already ends in _test, and also consider reading coreTestDataSourceName from config/settings.cfm since that is the setting the reporter actually configured.

   Confidence: `medium`

   The datasource-doubling half is high confidence (obvious off-by-one in string append), but the `--filter` half needs end-to-end investigation across the CLI → URL → `app-runner.cfm` boundary to identify which side is silently dropping the directory parameter — that pushes the overall classification down to medium.

   <!-- wheels-bot:triage:2489 -->
   <!-- wheels-bot:triage-class:bug -->
10. bpamiri

    closed this as completedin #2602on May 12, 2026
11. bpamiri

    added a commit that references this issue on May 12, 2026

    fix(cli): wheels test preamble stops doubling _test for coreTestDataS…

    ...

    1723dd5