Repository navigation
WheelsTest: spec files that skip describe() or error during load fail silently in the runner #2829
Description
Symptom
When a spec file under tests/specs/ either:
- has
it()calls at the top ofrun()without adescribe()wrapper, OR - errors on load (e.g., the spec references
application.wo.Xfrom a global include that doesn't exist yet)
…the test runner reports nothing about it. The summary line shows:
0 passed
(for case 1, when running only the malformed spec via --spec=path/to/X)
or:
48 passed, 0 failed, -1 error(s) (9.13s)
(for case 2, when running alongside passing specs — note the -1 error count).
No filename, no error message, no indication that a spec file failed to load.
Reproduction
Case 1 — it() at top of run(), no describe()
component extends="wheels.WheelsTest" {
function run() {
it("does a thing", () => {
expect(true).toBeTrue();
});
}
}Running wheels test --spec=path/to/that/file reports 0 passed. Wrapping the it() in describe("X", () => { ... }) makes it run normally. The runner gives no hint that the bare it() form is invalid.
Case 2 — Spec errors during load
component extends="wheels.WheelsTest" {
function beforeAll() {
variables.helper = application.wo.functionDoesNotExist;
}
function run() {
describe("foo", () => {
it("does a thing", () => {
expect(true).toBeTrue();
});
});
}
}If the spec file throws during load (here application.wo.functionDoesNotExist is undefined), the runner reports -1 error(s) in the summary count rather than printing the actual exception and the offending file name.
Why it matters
Both shapes are easy to land accidentally during iteration (especially the application.wo.X case during TDD when you're writing the spec before the helper). The silent failure costs minutes of "wait, why isn't my new test running?" debugging per occurrence, and the -1 error(s) count is itself a clue that something is wrong but unhelpful as a starting point for diagnosis.
Suggestion
For case 1: at minimum, log a warning when a spec file's run() contains it() calls that didn't end up registered. Ideally, register them under an auto-generated top-level describe (e.g. <filename>) so they DO run, since that's likely what the author intended.
For case 2: when a spec file throws during load, print the file path + the exception message + a short stack frame, and count it as an error (positive, with file context) rather than a -1.
Discovered during
Phase 1 of an internal RBAC portal build on Wheels 4.0.2 / Lucee 7 — full context in https://github.com/paiindustries/titan/pull/3337 (private, but happy to extract specific snippets if useful).
Activity
wheels-bot commentedon May 28, 2026
Wheels Bot — Triage
Classified as bug.
Suspected layer
cli— see.ai/wheels/testing/for the runner/BDD reference (the layer list doesn't have a dedicatedtestingentry; the user-visible summary line and exit aggregation live incli/lucli/services/TestRunner.cfc, while the BDDdescribe/itDSL itself lives invendor/wheels/WheelsTest.cfcandvendor/wheels/wheelstest/).Fix sketch
Two adjacent defects:
Case 1 (bare
it()at top ofrun()is silently dropped): the BDD DSL invendor/wheels/WheelsTest.cfcregistersit()callbacks against a current-describe context. Whenrun()invokesit()without an enclosingdescribe(...), the call has no parent to attach to and gets discarded. Fix shape: either (a) throw a clearWheels.Test.OrphanItexception naming the file whenit()is called outside adescribe, or (b) auto-wrap orphanit()calls under a synthetic top-level describe named after the spec file. The suggestion box in the issue argues for (b); the safer mechanical fix is (a). Pick one — this is a small design call propose-fix should make explicit.Case 2 (
-1 error(s)for load-time exceptions): the summary aggregation incli/lucli/services/TestRunner.cfc(and/or the harness invendor/wheels/tests/TestRunner.cfcfor the framework suite) is computingerrors = total - pass - failafter a load-time throw lefttotallower thanpass + fail. Fix shape: catch per-spec load exceptions, record{file, message, stackFrame}against an expliciterrorslist, and surface filename + message in the summary instead of relying on subtraction.Confidence:
mediumTwo distinct defects in one report and Case 1 has a design call (warn vs. auto-wrap); spec runner code paths are exercised on every engine so any DSL change wants a cross-engine sanity check before landing.
<!-- wheels-bot:triage:2829 -->
<!-- wheels-bot:triage-class:bug -->
<!-- wheels-bot:triage-confidence:medium -->
wheels-bot commentedon May 28, 2026
Wheels Bot — Fix proposed
Draft PR: #2831
Spec:
vendor/wheels/tests/specs/wheelstest/BDDRunnerErrorReportingSpec.cfc— failing → passing (via fixturesOrphanItFixture.cfcandBeforeAllErrorFixture.cfcundervendor/wheels/tests/resources/wheelstest/)Implementation:
vendor/wheels/wheelstest/system/runners/BDDRunner.cfc— bundle stats start beforetarget.run(), the outer try now wraps the suite-descriptor pass and the bundle execution together, and the catch records a positive error count.A human review is required before merge. Reviewer A and Reviewer B will weigh in shortly. Doc updates (MDX guides,
.ai/wheels/,CLAUDE.md) are handled separately bybot-update-docs.yml.fix(test): skip server-dependent migrate/test command specs (no serve…
fix(test): repair CLI suite failures masked by the BDDRunner error se…
fix(wheelstest): capture spec-load errors against the bundle and emit…
fix(wheelstest): capture spec-load errors against the bundle and emit…
fix(cli): describe packages install as an alias of add (#4211)