Skip to content

Commit be85bdb

Browse files
authored
test: add route to page object registry and route coverage QA stats (#47691)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** Adds test/e2e/route-registry, which maps every route in ROUTES to its E2E page object, or to null when there isn't one yet. Tools can use it to work out which screen a route is (for example mm describe-screen), and it shows which routes have no page object. The nightly QA Stats workflow uploads the report as the route-coverage-report artifact and adds an e2e_routes section to qa-stats.json. When adding, renaming or removing a route, update the registry, using null if there's no page object yet. route-registry.test.ts fails and lists the paths to fix if they drift apart. A page object is not required to not make it disruptive. Adds non-blocking review guidance (Bugbot rule 3.10 and a note in test/e2e/AGENTS.md) suggesting that new page objects for full screens with their own route be mapped in the route registry, while modals and components are skipped. ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** 1. Run `yarn test:e2e:route-coverage` to get the stats ## **Screenshots/Recordings** <img width="1704" height="1372" alt="image" src="https://github.com/user-attachments/assets/4f938cc6-a5de-4e1e-be06-707075ae11e0" /> <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [x] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Extension Coding Standards](https://github.com/MetaMask/metamask-extension/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [x] I've completed the PR template to the best of my ability - [x] I’ve included tests if applicable - [x] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [x] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-extension/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches nightly QA metrics, adds a large route registry map, and relies on static analysis staying aligned with routing changes—mis-entries could skew dashboards until caught by tests. > > **Overview** > **Route to page object coverage** adds `test/e2e/route-registry`, a shared `ROUTE_PAGE_OBJECT_REGISTRY`, `getRouteCoverage` helper, focused Jest suites, and a `yarn test:e2e:route-coverage` CLI (`route-coverage-report.ts`) that prints plus JSON reports under `test/test-results/e2e`. > > **QA automation** now runs that script in `qa-stats.yml`, uploads `route-coverage-report`, extends `collect-qa-stats.mts` with an `e2e_routes` namespace, and wires the route-registry tests into `jest.config.js`. > > Tooling bumps include the new Yarn script, and `eslint-suppressions.json` grows slightly to match the expanded collector. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e0dc977. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent c3a708b commit be85bdb

13 files changed

Lines changed: 640 additions & 4 deletions

‎.cursor/BUGBOT.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ Use `mms-extension-testing` unit reference for unit-test quality and bug detecti
3939
- Check for proper imports from the E2E framework
4040
- Verify Page Object Model pattern is used
4141
- Ensure tests are written in TypeScript (.spec.ts)
42-
- **ALWAYS** apply the Page Object Model rules in sections **3.3–3.9** to every changed file under `test/e2e/`. These sections are written for **local review** (CODEBOT / `/review` / local Bugbot run) and for agent self-check while writing tests — see `mms-extension-testing` → `references/e2e/pom-antipatterns.md`.
42+
- **ALWAYS** apply the Page Object Model rules in sections **3.3–3.10** to every changed file under `test/e2e/`. These sections are written for **local review** (CODEBOT / `/review` / local Bugbot run) and for agent self-check while writing tests — see `mms-extension-testing` → `references/e2e/pom-antipatterns.md`.
4343

4444
Use `mms-extension-testing` E2E references for test quality and POM enforcement.
4545

@@ -157,6 +157,13 @@ When both conditions hold, inspect the fixture builder chain in the same `withFi
157157
- Do not report `try`/`catch` inside a `driver.wait` polling callback that returns `false`, or framework helpers under `test/e2e/webdriver/`.
158158
- Severity: **MEDIUM**. Comment: "Do not swallow failures in page objects or flows. Let the error surface, and use driver wait helpers and `check*` methods with clear error messages."
159159

160+
#### 3.10 New page object and the route registry (Non-Blocking)
161+
162+
- Scope: new files under `test/e2e/page-objects/pages/**/*.{ts,js}`.
163+
- Detect: the diff adds a page object class and does not change `test/e2e/route-registry/route-registry.ts`.
164+
- Do not report page objects for modals, components, navbars, headers or any other part of a screen that no route renders on its own. Only full screens with their own route belong in the registry.
165+
- Severity: **LOW** (suggestion), non-blocking. Comment: "Does this page object cover a full screen with its own route? If so, consider mapping it in `test/e2e/route-registry/route-registry.ts` (replacing `null`) so route coverage and `mm describe-screen` pick it up. Ignore this if it is a modal or component."
166+
160167
### 4. Controller Guidelines
161168

162169
- **ALWAYS** load and reference [controller-guidelines](rules/controller-guidelines/RULE.md)

‎.github/scripts/collect-qa-stats.mts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -960,6 +960,39 @@ async function collectFeatureFlagCoverage(): Promise<Record<string, number>> {
960960
};
961961
}
962962

963+
// ---------------------------------------------------------------------------
964+
// Route page object coverage collector
965+
// ---------------------------------------------------------------------------
966+
967+
/**
968+
* Reports how many extension routes have an E2E page object, using the
969+
* shared report generator in test/e2e/scripts/route-coverage-report.ts.
970+
*
971+
* No artifact download required — the analysis compares `ROUTES` with the
972+
* route page object registry in test/e2e/route-registry.
973+
*/
974+
async function collectRouteCoverage(): Promise<Record<string, number>> {
975+
console.log('[e2e_routes] running static route coverage analysis...');
976+
977+
const { generateReport } =
978+
await import('../../test/e2e/scripts/route-coverage-report');
979+
980+
const { summary } = generateReport();
981+
982+
console.log(
983+
`[e2e_routes] total: ${summary.totalRoutes}, with page object: ${summary.withPageObject}, coverage: ${summary.coveragePercentage}%`,
984+
);
985+
986+
return {
987+
total_routes: summary.totalRoutes,
988+
routes_with_page_object: summary.withPageObject,
989+
routes_without_page_object: summary.withoutPageObject,
990+
unregistered_routes: summary.unregistered,
991+
stale_registry_entries: summary.staleEntries,
992+
coverage_percentage: summary.coveragePercentage,
993+
};
994+
}
995+
963996
// ---------------------------------------------------------------------------
964997
// Main
965998
// ---------------------------------------------------------------------------
@@ -974,6 +1007,7 @@ async function main(): Promise<void> {
9741007
{ namespace: 'benchmark', collect: collectBenchmarkScenarioCount },
9751008
{ namespace: 'metametrics', collect: collectMetametricsQaStats },
9761009
{ namespace: 'e2e_feature_flags', collect: collectFeatureFlagCoverage },
1010+
{ namespace: 'e2e_routes', collect: collectRouteCoverage },
9771011
];
9781012

9791013
for (const { namespace, collect } of collectors) {

‎.github/workflows/qa-stats.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ jobs:
2727
- name: Generate feature flag coverage report
2828
run: yarn test:e2e:feature-flag:coverage
2929

30+
- name: Generate route coverage report
31+
run: yarn test:e2e:route-coverage
32+
3033
- name: Collect QA stats
3134
run: yarn tsx .github/scripts/collect-qa-stats.mts
3235
env:
@@ -46,3 +49,10 @@ jobs:
4649
name: feature-flag-coverage-report
4750
path: ./test/test-results/e2e/feature-flag-coverage-report.json
4851
if-no-files-found: warn
52+
53+
- name: Upload route coverage report
54+
uses: actions/upload-artifact@v7
55+
with:
56+
name: route-coverage-report
57+
path: ./test/test-results/e2e/route-coverage-report.json
58+
if-no-files-found: warn

‎eslint-suppressions.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
},
6868
".github/scripts/collect-qa-stats.mts": {
6969
"@typescript-eslint/naming-convention": {
70-
"count": 22
70+
"count": 28
7171
},
7272
"no-plusplus": {
7373
"count": 3

‎jest.config.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ module.exports = {
9090
'<rootDir>/test/e2e/helpers/**/*.test.(js|ts|tsx)',
9191
'<rootDir>/test/e2e/benchmarks/**/*.test.(js|ts|tsx)',
9292
'<rootDir>/test/e2e/feature-flags/**/*.test.(js|ts|tsx)',
93+
'<rootDir>/test/e2e/route-registry/**/*.test.(js|ts|tsx)',
9394
'<rootDir>/test/e2e/playwright/llm-workflow/**/*.test.(js|ts|tsx)',
9495
'<rootDir>/test/release-testing-ai-analyzer/**/*.test.ts',
9596
],

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@
9898
"test:e2e:pw:report": "yarn playwright show-report public/playwright/playwright-reports/html",
9999
"test:e2e:smoke:local-node-ports": "tsx test/e2e/scripts/smoke-local-node-ports.ts",
100100
"test:e2e:feature-flag:coverage": "tsx test/e2e/scripts/feature-flag-coverage-report.ts",
101+
"test:e2e:route-coverage": "tsx test/e2e/scripts/route-coverage-report.ts",
101102
"test:e2e:chrome:rpc": "SELENIUM_BROWSER=chrome tsx test/e2e/run-all.mts --rpc",
102103
"test:e2e:chrome:multi-provider": "MULTIPROVIDER=true SELENIUM_BROWSER=chrome tsx test/e2e/run-all.mts --multi-provider",
103104
"test:e2e:firefox": "SELENIUM_BROWSER=firefox tsx test/e2e/run-all.mts",

‎test/e2e/AGENTS.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Instructions for AI coding agents working on E2E tests in the MetaMask Browser E
1212
- **E2E create:** `mms-extension-testing` → `references/e2e.md` → `references/e2e/writing-tests.md`
1313
- **E2E maintain (flake / bad practices):** `mms-extension-testing` → `references/e2e.md` → `references/e2e/maintain.md`
1414
- **E2E CI decision tree:** [.github/guidelines/E2E_DECISION_TREE.md](../../.github/guidelines/E2E_DECISION_TREE.md) — when E2E runs/skips, label effects, build-reuse logic
15-
- **E2E deprecated patterns & POM anti-patterns (local review):** [.cursor/BUGBOT.md](../../.cursor/BUGBOT.md) — sections **3.1–3.9**. Applied by **local** review (CODEBOT / `/review` / local Bugbot run) and agent self-check, **not** by Bugbot on the PR — see skill `references/e2e/pom-antipatterns.md`.
15+
- **E2E deprecated patterns & POM anti-patterns (local review):** [.cursor/BUGBOT.md](../../.cursor/BUGBOT.md) — sections **3.1–3.10**. Applied by **local** review (CODEBOT / `/review` / local Bugbot run) and agent self-check, **not** by Bugbot on the PR — see skill `references/e2e/pom-antipatterns.md`.
1616
- **Human E2E guidelines (POM best practices):** [contributor-docs extension E2E guidelines](https://github.com/MetaMask/contributor-docs/blob/main/docs/testing/e2e/extension-e2e-guidelines.md#best-practices)
1717
- **Testing philosophy:** [docs/testing.md](../../docs/testing.md)
1818
- **E2E Driver API:** [webdriver/README.md](./webdriver/README.md)
@@ -28,7 +28,7 @@ Deprecated standalone skills (`mms-e2e-testing`, `mms-e2e-flakiness-patterns`) r
2828

2929
## POM anti-patterns (quick checklist)
3030

31-
Write E2E correctly from the start using `mms-extension-testing`. Before submitting, run local review against [.cursor/BUGBOT.md](../../.cursor/BUGBOT.md) sections **3.3–3.9** and the skill checklist `references/e2e/pom-antipatterns.md`. Do **not** introduce:
31+
Write E2E correctly from the start using `mms-extension-testing`. Before submitting, run local review against [.cursor/BUGBOT.md](../../.cursor/BUGBOT.md) sections **3.3–3.10** and the skill checklist `references/e2e/pom-antipatterns.md`. Do **not** introduce:
3232

3333
| Avoid | Prefer |
3434
| ------------------------------------------------------------- | ----------------------------------------------------- |
@@ -40,4 +40,6 @@ Write E2E correctly from the start using `mms-extension-testing`. Before submitt
4040
| Page object importing/calling another page object | Flow that owns both page objects |
4141
| `try` / `catch` in page objects or flows | Let failures surface via waits / `check*` messages |
4242

43+
**Route registry:** when you add a page object for a full screen that has its own route, map it in [route-registry/route-registry.ts](./route-registry/route-registry.ts), replacing `null`. Skip this for page objects that cover modals or components. It is optional and never blocks a PR; only adding or removing routes is enforced, by `route-registry.test.ts`.
44+
4345
Run `yarn skills` after `yarn install` if `.cursor/rules/mms-extension-testing/` is missing locally.

‎test/e2e/route-registry/index.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
export { ROUTE_PAGE_OBJECT_REGISTRY } from './route-registry';
2+
export type {
3+
PageObjectClass,
4+
RoutePageObjectRegistry,
5+
} from './route-registry';
6+
export { getRouteCoverage } from './route-coverage';
7+
export type {
8+
RouteCoverageEntry,
9+
RouteCoverageReport,
10+
RouteCoverageStatus,
11+
} from './route-coverage';
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
import { getRouteCoverage } from './route-coverage';
2+
import type { RoutePageObjectRegistry } from './route-registry';
3+
4+
class HomePage {}
5+
6+
class SendPage {}
7+
8+
const ROUTES = [
9+
{ path: '/', label: 'Home' },
10+
{ path: '/send', label: 'Send Page' },
11+
{ path: '/activity', label: 'Activity' },
12+
{ path: '/lock', label: 'Lock Page' },
13+
];
14+
15+
describe('getRouteCoverage', () => {
16+
const registry: RoutePageObjectRegistry = {
17+
'/': HomePage,
18+
'/send': SendPage,
19+
'/activity': null,
20+
'/removed-route': HomePage,
21+
};
22+
23+
it('reports each route with its page object, none yet, or unregistered', () => {
24+
const { routes } = getRouteCoverage(ROUTES, registry);
25+
26+
expect(routes).toStrictEqual([
27+
{
28+
path: '/',
29+
label: 'Home',
30+
status: 'page-object',
31+
pageObject: 'HomePage',
32+
},
33+
{
34+
path: '/send',
35+
label: 'Send Page',
36+
status: 'page-object',
37+
pageObject: 'SendPage',
38+
},
39+
{ path: '/activity', label: 'Activity', status: 'no-page-object' },
40+
{ path: '/lock', label: 'Lock Page', status: 'unregistered' },
41+
]);
42+
});
43+
44+
it('lists registry entries for routes that no longer exist', () => {
45+
const { staleEntries } = getRouteCoverage(ROUTES, registry);
46+
47+
expect(staleEntries).toStrictEqual(['/removed-route']);
48+
});
49+
50+
it('summarizes the counts and the share of routes with a page object', () => {
51+
const { summary } = getRouteCoverage(ROUTES, registry);
52+
53+
expect(summary).toStrictEqual({
54+
totalRoutes: 4,
55+
withPageObject: 2,
56+
withoutPageObject: 1,
57+
unregistered: 1,
58+
staleEntries: 1,
59+
coveragePercentage: 50,
60+
});
61+
});
62+
63+
it('reports 0% coverage when there are no routes', () => {
64+
const { summary } = getRouteCoverage([], {});
65+
66+
expect(summary.coveragePercentage).toBe(0);
67+
});
68+
});
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import type { RoutePageObjectRegistry } from './route-registry';
2+
3+
export type RouteCoverageStatus =
4+
| 'page-object'
5+
| 'no-page-object'
6+
| 'unregistered';
7+
8+
export type RouteCoverageEntry = {
9+
path: string;
10+
label: string;
11+
status: RouteCoverageStatus;
12+
/** Class name of the page object, when there is one. */
13+
pageObject?: string;
14+
};
15+
16+
export type RouteCoverageReport = {
17+
generatedAt: string;
18+
summary: {
19+
totalRoutes: number;
20+
withPageObject: number;
21+
withoutPageObject: number;
22+
unregistered: number;
23+
staleEntries: number;
24+
coveragePercentage: number;
25+
};
26+
routes: RouteCoverageEntry[];
27+
/** Registry paths that are not in `ROUTES` (renamed or removed routes). */
28+
staleEntries: string[];
29+
};
30+
31+
/**
32+
* Compares the routes with the page object registry.
33+
*
34+
* @param routes - The routes to cover, normally `ROUTES`.
35+
* @param registry - Route path → page object (or `null`).
36+
* @returns Per-route status, stale registry entries and a summary.
37+
*/
38+
export function getRouteCoverage(
39+
routes: readonly { path: string; label: string }[],
40+
registry: RoutePageObjectRegistry,
41+
): RouteCoverageReport {
42+
const entries = routes.map(({ path, label }): RouteCoverageEntry => {
43+
if (!Object.hasOwn(registry, path)) {
44+
return { path, label, status: 'unregistered' };
45+
}
46+
const pageObject = registry[path];
47+
return pageObject
48+
? { path, label, status: 'page-object', pageObject: pageObject.name }
49+
: { path, label, status: 'no-page-object' };
50+
});
51+
52+
const routePaths = new Set(routes.map(({ path }) => path));
53+
const staleEntries = Object.keys(registry).filter(
54+
(path) => !routePaths.has(path),
55+
);
56+
const count = (status: RouteCoverageStatus) =>
57+
entries.filter((entry) => entry.status === status).length;
58+
const withPageObject = count('page-object');
59+
60+
return {
61+
generatedAt: new Date().toISOString(),
62+
summary: {
63+
totalRoutes: entries.length,
64+
withPageObject,
65+
withoutPageObject: count('no-page-object'),
66+
unregistered: count('unregistered'),
67+
staleEntries: staleEntries.length,
68+
coveragePercentage:
69+
entries.length > 0
70+
? Math.round((withPageObject / entries.length) * 100)
71+
: 0,
72+
},
73+
routes: entries,
74+
staleEntries,
75+
};
76+
}

0 commit comments

Comments
 (0)