Skip to content

[Due for payment 2026-08-12] ESLint CI: cache fallback hides lint violations and cold full-repo lint OOMs #96660

Closed

Description

@VickyStash

Problem

Two related problems in .github/workflows/lint.yml.

1. Cache fallback restores a stale cache, hiding violations

The restore step falls back to a catch-all key:

restore-keys: |
  ${{ runner.os }}-eslint-${{ hashFiles(...) }}-
  ${{ runner.os }}-eslint-        # any latest eslint cache, regardless of config

When a lint-config/rule change lands (e.g. bumping eslint-config-expensify), the exact key misses but the catch-all restores a cache built under the old rules. ESLint only caches files that lint clean and only re-lints files whose content or config hash changed — plugin source inside node_modules is not part of that hash. So files that were clean under the old rule are never re-evaluated under the new rule, and new violations silently pass CI. Only a cold full lint catches them — the exact run that OOMs.

2. Cold full-repo lint runs out of memory

Whenever the first npm run lint fails for any reason, the retry logic clears the cache and re-runs cold:

if ! npm run lint; then
  echo "Lint failed, clearing cache and retrying..."
  rm -rf node_modules/.cache/eslint
  npm run lint
fi

That cold, full-repo lint runs with --concurrency=auto, each allowed up to --max_old_space_size=8192. Combined demand exceeds runner RAM (~16GB) and a worker gets killed:

Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
##[error]Process completed with exit code 2.

This is pre-existing and independent of any single PR — it reproduces on unrelated commits whose only commonality is that the first lint pass failed. Examples:

All three: first cached lint fails → clearing cache and retrying... → cold lint → OOM.

Proposed fix

  1. Bust the cache only on lint config / lint dependency changes. Hash a lockfile slice containing only lint-affecting deps (anything matching eslint, which covers eslint, all eslint-*/@typescript-eslint plugins, and the eslint-config-expensify github dep) plus eslint.config.mjs and config/eslint/**. Drop the catch-all ${{ runner.os }}-eslint- fallback so a config/dep change gets a clean cold lint instead of a stale cache.
  2. Lower cold-lint concurrency (--concurrency=2 in scripts/lint.sh) so the full re-lint fits in runner memory (2 × 8GB ≈ 16GB). But we should verify how long the lint checks will take then.
Issue OwnerCurrent Issue Owner: @VickyStash

Activity

  1. self-assigned this
    on Jul 21, 2026
  2. VickyStash commented on Jul 23, 2026

    @VickyStash
    ContributorAuthor

    I'm going to be OOO till Monday, plan to work on this one next week

  3. VickyStash commented on Jul 31, 2026

    @VickyStash
    ContributorAuthor

    Cold full-repo lint OOMs were fixed in this PR: #96205 by moving the lint job to an 8vcpu/32GB runner and pinning ESLint to 2 workers with a 14GB heap cap (ESLINT_CONCURRENCY: 2, NODE_OPTIONS: --max_old_space_size=14336), so a cold, type-aware full-repo lint now fits in runner memory instead of spawning --concurrency=auto workers that together exceeded the old 16GB runner.

  4. VickyStash commented on Jul 31, 2026

    @VickyStash
    ContributorAuthor

    Here is the PR to fix cache fallback: #97551

  5. VickyStash commented on Jul 31, 2026

    @VickyStash
    ContributorAuthor

    I'll be OOO next week (back Aug 10) 🌴
    If any adjustments are needed before that, please ask @TMisiukiewicz so he can help to find someone

  6. added and removed
    ReviewingHas a PR in review
    on Aug 5, 2026
  7. changed the title [-]ESLint CI: cache fallback hides lint violations and cold full-repo lint OOMs[/-] [+][Due for payment 2026-08-12] ESLint CI: cache fallback hides lint violations and cold full-repo lint OOMs[/+] on Aug 5, 2026
  8. melvin-bot commented on Aug 5, 2026

    @melvin-bot

    @mountiny @VickyStash

    The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.50-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

    If no regressions arise, payment will be issued on 2026-08-12. 🎊

    The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]. If no checklist is needed for this issue, you can click: [no checklist button]

  9. mountiny commented on Aug 6, 2026

    @mountiny
    Contributor

    No C+ review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions