[PROPOSAL] Rule pattern engine — Semgrep-compatible YAML on tree-sitter AST (Layer 2 of #43 hybrid)

[Wolfvin]

Summary

Adopt a Semgrep-compatible YAML pattern language as CodeLens's Layer 2 rule engine (pattern matching + autofix + taint rules). Built on tree-sitter AST. This is the complementary Layer 2 to the .scm proposal in #43 (which is Layer 1: node discovery only).

Worker consensus (4 reports)

Worker	Source	Contribution
Opengrep	update!/CodeLens_Opengrep_Upgrade_Analysis.md #1	pattern / patterns / pattern-either / pattern-not / pattern-regex combinators + metavariables ($X, $...ARGS). Target <2x slowdown vs search regex.
Opengrep	same file #2	Metavariable capture + substitution ($X captures AST node, rest metavar captures zero-or-more args). Blocker for autofix from rule.
Opengrep	same file #3	Autofix from rule YAML (fix: field with metavar substitution, fix-regex for regex-based substitution).
Semgrep	update!/CodeLens_Upgrade_Issues_from_Semgrep.md CL-001	Phase 1: pattern / patterns / pattern-either / pattern-inside / pattern-not / pattern-regex + metavariables. Backward-compatible via auto-translation.
Semgrep	same file CL-002	Metavariable constraints: metavariable-regex, metavariable-pattern, metavariable-comparison, focus-metavariable.
OpenTaint	update!/CodeLens_vs_OpenTaint_Upgrade_Analysis.md B2	Optional mode: taint / mode: join for backward-compat. Semgrep rule converter (best-effort).

Proposed scope (P0, 3-5 weeks)

Phase 1 — Core pattern language:

• pattern, patterns, pattern-either, pattern-not, pattern-regex
• Metavariables: $X (single node), $...ARGS (rest)
• Languages: Python, JS, TS, Rust (initial 4)
• New files: scripts/rule_pattern_parser.py, scripts/rule_matcher.py
• Backward compat: auto-translate existing sources/sinks/sanitizers YAML to new format

Phase 2 — Metavariable constraints (P1, 1-2 weeks):

• metavariable-regex, metavariable-pattern, metavariable-comparison, focus-metavariable

Phase 3 — Autofix from rule (P1, 1 week):

• fix: field with metavar substitution
• Reuse existing autofix_engine.py confidence framework (rule-driven defaults to confidence: 0.85, risk: safe)

Phase 4 — Taint mode + join rules (P2, 2-3 weeks):

• mode: taint (pattern-sources, pattern-sinks, pattern-sanitizers)
• mode: join (refs: + on: for cross-file)

Acceptance criteria

• Pattern matcher passes 80% of Semgrep community rule corpus (best-effort)
• Backward compat: existing python_security.yaml and javascript_security.yaml rules still fire
• <2x slowdown vs search regex on equivalent patterns
• Snapshot tests for 20+ patterns per supported language
• Documentation at references/rule-syntax.md

Relationship to other issues

• [PROPOSAL] Adopt tree-sitter query files (.scm) for node discovery — phased rollout, defer full migration until triggers met #43 (.scm proposal): Layer 1 (discovery). This issue is Layer 2 (rule matching). Both ship independently.
• [BUG-02] --deep flag runs HybridEngine twice — LSP findings double-counted #32 (--deep duplicate): unrelated, but taint mode (Phase 4) depends on a clean engine consolidation.