Skip to content

Add financial_llm_faithfulness metric for regulated financial AI evaluation - #783

Open
priyanka25aug wants to merge 1 commit into
huggingface:mainfrom
priyanka25aug:feat/financial-llm-faithfulness-metric
Open

Add financial_llm_faithfulness metric for regulated financial AI evaluation#783
priyanka25aug wants to merge 1 commit into
huggingface:mainfrom
priyanka25aug:feat/financial-llm-faithfulness-metric

Conversation

@priyanka25aug

Copy link
Copy Markdown

…uation## What this adds

A new metric financial_llm_faithfulness for evaluating LLM outputs in regulated financial environments (FCA, MiFID II, Basel III, SR 11-7).

Why

Financial AI teams need deterministic, offline evaluation that doesn't rely on external models or APIs — compliance-restricted environments often prohibit external calls. Existing metrics (BERTScore, ROUGE) measure linguistic similarity but don't capture the two failure modes that matter most in regulated finance:

  1. Numerical hallucination — the model invents rates, prices, or returns not grounded in the source
    1. Missing disclaimers — the output lacks required regulatory disclosures (FCA COBS 4 / MiFID II Art. 24)

What it measures

Property Description
numerical_faithfulness Fraction of numerical claims (%, bps, £/$, monetary values) grounded in the reference
disclaimer_present Whether a required risk disclaimer is detected
compliance_risk_score Composite risk score (0–1, lower is safer) for use as a CI/CD gate
hallucinated_values Specific values in the prediction not supported by the reference

Design decisions

  • No external model required — all checks are regex-based and deterministic; suitable for air-gapped compliance environments
    • Configurable disclaimer patterns — ships with 11 FCA/MiFID II defaults; firms can supply their own patterns
    • Tolerance parameter — allows ±1% rounding difference by default when comparing numerical values
    • CI/CD readymean_compliance_risk and overall_faithfulness are designed to be used as regression gate thresholds

Usage

import evaluate
metric = evaluate.load("financial_llm_faithfulness")
results = metric.compute(predictions=["The fund returned 8.5%..."], references=["The fund returned 8.5%..."])

Checklist

  • Metric follows the existing evaluate.Metric pattern
  • - [x] _info(), _compute() implemented
  • - [x] README metric card included with usage, outputs, and regulatory references
  • - [x] All tests pass (regex extraction, grounding check, disclaimer detection)
  • - [x] No external model or API dependencies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant