🔒 feat: add LLM application security review rules - #1453
cyberspace-cs wants to merge 1 commit into
Conversation
Add llm_security.md rule set for AI/LLM application code: - Prompt injection risks (user input, RAG context, tool outputs) - Unsafe LLM output handling (eval/exec, SQL injection, path traversal) - Agent tool safety (arbitrary command execution, sensitive ops, unbounded loops) - Sensitive data leakage (hardcoded keys, PII in prompts, plaintext logs) Registered for files matching *llm*, *agent*, or *prompt* patterns.
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s). |
Qiyuanqiii
left a comment
There was a problem hiding this comment.
Thanks for putting this together. Dedicated checks for LLM applications would be useful.
The main thing I'd like to see changed is the rule selection. The new patterns replace the existing language-specific rules for matching files, so we lose checks we already have. I've left an inline comment with examples from the PR build.
A couple of wording suggestions as well:
- For prompt injection, could we focus on what untrusted input can actually cause? Missing delimiters alone doesn't establish a vulnerability, and adding them doesn't make the input safe.
- For tool access, it would help to account for existing user authorization and enforced permission scopes. A missing approval prompt doesn't necessarily mean an action is unauthorized.
- For sensitive data, could we distinguish unauthorized disclosure from an approved use of a provider? Encryption and redaction address different risks.
I'd request changes for the rule-selection issue. The other points are suggestions to make the checks more precise; they don't each need to be treated as separate blockers.
| "**/*llm*.{py,ts,js,go,java}": "llm_security.md", | ||
| "**/*agent*.{py,ts,js,go,java}": "llm_security.md", | ||
| "**/*prompt*.{py,ts,js,go,java}": "llm_security.md", |
There was a problem hiding this comment.
Could we keep the language-specific rules when adding these checks? The resolver stops at the first match, so these entries replace the rules below rather than supplementing them. I checked the PR build: agent.go gets only llm_security.md, and user_agent.go matches too.
An opt-in rule with merge_system_rule: true might be one way to preserve both, without these built-in overrides. It would also be helpful to add a few tests for the intended matches.
|
I don't think this is a good idea. You should probably update the every rule doc to contain these content, avoid putting them together int a shared doc. |
背景
随着 LLM 应用的爆发式增长,AI 应用特有的安全风险(prompt injection、不安全的 LLM 输出处理、agent 工具滥用等)越来越突出。目前的安全规则主要覆盖传统 Web 安全(XSS、SQL 注入等),缺少针对 LLM/Agent 应用的专项检查。
新增规则集
新增
llm_security.md规则文件,覆盖四大类 LLM 应用安全风险:注册方式
在
system_rules.json的path_rule_map中注册,匹配以下文件模式:**/*llm*.{py,ts,js,go,java}**/*agent*.{py,ts,js,go,java}**/*prompt*.{py,ts,js,go,java}设计考虑
价值
这个规则集填补了 AI 应用安全审查的空白,让 OCR 工具能更好地服务于 AI 开发者。