Skip to content

fix(parse): preserve filters for local git repositories - #3190

Merged
qin-ctx merged 1 commit into
volcengine:mainfrom
huangruiteng:codex/fix-3187-code-repo-filters
Jul 13, 2026
Merged

qin-ctx merged 1 commit into
volcengine:mainfrom
huangruiteng:codex/fix-3187-code-repo-filters

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

动机

修复 #3187

DirectoryParser 识别到本地 Git 仓库后,会把 ignore_dirsincludeexclude 传给 CodeRepositoryParser,但后者在调用共享目录上传逻辑前丢失了这三个参数。因此,同一个本地目录仅因存在 .git/,就会生成不同的资源树;watch 刷新也会重复这一不一致行为。

改动思路

复用普通目录扫描器已有的过滤语义,而不是为 Git 仓库定义第二套过滤语言:让 CodeRepositoryParser 将三个过滤参数完整传到共享 uploader,再由 uploader 统一组合内置忽略目录、.gitignore、include/exclude glob、扩展名和文件大小限制。

关键调用链 / 伪代码

DirectoryParser(local_git_repo, filters)
  -> CodeRepositoryParser.parse(filters)
  -> upload_directory(filters)
  -> built_in_ignores + .gitignore
  -> include(path/name) and exclude(path/name)
  -> upload only matching files

watch task 已经把这些参数保存在 processor_kwargs,因此后续刷新会自动走修复后的同一路径。

具体改动

  • 经由 CodeRepositoryParser 继续传递 ignore_dirsincludeexclude
  • 在共享 repository uploader 中应用现有 DirectoryParser 的 path/glob 过滤语义;
  • 保持内置忽略目录、.gitignore、忽略扩展名、文件大小限制和并发上传行为不变。

修复后复现

可以直接对任意本地 Git checkout 调用公开 parser API:

result = await DirectoryParser().parse(
    "/path/to/local/repo",
    ignore_dirs="08_Attachments",
    include="*.md",
    exclude="*.excalidraw.md",
)
assert result.parser_name == "CodeRepositoryParser"

在回归 fixture 中,修复后上传结果应只包含 notes/article.mdmain.py08_Attachments/diagram.mdnotes/private.excalidraw.md 均应被排除。

验证

origin/main revision 5bfa9b617ecff478f825ca435a35bc4222b30582 上,新回归测试先失败:即使设置了

ignore_dirs=08_Attachments
include=*.md
exclude=*.excalidraw.md

fixture 仍上传 4 个文件。应用修复后只保留 notes/article.md

  • tests/test_upload_utils.pytests/parse/test_directory_scan.py 及 focused Git repository regression:82 passed;
  • 所有改动 Python 文件 Ruff passed;
  • py_compile passed;
  • git diff --check passed。

对主干的风险与未覆盖

风险较低且仅限 repository upload filtering。未显式传入过滤参数的调用保持原行为;自定义 ignore_dirs 继续与内置忽略项叠加,与普通目录 parser 的契约一致。

@huangruiteng

Copy link
Copy Markdown
Collaborator Author

@qin-ctx @MaojiaSheng could you please review this focused fix when convenient?

It fixes #3187: local Git repositories silently dropped ignore_dirs, include, and exclude, so watched imports indexed files the user explicitly filtered out. The patch forwards the existing directory-filter contract into repository uploads; the regression uploaded 4 files on current main and only the expected 1 file after the fix. Related parser/upload coverage is 82 tests passing.

@qin-ctx
qin-ctx merged commit cbcec52 into volcengine:main Jul 13, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants