Issue Origin
Observed and reproduced in a real environment.
Bug Description
Scheduled refreshes of a watched local directory copy the complete source tree into OpenViking's temporary staged-source storage before applying ignore_dirs, include, or exclude.
This defeats an important purpose of those filters. Excluded content may not be parsed or embedded, but OpenViking still reads it from disk and writes it into viking://temp/... first.
A common deployment has one logical knowledge tree containing Markdown documentation alongside large datasets, build outputs, virtual environments, media, or generated artifacts. The desired resource is the documentation tree, with the large adjacent subtrees excluded. Users should not have to split that logical resource into many individual watches merely to prevent excluded data from being staged.
In a real deployment, a watched tree contained only a relatively small amount of eligible Markdown but approximately 57 GB of excluded binary market data. A scheduled refresh created a roughly 56 GB temporary source copy. The server cgroup peaked around 35 GB of memory, primarily filesystem page cache caused by the copy, and sustained heavy CPU and disk I/O. Parsing filters eventually excluded the binary files, but only after the expensive staging operation.
The current main implementation appears to explain the behavior:
ResourceService calls stage_source(prepared, viking_fs=..., ctx=...) without passing the normalized processor filters.stage_source() calls _copy_local_tree() for a directory._copy_local_tree() walks every non-symlink directory and reads/writes every regular file; it has no ignore_dirs, include, or exclude inputs.
Relevant code:
openviking/service/resource_service.py around the stage_source() callopenviking/resource/staged_source.py, especially stage_source() and _copy_local_tree()
Steps to Reproduce
- Run OpenViking Server and CLI on the same Linux host.
- Create a local source tree containing one eligible document and one large excluded subtree:
/tmp/ov-watch-repro/
├── docs/
│ └── keep.md
└── large-data/
└── ignored.bin
- Make
ignored.bin large enough for staging activity to be visible, then register the directory as a watched resource:
mkdir -p /tmp/ov-watch-repro/docs /tmp/ov-watch-repro/large-data
printf '# Keep me\n' > /tmp/ov-watch-repro/docs/keep.md
truncate -s 1G /tmp/ov-watch-repro/large-data/ignored.bin
ov add-resource /tmp/ov-watch-repro \
--to viking://resources/ov-watch-repro \
--watch-interval 1 \
--include '*.md' \
--ignore-dirs 'large-data' \
--wait
- Keep the local source path available to the server and wait for the scheduled watch refresh (or restart the server after the watch is due).
- Observe the temporary source tree and server disk/I/O/cache usage during refresh.
The initial CLI upload may be correctly filtered because #2070 added filtering during client-side ZIP creation. The problem described here occurs during the subsequent server-side watch refresh.
Expected Behavior
Filtering should happen at the ingestion boundary, before files are read or copied into staged-source storage:
ignore_dirs should prune matching directories before os.walk descends into them.include should prevent non-matching files from being copied.exclude should prevent matching files from being copied.- Excluded content should never appear below
viking://temp/..., contribute to staging size, or generate source-read/page-cache pressure.
- Initial imports and scheduled refreshes should have consistent filtering semantics.
- The immutable staged snapshot behavior should remain; the snapshot should simply contain only the selected source set.
Ideally the staging implementation should reuse the same normalized path-matching semantics as directory scanning/upload, rather than introducing a separate filter language.
Suggested regression coverage:
- A watched directory with
ignore_dirs="large-data" never copies a file inside that subtree into staged storage.
include="*.md" and exclude="private*.md" are honored before read_bytes() / write_file_bytes().- Scheduled refresh produces the same selected source set as the initial import.
- Existing symlink and path-safety behavior remains unchanged.
Actual Behavior
The full local directory is copied into task-owned temporary VikingFS storage. Filters are applied later during parsing, after the excluded files have already incurred disk reads, temporary writes, staging space, filesystem-cache growth, and cleanup work.
Minimal Reproducible Example
See the shell reproduction above. The essential condition is a local-directory watch where processor_kwargs contains ignore_dirs/include/exclude and the excluded subtree is large enough to observe during the scheduled server-side refresh.
Error Logs
There may be no application error. The refresh can appear successful while temporarily consuming substantial disk, CPU, and page cache. This makes the failure mode especially difficult to diagnose.
OpenViking Version
0.4.17.1; behavior also appears present on current main at the time of filing.
Python Version
3.12.3
Operating System
Linux
Model Backend
Other (the behavior occurs before model processing and is backend-independent).
Additional Context
Related fixes address adjacent but different paths:
Neither prevents the server-side watched-local-directory path from copying the complete tree in stage_source() before parsing.
Issue Origin
Observed and reproduced in a real environment.
Bug Description
Scheduled refreshes of a watched local directory copy the complete source tree into OpenViking's temporary staged-source storage before applying
ignore_dirs,include, orexclude.This defeats an important purpose of those filters. Excluded content may not be parsed or embedded, but OpenViking still reads it from disk and writes it into
viking://temp/...first.A common deployment has one logical knowledge tree containing Markdown documentation alongside large datasets, build outputs, virtual environments, media, or generated artifacts. The desired resource is the documentation tree, with the large adjacent subtrees excluded. Users should not have to split that logical resource into many individual watches merely to prevent excluded data from being staged.
In a real deployment, a watched tree contained only a relatively small amount of eligible Markdown but approximately 57 GB of excluded binary market data. A scheduled refresh created a roughly 56 GB temporary source copy. The server cgroup peaked around 35 GB of memory, primarily filesystem page cache caused by the copy, and sustained heavy CPU and disk I/O. Parsing filters eventually excluded the binary files, but only after the expensive staging operation.
The current
mainimplementation appears to explain the behavior:ResourceServicecallsstage_source(prepared, viking_fs=..., ctx=...)without passing the normalized processor filters.stage_source()calls_copy_local_tree()for a directory._copy_local_tree()walks every non-symlink directory and reads/writes every regular file; it has noignore_dirs,include, orexcludeinputs.Relevant code:
openviking/service/resource_service.pyaround thestage_source()callopenviking/resource/staged_source.py, especiallystage_source()and_copy_local_tree()Steps to Reproduce
ignored.binlarge enough for staging activity to be visible, then register the directory as a watched resource:The initial CLI upload may be correctly filtered because #2070 added filtering during client-side ZIP creation. The problem described here occurs during the subsequent server-side watch refresh.
Expected Behavior
Filtering should happen at the ingestion boundary, before files are read or copied into staged-source storage:
ignore_dirsshould prune matching directories beforeos.walkdescends into them.includeshould prevent non-matching files from being copied.excludeshould prevent matching files from being copied.viking://temp/..., contribute to staging size, or generate source-read/page-cache pressure.Ideally the staging implementation should reuse the same normalized path-matching semantics as directory scanning/upload, rather than introducing a separate filter language.
Suggested regression coverage:
ignore_dirs="large-data"never copies a file inside that subtree into staged storage.include="*.md"andexclude="private*.md"are honored beforeread_bytes()/write_file_bytes().Actual Behavior
The full local directory is copied into task-owned temporary VikingFS storage. Filters are applied later during parsing, after the excluded files have already incurred disk reads, temporary writes, staging space, filesystem-cache growth, and cleanup work.
Minimal Reproducible Example
See the shell reproduction above. The essential condition is a local-directory watch where
processor_kwargscontainsignore_dirs/include/excludeand the excluded subtree is large enough to observe during the scheduled server-side refresh.Error Logs
There may be no application error. The refresh can appear successful while temporarily consuming substantial disk, CPU, and page cache. This makes the failure mode especially difficult to diagnose.
OpenViking Version
0.4.17.1; behavior also appears present on current
mainat the time of filing.Python Version
3.12.3
Operating System
Linux
Model Backend
Other (the behavior occurs before model processing and is backend-independent).
Additional Context
Related fixes address adjacent but different paths:
--ignore-dirswhile the CLI creates the initial upload ZIP.ignore_dirs,include, andexcludethrough the Git-repository parser/uploader.Neither prevents the server-side watched-local-directory path from copying the complete tree in
stage_source()before parsing.