Repository navigation
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a bulk-import workflow to the Moss CLI that supports CSV/JSON/JSONL inputs where the source data doesn’t use the standard id / text field names, addressing the usability request in #379 and making non-code document loading more discoverable.
Changes:
- Added
moss documents import(andmoss doc import) to bulk-import documents with--id-column,--text-column, and metadata-column mapping. - Enhanced the document loader to support column mapping, better CSV handling (including multiline quoted fields), and more actionable validation errors.
- Added focused tests plus documentation and changelog entries describing the new workflow.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/moss-cli/tests/test_documents_import.py | Adds comprehensive parser + CLI coverage for mapped imports across CSV/JSON/JSONL and stdin. |
| packages/moss-cli/src/moss_cli/main.py | Registers documents as an alias command group for doc to make the feature discoverable. |
| packages/moss-cli/src/moss_cli/documents.py | Implements mapping-aware parsing/validation for CSV/JSON/JSONL while preserving legacy document formats. |
| packages/moss-cli/src/moss_cli/commands/doc.py | Adds the import command and shared submission logic; parses mapping flags and validates input before client creation. |
| packages/moss-cli/README.md | Documents the new import workflow and examples for mapped column usage. |
| packages/moss-cli/CHANGELOG.md | Notes the new bulk import feature in Unreleased. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| columns: List[str] = [] | ||
| for value in values: | ||
| columns.extend(name.strip() for name in value.split(",")) | ||
| return columns or None |
| if not isinstance(value, dict): | ||
| raise typer.BadParameter(f"{label}: 'metadata' must be a JSON object") | ||
| metadata = { | ||
| str(key): _metadata_value(item) |
There was a problem hiding this comment.
BLOCKING ```python
metadata = {
str(key): _metadata_value(item)
This path also handles the existing `metadata` field when users run `moss doc add` or import without `--metadata-column`. Previously that object was passed through after JSON parsing, but this now coerces values like `1`, `true`, and nested objects into strings and drops `null`, silently changing stored metadata. Keep `_metadata_value()` only for `_mapped_metadata()`, and have `_parse_metadata()` return the parsed metadata object unchanged after validating it is a dict.
Codex reviewThe import command is mostly scoped cleanly, but the shared document loader now mutates legacy metadata for existing |
|
@Mahin2076 resolve the comments |
Pull Request Checklist
Description
Adds a discoverable bulk-import workflow for CSV, JSON, and JSONL files whose source columns do not already use Moss's
idandtextnames.moss documents importcommand and the shortermoss doc importform.--id-column,--text-column, repeatable--metadata-column, and comma-separated--metadata-columnsmappings.Fixes #379
Testing
pytest packages/moss-cli/tests/test_documents_import.py -q— 29 passed on Python 3.13 and 3.14ruff check . --extend-exclude '**/*.ipynb'mypy packages/moss-cli/src/moss_cli/documents.py packages/moss-cli/src/moss_cli/commands/doc.py --ignore-missing-importsblack --checkandisort --check-onlyon changed Python filestwine checkpassed for bothmoss documents import --helppassedThe full Moss CLI suite reports 57 passed and 2 failed under the latest Typer. The same two untouched shell-completion tests (
test_completions_bash_outputs_scriptandtest_completions_zsh_outputs_script) also fail onorigin/mainin the same environment because Typer's completion API changed; this PR does not alter that code.Type of Change