Repository navigation
feat(rag): add pdf-inspector fallback, optimize batch embeddings, and fix migration - #34
Merged
Merged
Conversation
… fix migration - Add pdf-inspector fallback in ingest router for PDFs with low confidence or failed streams - Bump MAX_FILE_BYTES to 75 MB to support large monograph and book uploads - Optimize local neural embeddings with asyncio.to_thread and batch_size 64 - Fix slowapi rate limiter initialization to honor settings.slowapi_enabled - Fix migration 005_rank_by_attribute.sql to add quality_score column before hybrid_search_v6 - Improve Docker Compose v2 compatibility in validate_rag_local.sh
sanjeevafk
force-pushed
the
feat/pdf-inspector-eval
branch
from
October 5, 2026 19:52
3288876 to
218c688
Compare
sanjeevafk
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses document ingestion resilience, embedding throughput, rate limiting configuration, and database migration correctness:
Parser Resilience &
pdf-inspectorFallback:pdf-inspectoras an automated high-accuracy fallback inapi/routers/ingest.pywhendepth_enginereturns extraction confidence < 0.8 or encounters unparsed PDF text streams.MAX_FILE_BYTESceiling from 50 MB to 75 MB to support large document and book uploads.Async Neural Embeddings Optimization:
SentenceTransformer.encodeinasyncio.to_threadwithbatch_size=64to maximize multi-core CPU throughput and prevent event loop blocking.Rate Limiting & Migration Fixes:
api/main.pyslowapi Limiter initialization to honorsettings.slowapi_enabled.db/migrations/005_rank_by_attribute.sqlto ensurequality_scorecolumn is added toknowledge_chunksprior to creatinghybrid_search_v6.scripts/validate_rag_local.shfor Docker Compose v2 compatibility.Changed Files (5 files)
api/main.pyapi/routers/ingest.pyapi/services/rag/embeddings.pydb/migrations/005_rank_by_attribute.sqlscripts/validate_rag_local.shTest Verification
pytest tests/unit/).