Support the search and vector APIs (alpha) from Dapr 1.19 #1879

Summary

Dapr 1.19 adds the search and vector building blocks as alpha APIs. Search stores, reads and queries documents in a full-text search engine. Vector stores, reads and runs similarity queries over dense vectors. The Java SDK has no client API for either yet. The Go SDK implementation (dapr/go-sdk#838) is the reference for this work.

Runtime API

gRPC, in the Dapr service (search.proto, vector.proto). All calls are unary.

Search:

• CreateIndexAlpha1, GetIndexAlpha1, ListIndexesAlpha1, DeleteIndexAlpha1
• IndexDocumentsAlpha1 upserts documents by ID
• GetDocumentsAlpha1 returns documents by ID, in request order
• DeleteDocumentsAlpha1
• SearchAlpha1

Vector:

• CreateCollectionAlpha1 (with dimensions and distance metric), GetCollectionAlpha1, ListCollectionsAlpha1, DeleteCollectionAlpha1
• UpsertVectorsAlpha1 upserts records by ID
• GetVectorsAlpha1 returns records by ID, in request order, and omits missing IDs
• DeleteVectorsAlpha1
• QueryVectorsAlpha1
• BatchQueryVectorsAlpha1 runs several queries against one collection. Each query succeeds or fails on its own.

Both blocks share IndexingOptionsAlpha1 on writes and FailedItem in write responses. A write can return when the provider accepts it, or wait up to a timeout for indexing to finish.

HTTP: see the Search API and Vector API references.

Components in 1.19: search.meilisearch, search.in-memory, vector.meilisearch and vector.in-memory.

Reference: the Go SDK

dapr/go-sdk#838 (draft) adds client/search.go and client/vector.go:

• Search methods: CreateIndexAlpha1, GetIndexAlpha1, ListIndexesAlpha1, DeleteIndexAlpha1, IndexDocumentsAlpha1, GetDocumentsAlpha1, DeleteDocumentsAlpha1 and SearchAlpha1.
• Vector methods: CreateCollectionAlpha1, GetCollectionAlpha1, ListCollectionsAlpha1, DeleteCollectionAlpha1, UpsertVectorsAlpha1, DeleteVectorsAlpha1, GetVectorsAlpha1, QueryVectorsAlpha1 and BatchQueryVectorsAlpha1.
• Shared types: IndexingOption (WithReturnOnIndexAcceptance, WithWaitForIndexCompletion(timeout, onTimeout)), IndexAck, IndexingResponse and FailedItem.
• Search types: SearchDocument, SearchIndex, SearchQuery, SortClause, SearchHit, SearchResponse and TotalHitsRelation.
• Vector types: VectorCollection, VectorRecord, DistanceMetric, VectorQuery, VectorMatch, VectorQueryResponse and BatchQueryResult.
• Each method checks that the store name and the index or collection name are not empty before it calls the sidecar.

The PR is still a draft, so its API can change before it merges.

Other SDKs: the .NET SDK has Dapr.Search (dapr/dotnet-sdk#1944) and Dapr.Vector (dapr/dotnet-sdk#1945). The JavaScript SDK added vector in dapr/js-sdk#928.

Related links: the runtime implementation (dapr/dapr#9974) and the components (dapr/components-contrib#4613).

Scope

• Generated types: search.proto and vector.proto are already in the proto downloads in sdk-autogen/pom.xml.
• Add the search methods to DaprPreviewClient, where the SDK keeps alpha APIs, and implement them in DaprClientImpl.
• Add the vector methods to DaprPreviewClient and implement them in DaprClientImpl.
• Add domain types for documents, queries, records and results, so the public API does not expose the generated proto types.
• Add tests, and an example for each block that uses the search.in-memory and vector.in-memory components.
• Add docs to the Java SDK section of dapr/docs.

The two blocks can ship in separate PRs. The APIs are alpha, so their names and behavior can change in a later release.