Memory Refactor: Simplify Memory Interfaces, Remove Unnecessary Generics, Move Vector Stores to Connectors - #308

Motivation and Context

Existing memory interfaces are overly complicated, leading to confusion about how to implement a custom Memory Store.

This PR condenses IMemoryStore + IEmbeddingIndex + IDataStore -> IMemoryStore and eliminates the use of generics spread by the Embedding struct. It makes the codebase much easier to read and IMemoryStore easier to implement. As part of the effort, SqliteMemoryStore was properly implemented (it was not previously).

Additionally, this PR moves the MemoryStore 'skills' to the connectors namespace.

Addresses several points called out in #202

Description

• QdrantMemoryStore was moved from Skills to Connectors. Unit tests were added for the memory store object. Created an IQdrantVectorDbClient interface to better enable testing of the memory store through mocks.
• Modified QdrantVectorDbClient calls to default batched calls to reduced the amount of HTTP traffic when possible. GetVectorById -> GetVectorsById, UpsertVector -> UpsertVectors, DeleteVector -> DeleteVectors.
• Generally, for a vector store, the unique id passed by the user to associate with a saved memory will become the indexed ID for the vector store as well. However, Qdrant enforces a user-provided GUID or hex string as the index key. Added a nullable key field to MemoryRecord to support vector storage services that act similarly. Depending on the backing storage, this key is not necessary to pass to subsequent calls. SemanticTextMemory does not utilized this parameter. For now, this is only for direct interaction with the QdrantMemoryStore. Note: It is understood that Qdrant will be enabling service-side GUID generation in the near future, significantly reducing the stress on the client to produce unique values for every data entry.
• Eliminated the dependency on the core kernel in the connectors by replacing Embeddings struct references with IEnumerable<float>.
• SqliteMemoryStore was moved from Skills to Connectors. SqliteMemoryStore now properly implements IMemoryStore and the unit tests were updated to reflect this fact.
• The generic TEmbedding was removed from all Kernel and Connector files. If a memory store expects a non-FP32 vector, it is currently up to the IMemoryStore implementation to perform the necessary conversions.
• DataEntry is now a base class that enforces nullable Key and Timestamp fields. It is not strictly used for anything other than MemoryRecord. It may serve useful for connecting to data stores that are for structures other than embeddings, or vector stores that calculate embeddings server-side.
• Removed existing collection checks for Upsert calls. Depending on the implementation, this has the potential to drastically increase HTTP traffic or local compute resources when making batch calls. The call has been moved to SemanticTextMemory for now to avoid additional breaking changes, but will be addressed in a better fashion in a follow-up PR.
• Added a very simple example to KernelSyntaxExamples on how to create a read-only IMemoryStore that imports vectors from a json string.
• Renamed VectorDbException -> QdrantMemoryException. Ensured that these exceptions can only be thrown by the QdrantMemoryStore and can accept an inner exception as a paramter.

Additional small edits:

• removed unnecessary namespace from KernelSyntaxExamples Example21 and Example22
• Removed unused references in RestApiOperations.cs and JsonPathSkill.cs

---------------- OUT OF SCOPE, COMING SOON ----------------

• The addition of a value string to MemoryRecord. This is very important for metadata flexibility, but it will come in a follow-up PR.
• The ability to optionally return the vector to the user. In many cases, the vector is not important for get/search calls, thus it makes no sense to pass such large amounts of data around the kernel application.
• Disambiguation of SemanticTextMemory interfaces, classes, and constructors.
• Unit tests for QdrantVectorDbClient. Qdrant will be making breaking changes to their API very soon. TBD whether integration tests will be added as well.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
• The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with dotnet format
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄