Skip to content

Qdrant: fix skipping upserts - #1208

Merged
Shawn Callegari (shawncal) merged 1 commit into
microsoft:mainfrom
shawncal:qdrant-upsert-fix
May 30, 2023
Merged

Shawn Callegari (shawncal) merged 1 commit into
microsoft:mainfrom
shawncal:qdrant-upsert-fix

Conversation

@shawncal

Copy link
Copy Markdown
Contributor

Description

In the Qdrant connector, during Upsert(), it checks whether the vector already exists in the db; if so, it calls continue and skips upserting. The expected behavior of Upsert() is Insert+Update, but currently only the insert is working updates are explicitly skipped.

Resolves #1035

Also includes some minor cleanup on the internal request builder classes:

  • DeleteRange
  • UpsertRange
    ...allow you to create the Builder, add all operations, and Build() in a single statement, so you never have to hold an instance of the Builder as a var just to operate a loop.

Contribution Checklist

@shawncal

Copy link
Copy Markdown
Contributor Author

Tara W (@tawalke) This code is UNTESTED (hence the "Draft" status). Can you please take over this PR, test with a qdrant instance?

@github-actions github-actions Bot added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel labels May 25, 2023
@shawncal
Shawn Callegari (shawncal) marked this pull request as ready for review May 30, 2023 18:51
@shawncal
Shawn Callegari (shawncal) merged commit 40de0e5 into microsoft:main May 30, 2023
Shawn Callegari (shawncal) added a commit to shawncal/semantic-kernel that referenced this pull request Jul 6, 2023
### Description
In the Qdrant connector, during Upsert(), it checks whether the vector already exists in the db; if so, it calls `continue` and skips upserting. The expected behavior of Upsert() is Insert+Update, but currently only the insert is working updates are explicitly skipped.

Resolves microsoft#1035

Also includes some minor cleanup on the internal request builder classes:
- DeleteRange
- UpsertRange
...allow you to create the Builder, add all operations, and Build() in a single statement, so you never have to hold an instance of the Builder as a var just to operate a loop.
@shawncal
Shawn Callegari (shawncal) deleted the qdrant-upsert-fix branch July 18, 2023 20:03
Caesar / 谢佳辰 · Jiachen Xie (er-s-an) pushed a commit to er-s-an/semantic-kernel that referenced this pull request Oct 26, 2026
…uth (microsoft#14555)

Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.13.0 to 2.15.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jpadilla/pyjwt/releases">pyjwt's
releases</a>.</em></p>
<blockquote>
<h2>2.15.0</h2>
<p>See the <a
href="https://github.com/jpadilla/pyjwt/blob/2.15.0/CHANGELOG.rst">2.15.0
changelog</a> for complete release details.</p>
<h2>2.14.0</h2>
<p>See the <a
href="https://github.com/jpadilla/pyjwt/blob/2.14.0/CHANGELOG.rst">2.14.0
changelog</a> for the complete release details and related security
advisories.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst">pyjwt's
changelog</a>.</em></p>
<blockquote>
<h2><code>v2.15.0
&lt;https://github.com/jpadilla/pyjwt/compare/2.14.0...2.15.0&gt;</code>__</h2>
<p>Security</p>
<pre><code>
- Wrap recursion errors from deeply nested JWT payloads in
``DecodeError``
  instead of exposing a raw ``RecursionError``.
<p>Added</p>
<pre><code>
- Support Python 3.15 by @kytta in
`[microsoft#1202](jpadilla/pyjwt#1202)
&amp;lt;https://github.com/jpadilla/pyjwt/pull/1202&amp;gt;`__

Changed
</code></pre>
<ul>
<li><code>JWKSetCache</code> now stores the parsed <code>PyJWKSet</code>
rather than the raw JWKS
payload, so a cache hit no longer re-parses every key.
<code>JWKSetCache.put()</code>
accepts either form and raises <code>PyJWKSetError</code> for anything
else. As a
result, <code>PyJWKClient.get_jwk_set()</code> returns the same
<code>PyJWKSet</code> instance
for as long as it stays cached, rather than a freshly built one per call
in
<code>[microsoft#1208](jpadilla/pyjwt#1208)
&amp;lt;https://github.com/jpadilla/pyjwt/pull/1208&amp;gt;</code>__</li>
<li><code>PyJWKClient.fetch_data()</code> now raises
<code>PyJWKClientError(&amp;quot;The JWKS endpoint did not return a JSON
object&amp;quot;)</code> when
the endpoint response is not a JSON object, instead of returning it for
<code>get_jwk_set()</code> to reject. Callers reaching the JWKS through
<code>get_jwk_set()</code> see the same error as before in
<code>[microsoft#1208](jpadilla/pyjwt#1208)
&amp;lt;https://github.com/jpadilla/pyjwt/pull/1208&amp;gt;</code>__</li>
</ul>
<p>Fixed</p>
<pre><code>
- Return cached ``PyJWKSet`` values from ``PyJWKClient.get_jwk_set()``
instead
of raising ``PyJWKClientError(&amp;quot;The JWKS endpoint did not return
a JSON
object&amp;quot;)``. ``JWKSetCache.put()`` documents ``PyJWKSet`` as the
cached value,
so callers pre-populating the cache to avoid a network round-trip could
not
read it back in `[microsoft#914](jpadilla/pyjwt#914)
&amp;lt;https://github.com/jpadilla/pyjwt/issues/914&amp;gt;`__ and
`[microsoft#1208](jpadilla/pyjwt#1208)
&amp;lt;https://github.com/jpadilla/pyjwt/pull/1208&amp;gt;`__
- ``PyJWKClient.get_jwk_set()`` now caches the key set it returns, so a
``fetch_data()`` override that filters or transforms the JWKS is no
longer
  undone by the next cache hit in
`[microsoft#1208](jpadilla/pyjwt#1208)
&amp;lt;https://github.com/jpadilla/pyjwt/pull/1208&amp;gt;`__
- Raise the documented ``PyJWTError`` subclass instead of leaking a
  ``TypeError`` when the ``exp``, ``nbf``, or ``iat`` claim decodes to a
  non-numeric, non-string value such as a list, dict, or ``null``.
- Reject OKP JWK private keys when their public ``x`` component does not
  match the private ``d`` component.
- Treat malformed JWK Set members as unusable keys rather than letting
``AttributeError`` or ``TypeError`` escape ``PyJWKSet``. A member that
is not
&amp;lt;/tr&amp;gt;&amp;lt;/table&amp;gt; 
&lt;/code&gt;&lt;/pre&gt;
&lt;/blockquote&gt;
&lt;p&gt;... (truncated)&lt;/p&gt;
&lt;/details&gt;
&lt;details&gt;
&lt;summary&gt;Commits&lt;/summary&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@1d41a6478e1562e68ff667fcd703356acf085f68&quot;&gt;&lt;code&gt;1d41a64&lt;/code&gt;&lt;/a&gt;
chore: prepare 2.15.0 release&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@9bc06658f875b9b40091539140bbbdc4639161c3&quot;&gt;&lt;code&gt;9bc0665&lt;/code&gt;&lt;/a&gt;
fix: make recursive payload tests deterministic&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@5fde08a6cf906aa7698de2d6391d88b73006b17b&quot;&gt;&lt;code&gt;5fde08a&lt;/code&gt;&lt;/a&gt;
fix: normalize recursive JWT payload errors&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@171062d2d734315272a901100aa4b109f2fc3c19&quot;&gt;&lt;code&gt;171062d&lt;/code&gt;&lt;/a&gt;
utils: mention bytes in force_bytes type error (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1173&quot;&gt;#1173&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@c9d4d5375bf464ef363506fed9eb6e7f33217ab6&quot;&gt;&lt;code&gt;c9d4d53&lt;/code&gt;&lt;/a&gt;
docs/conf: drop duplicate 'and' from read() docstring (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1174&quot;&gt;#1174&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@2763752196113e1473b0ed7905aa6034aedfbe53&quot;&gt;&lt;code&gt;2763752&lt;/code&gt;&lt;/a&gt;
Add support for Python 3.15 (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1202&quot;&gt;#1202&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@4adcd02722f5011c60079d3978dfc167b9a8eaa5&quot;&gt;&lt;code&gt;4adcd02&lt;/code&gt;&lt;/a&gt;
Catch http.client.HTTPException in PyJWKClient.fetch_data (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1201&quot;&gt;#1201&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@9e501d993b4d3e7dba14bdb1722b1d993ac75097&quot;&gt;&lt;code&gt;9e501d9&lt;/code&gt;&lt;/a&gt;
fix: correct docstring typo in _validate_jti (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1179&quot;&gt;#1179&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@4047c44d51950ffda15f40054508d3f17c43b1e2&quot;&gt;&lt;code&gt;4047c44&lt;/code&gt;&lt;/a&gt;
docs: clarify JWK certificate member handling (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1212&quot;&gt;#1212&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;&lt;a
href=&quot;jpadilla/pyjwt@f4e2b59f543cc82d46d9d69922bba59e804216b9&quot;&gt;&lt;code&gt;f4e2b59&lt;/code&gt;&lt;/a&gt;
[pre-commit.ci] pre-commit autoupdate (&lt;a
href=&quot;https://redirect.github.com/jpadilla/pyjwt/issues/1210&quot;&gt;#1210&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Additional commits viewable in &lt;a
href=&quot;jpadilla/pyjwt@2.13.0...2.15.0&quot;&gt;compare
view&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/details&gt;

&lt;br /&gt;</code></pre>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Issues or pull requests impacting the core kernel .NET Issue or Pull requests regarding .NET code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpsertVectorsAsync in the QdrantVectorDbClient class

3 participants