fix: Warm up the whole DynamoDB connection pool by CyberRik · Pull Request #6985 · feast-dev/feast

CyberRik · 2026-10-09T07:27:31Z

What this PR does / why we need it:

warmup_connections is documented to establish the connection pool at startup, but initialize() made a single describe_limits() call, so only one connection was opened. Concurrent feature requests after startup still paid the cold-start cost on every other connection.

This issues max_pool_connections calls concurrently with asyncio.gather(..., return_exceptions=True). While one call is in flight its connection is busy, so each call needs its own connection. A sequential loop would reuse one connection, which is why the calls run together. A failed call is logged with the count of failures and does not stop the others, so startup behaves as before when warmup fails.

The warmup now calls describe_endpoints() instead of describe_limits(). AWS documents that DescribeLimits is throttled when called more than once a minute, so a burst of calls (or several pods starting at once) would be throttled. With the default adaptive retry mode, that throttling would also slow down the client used for the first real reads. DescribeEndpoints is not tied to a table and has no such limit. Behind a VPC endpoint it needs dynamodb:DescribeEndpoints; if that is denied, the connection is still opened and a warning is logged.

The config docstring, the DynamoDB reference page and the performance tuning guide now say how many calls are made and which call is used.

As noted on the issue, if warming the full pool by default is too much, I can add a separate count option instead.

Which issue(s) this PR fixes:

Fixes #6964

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

test_dynamodb_online_store_warmup_connections now checks that max_pool_connections calls are made and that all of them are in flight at the same time, and that one failed call is logged while the others still run. On master it fails with assert 1 == 8. All 47 tests in test_dynamodb_online_store.py pass; ruff and mypy are clean on the changed files. I don't have an AWS account to test the burst against real DynamoDB, so a check on a real deployment would be welcome.

warmup_connections made a single describe_limits call, so only one connection of the pool was opened. Issue max_pool_connections calls concurrently, so that each one needs its own connection. A failed call is logged and does not stop the others. Signed-off-by: Ritankar Mondal <[email protected]>

dbbvitor · 2026-10-09T17:45:14Z

Nice work.
Won't this hit the DescribeLimits throttle threshold?

The official docs say:

DescribeLimits should only be called periodically. You can expect throttling errors if you call it more than once in a minute.

I already observed this throttling behaviour when multiple pods scale up or cycle.

DescribeLimits is throttled when called more than once a minute, so warming the pool with max_pool_connections concurrent calls would be throttled, and with adaptive retries it would also slow the client down for the first reads. DescribeEndpoints is not tied to a table and has no such limit. Signed-off-by: Ritankar Mondal <[email protected]>

CyberRik · 2026-10-09T19:53:28Z

Good catch, thanks. @nanohanno pointed out the same thing on #6964. I've switched the warmup to describe_endpoints(), which isn't tied to a table and doesn't have the once-a-minute limit (more details in the issue thread).

I don't have an AWS setup to test a burst of calls against, so if you're able to try it on your pods when they scale up, that would really help.

codecov-commenter · 2026-10-10T03:28:22Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.59%. Comparing base (af20fe4) to head (3d9e34f).
⚠️ Report is 14 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6985      +/-   ##
==========================================
+ Coverage   49.92%   50.59%   +0.67%     
==========================================
  Files         443      451       +8     
  Lines       55511    56447     +936     
  Branches     8096     8228     +132     
==========================================
+ Hits        27714    28562     +848     
- Misses      25874    25923      +49     
- Partials     1923     1962      +39

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.04% <100.00%> (+0.69%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/infra/online_stores/dynamodb.py	`65.34% <100.00%> (ø)`

... and 23 files with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af20fe4...3d9e34f. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…-pool

CyberRik · 2026-11-09T09:09:52Z

I've merged the latest master into this branch - it had drifted a few weeks behind, so it's now up to date and conflict-free.

The CI workflows are sitting in the action_required state, waiting on maintainer approval to run. Could a maintainer approve the run and take a look when you get a chance? The change is self-contained, it warms up the full DynamoDB connection pool with concurrent describe_endpoints calls instead of a single describe_limits, and it's covered by unit tests. Happy to adjust anything if needed. Thanks!

shuchu

lgtm

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

* fix: Warm up the whole DynamoDB connection pool warmup_connections made a single describe_limits call, so only one connection of the pool was opened. Issue max_pool_connections calls concurrently, so that each one needs its own connection. A failed call is logged and does not stop the others. Signed-off-by: Ritankar Mondal <[email protected]> * fix: Use describe_endpoints for DynamoDB connection warmup DescribeLimits is throttled when called more than once a minute, so warming the pool with max_pool_connections concurrent calls would be throttled, and with adaptive retries it would also slow the client down for the first reads. DescribeEndpoints is not tied to a table and has no such limit. Signed-off-by: Ritankar Mondal <[email protected]> --------- Signed-off-by: Ritankar Mondal <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Signed-off-by: Shuchu Han <[email protected]>

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

CyberRik requested a review from a team as a code owner October 9, 2026 07:27

CyberRik added 3 commits October 11, 2026 19:29

Merge branch 'master' into fix/dynamodb-warmup-pool

e462d4d

Merge branch 'master' into fix/dynamodb-warmup-pool

f2af244

Merge remote-tracking branch 'origin/master' into fix/dynamodb-warmup…

a3d57cb

…-pool

Merge branch 'master' into fix/dynamodb-warmup-pool

3dd7748

shuchu approved these changes Nov 10, 2026

View reviewed changes

shuchu merged commit 8ce548c into feast-dev:master Nov 10, 2026
2 checks passed

patelchaitany mentioned this pull request Nov 10, 2026

chore: Sync upstream feast-dev/feast master opendatahub-io/feast#288

Merged

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Navigation Menu

Repository navigation

fix: Warm up the whole DynamoDB connection pool - #6985

fix: Warm up the whole DynamoDB connection pool#6985
shuchu merged 6 commits into
feast-dev:masterfrom
CyberRik:fix/dynamodb-warmup-pool

CyberRik commented Oct 9, 2026 •

edited

Loading

Uh oh!

dbbvitor commented Oct 9, 2026

Uh oh!

CyberRik commented Oct 9, 2026

Uh oh!

codecov-commenter commented Oct 10, 2026 •

edited

Loading

Uh oh!

CyberRik commented Nov 9, 2026 •

edited

Loading

Uh oh!

shuchu left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CyberRik commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

dbbvitor commented Oct 9, 2026

Uh oh!

CyberRik commented Oct 9, 2026

Uh oh!

codecov-commenter commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CyberRik commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CyberRik commented Oct 9, 2026 •

edited

Loading

codecov-commenter commented Oct 10, 2026 •

edited

Loading

CyberRik commented Nov 9, 2026 •

edited

Loading