You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4342d12
Browse filesBrowse the repository at this point in the historyBrowse files
feat: Batch DynamoDB online reads across feature views
The generic OnlineStore._read_features_per_fv loop reads one feature view after
another, so a sync request spanning N feature views costs at least N sequential
BatchGetItem round trips, and an async one at least N concurrent requests. Redis
and Postgres already override that seam to batch; DynamoDB now does too.
Each feature view lives in its own DynamoDB table, and a single BatchGetItem
request can read from several tables at once. The override resolves every view's
entity ids up front, packs all of the keys into as few requests as possible
(at most batch_size keys each, across all of their tables), sends those requests
in parallel (a thread pool capped at max_read_workers on the sync path,
asyncio.gather on the async path) and then splits the returned items back apart
by table and entity id. Each table keeps its own ProjectionExpression and
ConsistentRead setting.
BatchGetItem rejects a request that repeats a key within a table, so views that
map to the same table, such as one feature view requested under two aliases, are
read once with the union of their requested features.
Following the Postgres implementation, a single-view request keeps the generic
path, and requests larger than the new max_batched_result_rows setting (entity
rows x feature views, default 2048, 0 disables batching) also fall back to it.
Past that size each view already fills requests of its own, so packing saves at
most one partially filled request per view while holding every view's items at
once.
The item parsing in _process_batch_get_response moves into a shared _to_read_row
helper so the batched and per-view paths decode items identically.
Refs feast-dev#3259
Signed-off-by: NopeCharon <[email protected]>
Copy file name to clipboardExpand all lines: docs/how-to-guides/online-server-performance-tuning.md
+6-3Lines changed: 6 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,8 @@ When the server processes a `get_online_features()` call, it groups the requeste
35
35
**Redis exception:** The Redis online store overrides `_read_features_per_fv()` to batch all `HMGET` commands across every feature view into a **single pipeline execution**. Because all feature views for the same entity share one Redis hash key, the number of Redis round trips is always **1**, regardless of how many feature views the request touches. This means the "fewer feature views" guideline is less critical for Redis than for other stores — but consolidating feature views still reduces serialization and protobuf overhead at the application layer.
36
36
37
37
**PostgreSQL exception:** PostgreSQL stores each feature view in its own table and overrides `_read_features_per_fv()` to combine the per-view reads into a **single `UNION ALL` statement**, so a request touching any number of feature views costs **1** query on both the sync and async paths. This helps most where round trips dominate — a handful of entities against a remote database. Requests larger than `max_batched_result_rows` (entity rows × feature views, default 2048) fall back to one query per view, because at that size the saved round trips no longer offset holding every view's rows at once. A single-view request is never batched, since there is nothing to combine. Set `max_batched_result_rows: 0` in the online store config to turn batching off. As with Redis, consolidating feature views still reduces application-layer overhead.
38
+
39
+
**DynamoDB exception:** DynamoDB stores each feature view in its own table and overrides `_read_features_per_fv()` to pack the keys of every feature view into **shared `BatchGetItem` requests**, since one request can read from several tables. A request touching any number of feature views costs as few as **1** request (up to `batch_size` keys per request, sent in parallel) on both the sync and async paths. As with PostgreSQL, requests larger than `max_batched_result_rows` (entity rows × feature views, default 2048) read one view at a time, single-view requests are never batched, and `max_batched_result_rows: 0` turns batching off.
38
40
{% endhint %}
39
41
40
42
### Feature services are free (and can be faster)
@@ -89,7 +91,7 @@ Requesting just `combined_score` triggers reads from **both** `driver_stats_fv`
89
91
90
92
## Pre-computed feature vectors
91
93
92
-
When a `get_online_features()` request touches multiple feature views, the server issues a separate store read per feature view. For services spanning 5–15+ feature views, this fan-out dominates latency — even where the store batches its reads (Redis, PostgreSQL), the protobuf deserialization and response-building overhead grows linearly with the number of views.
94
+
When a `get_online_features()` request touches multiple feature views, the server issues a separate store read per feature view. For services spanning 5–15+ feature views, this fan-out dominates latency — even where the store batches its reads (Redis, PostgreSQL, DynamoDB), the protobuf deserialization and response-building overhead grows linearly with the number of views.
93
95
94
96
**Pre-computed feature vectors** solve this by storing all of a feature service's features for each entity as a single serialized blob. At read time, the server fetches one blob per entity instead of N reads per feature view, reducing the operation to O(1).
95
97
@@ -281,7 +283,7 @@ The online store is the single largest factor in `get_online_features()` latency
281
283
| Store | Typical p50 latency | Async read | Best for | Key trade-off |
| **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput; all FV reads batched into 1 pipeline | Requires in-memory capacity for your dataset |
284
-
| **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS | Pay-per-request cost; batch API limits (100 items) |
286
+
| **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS; FV reads packed into shared batch requests | Pay-per-request cost; batch API limits (100 items) |
285
287
| **PostgreSQL** | 3–10 ms | No (threadpool) | Teams with existing Postgres infra; all FV reads batched into 1 query | Connection pooling needed at scale |
286
288
| **MongoDB** | 2–5 ms | Yes | Flexible schema, async-native | Requires index tuning for large datasets |
287
289
| **Aerospike** | < 1 ms | No (threadpool) | Ultra-low latency, hybrid memory (RAM + SSD), large datasets | Namespace must be pre-configured on the cluster |
@@ -308,7 +310,7 @@ The feature server can read from the online store using either an **async** or *
| **PostgreSQL** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path. `_read_features_per_fv` is overridden to batch all feature view reads into a single `UNION ALL` query |
314
316
| **Redis** | Implemented | **Yes** | `online_read_async` and `online_write_batch_async` both implemented; uses sync/threadpool path for `get_online_features` (`_read_features_per_fv` overridden with batched single pipeline) |
@@ -343,6 +345,7 @@ Key knobs:
343
345
344
346
- **`batch_size`**: DynamoDB's `BatchGetItem` accepts up to 100 items per request. For 500 entities, this means 5 batches. Keep at 100 unless hitting the 16 MB response limit.
345
347
- **`max_read_workers`**: Controls parallelism for batch reads. With 10 workers, those 5 batches run concurrently (~10 ms) instead of sequentially (~50 ms).
348
+
- **`max_batched_result_rows`**: Requests spanning several feature views pack every view's keys into shared `BatchGetItem` requests, up to this many entity rows × feature views (default 2048). Set to `0` to read one feature view at a time.
346
349
- **`consistent_reads: false`**: Eventually consistent reads are faster and cheaper. Use `true` only if you need read-after-write consistency.
347
350
- **`warmup_connections: true`**: Pre-warms the DynamoDB connection pool on server startup by making `max_pool_connections` concurrent lightweight calls (`describe_endpoints`). This avoids a cold-start latency penalty (~20ms) on the first feature requests, including concurrent ones.
348
351
- **`max_pool_connections`**: Increase for high-throughput deployments to improve HTTP connection reuse to the DynamoDB endpoint.
| `batch_size` | int | `100` | Number of items per BatchGetItem/BatchWriteItem request (max 100) |
51
51
| `max_read_workers` | int | `10` | Maximum parallel threads for batch read operations. Higher values improve throughput for large batch reads but increase resource usage |
52
+
| `max_batched_result_rows` | int | `2048` | Largest request, in entity rows × feature views, whose feature views are read through shared BatchGetItem requests. Set to `0` to always read one feature view at a time |
52
53
| `consistent_reads` | bool | `false` | Whether to use strongly consistent reads (higher latency, guaranteed latest data) |
53
54
| `warmup_connections` | bool | `false` | Whether to pre-warm the async connection pool on startup with `max_pool_connections` concurrent lightweight calls (`describe_endpoints`) |
54
55
| `tags` | dict | `null` | AWS resource tags added to each table |
@@ -63,6 +64,8 @@ online_store:
63
64
64
65
For high-throughput workloads with large entity counts, increase `max_read_workers` (up to 20-30) based on your DynamoDB capacity and network conditions.
65
66
67
+
**Batched Multi-Feature-View Reads**: Each feature view is stored in its own table, and a single BatchGetItem request can read from several tables. When a request spans several feature views, the keys of every view are packed into the same BatchGetItem requests (up to `batch_size` keys each), which are sent in parallel. A request for 10 feature views and 5 entities therefore costs **1** BatchGetItem call instead of 10, on both the sync and async read paths. Requests larger than `max_batched_result_rows` (entity rows × feature views) read one feature view at a time instead, because at that size each view already fills requests of its own and holding every view's items at once only raises peak memory. A single-view request is never batched. Set `max_batched_result_rows: 0` to turn batching off.
68
+
66
69
**Batch Size**: Increase `batch_size` up to 100 to reduce the number of API calls. However, larger batches may hit DynamoDB's 16MB response limit for tables with large feature values.
67
70
68
71
**Connection Warmup**: The DynamoDB async client does not establish actual TCP/TLS connections to the AWS endpoint on initialization. The very first feature retrieval request is penalized with a cold-start overhead (~20ms). Setting `warmup_connections: true` opens up to `max_pool_connections` connections during server startup, one per concurrent `describe_endpoints` call. When DynamoDB is reached through a VPC endpoint, these calls need the `dynamodb:DescribeEndpoints` permission; without it the connections are still opened, but a warning is logged.
0 commit comments