Repository navigation
Concurrent request execution is exponentially slow #297
Description
Describe the bug
Concurrently executing thousands of queries is super slow in dart. For below code it took around ~2s for nodejs version to finish executing and around ~57s for dart version.
To Reproduce
The code should call production (actual firestore) APIs and not a local emulator to see the issue.
Dart (Super Slow):
import 'package:firebase_admin_sdk/firebase_admin_sdk.dart';
import 'package:firebase_functions/firebase_functions.dart';
void main() {
runFunctions((firebase) {
firebase.https.onRequest(name: 'helloWorld', (request) async {
final arr = List.generate(3000, (index) => index);
final firestore = FirebaseApp.instance.firestore();
await Future.wait(
arr.map((index) {
return firestore
.collection('numbers')
.where('number', .equal, index)
.limit(1)
.get();
}),
eagerError: true,
);
return .ok('Yay! Client Exception did not occur.');
});
});
}
Typescript (Fast):
import {initializeApp} from "firebase-admin/app";
import {getFirestore} from "firebase-admin/firestore";
import {onRequest} from "firebase-functions/https";
initializeApp();
export const helloWorld = onRequest(async (_request, response) => {
const arr = Array.from({length: 3000}, (_, index) => index);
const firestore = getFirestore();
await Promise.all(
arr.map((index) => {
return firestore
.collection("numbers")
.where("number", "==", index)
.limit(1)
.get();
}),
);
response.status(200).send("Yay! Client Exception did not occur.");
});Expected behavior
The speed of executing thousands of queries concurrently in dart should be as fast as if not faster than nodejs version.
Activity
brianquinlan commentedon Jul 6, 2026
@lahirumaramba wrote:
I don't know if we just HTTP/2 for messaging. Setting the maximum number of connections is very easy using
maxConnectionsPerHost.Maybe tomorrow I'll try to repro and see if switching to an HTTP/3 implementation improves performance.
mosuem commentedon Jul 6, 2026
See also the AI-generated #298 for the max connections - @demolaf is investigating in more detail though.
brianquinlan commentedon Jul 7, 2026
I wasn't able to reproduce this on my MacBook until I introduced 50ms of network delay.
brianquinlan commentedon Jul 7, 2026
Here are the results that I get on my macOS machine:
maxConnections=5)demolaf commentedon Jul 8, 2026
@brianquinlan he's piping through cloud functions, so maybe a lack of resources on the platform (Cloud Run)? My deployed function at 3000 queries produced the below (by default):
Response - Service unavailable.
This isn't just slow — it crashes outright. Default memory (256 MiB) is exceeded at 2000+ concurrent queries (works fine at 1000, ~2.5s); the container gets OOM-killed mid-request:
{ "textPayload": "The request failed because either the HTTP response was malformed or connection to the instance had an error. Additional troubleshooting documentation can be found at: https://cloud.google.com/run/docs/troubleshooting#malformed-response-or-connection-error\nWhile handling this request, the container instance was found to be using too much memory and was terminated. This is likely to cause a new container instance to be used for the next request to this revision. If you see this message frequently, you may have a memory leak in your code or may need more memory. Consider creating a new revision with more memory.", "insertId": "6a4e144a0007936c1f83dfcd", "httpRequest": { "requestMethod": "GET", "requestUrl": "https://issue297-repro-c37rgip73q-uc.a.run.app/issue297repro?n=3000", "requestSize": "289", "status": 503, "responseSize": "67", "userAgent": "curl/8.7.1", "remoteIp": "102.216.202.19", "serverIp": "34.143.79.2", "latency": "5.792330713s", "protocol": "HTTP/1.1" }, "resource": { "type": "cloud_run_revision", "labels": { "configuration_name": "issue297-repro", "project_id": "dart-firebase-admin", "revision_name": "issue297-repro-00001-tn5", "location": "us-central1", "service_name": "issue297-repro" } }, "timestamp": "2026-07-08T09:11:32.694130Z", "severity": "ERROR", "labels": { "instanceId": "002f8ffd47aa5d4f0104c7c8bd0277e3171d1f5fea6c5cf3a091a960a532b25bf8d5badb470aeb2b7e7c51a663c52fce44c76d591eb4bfadec021c6792235c179cd11f3c37b86461d0a94025b94fff", "goog-managed-by": "firebase-functions" }, "logName": "projects/dart-firebase-admin/logs/run.googleapis.com%2Frequests", "trace": "projects/dart-firebase-admin/traces/ba025a4c1a5b84d061c1ab4aca24bb96", "receiveTimestamp": "2026-07-08T09:11:38.504277509Z", "spanId": "1b4ce013f30d31c7", "traceSampled": true }The follow-up system log names the actual number:
{ "textPayload": "Memory limit of 256 MiB exceeded with 266 MiB used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits", "insertId": "6a4e144a000757e9d7ea0eaf", "resource": { "type": "cloud_run_revision", "labels": { "revision_name": "issue297-repro-00001-tn5", "project_id": "dart-firebase-admin", "configuration_name": "issue297-repro", "service_name": "issue297-repro", "location": "us-central1" } }, "timestamp": "2026-07-08T09:11:38.481257Z", "severity": "ERROR", "labels": { "goog-managed-by": "firebase-functions", "instanceId": "002f8ffd47aa5d4f0104c7c8bd0277e3171d1f5fea6c5cf3a091a960a532b25bf8d5badb470aeb2b7e7c51a663c52fce44c76d591eb4bfadec021c6792235c179cd11f3c37b86461d0a94025b94fff", "run.googleapis.com/base_image_versions": "us-docker.pkg.dev/serverless-runtimes/google-24/runtimes/osonly24:osonly24_20260607__RC00" }, "logName": "projects/dart-firebase-admin/logs/run.googleapis.com%2Fvarlog%2Fsystem", "receiveTimestamp": "2026-07-08T09:11:38.487919294Z", "errorGroups": [ { "id": "CKnV1_TJwMrDQw" } ] }OutdatedGuy commentedon Jul 8, 2026
@demolaf I forgot to mention, but I'm running the cloud function through local firebase emulator, and not through deployed Cloud Run. I haven't tried it with deployed functions.
I'm running only functions through firebase emulator and these functions are calling production APIs for Firestore.
brianquinlan commentedon Jul 8, 2026
@OutdatedGuy what platform are you on?
demolaf commentedon Jul 8, 2026
@brianquinlan maybe the 57s was specific to network/machine conditions? I'm getting roughly the same values.
Using functions emulator only:
OutdatedGuy commentedon Jul 8, 2026
I am on MacBook M1 Pro
brianquinlan commentedon Jul 8, 2026
@OutdatedGuy Was the firestore database that were testing against populated in any way?
Because I was also testing on a Mac and my numbers were very different than yours. Oh, what Dart version are you using?
OutdatedGuy commentedon Jul 8, 2026
Yes, the collection of the production database has 5k+ documents, each document having around 15 fields with string values.
I'm on Dart 3.12.2
brianquinlan commentedon Jul 13, 2026
Hi @OutdatedGuy, can you provide a self-contained example that we can run to reproduce this issue?
OutdatedGuy commentedon Jul 13, 2026
@brianquinlan Here you go, https://github.com/OutdatedGuy/firebase-admin-dart-issues-297
I have included the steps to reproduce, my testing results and logs in the readme
brianquinlan commentedon Jul 16, 2026
@demolaf are you going to take a look at the repo?
demolaf commentedon Jul 17, 2026
Reproduced the same issue from the repro sample provided, see the next comment which tested against an unpublished
google_cloud_firestorepackage.hello-world)helloWorld)Firebase Emulator Logs
<PROJECT_DIR>/dartfunctionsfor Cloud Functions...hello-world.http://127.0.0.1:5001/<PROJECT_ID>/us-central1/hello-world).<PROJECT_DIR>/tsfunctionsfor Cloud Functions...helloWorld.http://127.0.0.1:5001/<PROJECT_ID>/us-central1/helloWorld).Dart (
hello-world) executionsRun 1 — 10:21:23 to 10:21:34 (11347.582792ms)
{ "severity": "ERROR", "message": "ClientException: Connection closed before full header was received, uri=https://firestore.googleapis.com/v1/projects/<PROJECT_ID>/databases/(default)/documents:runQuery" }Stack:
package:http/src/io_client.dart 229:7 IOClient.send→package:googleapis_auth/src/auth_http_utils.dart 43:22 AuthenticatedClient.send→package:google_cloud_rpc/service_client.dart 149:22 ServiceClient._makeStreamingRequestRun 2 — 10:21:49 to 10:21:59 (10503.417042ms)
{ "severity": "ERROR", "message": "ClientException: Connection closed before full header was received, uri=https://firestore.googleapis.com/v1/projects/<PROJECT_ID>/databases/(default)/documents:runQuery" }Stack: same as Run 1.
Run 3 — 10:22:02 to 10:22:13 (11295.328792ms)
{ "severity": "ERROR", "message": "HandshakeException: Connection terminated during handshake" }Node (
helloWorld) executionsRun 1 — 10:22:20 to 10:22:52 (32156.1335ms)
Run 2 — 10:22:58 to 10:23:08 (9896.514167ms)
No warnings or errors.
Run 3 — 10:23:11 to 10:23:21 (10727.0925ms)
No warnings or errors.
demolaf commentedon Jul 17, 2026
@brianquinlan
Working from the local
google_cloud_firestoreusing pubspec overrides with the same repro produces a different result. NoHandshakeExceptioneither.I re-ran @OutdatedGuy's repro (firebase-admin-dart-issues-297) exactly as-is, except I pointed
dartfunctions/pubspec_overrides.yamlat local (againstmain) paths for both packages instead of a git ref onfirebase_admin_sdkalone:With both packages genuinely pointed at the fixed code, I ran the same 1500-concurrent-query repro 6 times against production Firestore:
Zero errors across all 6 — no
ClientException, noHandshakeException. Previously this same repro reliably failed withClientException: Connection closed before full header was receivedaround run 1.Dart is still consistently 2-3x slower than Node (2.2-7s vs 1.1-2.1s), so the underlying performance gap still exists — but the crash is gone once PR #294 is actually in the dependency graph.
OutdatedGuy commentedon Jul 17, 2026
I tried @demolaf's changes, and crashes stopped. Seems like I was supposed to override
google_cloud_firestoreand notfirebase_admin_sdk🤦♂️@demolaf can you check by increasing query count to 3000 from 1500 and then to 5000. Cuz I believe it's exponential rather than linear. I hit Quota limits while testing the 5K once, so can't check that one, but here are my test results for 1500 & 3000 queries after the crash fixed:
I have updated my repo with the crash fixed version and new testing results.
demolaf commentedon Jul 17, 2026
hello-world)helloWorld)hello-world)helloWorld)demolaf commentedon Jul 23, 2026
Working on a solution using
http/2with pooled connections (open new connections after max 100 requests/streams), similar to how node'sClientPoolworks.The current problem with
http/1- opens a connection (TCP/TLS Handshake) on every request, so if there are 3000 it would open 3000 connections all at the same time which can cause out of memory (OOM errors) and slow responses.http/2solves this by allowing multiple requests in a single connection (one TCP/TLS Handshake) and with pooled (open a new connection when an active connection is full at 100 requests/streams) connections allows processing concurrent requests simultaneously.demolaf commentedon Jul 23, 2026
@OutdatedGuy can you try testing from this PR #305 and share your benchmark results?
OutdatedGuy commentedon Jul 23, 2026
@demolaf worked like a charm. Awesome work 🙌🏼
demolaf commentedon Jul 24, 2026
@OutdatedGuy awesome, thanks for catching this.