Skip to content
This repository was archived by the owner on Mar 4, 2026. It is now read-only.
This repository was archived by the owner on Mar 4, 2026. It is now read-only.

Too many documents returned from query.stream with limit after retry #2200

Closed
#2203
Closed
Too many documents returned from query.stream with limit after retry#2200
#2203
Assignees
ehsannas
Labels
api: firestoreIssues related to the googleapis/nodejs-firestore API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
@wvanderdeijl

Description

@wvanderdeijl
wvanderdeijl
opened on Sep 27, 2024

Environment details

  • OS: macOS 15.0
  • Node.js version: 20.11.0
  • npm version: 10.2.4
  • @google-cloud/firestore version: 7.10.0

Steps to reproduce

We sometimes see that a streaming query returns more rows than the limit set in the query. In production this is only intermittent, but we were able to reproduce this in a demo script.

I think the issue is around

nodejs-firestore/dev/src/reference/query-util.ts

Line 333 in 1775ce6

if (this._queryOptions.requireConsistency) {

Here the query is re-executed after an error, but there doesn't seem to be any code to reduce the limit of the query with the number of documents already fetched, nor could I find any code that ends the stream once the total expected number of documents was fetched.

We used the following script to reproduce:

import { Firestore, setLogFunction } from '@google-cloud/firestore';
import * as assert from 'assert';

const firestore = new Firestore({ projectId: 'my-project' });

(async () => {
    console.log('creating documents');
    const bw = firestore.bulkWriter();
    for (let i = 0; i < 100_000; i++) {
        bw.create(firestore.collection('docs').doc(), {foo: i});
    }
    await bw.close();

    setLogFunction(msg => process.stdout.write(msg + '\n'));

    console.log('start counting at', new Date().toLocaleTimeString());
    const LIMIT = 99_900;
    let count = 0;
    const nums = [];
    for await (const doc of firestore.collection('docs').orderBy('foo').limit(LIMIT).stream()) {
        nums.push((doc as unknown as FirebaseFirestore.QueryDocumentSnapshot<{foo:number}>).data().foo);
        count++;
        await new Promise(resolve => setTimeout(resolve, 2)); // wait a bit so we run into Query timeout
    }
    console.log('done counting at', new Date().toLocaleTimeString());
    assert.strictEqual(count, LIMIT);
})().catch(e => {
    console.log(e);
    process.exit(1);
});

This fails with the assertion about the count since we fetched too many documents:

done counting at 8:24:37 AM
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

100000 !== 99900

    at /Users/***/firebase-stream-limit/index.ts:26:12
    at Generator.next (<anonymous>)
    at fulfilled (/Users/***/firebase-stream-limit/index.ts:28:58) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 100000,
  expected: 99900,
  operator: 'strictEqual'
}

Here are the interesting parts of the debug logging where you can see that the query is retried after the failure with the original limit (thus returning a full set again):


> [email protected] start
> ts-node index.ts

creating documents
start counting at 8:20:41 AM
Firestore (7.10.0) 2024-09-27T06:20:41.520Z ca8b8 [ClientPool.acquire]: Re-using existing client with 100 remaining operations
Firestore (7.10.0) 2024-09-27T06:20:41.521Z ca8b8 [Firestore.requestStream]: Sending request: {"parent":"projects/my-project/databases/(default)/documents","structuredQuery":{"from":[{"collectionId":"docs"}],"orderBy":[{"field":{"fieldPath":"foo"},"direction":"ASCENDING"}],"limit":{"value":99900}}}
Firestore (7.10.0) 2024-09-27T06:20:41.685Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"0","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/pjbfu4L1wo6qNE4cDVKA","createTime":{"seconds":"1727417837","nanos":522716000},"updateTime":{"seconds":"1727417837","nanos":522716000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:20:41.685Z ca8b8 [Firestore._initializeStream]: Stream ready
Firestore (7.10.0) 2024-09-27T06:20:41.686Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"1","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/zgPDCNwRrBc0igloi7Ki","createTime":{"seconds":"1727417837","nanos":510459000},"updateTime":{"seconds":"1727417837","nanos":510459000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:20:41.686Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"2","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/x84n3UYkEGmDPXwQAPr7","createTime":{"seconds":"1727417837","nanos":525929000},"updateTime":{"seconds":"1727417837","nanos":525929000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
... snip ...
Firestore (7.10.0) 2024-09-27T06:22:54.574Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57132","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/FGX0a1BDXyhLUbbUytGL","createTime":{"seconds":"1727417954","nanos":94545000},"updateTime":{"seconds":"1727417954","nanos":94545000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.574Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57133","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/jOFhHA93ODyQMcX7TqBj","createTime":{"seconds":"1727417954","nanos":94623000},"updateTime":{"seconds":"1727417954","nanos":94623000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.620Z ca8b8 [Firestore._initializeStream]: Received stream error: Error: 14 UNAVAILABLE: Query timed out. Please try either limiting the entities scanned, or run with an updated index configuration.
    at callErrorFromStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/call.ts:82:17)
    at Object.onReceiveStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client.ts:612:51)
    at Object.onReceiveStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
    at /***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeServerStreamRequest (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client.ts:595:42)
    at ServiceClientImpl.<anonymous> (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
    at /***/firebase-stream-limit/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:237:29
    at /***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:38:28
    at /***/firebase-stream-limit/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at Object.request (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streaming.js:376:40)
    at makeRequest (/***/firebase-stream-limit/node_modules/retry-request/index.js:159:28)
    at retryRequest (/***/firebase-stream-limit/node_modules/retry-request/index.js:119:5)
    at StreamProxy.setStream (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streaming.js:367:37)
    at StreamingApiCaller.call (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:54:16) {
  code: 14,
  details: 'Query timed out. Please try either limiting the entities scanned, or run with an updated index configuration.',
  metadata: Metadata {
    internalRepr: Map(2) {
      'content-disposition' => [Array],
      'x-debug-tracking-id' => [Array]
    },
    options: {}
  }
}
Firestore (7.10.0) 2024-09-27T06:22:54.620Z ca8b8 [Firestore._initializeStream]: Received stream end
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57134","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/6TZHik0xJpRpbvcLv47Z","createTime":{"seconds":"1727417954","nanos":94286000},"updateTime":{"seconds":"1727417954","nanos":94286000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57135","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/K1xcEA0em6lO4utu4rbX","createTime":{"seconds":"1727417954","nanos":94719000},"updateTime":{"seconds":"1727417954","nanos":94719000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57136","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/tchNu5hT2DmUInubLbTk","createTime":{"seconds":"1727417954","nanos":94546000},"updateTime":{"seconds":"1727417954","nanos":94546000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57137","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/Y7cEhEbxmZzPYgUpf7XM","createTime":{"seconds":"1727417954","nanos":94623000},"updateTime":{"seconds":"1727417954","nanos":94623000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57138","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/WvxucXpw7PeDy4UaNpdF","createTime":{"seconds":"1727417954","nanos":94285000},"updateTime":{"seconds":"1727417954","nanos":94285000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57139","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/GKdiNBQDH434h3aw2eE6","createTime":{"seconds":"1727417954","nanos":94721000},"updateTime":{"seconds":"1727417954","nanos":94721000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57140","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/ogTO2SAvqnDCJpGVxgNu","createTime":{"seconds":"1727417954","nanos":171161000},"updateTime":{"seconds":"1727417954","nanos":171161000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57141","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/Tr7dp9DfKSTt6IGFQWh7","createTime":{"seconds":"1727417954","nanos":170661000},"updateTime":{"seconds":"1727417954","nanos":170661000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57142","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/9oBusX6dIh3S4FWb8qIx","createTime":{"seconds":"1727417954","nanos":173825000},"updateTime":{"seconds":"1727417954","nanos":173825000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57143","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/gFYG0aaBJCJ6oAh8mgqk","createTime":{"seconds":"1727417954","nanos":171096000},"updateTime":{"seconds":"1727417954","nanos":171096000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57144","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/qfV78gAk0ptgcCemc1vW","createTime":{"seconds":"1727417954","nanos":171158000},"updateTime":{"seconds":"1727417954","nanos":171158000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57145","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/qzEEJ6FWFwRBg97fQm7x","createTime":{"seconds":"1727417954","nanos":171162000},"updateTime":{"seconds":"1727417954","nanos":171162000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57146","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/sIgDOgs2mKmSJMgr5HRM","createTime":{"seconds":"1727417954","nanos":171094000},"updateTime":{"seconds":"1727417954","nanos":171094000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57147","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/Vr9e6Jrr6IZCPelijXQm","createTime":{"seconds":"1727417954","nanos":171157000},"updateTime":{"seconds":"1727417954","nanos":171157000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57148","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/I98U8HPdV6unqk9dRg03","createTime":{"seconds":"1727417954","nanos":173828000},"updateTime":{"seconds":"1727417954","nanos":173828000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57149","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/DF7Y6J1MbG9hgn04nZr9","createTime":{"seconds":"1727417954","nanos":173827000},"updateTime":{"seconds":"1727417954","nanos":173827000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.621Z ca8b8 [QueryUtil._stream]: Query failed with retryable stream error: Error: 14 UNAVAILABLE: Query timed out. Please try either limiting the entities scanned, or run with an updated index configuration.
    at callErrorFromStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/call.ts:82:17)
    at Object.onReceiveStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client.ts:612:51)
    at Object.onReceiveStatus (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
    at /***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeServerStreamRequest (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/client.ts:595:42)
    at ServiceClientImpl.<anonymous> (/***/firebase-stream-limit/node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
    at /***/firebase-stream-limit/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:237:29
    at /***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:38:28
    at /***/firebase-stream-limit/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at Object.request (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streaming.js:376:40)
    at makeRequest (/***/firebase-stream-limit/node_modules/retry-request/index.js:159:28)
    at retryRequest (/***/firebase-stream-limit/node_modules/retry-request/index.js:119:5)
    at StreamProxy.setStream (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streaming.js:367:37)
    at StreamingApiCaller.call (/***/firebase-stream-limit/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:54:16) {
  code: 14,
  details: 'Query timed out. Please try either limiting the entities scanned, or run with an updated index configuration.',
  metadata: Metadata {
    internalRepr: Map(2) {
      'content-disposition' => [Array],
      'x-debug-tracking-id' => [Array]
    },
    options: {}
  }
}
Firestore (7.10.0) 2024-09-27T06:22:54.633Z ca8b8 [Query._stream]: Query failed with retryable stream error and progress was made receiving documents, so the stream is being retried.
Firestore (7.10.0) 2024-09-27T06:22:54.635Z ca8b8 [ClientPool.acquire]: Re-using existing client with 100 remaining operations
Firestore (7.10.0) 2024-09-27T06:22:54.635Z ca8b8 [Firestore.requestStream]: Sending request: {"parent":"projects/my-project/databases/(default)/documents","structuredQuery":{"from":[{"collectionId":"docs"}],"orderBy":[{"field":{"fieldPath":"foo"},"direction":"ASCENDING"},{"field":{"fieldPath":"__name__"},"direction":"ASCENDING"}],"startAt":{"values":[{"integerValue":57102},{"referenceValue":"projects/my-project/databases/(default)/documents/docs/Ms3o0n1hD9i89iyb6J4d"}]},"limit":{"value":99900}},"readTime":{"seconds":"1727418041","nanos":608088000}}
Firestore (7.10.0) 2024-09-27T06:22:54.861Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57103","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/LzyXk2goXSxOLCnRAiZy","createTime":{"seconds":"1727417954","nanos":46109000},"updateTime":{"seconds":"1727417954","nanos":46109000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.861Z ca8b8 [Firestore._initializeStream]: Stream ready
Firestore (7.10.0) 2024-09-27T06:22:54.863Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57104","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/eF3xw4tbYxLiyCAokuZ6","createTime":{"seconds":"1727417954","nanos":46402000},"updateTime":{"seconds":"1727417954","nanos":46402000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.863Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57105","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/nTcLERejcftalgcLJxX0","createTime":{"seconds":"1727417954","nanos":46105000},"updateTime":{"seconds":"1727417954","nanos":46105000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:22:54.863Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"57106","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/eue8Elmhpywx1OwqfVk0","createTime":{"seconds":"1727417954","nanos":46742000},"updateTime":{"seconds":"1727417954","nanos":46742000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
... snip ...
Firestore (7.10.0) 2024-09-27T06:24:36.863Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"99997","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/Sd5zdJRlmLAmQGbOSlhG","createTime":{"seconds":"1727418041","nanos":542622000},"updateTime":{"seconds":"1727418041","nanos":542622000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:24:36.900Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"99998","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/uFTLuMS5LaK3LPGVz5GG","createTime":{"seconds":"1727418041","nanos":542425000},"updateTime":{"seconds":"1727418041","nanos":542425000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:24:36.900Z ca8b8 [Firestore.requestStream]: Received response: {"document":{"fields":{"foo":{"integerValue":"99999","valueType":"integerValue"}},"name":"projects/my-project/databases/(default)/documents/docs/fWI0Yaxu4hoQBPDXLOlx","createTime":{"seconds":"1727418041","nanos":542480000},"updateTime":{"seconds":"1727418041","nanos":542480000}},"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1727418041","nanos":608088000},"skippedResults":0,"explainMetrics":null}
Firestore (7.10.0) 2024-09-27T06:24:36.900Z ca8b8 [Firestore._initializeStream]: Received stream end
done counting at 8:24:37 AM
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

100000 !== 99900

    at /***/firebase-stream-limit/index.ts:26:12
    at Generator.next (<anonymous>)
    at fulfilled (/***/firebase-stream-limit/index.ts:28:58) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 100000,
  expected: 99900,
  operator: 'strictEqual'
}

Notice the "Sending request" line at "2024-09-27T06:22:54.635Z" which sends the query again with the original limit of 99900

At first, we thought this was related to a previous issue we filed regarding duplicate documents in a retried stream, but after investigation this appears to have a very different cause.

As a workaround we can limit the stream in client code, but it would be nice if this is handled by the @google-cloud/firestore library. I even feel that if you are very unlucky with the errors you could get a stream that never ends as it only seems to end if a full set to the supplied limit is received in one batch without any intermittent errors.

Metadata

Metadata

Assignees

Labels

api: firestoreIssues related to the googleapis/nodejs-firestore API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions