Skip to content

Firestore pagination with a map field using the Python Admin SDK ends with error #281

@torresalmonte

Description

@torresalmonte

Describe your environment

  • Operating System version: Debian GNU/Linux 9.8 (stretch)
  • Firebase SDK version: 2.16.0
  • Library version: 2.16.0
  • Firebase Product: firestore

Describe the problem

I'm trying to paginate a collection using a map field, but I get an error when trying to fetch the second page:

l050ZZ1bAsBddd7hE1GD => {'wordcount': {'page3': 130, 'page2': 120, 'page1': 100}, 'availability_date': DatetimeWithNanoseconds(2019, 3, 4, 6, 0, tzinfo=<UTC>), 'count': 100}
JCn6BxwfEDIvirnkMyOf => {'wordcount': {'page3': 180, 'page2': 120, 'page1': 150}, 'availability_date': DatetimeWithNanoseconds(2019, 3, 4, 6, 0, tzinfo=<UTC>), 'count': 150}
nfW0ybNz4WPGWtikfVK2 => {'count': 200, 'wordcount': {'page3': 50, 'page2': 100, 'page1': 200}}
last value: 200
Traceback (most recent call last):
  File "/home/ctorres/.local/lib/python3.5/site-packages/google/cloud/firestore_v1/query.py", line 642, in _normalize_cursor
    values.append(field_path_module.get_nested_value(order_key, data))
  File "/home/ctorres/.local/lib/python3.5/site-packages/google/cloud/firestore_v1/field_path.py", line 243, in get_nested_value
    raise KeyError(msg)
KeyError: "'wordcount' is not contained in the data"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "firestore_pagination.py", line 28, in <module>
    docs_list = list(docs)
  File "/home/ctorres/.local/lib/python3.5/site-packages/google/cloud/firestore_v1/query.py", line 736, in stream
    self._to_protobuf(),
  File "/home/ctorres/.local/lib/python3.5/site-packages/google/cloud/firestore_v1/query.py", line 675, in _to_protobuf
    start_at = self._normalize_cursor(self._start_at, orders)
  File "/home/ctorres/.local/lib/python3.5/site-packages/google/cloud/firestore_v1/query.py", line 645, in _normalize_cursor
    raise ValueError(msg)
ValueError: The "order by" field path 'wordcount.page1' is not present in the cursor data {'wordcount.page1': 200}. All fields sent to ``order_by()`` must be present in the fields if passed to one of ``start_at()`` / ``start_after()`` / ``end_before()`` / ``end_at()`` to define a cursor.

Attached code:
requirements.txt
firestore_pagination.txt

Steps to reproduce:

  • Create a collection with several documents including a map field
  • Rename firestore_pagination.txt to firestore_pagination.py
  • Create a new virtualenvironment
  • Install requirements.txt
  • Set the variable GOOGLE_APPLICATION_CREDENTIALS to a valid service account file
  • Run the script

What happened? How can we make the problem occur?
This could be a description, log/console output, etc.

The equivalent query for nodejs works without issue, so I'm assuming that this is supported although it's not explicitly mentioned in the documentation.

Relevant Code:

firestore_pagination.py

import firebase_admin
from firebase_admin import firestore

firebase_admin.initialize_app()

db = firestore.client()

pages_ref = db.collection('pages')

first_query = pages_ref.order_by('wordcount.page1').limit(3)
docs = first_query.get()

docs_list = list(docs)

for doc in docs_list:
    print(u'{} => {}'.format(doc.id, doc.to_dict()))

last_value = docs_list[-1].to_dict()[u'wordcount'][u'page1']

print('last value: {}'.format(last_value))

next_query = pages_ref.order_by(u'wordcount.page1').start_after({u'wordcount.page1': last_value}).limit(3)
docs = next_query.get()

docs_list = list(docs)
print(len(docs_list))

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions