Implement pagination to localstacks SQS list_queues method #12397

DrisDary · 2025-03-17T13:23:40Z

Motivation

The following PR addresses the bug: #11774

Currently, localstacks sqs list_queues method does not implement pagination and cannot display results outside of a given 'MaxResults'. No 'NextToken' value is generated.

Expected behaviour: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ListQueues.html

Changes

The localstack sqs list_queues method now implements pagination by returning and accepting a NextToken value. If you do not set MaxResults, then the response only returns a maximum of 1000 results and all other values are retrieved by using the NextToken value from the previous response.

Testing

create a list of 20 queues:

for i in $(seq 1 20); do
  awslocal sqs create-queue --queue-name "Queue$i"
  echo "Created Queue$i"
done

verify creation of queues:

awslocal sqs list-queues

Apply a MaxResult value.This should result in only the first 4 queues and a NextToken value:

awslocal sqs list-queues --max-results 4

Pass the value of the above NextToken in the response above to see the next batch of queues:

awslocal sqs list-queues --max-results 4 --next-token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

TODO

Implement error handling and error testing as described by https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ListQueues.html

2) Applied parity test to changes from 1)

localstack-bot

Welcome to LocalStack! Thanks for raising your first Pull Request and landing in your contributions. Our team will reach out with any reviews or feedbacks that we have shortly. We recommend joining our Slack Community and share your PR on the #community channel to share your contributions with us. Please make sure you are following our contributing guidelines and our Code of Conduct.

localstack-bot · 2025-03-17T13:23:56Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

DrisDary · 2025-03-17T13:30:56Z

I have read the CLA Document and I hereby sign the CLA

2) applied transfomer

alexrashed

Thanks a lot for your contribution, @DrisDary!
The changes are looking really good! We just need to address some comments / discuss some changes, afterwards this is good to get merged into LocalStack! 🚀

.gitignore

Makefile

localstack-core/localstack/services/sqs/constants.py

alexrashed · 2025-03-19T08:17:23Z

localstack-core/localstack/utils/collections.py

            page_size = self.DEFAULT_PAGE_SIZE

-        if len(result_list) <= page_size:
+        if len(result_list) <= page_size and next_token is None:


suggestion: ‏Could you maybe add a comment here to explain what these two checks are doing?

Sure. Just FYI, if length of list is less than the permitted max default size and next_token is not found, then return the last N remaining elements of page where N is length of list. Without the next_token condition, there may be cases where elements from previous pages could be returned.

Oh, interesting! So basically, what you are saying is that if I use next_token on an endpoint where there are less entries than the defined max, the next_token won't be considered.
What is the handling of AWS in this case? Have you verified that AWS respects the next token in this case?
Very good catch!

suppose you have awslocal sqs list-queues

{ "QueueUrls": [ "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-0", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-1", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-2", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-3", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-4", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-5", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-6", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-7", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-8", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-9" ] }

and you executed awslocal sqs list-queues --max-results 2 then you would get

{ "QueueUrls": [ "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-0", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-1" ], "NextToken": "aHR0cDovL3Nxcy48cmVnaW9uPi5sb2NhbGhvc3QubG9jYWxzdGFjay5jbG91ZDo0NTY2LzExMTExMTExMTExMS83ZjdkZjBmNS10ZXN0LXF1ZXVlLTE=" }

if after you execute list-queues --max-results 10 --next-token "aHR0cDovL3Nxcy48cmVnaW9uPi5sb2NhbGhvc3QubG9jYWxzdGFjay5jbG91ZDo0NTY2LzExMTExMTExMTExMS83ZjdkZjBmNS10ZXN0LXF1ZXVlLTE=" then you now get

{ "QueueUrls": [ "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-2", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-3", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-4", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-5", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-6", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-7", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-8", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-9" ] }

whereas before you would get

{ "QueueUrls": [ "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-0", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-1", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-2", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-3", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-4", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-5", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-6", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-7", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-8", "http://sqs.<region>.localhost.localstack.cloud:4566/111111111111/7f7df0f5-test-queue-9" ] }

which of course is incorrect.

The addition of 'and next_token is None:' corrects this.

Thanks a lot for the detailed explanation! This is a really great catch and will actually affect quite a few other services! 🤩

tests/aws/services/sqs/test_sqs.py

tests/aws/services/sqs/test_sqs.validation.json

2) Applied comments 3) updated snapshot test

alexrashed

Thanks a lot for addressing all the comments! I added another round, I think you already fixed the rebase issues, but the test still needs to be adjusted. Afterwards, we should be good to go! 🚀

tests/aws/services/sqs/test_sqs.py

tests/aws/services/sqs/test_sqs.snapshot.json

tests/aws/services/sqs/test_sqs.py

tests/aws/services/sqs/test_sqs.validation.json

alexrashed · 2025-03-19T16:57:00Z

localstack-core/localstack/utils/collections.py

            page_size = self.DEFAULT_PAGE_SIZE

-        if len(result_list) <= page_size:
+        if len(result_list) <= page_size and next_token is None:


Oh, interesting! So basically, what you are saying is that if I use next_token on an endpoint where there are less entries than the defined max, the next_token won't be considered.
What is the handling of AWS in this case? Have you verified that AWS respects the next token in this case?
Very good catch!

Makefile

tests/aws/services/sqs/test_sqs.snapshot.json

2) updated snapshots

Testing "no newline at end of file"

2)Added updated validation

alexrashed

Thanks so much for the great contribution! Really thorough testing and you even caught an issue in the pagination implementation shared with a lot of other services! 🚀

alexrashed · 2025-03-20T07:27:37Z

localstack-core/localstack/utils/collections.py

            page_size = self.DEFAULT_PAGE_SIZE

-        if len(result_list) <= page_size:
+        if len(result_list) <= page_size and next_token is None:


Thanks a lot for the detailed explanation! This is a really great catch and will actually affect quite a few other services! 🤩

DrisDary added 5 commits March 6, 2025 20:27

added .vs to gitignore

24b92eb

1) Applied pagination to localstack pagination.

2f62308

2) Applied parity test to changes from 1)

Merge branch 'master' of https://github.com/DrisDary/localstack

d68ea93

Applied make format

3022972

Merge branch 'localstack:master' into master

ba9114e

DrisDary requested review from baermat, gregfurman and thrau as code owners March 17, 2025 13:23

localstack-bot reviewed Mar 17, 2025

View reviewed changes

localstack-bot added a commit that referenced this pull request Mar 17, 2025

@DrisDary has signed the CLA in #12397

8437c91

alexrashed self-requested a review March 17, 2025 13:58

alexrashed self-assigned this Mar 17, 2025

alexrashed added the semver: patch Non-breaking changes which can be included in patch releases label Mar 17, 2025

alexrashed added this to the Playground milestone Mar 17, 2025

DrisDary added 5 commits March 18, 2025 14:15

1) corrected failed tests

0f092f6

2) applied transfomer

Merge branch 'master' of https://github.com/DrisDary/localstack

705a5d8

Merge branch 'localstack:master' into master

5e52a51

corrected format

b89f4d6

Merge branch 'master' of https://github.com/DrisDary/localstack

eee9cf5

alexrashed requested changes Mar 19, 2025

View reviewed changes

DrisDary added 4 commits March 19, 2025 15:14

1) Applied seeded uuid to test

d61389f

2) Applied comments 3) updated snapshot test

Merge branch 'master' into master

0f5f9a2

updated snapshot

7b5e9a6

Merge branch 'master' of https://github.com/DrisDary/localstack

54ab55b

alexrashed reviewed Mar 19, 2025

View reviewed changes

DrisDary added 3 commits March 19, 2025 19:28

Merge branch 'localstack:master' into master

21d8b84

1) Added listqueues prefix

af6c96f

2) updated snapshots

Delete Makefile

ef0ef69

Testing "no newline at end of file"

DrisDary added 4 commits March 19, 2025 20:33

Added Makefile

91a9767

1)Added updated snapshot

66b1f79

2)Added updated validation

Merge branch 'master' of https://github.com/DrisDary/localstack

9b31d98

Merge branch 'localstack:master' into master

f9b4447

alexrashed approved these changes Mar 20, 2025

View reviewed changes

thrau merged commit 68df8dc into localstack:master Mar 21, 2025
29 checks passed

Uh oh!

Implement pagination to localstacks SQS list_queues method #12397

Implement pagination to localstacks SQS list_queues method #12397

Uh oh!

Conversation

DrisDary commented Mar 17, 2025

Motivation

Changes

Testing

TODO

Uh oh!

localstack-bot left a comment

Choose a reason for hiding this comment

Uh oh!

localstack-bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrisDary commented Mar 17, 2025

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexrashed Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

DrisDary Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrashed Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

DrisDary Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrashed Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexrashed Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

alexrashed Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

localstack-bot commented Mar 17, 2025 •

edited

Loading

DrisDary Mar 19, 2025 •

edited

Loading

DrisDary Mar 19, 2025 •

edited

Loading