Skip to content

Logs: harden dict view iterations #13272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nik-localstack merged 1 commit into from
Oct 15, 2025
Merged

Logs: harden dict view iterations #13272

nik-localstack merged 1 commit into from
Oct 15, 2025

Conversation

@nik-localstack
Copy link
Contributor

@nik-localstack nik-localstack commented Oct 15, 2025 •
edited
Loading

Motivation

The following issue came up when looking in telemetry for issues with the new Batch service. Batch provider calls the DescribeLogGroups API when creating a new task.

exception while calling logs.DescribeLogGroups: dictionary changed size during iterationTraceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 116, in __call__
    handler(chain, context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 86, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 155, in invoke
    return self.dispatch_request(serializer, context, instance)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 169, in dispatch_request
    result = handler(context, instance) or {}
             ~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/forwarder.py", line 137, in _call
    return handler(context, req)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/api/core.py", line 181, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/services/logs/provider.py", line 108, in describe_log_groups
    copy_groups = copy.deepcopy(region_backend.groups)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 163, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.13/copy.py", line 260, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 221, in _deepcopy_dict
    for key, value in x.items():
                      ~~~~~~~^^
localstack.aws.api.core.CommonServiceException: exception while calling logs.DescribeLogGroups: dictionary changed size during iteration

The error occurs because the dictionary region_backend.groups is being modified by another thread while copy.deepcopy() is trying to iterate over it. This is a classic race condition where concurrent modifications to the dictionary cause the "dictionary changed size during iteration" error.

Changes

  • Fixes the race condition, by creating a shallow copy first to avoid the race condition, then do the deep copy:

@nik-localstack nik-localstack self-assigned this Oct 15, 2025
@nik-localstack nik-localstack added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Oct 15, 2025
@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results - Preflight, Unit

22 337 tests  ±0   20 587 ✅ ±0   15m 20s ⏱️ -31s
     1 suites ±0    1 750 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 20s ⏱️ -4s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   53m 43s ⏱️ - 1h 7m 10s
2 016 tests  - 2 801  1 923 ✅  - 2 557  93 💤  - 244  0 ❌ ±0 
2 018 runs   - 2 801  1 923 ✅  - 2 557  95 💤  - 244  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 2801 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results (amd64) - Integration, Bootstrap

    5 files  ±    0      5 suites  ±0   1h 17m 4s ⏱️ - 1h 24m 15s
2 040 tests  - 3 151  1 949 ✅  - 2 745  91 💤  - 406  0 ❌ ±0 
2 046 runs   - 3 151  1 949 ✅  - 2 745  97 💤  - 406  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 3151 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@nik-localstack nik-localstack marked this pull request as ready for review October 15, 2025 11:14
pinzon
pinzon approved these changes Oct 15, 2025
View reviewed changes
Copy link
Member

@pinzon pinzon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nik-localstack nik-localstack merged commit 814a597 into main Oct 15, 2025
56 checks passed
@nik-localstack nik-localstack deleted the logs/improve-dictionary-handling branch October 15, 2025 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pinzon pinzon pinzon approved these changes

@steffyP steffyP Awaiting requested review from steffyP steffyP is a code owner

Assignees

@nik-localstack nik-localstack

Labels

docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nik-localstack @pinzon