Actions
Bug #65654
openrun-bucket-check.sh: failed assert len(json_out) == len(unlinked_keys)
% Done:
0%
Source:
Backport:
reef squid tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Description
https://qa-proxy.ceph.com/teuthology/suriarte-2024-04-23_15:04:03-rgw-rgw-update-boost-redis-distro-default-smithi/7669974/teuthology.log
https://qa-proxy.ceph.com/teuthology/suriarte-2024-04-23_15:04:03-rgw-rgw-update-boost-redis-distro-default-smithi/7669997/teuthology.log
2024-04-23T19:08:46.262 INFO:tasks.workunit.client.0.smithi145.stdout:Could not connect to the endpoint URL: "http://localhost:80/" 2024-04-23T19:08:46.262 INFO:tasks.workunit.client.0.smithi145.stdout:Could not connect to the endpoint URL: "http://localhost:8000/" 2024-04-23T19:08:46.262 INFO:tasks.workunit.client.0.smithi145.stdout:connected to https://localhost:443 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr:Traceback (most recent call last): 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 193, in <module> 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr: main() 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 176, in main 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr: assert len(json_out) == len(unlinked_keys) 2024-04-23T19:08:46.263 INFO:tasks.workunit.client.0.smithi145.stderr:AssertionError
Updated by Cory Snyder almost 2 years ago
It looks like the test reproduced and caught another scenario where unlinked objects get left behind. Still looking at logs to identify precisely how it happened.
Updated by J. Eric Ivancich over 1 year ago
Hey Cory, just wanted to ping you on this urgent bug.
Updated by Casey Bodley over 1 year ago
Updated by J. Eric Ivancich over 1 year ago ยท Edited
- Status changed from New to Need More Info
Because this failure showed up in the squid-release branch, I looked into it. I was unable to reproduce. The logic in the test looks right. I then reviewed the code in the two RGWBucket::check_index_unlinked functions, thinking that perhaps if the check occurred during the same second that the unlinked keys were created, then it wouldn't work (i.e., a race) but that logic looks good.
I'm tempted to mark this as 'can't reproduce'.
Updated by Casey Bodley over 1 year ago
2024-09-25T13:39:19.558 INFO:tasks.workunit.client.0.smithi092.stderr:Traceback (most recent call last): 2024-09-25T13:39:19.558 INFO:tasks.workunit.client.0.smithi092.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 193, in <module> 2024-09-25T13:39:19.559 INFO:tasks.workunit.client.0.smithi092.stderr: main() 2024-09-25T13:39:19.559 INFO:tasks.workunit.client.0.smithi092.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 176, in main 2024-09-25T13:39:19.559 INFO:tasks.workunit.client.0.smithi092.stderr: assert len(json_out) == len(unlinked_keys) 2024-09-25T13:39:19.559 INFO:tasks.workunit.client.0.smithi092.stderr:AssertionError
Updated by Casey Bodley over 1 year ago
opened https://github.com/ceph/ceph/pull/60001 to log json_out and unlinked_keys before this comparison
Updated by Casey Bodley about 1 year ago
captured a failure in https://qa-proxy.ceph.com/teuthology/cbodley-2024-12-05_21:18:11-rgw-wip-cbodley-testing-distro-default-smithi/8022557/teuthology.log
2024-12-05T21:48:39.960 INFO:tasks.workunit.client.0.smithi157.stderr:"bucket check unlinked" returned [{'name': 'd', 'instance': 'LmFz8AJZ6VuWE7c18PpuTG6d6aoSMWD'}, {'name': 'a', 'instance': ''}, {'name': 'c', 'instance': 'nbYeCc9hgNHVXQDDBKbIhgL.-sZVo7q'}, {'name': 'f', 'instance': 'EPZ.wOA7r8sYBIInto.RkJpdepNhFdG'}, {'name': 'e', 'instance': 'iQkU4rjTIegfw2LxfUueH4GH7EXCCbm'}], expecting ['c', 'd', 'e', 'f']
2024-12-05T21:48:39.961 INFO:tasks.workunit.client.0.smithi157.stdout:Could not connect to the endpoint URL: "http://localhost:80/"
2024-12-05T21:48:39.961 INFO:tasks.workunit.client.0.smithi157.stdout:Could not connect to the endpoint URL: "http://localhost:8000/"
2024-12-05T21:48:39.961 INFO:tasks.workunit.client.0.smithi157.stdout:connected to https://localhost:443
2024-12-05T21:48:39.961 INFO:tasks.workunit.client.0.smithi157.stderr:Traceback (most recent call last):
2024-12-05T21:48:39.961 INFO:tasks.workunit.client.0.smithi157.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 194, in <module>
2024-12-05T21:48:39.962 INFO:tasks.workunit.client.0.smithi157.stderr: main()
2024-12-05T21:48:39.962 INFO:tasks.workunit.client.0.smithi157.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 177, in main
2024-12-05T21:48:39.962 INFO:tasks.workunit.client.0.smithi157.stderr: assert len(json_out) == len(unlinked_keys)
2024-12-05T21:48:39.962 INFO:tasks.workunit.client.0.smithi157.stderr:AssertionError
2024-12-05T21:48:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1
seems that the result included an unexpected entry {'name': 'a', 'instance': ''}
Updated by Casey Bodley 12 months ago
- Status changed from Need More Info to New
2025-02-26T06:33:24.546 INFO:tasks.workunit.client.0.smithi055.stderr:"bucket check unlinked" returned [{'name': 'a', 'instance': ''}, {'name': 'c', 'instance': 'ZjT7ZE.DH-2XwTCafm8Kyer3kQzmsDL'}, {'name': 'd', 'instance': 'le6gN5d.m6l7sRLryWzfk8xHCqiPEgz'}, {'name': 'e', 'instance': 'b3sLK8qQ5vYnMlRpOKZfZPhrKjxfuR9'}, {'name': 'f', 'instance': 'i5X9R13WbZnobobMIDihybOYpTm4VJW'}], expecting ['c', 'd', 'e', 'f']
2025-02-26T06:33:24.546 INFO:tasks.workunit.client.0.smithi055.stdout:connected to http://localhost:80
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr:Traceback (most recent call last):
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 194, in <module>
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr: main()
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr: File "/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py", line 177, in main
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr: assert len(json_out) == len(unlinked_keys)
2025-02-26T06:33:24.547 INFO:tasks.workunit.client.0.smithi055.stderr:AssertionError
Updated by J. Eric Ivancich 3 months ago
- Status changed from New to In Progress
Actions