Project

General

Profile

Actions
Copy link

Bug #69236

open

md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2)

Added by Laura Flores about 1 year ago. Updated 4 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Ronen Friedman
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
squid,reef,quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
61289
Tags (freeform):
backport_processed
Merge Commit:
091c7c5400f49e0ca16e8462ede8754b81f17b96
Fixed In:
v19.3.0-6889-g091c7c5400
Released In:
v20.2.0~1350
Upkeep Timestamp:
2025-11-01T01:01:15+00:00
Tags:
main-failures

Description

/a/yuriw-2024-12-03_16:16:51-rados-wip-yuri6-testing-2024-12-02-1528-distro-default-smithi/8018937

2024-12-03T19:10:12.083 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub:  sleep 1
2024-12-03T19:10:13.084 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub:  (( i++ ))
2024-12-03T19:10:13.084 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub:  (( i < 300 ))
2024-12-03T19:10:13.085 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub:  get_last_scrub_stamp 1.0 last_scrub_stamp
2024-12-03T19:10:13.085 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp:  local pgid=1.0
2024-12-03T19:10:13.085 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp:  local sname=last_scrub_stamp
2024-12-03T19:10:13.085 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp:  ceph --format json pg dump pgs
2024-12-03T19:10:13.085 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp:  jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp'
2024-12-03T19:10:13.373 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub:  test 2024-12-03T19:10:10.899584+0000 '>' 2024-12-03T19:09:03.839269+0000
2024-12-03T19:10:13.373 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub:  return 0
2024-12-03T19:10:13.374 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps:  grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log
2024-12-03T19:10:13.374 INFO:tasks.workunit.client.0.smithi103.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps:  wc -l
2024-12-03T19:10:13.377 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps:  test 1 = 2
2024-12-03T19:10:13.377 INFO:tasks.workunit.client.0.smithi103.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps:  return 1

Related issues 3 (1 open — 2 closed)

Copied to RADOS - Backport #69530: squid: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2)ResolvedRonen FriedmanActions
Copied to RADOS - Backport #69531: reef: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2)ResolvedRonen FriedmanActions
Copied to RADOS - Backport #69532: quincy: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2)Fix Under ReviewRonen FriedmanActions
Actions
Copy link
 #1

Updated by Radoslaw Zarzynski about 1 year ago

  • Assignee set to Ronen Friedman

Hi Ronen! Would you mind checking whether it's a problem with the scrub itself or scrub simply unveils something underneath?

Actions
Copy link
 #2

Updated by Ronen Friedman about 1 year ago

Turns out it's a problem with handling changes in configuration parameters.
I am not sure why I need to explicitly 'watch' the confs, as I am accessing the values directly.
Will check that (and for now - will add a watch).

Actions
Copy link
 #3

Updated by Aishwarya Mathuria about 1 year ago

/a/skanta-2024-12-05_07:34:09-rados-wip-bharath2-testing-2024-12-04-1214-distro-default-smithi/8021698

Actions
Copy link
 #4

Updated by Ronen Friedman about 1 year ago

  • Subject changed from TEST_scrub_snaps: test 1 = 2 to md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2)
Actions
Copy link
 #5

Updated by Ronen Friedman about 1 year ago

  • Status changed from New to Fix Under Review
  • Backport set to squid,reef,quincy
  • Severity changed from 3 - minor to 2 - major
  • Pull request ID set to 61289

Radek was right in suspecting something deeper. The actual problem is a bug in the cacher
(md_config_t) implementation. See a fuller explanation in the commit message of the suggested fix.

Actions
Copy link
 #6

Updated by Laura Flores about 1 year ago

/a/yuriw-2024-12-18_15:56:21-rados-wip-yuri6-testing-2024-12-17-1653-distro-default-smithi/8043290

Actions
Copy link
 #7

Updated by Naveen Naidu about 1 year ago

/a/skanta-2024-12-11_23:59:30-rados-wip-bharath9-testing-2024-12-10-1652-distro-default-smithi/8031120

Actions
Copy link
 #8

Updated by Radoslaw Zarzynski about 1 year ago

scrub note: approved the PR

Actions
Copy link
 #9

Updated by Ronen Friedman about 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #10

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69530: squid: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2) added
Actions
Copy link
 #11

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69531: reef: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2) added
Actions
Copy link
 #12

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69532: quincy: md_config_t objects fail to register observers (was: TEST_scrub_snaps: test 1 = 2) added
Actions
Copy link
 #13

Updated by Upkeep Bot about 1 year ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #14

Updated by Shraddha Agrawal about 1 year ago

/a/skanta-2024-12-26_01:49:37-rados-wip-bharath12-testing-2024-12-24-0842-distro-default-smithi/8053511

Actions
Copy link
 #15

Updated by Ronen Friedman 12 months ago ยท Edited

Not fixing Quincy. The rest are done.

Actions
Copy link
 #16

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to 091c7c5400f49e0ca16e8462ede8754b81f17b96
  • Fixed In set to v19.3.0-6889-g091c7c5400f
  • Upkeep Timestamp set to 2025-07-08T18:12:46+00:00
Actions
Copy link
 #17

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-6889-g091c7c5400f to v19.3.0-6889-g091c7c5400f4
  • Upkeep Timestamp changed from 2025-07-08T18:12:46+00:00 to 2025-07-14T15:22:56+00:00
Actions
Copy link
 #18

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-6889-g091c7c5400f4 to v19.3.0-6889-g091c7c5400
  • Upkeep Timestamp changed from 2025-07-14T15:22:56+00:00 to 2025-07-14T21:08:49+00:00
Actions
Copy link
 #19

Updated by Upkeep Bot 4 months ago

  • Released In set to v20.2.0~1350
  • Upkeep Timestamp changed from 2025-07-14T21:08:49+00:00 to 2025-11-01T01:01:15+00:00
Actions
Copy link

Also available in: Atom PDF