Skip to content

Comments

test/test_mempool: skip check_shard_select test on ARM64#56448

Merged
rosinL merged 1 commit intoceph:mainfrom
Svelar:arm_pass_check_shard_select
Apr 22, 2024
Merged

test/test_mempool: skip check_shard_select test on ARM64#56448
rosinL merged 1 commit intoceph:mainfrom
Svelar:arm_pass_check_shard_select

Conversation

@Svelar
Copy link
Member

@Svelar Svelar commented Mar 25, 2024

All ARM64 CI jobs are failed because of this failed test. How to address the shifting value is under active discussions, need to find a way to circumvent it first.

See:
Ticket
Possible resolutions:
#55249 Modify shifting value to another static value.
#55696 Replace the current hash of pthread_self() with a round robin assignment of threads to shards.

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@Svelar
test/test_mempool: skip check_shard_select test on ARM64
456fdff 
All ARM64 CI jobs are failed because of this failed test. How to address the shifting value is under active discussions, need to find a way to circumvent it first.

Signed-off-by: Rongqi Sun <[email protected]>
@github-actions github-actions bot added the tests label Mar 25, 2024
@Svelar Svelar requested a review from a team March 25, 2024 12:20
@Svelar Svelar added aarch64 CI Continuous Integration arm64 labels Mar 26, 2024
@ljflores ljflores requested a review from aclamk April 1, 2024 18:31
@ljflores
Copy link
Member

ljflores commented Apr 1, 2024

@Svelar

RE:

Possible resolutions:
#55249 Modify shifting value to another static value.
#55696 Replace the current hash of pthread_self() with a round robin assignment of threads to shards.

Is this PR a third solution?

@Svelar
Copy link
Member Author

Svelar commented Apr 2, 2024

@Svelar

RE:

Possible resolutions:
#55249 Modify shifting value to another static value.
#55696 Replace the current hash of pthread_self() with a round robin assignment of threads to shards.

Is this PR a third solution?

Hi @ljflores,
Not really. It's more like blocking CI failed temporarily. Once either of the above or new resolutions fixed the issue, I would reopen this test for aarch64.

@Svelar
Copy link
Member Author

Svelar commented Apr 12, 2024

jenkins test make check arm64

1 similar comment
@rosinL
Copy link
Member

rosinL commented Apr 19, 2024

jenkins test make check arm64

@rosinL
Copy link
Member

rosinL commented Apr 19, 2024 •
edited
Loading

LGTM, make check(arm64) always failed as the testcase mempool.check_shard_select blurred the make check result, we can make this as a temporary fix to skip the testcase until the proper fix merged.

@rosinL rosinL self-requested a review April 19, 2024 09:36
@rosinL
Copy link
Member

rosinL commented Apr 22, 2024

jenkins test make check arm64

@rosinL rosinL changed the title [RFC] test/test_mempool: skip check_shard_select test on ARM64 test/test_mempool: skip check_shard_select test on ARM64 Apr 22, 2024
@rosinL rosinL merged commit 3453230 into ceph:main Apr 22, 2024
@Svelar
Copy link
Member Author

Svelar commented Apr 22, 2024 •
edited
Loading

Thanks @rosinL . I will file a new PR to reopen this UT once threads to shards issue is addressed.

@bazaah bazaah mentioned this pull request May 20, 2024
Closed
14 tasks
bazaah added a commit to bazaah/aur-ceph that referenced this pull request May 25, 2024
@bazaah
repo: add test-mempool-shard-select-disable
3a2a24a 
this is basically just a performance issue that is triggering on our
build server.

Should be reverted once the upstream decides how to fix it.

Quoting myself:

> looks like just a perf regression the upstream hasn't decided how
> to fix yet: ceph/ceph#56448, ceph/ceph#55249, ceph/ceph#55696

References: #23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rosinL rosinL rosinL approved these changes

@dmick dmick Awaiting requested review from dmick

@akraitman akraitman Awaiting requested review from akraitman

@ljflores ljflores Awaiting requested review from ljflores

@bill-scales bill-scales Awaiting requested review from bill-scales

@aclamk aclamk Awaiting requested review from aclamk

Assignees

No one assigned

Labels

aarch64 arm64 CI Continuous Integration tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Svelar @ljflores @rosinL