So this has obviously created havoc for the other component suites. I think the right thing to do here is:

• revert most of the qa/ suite changes
• add a developer config to exclude pools that are empty which is turned on for all qa tests, probably in qa/tasks/ceph.conf/template

That config can be selectively turned back on for a dedicated set of tests for this particular change.

cc @cbodley @idryomov @vshankar

I also strongly dislike ignoring warnings. This should not be a routine way to fix these types of issues even if this warning is particularly unhelpful for QA.

• add a developer config to exclude pools that are empty which is turned on for all qa tests, probably in qa/tasks/ceph.conf/template

That config can be selectively turned back on for a dedicated set of tests for this particular change.

@batrick I don't see how that is substantially different from just ignoring the health alert -- instead of the "ignoring" part being conditional, you would be making the raising of the health alert conditional, that's it.

IMO the right way to address this is to make RADOS not raise a bogus health alert independent of the configuration, so that it's not seen by regular users/operators too. It shouldn't be just developers running their tests with a special knob set who get the sane behavior. I suggested the following on the dev list:

[...] perhaps we could treat each new POOL_APP_NOT_ENABLED as muted behind the scenes for a couple of minutes? Essentially just delay raising the alert to give admins enough time to run "ceph osd pool application enable" or our own tools to call rados_application_enable() API.

• add a developer config to exclude pools that are empty which is turned on for all qa tests, probably in qa/tasks/ceph.conf/template

That config can be selectively turned back on for a dedicated set of tests for this particular change.

@batrick I don't see how that is substantially different from just ignoring the health alert -- instead of the "ignoring" part being conditional, you would be making the raising of the health alert conditional, that's it.

The difference is we don't need to play whack-a-mole by adding the warning to ignorelist for dozens of suites. As this PR does. Of course, playing that game means we miss things which is why we're having this discussion :)

IMO the right way to address this is to make RADOS not raise a bogus health alert independent of the configuration, so that it's not seen by regular users/operators too. It shouldn't be just developers running their tests with a special knob set who get the sane behavior. I suggested the following on the dev list:

[...] perhaps we could treat each new POOL_APP_NOT_ENABLED as muted behind the scenes for a couple of minutes? Essentially just delay raising the alert to give admins enough time to run "ceph osd pool application enable" or our own tools to call rados_application_enable() API.

That also seems reasonable but I think it'll still break many QA tests.

The difference is we don't need to play whack-a-mole by adding the warning to ignorelist for dozens of suites.

I agree that the config opt would be basically an ingorelist at the C++ level. I would prefer to not put extra, QA-driven complexity here.

If ignoring is the way, then we should consider sharing log-ignorelist across the suites. At first glance it looks like a change in the ceph.py task.

How big the impact really is? Noise in suites, right? Aren't the failures pretty schematic (like "POOL_APP_NOT_ENABLED has been found"), and thus easy to discriminate?

I'm asking these questions as the patch fixes a real issue that popped up at a user's site.

How big the impact really is? Noise in suites, right? Aren't the failures pretty schematic (like "POOL_APP_NOT_ENABLED has been found"), and thus easy to discriminate?

I'm asking these questions as the patch fixes a real issue that popped up at a user's site.

The patch can be resubmitted. I think there are two main issues here:

• This PR broke a lot of QA tests. That should have blocked merge and is justification enough to revert.
• The QA changes are widespread and should have been consolidated into e.g. ceph.py as you mentioned (as one possibility) Those need reverted even if this change is not reverted. I think the entire PR should be reverted so it can be re-[done correctly.

[...] perhaps we could treat each new POOL_APP_NOT_ENABLED as muted behind the scenes for a couple of minutes? Essentially just delay raising the alert to give admins enough time to run "ceph osd pool application enable" or our own tools to call rados_application_enable() API.

That also seems reasonable but I think it'll still break many QA tests.

@batrick Can you elaborate on the "many"? The way I see it, the tests that don't set an application on the pools they create but then write to them would already have POOL_APP_NOT_ENABLED in the ignorelist. The tests that don't write might break -- assuming the pools in question live long enough (i.e. not removed within X minutes before the alert is raised). But this latter variety should be rare?

[...] perhaps we could treat each new POOL_APP_NOT_ENABLED as muted behind the scenes for a couple of minutes? Essentially just delay raising the alert to give admins enough time to run "ceph osd pool application enable" or our own tools to call rados_application_enable() API.

That also seems reasonable but I think it'll still break many QA tests.

@batrick Can you elaborate on the "many"? The way I see it, the tests that don't set an application on the pools they create but then write to them would already have POOL_APP_NOT_ENABLED in the ignorelist. The tests that don't write might break -- assuming the pools in question live long enough (i.e. not removed within X minutes before the alert is raised). But this latter variety should be rare?

I think you're right. I retract my statement.

#53336