Context

• Some users don't want all multisite XML links visible in the robots.txt this PR provides a way to filter them.

Summary

This PR can be summarized in the following changelog entry:

• Adds a filter to remove subdirectory sites from the robots.txt

Relevant technical choices:

• The choice was made to only make a turn on or off feature, instead of a filter to actually filter the list of subsites.

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

• Start the multisite docker (not multidomain)
• Open https://multisitedomain.wordpress.test/robots.txt you should see 3 entries
• Sitemap: http://multisitedomain.wordpress.test/sitemap_index.xml
  Sitemap: https://multisitedomain.wordpress.test/sitemap_index.xml
  Sitemap: https://multisitedomain.wordpress.test/site2/sitemap_index.xml
• add add_filter( 'wpseo_should_add_subdirectory_multisite_xml_sitemaps', __return_false ); as a code snippet and refresh the .txt file. It should now only show Sitemap: http://multisitedomain.wordpress.test/sitemap_index.xml
• Stop multisite docker and start multidomain docker
• Open the robots.txt with and without the code snippet there should one be 1 entry Sitemap: http://multisitedomain.wordpress.test/sitemap_index.xml
• Stop multisite docker and start basic docker
• Open the robots.txt with and without the code snippet there should one be 1 entry Sitemap: http://basic.wordpress.test/sitemap_index.xml

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Block/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

QA can test this PR by following these steps:

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change.

Quality assurance

• I have tested this code to the best of my abilities
• I have added unit tests to verify the code works as intended
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.

Fixes #
PC-393