The Apache Sling Sitemap module is an extension for Apache Sling that helps to serve XML Sitemaps to Search Engines. It was designed to cover various uses cases from small sites serving sitemaps on-demand, large sites generating them in the background to even sites that collect 3rd party data to include dynamically rendered pages.
- A simple, builder-like API to create Sitemaps, that hides all the XML specifics from the implementation
- Support for on-demand and background generation w/ continuation after job interruption
- Support for nested sitemaps, that are automatically collected into a sitemap indexes
- Support for auto-balancing sitemaps into multiple files for background generation
The Sling Sitemap module is rather abstract, as it highly depends on the content structure of the application it is used in. To get started a few things must be done:
- Implement at least one
SitemapGenerator. The abstractResourceTreeSitemapGeneratormay be a good starting point for any generator walking the resource tree. - Configure the
SitemapServletto register the resource type(s) which may be a sitemap root resource. - Configure a service user mapping for
org.apache.sling.sitemap:sitemap-readergranting read access to the content. - Configure another service user mapping one for
org.apache.sling.sitemap:sitemap-writergranting read access to the content and write access to the storage path for background generation (per default /var/sitemaps). - Finally, either configure a
SitemapSchedulerto create a job for background generation, or implement theSitemapGeneratorto serve the sitemaps on-demand.
In order to serve a sitemap a resource must be marked as sitemap root resource. This is done by adding
a sling:sitemapRoot = true property either to the resource or its jcr:content child.
When multiple resources in a resource tree are marked as sitemap roots, the one closest to the repository root is considered to top level sitemap root and serves a sitemap-index additionally to the sitemap.
/content/site/ch/
+ de-ch/
- sitemapRoot = true
+ faqs/
+ news/
+ products/
- sitemapRoot = true
+ fr-ch/
- sitemapRoot = true
...
In the example above the paths de-ch/ and fr-ch/ are both top level sitemap roots and de-ch/products/ is a nested
sitemap root. The sitemaps will be served as following, assuming the appropriate mappings being in place:
https://site.ch/de-ch.sitemap-index.xml
https://site.ch/de-ch.sitemap.xml
https://site.ch/de-ch.sitemap.products-sitemap.xml
https://site.ch/fr-ch.sitemap-index.xml
https://site.ch/fr-ch.sitemap.xml
The module does not ship a specific SitemapGenerator implementation. Products/Projects using the Apache Sling Sitemap
module must implement an appropriate SitemapGenerator that fits their content model. An abstract
ResourcceTreeSitemapGenator implementation is available to cover the most common use cases.
Each SitemapGenerator may produce multiple sitemaps for a given sitemap root. For example, a default sitemap and a news
specific sitemap that contains only up to 1000 urls that were changed in the past 2 days. Or as another example,
consider an eCommerce site, that generates a product sitemap for each top level category of a catalog. To enable that,
a SitemapGenerator can return 0..n names for a given resource, each name representing a single sitemap at the given
resource.
SitemapGenerator implementations need to be registered as OSGI services. In case of an overlap, it depends on
the service.ranking which SitemapGenerators are used for a particular sitemap root. For example, it may be that two
SitemapGenerators can generate a default sitemap for a sitemap root, but the second one can also generate a news
sitemap. The first, higher-ranked SitemapGenerator will be used for the default sitemap, but the second one will still
be taken into account for the news sitemap.
A configurable scheduler can be configured to trigger background generation. Multiple schedules can be created for different sitemap names or generators as described before. This enables the background generation to align on the cadence in which particular content may be updated, for example product catalogues that sync on a regular basis.
For each sitemap root in the repository and for each sitemap name returned for them, a job with the topic
org/apache/sling/sitemap/build will be queued. The SitemapGeneratorExecutor implementation consumes those jobs and
calls the corresponding SitemapGeneator. It is recommended to create an unordered queue for those jobs so that they
can be distributed across multiple instances within a cluster.
The SitemapGeneratorExecutor provides an execution context to the SitemapGenerator which it may use to keep track of
progress. The implementation on the other hand will persist this state along with the already written sitemap after
a configurable amount of urls has been added. This allows to resume jobs after an instance gets restarted or discarded
in a dynamic cluster. Per default the SitemapGeneratorExecutor is configured with a chunk size of Integer.MAX_VALUE,
which effectively means that no checkpoints will be written. When using this feature make sure to find a good balance
between write overhead and performance gain for those particular cases.
Background generation supports auto-balancing according to configurable limits for size (in bytes), and the number of
urls in a single sitemap file. This is transparently handled by the SitemapGeneratorExecutor, providing a Sitemap
instance which pipes added urls to multiple files when needed. As a consequence returning sitemap files from storage for
a given name and sitemap root may result in multiple return values.
For smaller sites, calculating sitemaps in the background may not be necessary and serving sitemaps when they get
requested may even result in higher accuracy. On the other hand serving a sitemap on-demand within the timeout of
different crawlers highly depends on the amount of content and the SitemapGenerator implementation(s) used. Because of
that, serving sitemaps on-demand must be explicitly enabled.
To enable serving sitemaps on-demand, a SitemapGenerator must indicate that a particular sitemap name should be served
on demand. Alternatively the SitemapGeneratorManagerImpl can be configured to force all sitemaps to be served
on-demand. In both cases the SitemapServlet changes its behaviour slightly:
- When serving a sitemap-index, it queries for all sitemap roots and adds the sitemaps of those, that should be served on-demand. Additionally, all sitemaps form the top level sitemap root's storage location are added if not already present.
- For serving sitemap files it first checks if the sitemap should be served on-demand, if not it falls back to the sitemap file from the top level sitemap root's storage location.
On-demand generation does NOT support auto-balancing. The configured limits will be ignored.
The builder-like API supports adding sitemap extensions (image, video, news, alternate languages ...) by implementing an
ExtensionProvider. This provider has to be registered specifying namespace, prefix, and localName of the xml
element the extension adds to an url object.
In order to hide the implementation detail from the consumer API, the ExtensionProvider works with an abstract
AbstractExtension class, and the consumer API only with an Extension marker interface. To implement an Extension:
- Extend the marker interface with methods to capture the extension specific data
- Implement this extension interface by extending the
AbstractExtensionclass - Provide an instance of the extension implementation by implementing an
ExtensionProvider - Make sure to register the
ExtensionProviderwith theextension.interfaceset to the fqn of the extension interface
An example extension implementation can be found with the AlternateLanguageExtension.
For the following implemented extensions, refer to the respective interfaces in o.a.s.sitemap.builder.extensions: