Skip to content

Log all the handlers for a stage before they start / stop.#18662

Merged
abhishekrb19 merged 2 commits into
masterfrom
more_lifecycle_logging
Oct 21, 2025
Merged

Log all the handlers for a stage before they start / stop.#18662
abhishekrb19 merged 2 commits into
masterfrom
more_lifecycle_logging

Conversation

@abhishekrb19

@abhishekrb19 abhishekrb19 commented Oct 18, 2025

Copy link
Copy Markdown
Contributor

This logs all handlers during the start and stop lifecycle stages. It’s useful for understanding the execution order ahead of time, especially to identify which handlers didn’t run when a service stops. For example, some handlers like SupervisorManager.stop() can take a while to stop, blocking other subsequent handlers from stopping if a certain timeout is exceeded.

A sample of the start lifecycle logs:

2025-10-18T00:39:38,019 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [NORMAL] for handlers [[LoggingEmitter, ServiceEmitter, org.apache.druid.curator.CuratorModule$1@3e89f5bc, org.apache.druid.curator.discovery.DiscoveryModule$5@259aec72, DerbyConnector, org.apache.druid.guice.JacksonConfigManagerModule$1@36d39655, ConfigManager, SQLAuditManager, JvmMonitor, MetadataTaskStorage, IndexerSQLMetadataStorageCoordinator, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@26be0d23, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@576b385d, SegmentAllocationQueue, SQLMetadataSupervisorManager, org.apache.druid.java.util.http.client.HttpClientInit$1@2560a7be, org.apache.druid.java.util.http.client.HttpClientInit$2@fabfc79, NettyHttpClient, CuratorDruidNodeDiscoveryProvider, TaskMaster, org.apache.druid.metadata.SqlSegmentsMetadataManagerProvider$1@59aa2fba, SqlSegmentsMetadataManagerV2, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@578d5d02, OverlordCompactionScheduler, DiscoveryServiceLocator, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@77f04d1a, DruidOverlord, BasicMonitorScheduler, HdfsStorageAuthentication, HdfsStorageAvailabilityChecker, org.apache.druid.java.util.http.client.HttpClientInit$1@27682fa9, org.apache.druid.java.util.http.client.HttpClientInit$2@2fb70301, NettyHttpClient, HttpServerInventoryView, CoordinatorServerView, org.apache.druid.metadata.SQLMetadataRuleManagerProvider$1@126f2eb8, SQLMetadataRuleManager, DiscoveryServiceLocator, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@67823972, org.apache.druid.java.util.common.concurrent.ExecutorServices$1@1f1574c, CoordinatorDynamicConfigSyncer, DruidCoordinator]]
2025-10-18T00:39:38,023 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Starting lifecycle [LoggingEmitter#start]

A sample of the stop lifecycle logs:

2025-10-18T00:41:46,863 INFO [Thread-57] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [task-master] stage [NORMAL] for handlers [[org.apache.druid.indexing.overlord.DruidOverlord$1$2@111b1ea8, OverlordDutyExecutor, SupervisorManager, org.apache.druid.indexing.overlord.DruidOverlord$1$1@388e6093, TaskQueue, HttpRemoteTaskRunner]]
2025-10-18T00:41:46,871 INFO [Thread-57] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Stopping lifecyle [OverlordDutyExecutor#stop].
2025-10-18T00:41:46,871 INFO [Thread-57] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Stopping lifecyle [SupervisorManager#stop].
2025-10-18T00:41:46,872 INFO [Thread-57] org.apache.druid.indexing.overlord.supervisor.SupervisorManager - Stopping [6] supervisors
2025-10-18T00:41:46,877 INFO [Thread-57] org.apache.druid.indexing.overlord.supervisor.SupervisorManager - Waiting for [6] supervisors to shutdown

This PR has:

  • been self-reviewed.
  • been tested in a test Druid cluster.

This logs all the handlers at the start/stop lifecycle stage.
This is useful to know the order ahead of time, especially when a service
stops, some handlers don't execute.
@jtuglu1
jtuglu1 self-requested a review October 18, 2025 01:20

@jtuglu1 jtuglu1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending CI, minor comments.

}
catch (Exception e) {
log.warn(e, "Lifecycle [%s] encountered exception while stopping %s", name, handler);
log.warn(e, "Lifecycle [%s] encountered exception while stopping [%s]", name, handler);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: stopping handler [%s] to match other logs.

@Override
public String toString()
{
return o.getClass().getSimpleName();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are duplicate handlers with same simpleName, do we care about differentiating them?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth differentiating them if the handlers are ever invoked concurrently, in that case, we could include the object's hash code and update the AnnotationBasedHandler's start() and stop() methods too to include that info for individual handlers. Since they’re executed sequentially, it doesn’t matter much for now.

@abhishekrb19

Copy link
Copy Markdown
Contributor Author

Thanks for the review @jtuglu1!

@abhishekrb19
abhishekrb19 merged commit c04e2b8 into master Oct 21, 2025
50 of 51 checks passed
@abhishekrb19
abhishekrb19 deleted the more_lifecycle_logging branch October 21, 2025 01:07
@kgyrtkirk kgyrtkirk added this to the 36.0.0 milestone Jan 19, 2026
riovic918data pushed a commit to riovic918data/druid that referenced this pull request Jun 12, 2026
)

Log all the handlers for a lifecycle stage before they start / stop.

This logs all the handlers at the start/stop lifecycle stage.
This is useful to know the order ahead of time, especially when a service
stops, some handlers don't execute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants