We instrument the ingestion processors so we can measure the duration of each processor:
https://github.com/stellar/go/blob/release-horizon-v2.28.0/services/horizon/internal/ingest/group_processors.go#L114
However, we do not have any metrics capturing how long it takes to run the ingestion loaders:
https://github.com/stellar/go/blob/release-horizon-v2.28.0/services/horizon/internal/ingest/group_processors.go#L101-L105
We should add prometheus metrics for the duration of each loader so we can determine how much of ingestion is spent in the loaders vs the processors.
[@sreuland suggested] - I think at the same time we should add a new metric key as a gauge type for captive_core_txmeta_ledger_retrieval_duration to track the per-ledger wait times that horizon performs while reading from the tx meta pipe.
We instrument the ingestion processors so we can measure the duration of each processor:
https://github.com/stellar/go/blob/release-horizon-v2.28.0/services/horizon/internal/ingest/group_processors.go#L114
However, we do not have any metrics capturing how long it takes to run the ingestion loaders:
https://github.com/stellar/go/blob/release-horizon-v2.28.0/services/horizon/internal/ingest/group_processors.go#L101-L105
We should add prometheus metrics for the duration of each loader so we can determine how much of ingestion is spent in the loaders vs the processors.
[@sreuland suggested] - I think at the same time we should add a new metric key as a gauge type for
captive_core_txmeta_ledger_retrieval_durationto track the per-ledger wait times that horizon performs while reading from the tx meta pipe.