Proposal

Problem:
Currently, users running Prometheus on Azure do not have an option to natively to send Prometheus metrics via remote write to an Azure Monitor workspace, which is used to store Prometheus metrics in Azure. Users are currently required to use a sidecar container as mentioned here, which acts as a reverse proxy taking care of the Azure Active Directory (Azure AD) authentication to the Azure Monitor workspace for ingesting Prometheus metrics.

Though the sidecar solves the ingestion authentication problem, it adds additional overhead for people to send Prometheus metrics via remote write to an Azure Monitor workspace. Hence, the goal of this proposal is to reduce the overhead and improve the remote write experience for people wanting to send Prometheus metrics to an Azure Monitor workspace.

Proposed solution:
An introduction of a new HTTP.RoundTripper that takes care of fetching/refreshing the access token from Azure AD and subsequently pushing Prometheus remote write metrics to the specified Azure Monitor workspace. The behavior of this new HTTP.RoundTripper is going to be similar to the sidecar container mentioned previously.

With the introduction of a new HTTP.RoundTripper, we would also require introduction of new fields in the Prometheus remote write configuration. Below is an excerpt of the new remote write configuration fields that would be required to retrieve appropriate values from the users for functioning of the new HTTP.RoundTripper proposed:

remote_write: 
  - url: http://localhost:1234/api/v1/write 
    azure_ad_auth: 
      - client_id: <User Assigned Managed Identity client ID> 

The proposed remote write configuration(azure_ad_auth), like other authentication options offered by Prometheus remote write, is going to be optional and mutually exclusive.

NOTE: We did try doing a POC using the oauth2 authentication support provided by Prometheus. However, the Azure Metadata Instance service endpoint, which is used to retrieve the access token does not fully support the OAuth2 flow. Hence, we are suggesting the method proposed above to provide authentication support for Azure in Prometheus remote write. Furthermore, the auth proposed is via Azure managed identity. This does not expose any secrets to the application/user and the secret is secured by the platform. Due to this, Azure managed identity cannot support client credential flow.