Proposal: HPA using external metrics #1801

MaciekPytel · 2018-02-14T23:01:50Z

This proposal describes an extension to the HPA v2 API that would allow autoscaling based on metrics coming from outside kubernetes cluster. This is not possible using existing API.

Common use-cases are autoscaling pods using a hosted service (ex. hosted message queue) or autoscaling in systems that have (so far) only partly moved to Kubernetes.

#1802 describes External Metrics API used internally by HPA to retrieve metrics.

/sig-autoscaling

MaciekPytel · 2018-02-14T23:03:19Z

@kubernetes/sig-autoscaling-proposals
@kubernetes/api-reviewers

@DirectXMan12 - we discussed this during sig-autoscaling, can you take a look?

DirectXMan12

couple of nits inline.

You need to discuss more concretely how this maps to the external metrics API proposed in #1802.

I like the targetAverageValue change. No matter what happens with the rest of this PR, I've become convinced that we should land that, because it helps with certain aspects of queue-based scaling (it's insufficient on its own, IMO, but together with a "traffic intensity" metric it should work).

That being said: this needs to have a more serious discussion of alternatives, because there are some. I'm not saying that this isn't the solution (it very well might be the easiest to work with), but we need documentation on what alternatives we considered, and why they were rejected.

I'm going to play the Devil's Advocate here for a moment.

What, fundamentally, prevents you from associating your metrics with some cluster-level object, or the namespace object itself? Couldn't we add the metricSelector field to the pods and object metrics (to allow further drilling down of a particular metric), and then say "if you have external services that produce metrics, use relabeling to attach metadata to those services so that they're associate with some object, such as the "default" namespace?

What are the downsides to that approach? What does this approach buy/lose us in terms of reducing user complexity and confusion? What does this approach buy/lose us in terms of operator complexity and confusion?

DirectXMan12 · 2018-02-16T16:11:46Z

contributors/design-proposals/autoscaling/hpa-external-metrics.md

+        ResourceMetricSourceType MetricSourceType = "Resource"
+        // ExternalMetricSourceType is a global metric coming from
+        // outside of Kubernetes. It allows autoscaling based on information
+        // coming from components running outside of cluster


this is a bit of a documentation issue, but I think "coming from outside the cluster" is the wrong term here. I think it's better to stress that this is for the case when the metrics aren't immediately associatable with objects inside the cluster.

For instance, I could run a DBaaS that runs outside the cluster, but still provide metrics to track usage per namespace, and thus not have to use the external type.

Good point. Updated all comments as per your suggestion.

DirectXMan12 · 2018-02-16T16:13:23Z

contributors/design-proposals/autoscaling/hpa-external-metrics.md

+        MetricSelector metav1.LabelSelector `json:"metricSelector" protobuf:"bytes,2,name=metricSelector"`
+
+        // CurrentValue is the current value of the metric (as a quantity)
+        CurrentValue resource.Quantity `json:"currentValue" protobuf:"bytes,3,name=currentValue"`


This also needs currentAverageValue *resource.Quantity as an optional field.

DirectXMan12 · 2018-02-16T16:29:22Z

holding due to needing some further some further consideration of the alternatives

MaciekPytel · 2018-02-20T16:31:32Z

@DirectXMan12 I've added a section discussing considered alternatives. Can you take a look?

This proposal details an extension to the HPA that would allow autoscaling based on metrics coming from outside kubernetes cluster.

DirectXMan12 · 2018-02-21T21:04:23Z

So, after a bit discussion with @MaciekPytel offline, I'm inclined to accept this proposal, with a caveat. I'm still hesitant about potential confusion on the part of end-users (when do I use external vs object), and not entirely convinced that the better solution isn't association in the metrics adapters, so I think that we'll need to make super certain to collect user feedback and re-evaluate next release how we want to move forward with these changes.

I believe we still need an @kubernetes/api-reviewers on the sibling proposal, so we should hold off merge until that gets the ok.

eghobo · 2018-02-22T04:23:31Z

@MaciekPytel sorry for stupid question (I honestly read full proposal ;)). But how this different from existing API (e.g. in our use-case we wrote api-adapter for external system and use ObjectMetricSourceType in HPA)?

mwielgus · 2018-02-22T15:51:26Z

@eghobo With ObjectMetricSource users have to bind their metrics to some kubernetes object (thus possibly modify their definition in the monitoring solution). With external metrics we allow to use unmodified metrics (like pubsub queue lenghth, external load balancer qps, etc) in HPA.

mwielgus · 2018-02-22T15:51:37Z

/lgtm

k8s-ci-robot · 2018-02-22T15:51:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MaciekPytel, mwielgus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~contributors/design-proposals/autoscaling/OWNERS~~ [mwielgus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

MaciekPytel · 2018-02-22T15:52:34Z

@eghobo
Short answer: If you can identify a metric you want to use using Object, than use it. Otherwise you need External.

The goal is to allow autoscaling based on metrics that have no inherent relationship to any k8s object. For example if your application use a hosted service and you want to autoscale based on its metrics, which contain no reference to your kubernetes object.

The design of Object metric (even the name :) ) is suited to use metrics inherently associated with some k8s object. Target field in ObjectMetricSource is useless for identifying metrics that have no such association. As discussed in Using Object metric source for external metrics section there are some workarounds that allow using Object for autoscaling such metrics. However, regardless which workaround you choose, at the end of the day you will need to explicitly define the metric you want to use (ie. specify the same set of information you give to External metric). You may write adapter so that using a specific Target gives you some arbitrary metric of course, but that's stretching the Object metric API. Also that means everyone will also need to somehow do that in their adapter. Instead the idea is to just create a standard way of accessing such metrics.

DirectXMan12 · 2018-02-22T16:20:31Z

/hold

for api review

mwielgus · 2018-02-22T18:24:55Z

#60096 is approved by API reviewer (@thockin).

@DirectXMan12 Can we unhold?

DirectXMan12 · 2018-02-22T19:21:40Z

/unhold

yep

DirectXMan12 · 2018-02-22T19:23:07Z

(although I kind-of wanted approval on the external metrics API, too)

eghobo · 2018-02-23T04:17:35Z

@MaciekPytel thx for details, we are using adapter to access external metrics now. Sorry, but I am very skeptical about standard way of accessing such metrics. E.g. we have three monitoring systems Graphite/JMX/Prometheus in place now and there is no way to find standard way for them without writing adapter ):

Automatic merge from submit-queue (batch tested with PRs 60208, 60084, 60183, 59713, 60096). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add external metric type to HPA API **What this PR does / why we need it**: Add external metric type to HPA API proposed in xref kubernetes/community#1801 **Release note**: ```release-note Allows HorizontalPodAutoscaler to use global metrics not associated with any Kubernetes object (for example metrics from a hoster service running outside of Kubernetes cluster). ```

Automatic merge from submit-queue (batch tested with PRs 60208, 60084, 60183, 59713, 60096). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add external metric type to HPA API **What this PR does / why we need it**: Add external metric type to HPA API proposed in xref kubernetes/community#1801 **Release note**: ```release-note Allows HorizontalPodAutoscaler to use global metrics not associated with any Kubernetes object (for example metrics from a hoster service running outside of Kubernetes cluster). ``` Kubernetes-commit: b22b7853459db50a2c0619faa398f9fa1e98228e

Proposal: HPA using external metrics

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 14, 2018

k8s-github-robot added kind/design Categorizes issue or PR as related to design. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. labels Feb 14, 2018

k8s-ci-robot requested review from DirectXMan12 and mwielgus February 14, 2018 23:01

MaciekPytel mentioned this pull request Feb 14, 2018

Proposal: External Metrics API #1802

Merged

DirectXMan12 suggested changes Feb 16, 2018

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 16, 2018

MaciekPytel force-pushed the hpa_external branch 2 times, most recently from 9bbb929 to 4e93010 Compare February 20, 2018 16:28

MaciekPytel mentioned this pull request Feb 20, 2018

Add external metric type to HPA API kubernetes/kubernetes#60096

Merged

Proposal: HPA using external metrics

b2f8d0b

This proposal details an extension to the HPA that would allow autoscaling based on metrics coming from outside kubernetes cluster.

MaciekPytel force-pushed the hpa_external branch from 4e93010 to b2f8d0b Compare February 21, 2018 12:13

k8s-ci-robot assigned mwielgus Feb 22, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 22, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 22, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 22, 2018

k8s-ci-robot merged commit a20dbc3 into kubernetes:master Feb 22, 2018

MadhavJivrajani pushed a commit to MadhavJivrajani/community that referenced this pull request Nov 30, 2021

Merge pull request kubernetes#1801 from MaciekPytel/hpa_external

5f6b764

Proposal: HPA using external metrics

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal: HPA using external metrics #1801

Proposal: HPA using external metrics #1801

MaciekPytel commented Feb 14, 2018 •

edited

Loading

MaciekPytel commented Feb 14, 2018

DirectXMan12 left a comment •

edited

Loading

DirectXMan12 Feb 16, 2018

MaciekPytel Feb 20, 2018

DirectXMan12 Feb 16, 2018

MaciekPytel Feb 20, 2018

DirectXMan12 commented Feb 16, 2018

MaciekPytel commented Feb 20, 2018

DirectXMan12 commented Feb 21, 2018 •

edited

Loading

eghobo commented Feb 22, 2018

mwielgus commented Feb 22, 2018

mwielgus commented Feb 22, 2018

k8s-ci-robot commented Feb 22, 2018

MaciekPytel commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

mwielgus commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

eghobo commented Feb 23, 2018

Proposal: HPA using external metrics #1801

Proposal: HPA using external metrics #1801

Conversation

MaciekPytel commented Feb 14, 2018 • edited Loading

MaciekPytel commented Feb 14, 2018

DirectXMan12 left a comment • edited Loading

Choose a reason for hiding this comment

DirectXMan12 Feb 16, 2018

Choose a reason for hiding this comment

MaciekPytel Feb 20, 2018

Choose a reason for hiding this comment

DirectXMan12 Feb 16, 2018

Choose a reason for hiding this comment

MaciekPytel Feb 20, 2018

Choose a reason for hiding this comment

DirectXMan12 commented Feb 16, 2018

MaciekPytel commented Feb 20, 2018

DirectXMan12 commented Feb 21, 2018 • edited Loading

eghobo commented Feb 22, 2018

mwielgus commented Feb 22, 2018

mwielgus commented Feb 22, 2018

k8s-ci-robot commented Feb 22, 2018

MaciekPytel commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

mwielgus commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

DirectXMan12 commented Feb 22, 2018

eghobo commented Feb 23, 2018

MaciekPytel commented Feb 14, 2018 •

edited

Loading

DirectXMan12 left a comment •

edited

Loading

DirectXMan12 commented Feb 21, 2018 •

edited

Loading