[ASCII-1218] Only initialize needed configs for serverless#23006
Conversation
pgimalac
added
changelog/no-changelog
Bloop Bleep... Dogbot HereRegression Detector ResultsRun ID: b4dda9eb-4689-4aea-9a4a-3ca2239a3254 Performance changes are noted in the perf column of each table:
Experiments with missing or malformed data
Usually, this warning means that there is no usable optimization goal data for that experiment, which could be a result of misconfiguration. No significant changes in experiment optimization goalsConfidence level: 90.00% There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
|
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | file_to_blackhole | % cpu utilization | -1.73 | [-8.21, +4.75] |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | process_agent_standard_check_with_stats | memory utilization | +0.77 | [+0.74, +0.81] |
| ➖ | file_tree | memory utilization | +0.75 | [+0.65, +0.85] |
| ➖ | process_agent_standard_check | memory utilization | +0.48 | [+0.42, +0.53] |
| ➖ | idle | memory utilization | +0.45 | [+0.40, +0.49] |
| ➖ | process_agent_real_time_mode | memory utilization | +0.15 | [+0.12, +0.18] |
| ➖ | trace_agent_msgpack | ingress throughput | +0.02 | [+0.01, +0.02] |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.00, +0.00] |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.00, +0.00] |
| ➖ | trace_agent_json | ingress throughput | -0.00 | [-0.04, +0.03] |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.21 | [-1.63, +1.21] |
| ➖ | otel_to_otel_logs | ingress throughput | -0.32 | [-0.95, +0.30] |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.51 | [-0.56, -0.45] |
| ➖ | file_to_blackhole | % cpu utilization | -1.73 | [-8.21, +4.75] |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
ce234f6 to
b75de07
Compare
b75de07 to
8ce8e2f
Compare
pgimalac
changed the title
8ce8e2f to
a3fcc61
Compare
a3fcc61 to
f72c641
Compare
05a5cfc to
8b95277
Compare
Regression DetectorRegression Detector ResultsRun ID: 4bbf3c4b-e892-46ab-8266-905a0b57ef10 Performance changes are noted in the perf column of each table:
No significant changes in experiment optimization goalsConfidence level: 90.00% There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
|
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ✅ | file_to_blackhole | % cpu utilization | -40.85 | [-45.59, -36.12] |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI |
|---|---|---|---|---|
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +3.44 | [-18.01, +24.88] |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +1.15 | [-1.71, +4.02] |
| ➖ | basic_py_check | % cpu utilization | +0.33 | [-2.20, +2.86] |
| ➖ | process_agent_standard_check_with_stats | memory utilization | +0.21 | [+0.16, +0.27] |
| ➖ | idle | memory utilization | +0.11 | [+0.08, +0.15] |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.02 | [-0.03, +0.06] |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.20, +0.20] |
| ➖ | trace_agent_json | ingress throughput | -0.00 | [-0.01, +0.01] |
| ➖ | trace_agent_msgpack | ingress throughput | -0.00 | [-0.00, +0.00] |
| ➖ | process_agent_standard_check | memory utilization | -0.01 | [-0.07, +0.05] |
| ➖ | process_agent_real_time_mode | memory utilization | -0.02 | [-0.06, +0.01] |
| ➖ | otel_to_otel_logs | ingress throughput | -0.17 | [-0.57, +0.22] |
| ➖ | file_tree | memory utilization | -0.80 | [-0.90, -0.71] |
| ➖ | pycheck_1000_100byte_tags | % cpu utilization | -1.10 | [-5.86, +3.66] |
| ✅ | file_to_blackhole | % cpu utilization | -40.85 | [-45.59, -36.12] |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
…/serverless-trim-config
…/serverless-trim-config
|
/merge |
|
🚂 MergeQueue Pull request added to the queue. There are 2 builds ahead! (estimated merge in less than 1h) Use |
|
/merge |
|
🚂 MergeQueue This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals. Use |
|
🚂 MergeQueue Pull request added to the queue. There are 2 builds ahead! (estimated merge in less than 1h) Use |
4 participants
What does this PR do?
Uses build tags to avoid initializing all config values when building for serverless.
Motivation
Init duration is important for serverless, the
setuppackage takes from 10 to 20ms to perform its init, performing around 24000 allocations.Removing most of the configs (which aren't used for Serverless) drops it down to around 2.5ms and 8000 allocations (and even that is still way more than actually needed).
You can run the lambda extension with
GODEBUG=inittrace=1to check how long it takes (look at theclockduration).Before (we've seen durations between 10 and 20ms for that single package):
After:
Additional Notes
Review commit by commit if you want to make sure no config was removed or added in the process.
I considered that if any config from a "group" was reachable by the serverless agent then all configs of that group should be declared. Future work will refine this to remove some groups entirely from the serverless agent config setup.
I assumed people add new config options next to other ones they are related to, and so moving all related configs in a separate config ensures new related configs will be added in the proper place.
All configs are still declared in the
setup/config.gofile, so you don't have to look at different files.I enforced the number of serverless config components so that we can explicitly avoid adding entire new config group by accident. Adding a single config in eg.
dogstatsdis fine and shouldn't be prevented, but adding an entire group on the in the serverless agent needs to be inspected and validated properly.Possible Drawbacks / Trade-offs
We now don't declare all configs for all agents, so we can use a config that should be declared but isn't.
It could be interesting to warn when using an unknown config ?
Describe how to test/QA your changes
Check that the serverless agent still works and isn't using any unset config.