-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Refactor container stats collection #7469
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Hi @danielye11. Thanks for your PR. I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
pkg/cri/store/container/container.go
Outdated
|
|
||
| c := s.containers[id] | ||
| c.Stats = newContainerStats | ||
| if c.Stats != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need to check if c.Stats is nil here? Shouldn't it be initialized previously?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this checks for the case where old stats is nil here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also get a memory panic if I take it out
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, where do we actually assign c.Stats? Is it ever assigned?
As I see we are only returning *containerstorestats.ContainerStats in generatedContainerMetrics, but we are not actually storing the stats on the container object? Is that right? If that's the case, then c.Stats will always be nil...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, good point -- actual caching of the metrics should be fixed now.
|
CI is failing as all of your commits are missing valid |
|
Your commit messages and PR description don't really provide much context for the changes you'd like to contribute to containerd. Can you describe more of what you're trying to achieve? Is there an associated issue for this PR with more information? |
Updated description of PR |
|
Left a few small comments, but looking really good. Maybe time to move this out of draft? |
8459d3f to
1eda6ec
Compare
362cfa2 to
b330dc0
Compare
650ce29 to
45f4d5f
Compare
613cbd1 to
786b541
Compare
Change how ListPodSandboxStats and ListContainerStats retrieves metrics. Previously, metrics were fetched from cgroup and returned directly via a runtime struct (ListContainerStats). With the need of Prometheus exporting, we will now collect these metrics and cache them, fetching metrics from the cache and exporting them to whichever metric type we need. Cache the metrics added in stats object of container and sandbox and revamp getUsageNanoCores to not cache metrics inside that function. Add and update tests based on changes. Signed-off-by: Daniel Ye <[email protected]>
c3ec795 to
ee774ed
Compare
samuelkarp
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@danielye11, thanks for opening this PR! I took a first pass at it and left a few comments, but also I have two more overall comments:
- I think I'm missing the caching and expiration/eviction behavior here. I do see where stats are stored (in the
container.Store), but it looks like they're never updated and also never read. - It would be helpful for me as a reviewer to break this up into at least two commits: one where the structural changes (types, refactoring) are done and a second with the functional changes (new data + caching behavior).
| ContainerCPUStats | ||
| ContainerMemoryStats | ||
| ContainerFileSystemStats | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the rationale for struct embedding here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All stats whose additional metrics will eventually be returned via Prometheus format. Rationale would be to add the additional caching logic in the future, and since stats and metrics will be set in the same place (reading the same linux cgroup file) we can read from the container stats store or sandbox stats store.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe my question wasn't clear. What's the reason for embedding versus using fields?
// embedding
type ContainerStats struct {
ContainerCPUStats
ContainerMemoryStats
ContainerFileSystemStats
}
// fields
type ContainerStats struct {
CPU ContainerCPUStats
Memory ContainerMemoryStats
FileSystem ContainerFileSystemStats
}| // Timestamp in nanoseconds at which the information were collected. Must be > 0. | ||
| Timestamp int64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each of these three structs (ContainerCPUStats, ContainerMemoryStats, and ContainerFileSystemStats contain a Timestamp field. Since all three are then embedded into ContainerStats, only one of the three Timestamp fields will be visible as ContainerStats.Timestamp; the other two will be shadowed. This can be a source of confusing behavior; if we do need to have this embedding pattern, please move Timestamp out of the individual structs and up into ContainerStats directly.
| Timestamp int64 | ||
| // Cumulative CPU usage (sum across all cores) since object creation. | ||
| UsageCoreNanoSeconds uint64 | ||
| // Total CPU usage (sum of all cores) averaged over the sample window. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the sample window?
|
|
||
| type ContainerCPUStats struct { | ||
| // Timestamp in nanoseconds at which the information were collected. Must be > 0. | ||
| Timestamp int64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Existing code using the ContainerStats.Timestamp field are seeing a time.Time. This code changes it to an int64. Is there a specific reason why? Generally time.Time would be preferred as a standard type for representing time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CRI fields use int64, figured it would be easier than converting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is an internal struct that needs to be converted on the way out anyway, I think it'd make more sense to centralize the conversion and maintain the standard time.Time here.
| WorkingSetBytes uint64 | ||
| // Available memory for use. This is defined as the memory limit - workingSetBytes. | ||
| AvailableBytes uint64 | ||
| // Total memory in use. This includes all memory regardless of when it was accessed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I can parse "when it was accessed". Does this mean you're tracking the total number of bytes over the lifetime of a container (so maybe you mean the max usage?).
| case *v1.Metrics: | ||
| if metrics.Memory != nil && metrics.Memory.Usage != nil { | ||
| workingSetBytes := getWorkingSet(metrics.Memory) | ||
| func (c *criService) setContainerStats(containerID string, isSandbox bool, cs *containerstorestats.ContainerStats) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function appears to do nothing if there is already a stats.ContainerStats attached to the container or sandbox. Is that intentional?
| } | ||
|
|
||
| cpuStats, err := c.cpuContainerStats(meta.ID, false /* isSandbox */, s, protobuf.FromTimestamp(stats.Timestamp)) | ||
| c.setContainerStats(meta.ID, false, &cs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's odd to store a pointer to a struct, and then continue to mutate the struct (lines 115 and 121). It looks like this breaks the intent of the mutex in the sandbox.Store and container.Store, opening up the possibility of a data race.
| if isSandbox { | ||
| sandbox, err := c.sandboxStore.Get(containerID) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to get sandbox container: %s: %w", containerID, err) | ||
| } | ||
| case *v2.Metrics: | ||
| if metrics.Memory != nil { | ||
| workingSetBytes := getWorkingSetV2(metrics.Memory) | ||
| oldStats = sandbox.Stats |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't look like setContainerStats is ever called with isSandbox set to true. Can you remove the parameter and the codepath?
| } | ||
|
|
||
| func (c *criService) cpuContainerStats(ID string, isSandbox bool, stats interface{}, timestamp time.Time) (*runtime.CpuUsage, error) { | ||
| func (c *criService) generatedCPUContainerStats(ID string, isSandbox bool, stats interface{}, timestamp time.Time) (*containerstorestats.ContainerCPUStats, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ID and isSandbox parameters appear unused. Can you remove them?
| return &cs, nil | ||
| } | ||
|
|
||
| func (c *criService) generatedMemoryContainerStats(ID string, stats interface{}, timestamp time.Time) (*containerstorestats.ContainerMemoryStats, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ID parameter appears unused. Can you remove it?
Thanks for review, will address comments and split up PR. Just some more context, but for this PR we don't specifically need more advanced caching logic. There's not a big regression for adding more structs in container store and it will help a lot for adding prometheus exporting for extra pod and container metrics (which will be put in container stats and sandbox stats store). |
|
This PR is stale because it has been open 90 days with no activity. This PR will be closed in 7 days unless new comments are made or the stale label is removed. |
|
This PR was closed because it has been stalled for 7 days with no activity. |
Update the logic for container stats collection, storing the stats on the container object now. This is in preparation for the CRI changes described in KEP-2371, which adds a new gRPC call that returns unstructured Prometheus metrics. These will not be returned via a runtime struct, and instead will be added to stats. The additional unstructured metrics described in the above KEP will also be modified and retrieved from the container stats cache.
Directly tied to this issue