Implement Consumable Capacity and fine-grained CPU Allocation on the DRA driver by pravk03 · Pull Request #16 · kubernetes-sigs/dra-driver-cpu

pravk03 · 2025-08-21T20:28:35Z

This PR enables modeling CPU resources as a consumable capacity. The behavior is controlled by the flag (cpu-device-mode) and setting it to individual will revert the behavior to the current model where every CPU is exposed as a device in the ResourceClaim.

POC

Design Details and Rationale:

The PR introduces a cpu-device-mode flag. "Grouped" mode treats CPUs as consumable capacity to improve scalability by reducing API objects, while "individual" mode retains the current behavior of exposing every CPU as a device.
Reusing Kubelet Logic: The implementation imports CPU assignment logic directly from the Kubelet (cpumanager). This is a practical short-term solution which provides a mature, well-tested starting point and avoids reinventing the wheel.
Maintenance Strategy : The copied kubelet code is treated as an implementation detail that is subject to change, rather than a permanent architectural commitment. The long-term strategy (periodic sync/ fork) is still under discussion. Until then, any current deviations from the Kubelet code are documented.

pravk03 · 2025-08-21T20:37:25Z

@johnbelamaric @ffromani This is currently a prototype and this PR is not ready for review yet. I wanted to get feedback on the idea of using consumable capacity along with reusing cpu allocation logic from Kubelet to make the fine grained cpu assignment decisions on the Node (DRA driver) instead of the scheduler. This should also address #5.

I have captured more details in this doc.

pravk03 · 2025-08-21T20:41:15Z

/cc @catblade @dchen1107

k8s-ci-robot · 2025-08-21T20:41:18Z

@pravk03: GitHub didn't allow me to request PR reviews from the following users: catblade, dchen1107.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @catblade @dchen1107

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pravk03 · 2025-08-21T20:56:14Z

/assign @pravk03
/reviewers add @ffromani

ffromani

super quick and very partial review because I'm going to be OOO next week till Sep 1.

I'm super interested in the consumable capacity approach. I'm admittedly catching up with DRA ecosystem (and CDI) but unless @johnbelamaric / @klueska which have a much, much more deep understanding raise concerns about the general direction, this should be the way to go.
historically, cpumanager suffered some times quite a lot from the fact that it had to work with the limited and unflexible hardware representation cadvisor offered, which was a pain to extend and was a poor fit to model modern, complex CPU topologies. The cpu assignment logic also indirectly suffers from these limitations. IMO, it's ok to start replicating/copying cpumanager logic to quickly catch up, but we should treat it as a. implementation detail subject to change b. only the first bootstrap step. This should be made abundandtly clear in docs, and probably the code layout should also reflect that (move cpuinfo and cpuassignment and possibly driver in internal/?)

ffromani · 2025-08-22T09:46:56Z

 func init() {
 	flag.StringVar(&kubeconfig, "kubeconfig", "", "absolute path to the kubeconfig file")
 	flag.StringVar(&hostnameOverride, "hostname-override", "", "If non-empty, will be used as the name of the Node that kube-network-policies is running on. If unset, the node name is assumed to be the same as the node's hostname.")
+	flag.StringVar(&cpuDeviceMode, "cpu-device-mode", "grouped", "Sets the mode for exposing CPU devices. 'grouped' exposes a single device for a group of CPUs (currently grouping by socket). 'individual' exposes each CPU as a separate device.")


Let's evaluate if we want to keep the old method at all. I fail to see the advantages of the old method, besides the fact the new method depends on less mature APIs.

I think there is still value in the the "individual" mode. It can expose much more granular details (eg: P-cores and E-cores) which allows the workload to explicitly request in the ResourceClaim. With consumable capacity (currently) this level of detail is abstracted away within a group (like socket in this implementation).

Long term: The ideal solution could be a combination of consumable capacity and partitionable devices (kubernetes/enhancements#4815) to express hierarchical topology information. For instance, we could partition a node or socket into smaller, potentially overlapping sets based on attributes like core type or L3 cache. A workload could then request CPUs from a specific partition (e.g., "only E-cores"), and the DRA driver would allocate from that partition while also tracking consumption at the group level. FWIU, partitionable devices may not be fully compatible with consumable capacity at present, but it's a direction we could explore to achieve both fine-grained resource selection and scalability.

Yes, that's a really interesting option. I have a couple other use cases where I may want to use consumable capacity and partitionable devices together. The key API I think we need here to connect the two is a way to specify how to translate consumable capacity consumption into consumption from both that user-facing entry AND some counter sets. Alternatively, we may want to allow the base functionality of partitionable - counter sets - to be utilized without pre-enumeration of devices via a similar mapping. This essentially would allow us to encode the "rules" for how to generate partitions without having to enumerate them all. When there can be 1000s of partitions, this can be necessary. I want to explore this more and would love more details on this use case.

An alternative to using partitionable would be to use aggregation on the claim side. So, your claim would say "I need 15 CPUs from any number of sockets", as opposed to using AllocationMode: ExactCount, which would require the user know ahead of time the CPU/socket topology. We have talked about that as a feature on the GPU use case before ("I can take anywhere from 1-4 GPUs, so long as aggregate memory is at least 80Gi").

ffromani · 2025-08-22T09:48:00Z

 	flag.StringVar(&kubeconfig, "kubeconfig", "", "absolute path to the kubeconfig file")
 	flag.StringVar(&hostnameOverride, "hostname-override", "", "If non-empty, will be used as the name of the Node that kube-network-policies is running on. If unset, the node name is assumed to be the same as the node's hostname.")
+	flag.StringVar(&cpuDeviceMode, "cpu-device-mode", "grouped", "Sets the mode for exposing CPU devices. 'grouped' exposes a single device for a group of CPUs (currently grouping by socket). 'individual' exposes each CPU as a separate device.")
+	// TODO: We can add a --cpu-grouping-strategy flag to specify how the CPUs should be grouped (e.g., 'socket', 'l3cache', 'numa').


There always was a strong push to have this policy per-workload vs per-node with cpumanager and topology manager, so I'd be really wary of adding this option. We should IMO push harder for a cleaner/more flexible solution in the long run.

johnbelamaric · 2025-08-25T21:11:00Z

cc @sunya-ch

catblade · 2025-09-28T10:34:19Z

Hi, sorry I've been out on leave. Still out on leave until October 20, but if I'm reading this correctly, this has us back to talking as a CPU is a CPU with no distinguishing between types and locations.

For HPC and for other high performance workloads, schedulers like Slurm schedule according to position. I truly understand the desire to push this decision into the node, but part of the point of this work is to be able to handle multi node workloads where placement absolutely will matter..

Additionally, part of the reason we couldn't do blocks of cpus is sometimes heterogenous cpus are not in an easily predictable location.

It might be useful to have this as an option, however, as if you're not using that type of workload I'm the cluster, it may not be useful to have the scheduler work harder for placement.

pravk03 · 2025-09-30T18:29:16Z

@catblade Thanks for the feedback. I agree that for HPC and other high-performance workloads, topology-aware scheduling is critical, and we need to preserve the ability to distinguish between CPU types and their locations. The goal of this PR is not to replace the existing model but to offer a more scalable alternative for certain use cases. This PR introduces a --cpu-device-mode flag that allows the operator to choose between two modes individual and grouped.

The primary motivation for the grouped mode is to address the scalability limitations of the current approach, which can create a large number of API objects on nodes with many CPUs and increase scheduling latency. I see the grouped mode as a good solution for use cases where you need alignment between different DRA drivers or for workloads that are not as sensitive to topology, and where managing a large number of individual CPU devices is not necessary.

In the future, I believe that integrating consumable capacity with partitionable devices could allow for hierarchical topology definition and resource requests from specific partitions, achieving both scalability and the granular control.

catblade · 2025-10-03T02:58:50Z

@pravk03 I understand. And we'll have to be clever eventually even in grouped mode because number of cores can exceed the 256 limit per partition, even today. Maybe we can come up with something clever in a brainstorm.

ffromani · 2025-10-14T13:29:45Z

I'm obviously in favor of supporting the HPC usecase. I'm still trying to reconcile the declarative model which should be preserved with the benefits of exposing granular CPU and, well, allow workloads to pick the exact CPU they wish to run on. Where I can find again a document detailing why we have such strict and precise requirement? I probably already asked but I'm struggling to find it.

ffromani

initial review

ffromani · 2025-11-27T15:34:13Z

 	// L3CacheID is the L3 cache ID
 	L3CacheID int64 `json:"l3CacheID"`
+
+	// TODO: FIX
+	UncoreCacheID int `json:"uncoreCacheID"`


TODO is because we have a duplication, right?
I think we are at a junction:

we fork the cpumanager code: import and then start to adapt it to the driver needs. Then we can rename UncareCacheID to L3CacheID and move on

we want to import the cpumanager code with minimal or no changes. Then we should adapt the driver code and rename L3CacheID to UncoreCacheID.

For the time being, I don't see the shared code to be moving in a public package, one of the main reasons being the cpumanager code was never really made to be a library, so we need quite some review before to enable this.
Nevertheless, being able to update it with a trivial copy/paste seems desirable, so I think we should bite the bullet and rename L3CacheID to UncoreCacheID aligning to cpumanager naming.

Sidenote: I think the UncoreCache naming is actually slightly better than L3Cache in this specific context

Thanks for the suggestion and I agree with the above. Replacing L3CacheID with UncoreCacheID to maintain compatibility with code borrowed from kubelet codebase.

ffromani · 2025-11-27T15:46:45Z

+	// Process the last block of cpu info.
+	if len(cpuInfoLines) > 0 {
+		cpuInfo := parseCPUInfo(isHybrid, eCoreCpus, cpuInfoLines...)
+		if cpuInfo != nil {
+			cpuInfos = append(cpuInfos, *cpuInfo)
+		}
+	}


for the future (milestone 2 or so) we may want to try to use only sysfs and stop processing cpuinfo. This should lead to a simpler code and we should have all the informations we need, and then some.

ffromani · 2025-11-27T15:48:00Z

+func (d CPUDetails) KeepOnly(cpus cpuset.CPUSet) CPUDetails {
+	result := CPUDetails{}
+	for cpu, info := range d {


do we want to extract this code in a separate file, to make future imports from cpumanager codebase easier?

Makes sense. Moved these interface function (with cpu_assignment.go) to a separate file. Can further clean this up and use the same dir structure as kubelet code. Added a TODO for now and will take up all cleanup's in a followup PR.

ffromani · 2025-11-27T16:00:09Z

+			socketID := int64(i)
+			deviceName := fmt.Sprintf("%s%d", cpuDeviceSocketGroupedPrefix, i)
+			socketCPUSet := topo.CPUDetails.CPUsInSockets(i)
+			allocatableCPUs := socketCPUSet.Difference(cp.reservedCPUs)


This is important to make sure we use consistent IDs to make sure we account the reserved CPUs against the right pool. It may a source of nasty bugs later on if the kernel enumerates resources in a nontrivial way (yep, it actually happened: kubernetes/kubernetes#100145)

Updated to read the ID's from the topology directly.

ffromani · 2025-11-27T16:04:32Z

+			socketCPUSet := topo.CPUDetails.CPUsInSockets(i)
+			allocatableCPUs := socketCPUSet.Difference(cp.reservedCPUs)
+			availableCPUsInSocket := int64(allocatableCPUs.Size())
+			hyperThreadEnabled := topo.CPUsPerCore() > 1


this was a historical flow in the cpumanager code. We should fix this design issue and fetch the actual value from the system rather than try to guess.
We can check system-wide or per-core.
A system-wide check is already an improvement, and we need to read /sys/devices/system/cpu/smt/control

Added an explicit check based on devices/system/cpu/smt/control in cpuinfo package. I retained the fallback mechanism based on CPU and Core count if the file is not present. Do you think that is needed ?

ffromani · 2025-11-27T16:05:28Z

+				Attributes: map[resourceapi.QualifiedName]resourceapi.DeviceAttribute{
+					"dra.cpu/socketID":       {IntValue: &socketID},
+					"dra.cpu/numCPUs":        {IntValue: &availableCPUsInSocket},
+					"dra.cpu/hyperthreading": {BoolValue: &hyperThreadEnabled},


I'd use "smt" rather than the commercial name (hyperthreading).
I think we can and should expose the NUMANodeID anyway, can't we?
In general, shouldn't we expose the same attributes in both grouping modes?

renamed to dra.cpu/smtEnabled

Regading NUMANodeID, Since there could be mutiple NUMA domains in a socket, I decided not to expose it with socket grouping.

We could technically expose a list of NUMA IDs, but we currently lack primitives in ResourceClaim to match against a list.

ffromani

need to deep dive on e2e tests, but at glance look reasonnable.
It is not a blocker for this PR, but we need to start thinking about how to keep the test matrix as small as possible, because it is starting to become unwieldly.

swatisehgal · 2025-12-02T12:41:08Z

+// NOTE: This file is a copy of https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cpumanager/cpu_assignment.go
+// as of commit https://github.com/kubernetes/kubernetes/commit/fd5b2efa76e44c5ef523cd0711f5ed23eb7e6b1a with minor modifications
+


This is a a lot of code copied from Kubernetes core! How will this be maintained? What is the sync strategy going to be? Are there any divergences and have they been documented?

That’s a valid concern. We decided to start with the Kubelet code to leverage its maturity and existing test coverage. @ffromani and I briefly discussed about long term strategy, but it wasn't very clear at that point and definitely needs more discussion. We may not necessarily continue keeping in sync with kubelet code, as customizing the implementation to fit the DRA driver's specific use cases (and removing unused logic) could be more beneficial in the long term.

Currently, the divergences from kubelet code is minimal. I can add a comment in the code to document them.

In addition to what @pravk03 mentioned, the kubelet code is not ready nor designed to be consumed as library. I reckon the effort to make it so is not trivial, and is not clear yet if we should do that from kube side. The ecosystem is very fluid yet and until we stabilize, I think that our poor man's vendoring (borrowing files from kubelet) is actually the cheapest approach.
Surely this is a long term concern we need to address somehow.
I won't rule out that we end up rolling up our own independent allocator code.

Thanks both! This gives helpful context on why the kubelet code was brought in directly. I agree that starting from a mature, well-tested implementation is a sensible short-term choice, especially while the DRA ecosystem is still evolving.

At the same time, I think it would be good for us to define an explicit maintenance strategy so we don’t end up with silent drift or unexpected behavioural differences over time. Even something lightweight would help, for example:

documenting the initial divergence points and any future deviations as they happen

clarifying whether we plan to periodically re-sync with kubelet or intentionally fork and evolve independently

outlining the criteria that would signal it’s time to move away from borrowed kubelet code toward a standalone allocator

This doesn’t need to be finalised today, but having a small design note capturing the intended direction would give contributors and users better clarity and help avoid surprises when kubelet logic evolves.

Thanks, @swatisehgal. That makes sense. Updating the documentation is long overdue, so I will take that up in a follow-up PR after this one merges. I'll make sure to include the suggestions you outlined above.

pravk03 · 2025-12-03T07:51:53Z

Diff between kubelet code (commit kubernetes/kubernetes@fd5b2ef) and code in this repo -
cpu_assignment.go diff - https://gist.github.com/pravk03/e228285ab79dddfe46d0e38141dc7057
cpu_assignment_test.go diff - https://gist.github.com/pravk03/c3a358f638a957658a75df827bbffb16

ffromani

LGTM with minor comments, most notably the spurious file seems still around?
I'll have another final pass focusing on tests, then I think we can merge and possibly iterate a bit later to polish.

ffromani · 2025-12-04T09:49:10Z

+func (t *CPUTopology) CPUsPerCore() int {
+	if t.NumCores == 0 {
+		return 0
+	}
+	return t.NumCPUs / t.NumCores
+}
+
+func (t *CPUTopology) CPUsPerSocket() int {
+	if t.NumSockets == 0 {
+		return 0
+	}
+	return t.NumCPUs / t.NumSockets
+}
+
+func (t *CPUTopology) CPUsPerUncore() int {
+	if t.NumUncoreCache == 0 {
+		// Avoid division by zero. If there are no uncore caches, then
+		// no CPUs can be in one.
+		return 0
+	}
+	// Note: this is an approximation that assumes all uncore caches have the same number of CPUs.
+	return t.NumCPUs / t.NumUncoreCache
+}


I'm really torn here. On one hand we should not block here, but this is a really egregious oversimplification has been a subtle source of troubles in cpumanager. The easiest case to consider is a user offlining a set of CPUs asymmetrically, e.g. only on one NUMA node, or LLC complex. All the math goes haywire.

This is probably one of the best areas on which we should diverge from cpumanager code (which is pretty much unfixable at this point in time) and do better. I'm confident that this "do better" doesn't require a totalrewrite or a super complex representation.

I think the best way forward is be very aware of this fragility and plan improvements for the next milestones.

Makes sense. I added a todo and filed #35 for the followup work.

* pkg/cpumanager/cpu_assignment.go and tests are branched from kubelet [cpu_assignment.go](https://github.com/kubernetes/kubernetes/blob/4cf195304caa519be476b367267f6c656bce19f7/pkg/kubelet/cm/cpumanager/cpu_assignment.go) * The pkg/cpuinfo interface and data structures have been updated to provide the detailed topology information required by the imported CPU assignment logic, including methods for querying CPUs by NUMA, socket, and core. * Renamed NumaNode to NUMANodeID in CPUInfo for consistency.

ffromani · 2025-12-10T13:41:43Z

/approve
/lgtm

thanks for your good work @pravk03 . I think we have some more room for polishing, but that's better done once this PR is merged. So let's move on.

k8s-ci-robot · 2025-12-10T13:41:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ffromani, pravk03

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ffromani,pravk03]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ffromani · 2025-12-10T13:43:03Z

/hold

ffromani · 2025-12-10T13:43:34Z

/hold cancel

only the notification email had still the "WIP" prefix. We're good to merge.

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 21, 2025

k8s-ci-robot requested review from johnbelamaric and klueska August 21, 2025 20:28

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 21, 2025

k8s-ci-robot assigned pravk03 Aug 21, 2025

ffromani reviewed Aug 22, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 30, 2025

pravk03 force-pushed the consumable-capacity branch from 6c99e51 to bbe5e91 Compare October 11, 2025 00:52

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 11, 2025

pravk03 force-pushed the consumable-capacity branch 2 times, most recently from 08a6874 to 88e9a23 Compare October 11, 2025 01:16

pravk03 changed the title ~~[WIP] Implement Consumable Capacity and fine-grained CPU Allocation on the DRA driver~~ Implement Consumable Capacity and fine-grained CPU Allocation on the DRA driver Oct 14, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2025

pravk03 force-pushed the consumable-capacity branch from 88e9a23 to 78c9b37 Compare November 20, 2025 23:43

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 20, 2025

pravk03 force-pushed the consumable-capacity branch 4 times, most recently from dbd3676 to 969be76 Compare November 21, 2025 01:22

pacoxu mentioned this pull request Nov 25, 2025

A Blog aboud topology aware scheduling: smarter scheduling for AI workloads. From Device Plugin to DRA. pacoxu/AI-Infra#143

Closed

Copilot AI mentioned this pull request Nov 25, 2025

Add bilingual blog on topology-aware scheduling: Device Plugin to DRA pacoxu/AI-Infra#144

Merged

ffromani reviewed Nov 27, 2025

View reviewed changes

ffromani reviewed Nov 28, 2025

View reviewed changes

Comment thread pkg/driver/dra_hooks.go

ffromani reviewed Nov 28, 2025

View reviewed changes

Comment thread 1 Outdated

Comment thread Makefile

pravk03 force-pushed the consumable-capacity branch 2 times, most recently from a76eb94 to 2e686f6 Compare December 1, 2025 09:37

swatisehgal reviewed Dec 2, 2025

View reviewed changes

ffromani mentioned this pull request Dec 3, 2025

PLANNING: Release 0.1.0 #34

Closed

ffromani reviewed Dec 4, 2025

View reviewed changes

pravk03 added 4 commits December 4, 2025 13:43

Add support for consumable capacity and CPU assignment.

5c43e93

Add E2E test for grouped mode

930677a

Add explicit SMT check via sysfs in cpuinfo package

3aff81c

pravk03 force-pushed the consumable-capacity branch from 1ef2eb8 to 3aff81c Compare December 4, 2025 14:01

pravk03 mentioned this pull request Dec 4, 2025

Refactor CPU Topology Representation for Asymmetric/Heterogeneous Systems #35

Open

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 10, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 10, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 10, 2025

k8s-ci-robot merged commit 70fbc74 into kubernetes-sigs:main Dec 10, 2025
8 checks passed

pravk03 mentioned this pull request Jan 7, 2026

Docs: Update README for Grouped/Individual Modes #41

Merged

AutuSnow mentioned this pull request Feb 9, 2026

Refactor CPU Topology Representation for Asymmetric/Heterogeneous Systems #59

Closed

		// NOTE: This file is a copy of https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cpumanager/cpu_assignment.go
		// as of commit https://github.com/kubernetes/kubernetes/commit/fd5b2efa76e44c5ef523cd0711f5ed23eb7e6b1a with minor modifications

Conversation

pravk03 commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pravk03 commented Aug 21, 2025

Uh oh!

pravk03 commented Aug 21, 2025

Uh oh!

k8s-ci-robot commented Aug 21, 2025

Uh oh!

pravk03 commented Aug 21, 2025

Uh oh!

ffromani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnbelamaric Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johnbelamaric commented Aug 25, 2025

Uh oh!

catblade commented Sep 28, 2025

Uh oh!

pravk03 commented Sep 30, 2025

Uh oh!

catblade commented Oct 3, 2025

Uh oh!

ffromani commented Oct 14, 2025

Uh oh!

ffromani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ffromani Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ffromani left a comment

Choose a reason for hiding this comment

Uh oh!

pravk03 commented Aug 21, 2025 •

edited

Loading

johnbelamaric Aug 25, 2025 •

edited

Loading

ffromani Nov 27, 2025 •

edited

Loading