From what I've seen this is a recurring topic and solutions have already been provided to tackle cache memory consumption:

1. By configuring cache limitations by-namespace and/or by-label
2. By caching only partial meta-data

I'll explain the problems that I see, but perhaps I don't have a full/correct understanding of these things, in which case I'm happy to be contradicted.

There are still use cases that aren't covered: when queried objects aren't known beforehand, at cache setup time, and their namespace and labels aren't under control (e.g. user-supplied resource references). In those cases, operators would watch resources cluster-wide to get the desired ones - which is a big waste of memory and kube API traffic, especially on big clusters with tons of Configmaps and Secrets.

Caching partial meta-data certainly helps but isn't a panacea: first, because it's still watching a lot of unneeded objects - so there is still a compressible overhead; second, because it forces you to still run queries to the kube API to get the desired object content, if metadata isn't enough for you - which is kind of defeating the idea of a cache?; and perhaps last but I'm not sure about that one: it saves memory but does it saves kube API traffic as well? Said differently, if the metadata filtering is performed after getting informed via the Kube API, it won't prevent the traffic overhead - or is it a feature implemented server-side?

Anyway, I think it would be great to have an option to cache certain GVKs only per object reference rather than full GVKs at large.

I am currently testing such an approach in the operator I'm working on, this is here: https://github.com/jotak/network-observability-operator/tree/configmap-no-cache/pkg/narrowcache

Here's basically how it's used:

	 // This creates a narrowcache.Client for ConfigMaps and Secrets.
	 narrowCache := narrowcache.NewConfig(cfg, narrowcache.ConfigMaps, narrowcache.Secrets)
	 client, err := narrowCache.CreateClient(mgr.GetClient())
	 if err != nil {
		 setupLog.Error(err, "unable to create narrow cache client")
		 os.Exit(1)
	 }

This new cache layer is invoked for any call to client.Get on the configured GVK. For other
GVKs and other verbs, the underlying client is used.

Furthermore, cached objects are available as source.Source, so that they can be used as controller
watches to enqueue reconcile requests:

	 src, _ := client.GetSource(ctx, obj)
	 controller.Watch(
		 src,
		 handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, o client.Object) []reconcile.Request {
			// etc.
		 }),
	 )

This implementation is in certain aspects tailored for my use case - e.g. I'm not caring about List calls. But I think something along those lines would be great to have directly in controller-runtime. Especially as I duplicated some logic that already exists there.