What did you do?

Hello,
We have a consul that has integration and staging hosts on it. Services are like node-exporter and regroup all hosts in there. We're trying to move from node_meta to filter and we have this filter:

filter: 'NodeMeta.env == "stg"'

However, we find metrics for both environments. After digging calls being made by prometheus, we can see:

• List all services: https://consul.domain/v1/catalog/services?filter=NodeMeta.env+%3D%3D+%22stg%22&stale=&wait=120000ms (source code)
• Get service health: https://consul.domain/v1/health/service/node-exporter?index=291464323&stale=&tag=metrics&wait=120000ms: there is no filter there so that explains why I can see both environments (source code)

I can see that the watch function has watchedNodeMeta but we shouldn't use it anymore with Consul 1.14+.

I feel like we should add a watchedServiceFilter, like watchedFilter (here) so we can have a filter on services that have been discovered.

If you're ok with it, I can propose a PR 🙏
What do you think?

Thanks!

What did you expect to see?

No response

What did you see instead? Under which circumstances?

I get more services with filter than with node_meta

System information

No response

Prometheus version

/prometheus # prometheus --version
prometheus, version 3.1.0 (branch: HEAD, revision: 7086161a93b262aa0949dbf2aba15a5a7b13e0a3)
  build user:       root@74c225e2044f
  build date:       20250102-13:52:43
  go version:       go1.23.4
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

Alertmanager version

Alertmanager configuration file

Logs