XListenerSet: invalid ReferenceGrant requirement for Secret

```
    - lastTransitionTime: "2025-11-20T18:23:17Z"
      message: 'Some listeners are not programmed: https2: Reference not permitted
        by ReferenceGrant.'
      observedGeneration: 1
      reason: ListenersNotValid
      status: "True"
      type: Accepted
```

when I have:
* Gateway in ns1
* XListenreset in ns2
* Secret in ns2

This should not require a reference grant.

