What type of PR is this?
/kind cleanup

What this PR does / why we need it:
We missed this when reviewing the ListenerSet GEP. Although an optional port that would require implementations to assign a port could be legitimately useful, I think we haven't fully processed this concept. Moving this back to required allows us to make it optional in the future, where starting with optional would make it much more painful to transition to required.

Some potential concerns of an optional port on ListenerSet:

• Automatic port allocation could become a bit complicated for Gateway implementations to maintain
• Spec.Listeners.Port would no longer be a reliable source for understanding which Port a Listener was listening on
• Each Listener with an optional Port would require a status entry (not currently the case)
• If ListenerSet ever supported multiple parents, the same Listener might have different Ports assigned

With all that said, I think this is potentially a very useful idea, I just don't think we spent enough time on this while reviewing ListenerSet and would prefer to take the safest possible path forward towards RC2. This is not "never", just "not yet".

Does this PR introduce a user-facing change?:

XListenerSet Port is required. (It was already effectively required due to 0 being an invalid value, but had been marked optional).

/cc @dprotaso @youngnick