It comes out of a practical example where I've used semver to over more then then "Public API". Cases of things that need to be kept stable within a major-cycle:

• database schema (should be migrateable from any to any version within a major-cycle with the standard migration runner)
• file formats (only non breaking additions/changes allowed within major-cycle)
• network protocols
• API-calls it makes on other systems (just as much public API actually)

Maybe semver could be formulated in a way that it shows it naturally covers more the just "public API".