Rather than calling wp_get_object_terms, we should use get_the_terms, or check get_object_term_cache ourselves. This will require doing the sorting ourselves though; we may be able to optimise for the common use case of the defaults.