Raised from #2285

In JSON output mode, we should return error messages as JSON so that they can be handled accordingly.

This is a breaking change and can be included in v4 later this year.