@lidel another option I see is to completely remove support for ?format=json|cbor but that will also raise the question of: should we then also support CIDs whose codec is JSON|CBOR?

Let's add a test that ensures we keep read-only support for CIDs with json and cbor codec: when someone requests /ipfs/bagaaierayeva7pcl4bchk3udf3i5qm3eeqlbj6tzcmidv64uonglmi5nebca (CID with json codec) without any parameters, we want gateway to return same payload as raw block response, but with content-type: application/json based on the codec from CID.

@lidel we had the tests already 😃 (test_cmp_dag_get).

This is getting really hairy. I am writing more tests, as I've found some new papercuts and missing cases that we should have regression tests for:

• Requesting CID with json codec with Accept: aplication/json should work, but does not
• Requesting CID with dag-json codec with Accept: aplication/json should work (without conversion of bytes), but does not

I will prepare a fix and push together with tests.

Pushed two fixes and tests in c3ebe80
It should solve the problem from https://github.com/protocol/bifrost-infra/issues/2290 and be ready for 0.18.

@hacdias when you start on Friday, take a look at the above changes as a sanity check and lmk if anything is weird or needs to be better commented.

(interop/webui tests are failing due to unrelated reason – tracking in #9570)

@lidel what you pushed works if a file has JSON content (or whatever content it has). However, it does not work for a CID that has the codec JSON (or CBOR for that matter). The fault in the tests lies in that FILE_JSON_CID is just a regular UnixFS/Raw CID and not a JSON CID.

I pushed a fix and another test... I'm also not very happy with this. It's becoming increasingly complex and I feel like there must be a way of simplifying this. I will give it some thought during today's morning and see if I can simplify it somehow.

Thanks for catching this @hacdias

Maybe we could simplify things a bit by adding CID codec check at the end of customResponseFormat func, and set explicit one when CID is json cbor dag-json or dag-cbor? at least that will happen it one place.

I will circle back to this PR after meetings, but don't want to block the release, we have test coverage now which is the most important part.

We may:

• merge it as-is to fix the JSON problem for 0.18, and refactor/simplify later (fill issue with ideas how to simplify it)
  • if we do this, we need to update https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.18.md to reflect changes in this PR
• revert entire JSON/CBOR support, and move it to Kubo 0.19

I feel this PR makes it safe to ship in 0.18, but lmk

@lidel I think we should merge as-is and simplify later if possible.

I'm good with us doing the followups in the next release (assuming we do the changelog update and have the followup issue for 0.19). Thanks for handling.

(writing updates to release notes section, will push and merge before Monday)