feat(gateway): implement IPIP-402 extensions for gateway CAR requests by aschmahmann · Pull Request #303 · ipfs/boxo

aschmahmann · 2023-05-16T13:33:05Z

Fixes #283.

Requires fix: do not read extra block if offset = at+childSize go-unixfsnode#50
IPIP: IPIP-402: Partial CAR Support on Trustless Gateways specs#402
Kubo PR: feat(gateway): support for IPIP-402 CAR params kubo#9914
Conformance Tests: feat: tests for IPIP-402 CAR parameters gateway-conformance#56

gateway/gateway.go

gateway/blocks_gateway.go

codecov · 2023-05-16T13:44:20Z

Codecov Report

Merging #303 (7be4474) into main (a85e005) will decrease coverage by 0.08%.
The diff coverage is 56.82%.

@@            Coverage Diff             @@
##             main     #303      +/-   ##
==========================================
- Coverage   50.15%   50.08%   -0.08%     
==========================================
  Files         281      278       -3     
  Lines       33763    33983     +220     
==========================================
+ Hits        16935    17019      +84     
- Misses      15046    15173     +127     
- Partials     1782     1791       +9

Impacted Files	Coverage Δ
examples/gateway/car/main.go	`11.53% <0.00%> (ø)`
examples/gateway/proxy/main.go	`0.00% <0.00%> (ø)`
gateway/dns.go	`0.00% <ø> (ø)`
gateway/handler_ipns_record.go	`18.46% <0.00%> (-2.23%)`	⬇️
gateway/handler_unixfs__redirects.go	`37.63% <36.36%> (ø)`
gateway/blocks_backend.go	`44.38% <44.38%> (ø)`
gateway/handler_defaults.go	`40.26% <50.00%> (+1.85%)`	⬆️
gateway/handler_unixfs_dir.go	`64.28% <50.00%> (ø)`
gateway/metrics.go	`75.00% <69.69%> (-0.22%)`	⬇️
gateway/handler_car.go	`64.88% <74.68%> (+6.55%)`	⬆️
... and 9 more

... and 12 files with indirect coverage changes

gateway/blocks_gateway.go

gateway/handler_car.go

gateway/handler.go

gateway/handler_car.go

gateway/handler.go

gateway/handler_car.go

gateway/handler_car_test.go

gateway/blocks_backend.go

addressed

Co-authored-by: Henrique Dias <[email protected]>

- Improve documentation of public API that to conform with pkg.go.dev stylings. Can be previewed with: `go install golang.org/x/pkgsite/cmd/pkgsite@latest && pkgsite` Note that there's a bug that makes the comments in struct fields not formatted: golang/go#56004 - Renamed a few things for consistency and clarity: - `BlocksGateway` to `BlocksBackend` (it is an `IPFSBackend`) - `api` to `backend` (it is an `IPFSBackend`) - `WithHostname` to `NewHostnameHandler` - `Specification` to `PublicGateway` - `ErrorResponse` to `ErrorStatusCode` - `DagEntityByteRange` to `DagByteRange` (there isn't other) - Unexported `ServeContent`, which was never meant to be public - Removed `webRequestError`, which is already handled by `webError`. Added a test to ensure that is the case. - Removed return types from functions that always returned nil. - Exported `DagScopeEntity`, `DagScopeAll`, `DagScopeBlock` as they are `DagScope` - Change some usages of global `log` variable to request-scoped `logger` for more context - Handle CAR before maybe resolving path (CAR handles it differently) - Response formats as constants (avoids typos, etc)

- this also allows to return the PathMetadata, which also allows to set the correct X-Ipfs-Root header.

hacdias · 2023-06-08T15:15:35Z

Plan is:

@hacdias Merge this, update in Kubo.
@laurentsenta merge conformance, release conformance.
@hacdias release Boxo update.

BigLep · 2023-06-08T22:39:05Z

2023-06-08 maintainer conversation:
Expanding #303 (comment)
Adin will handle the proactive communication to bedrock about what is happening.
@hacdias intends to do a 0.9.1 Boxo release on 2023-06-09 with this work included.

We'll let the spec conversation around CAR roots drive if there are any CAR spec changes and ipld/go-car updates required.

aschmahmann requested review from hacdias and lidel as code owners May 16, 2023 13:33

aschmahmann force-pushed the feat/gateway-ipip-402 branch 2 times, most recently from 0b2dac0 to 8f39b1a Compare May 16, 2023 13:34

aschmahmann commented May 16, 2023

View reviewed changes

gateway/gateway.go Outdated Show resolved Hide resolved

Jorropo reviewed May 16, 2023

View reviewed changes

gateway/blocks_gateway.go Outdated Show resolved Hide resolved

Jorropo reviewed May 16, 2023

View reviewed changes

gateway/blocks_gateway.go Outdated Show resolved Hide resolved

aschmahmann force-pushed the feat/gateway-ipip-402 branch 2 times, most recently from a400896 to 42f28cb Compare May 16, 2023 14:23

aschmahmann commented May 16, 2023

View reviewed changes

gateway/blocks_gateway.go Outdated Show resolved Hide resolved

gateway/blocks_gateway.go Show resolved Hide resolved

gateway/blocks_gateway.go Show resolved Hide resolved

gateway/blocks_gateway.go Show resolved Hide resolved

hacdias reviewed May 16, 2023

View reviewed changes

gateway/handler_car.go Outdated Show resolved Hide resolved

aschmahmann force-pushed the feat/gateway-ipip-402 branch from 42f28cb to cd4653b Compare May 16, 2023 14:55

aschmahmann commented May 16, 2023

View reviewed changes

gateway/handler.go Outdated Show resolved Hide resolved

BigLep assigned aschmahmann May 16, 2023

lidel mentioned this pull request May 16, 2023

IPIP-402: Partial CAR Support on Trustless Gateways ipfs/specs#402

Merged

aschmahmann commented May 16, 2023

View reviewed changes

gateway/handler_car.go Outdated Show resolved Hide resolved

hacdias mentioned this pull request May 17, 2023

refactor: use IPIP-402 param names when GRAPH_BACKEND=true ipfs-inactive/bifrost-gateway#108

Merged

hacdias force-pushed the feat/gateway-ipip-402 branch from 6520009 to 37c40dd Compare May 17, 2023 08:36

hacdias mentioned this pull request May 17, 2023

feat(gateway): support for IPIP-402 CAR params ipfs/kubo#9881

Closed

lidel reviewed May 17, 2023

View reviewed changes

gateway/handler_car.go Outdated Show resolved Hide resolved

gateway/handler.go Outdated Show resolved Hide resolved

gateway/handler_car.go Outdated Show resolved Hide resolved

aschmahmann mentioned this pull request May 19, 2023

feat: tests for IPIP-402 CAR parameters ipfs/gateway-conformance#56

Merged

5 tasks

hacdias force-pushed the feat/gateway-ipip-402 branch from d62808d to 03debb9 Compare May 25, 2023 08:58

lidel reviewed May 29, 2023

View reviewed changes

gateway/handler_car.go Outdated Show resolved Hide resolved

gateway/handler_car.go Outdated Show resolved Hide resolved

gateway/handler_car.go Outdated Show resolved Hide resolved

gateway/handler_car.go Outdated Show resolved Hide resolved

hacdias force-pushed the feat/gateway-ipip-402 branch 6 times, most recently from 7647b01 to cdc0436 Compare June 1, 2023 09:02

rvagg mentioned this pull request Jun 2, 2023

feat: add stable etag filecoin-project/lassie#285

Merged

hacdias force-pushed the feat/gateway-ipip-402 branch 2 times, most recently from fe573f5 to 817991c Compare June 2, 2023 08:50

hacdias mentioned this pull request Jun 2, 2023

fix: do not read extra block if offset = at+childSize ipfs/go-unixfsnode#50

Merged

hacdias requested a review from a team as a code owner June 2, 2023 14:17

hacdias requested a review from lidel June 2, 2023 14:53

hacdias force-pushed the feat/gateway-ipip-402 branch 7 times, most recently from 4544260 to 07a0bef Compare June 5, 2023 09:24

hacdias mentioned this pull request Jun 7, 2023

fix(gateway): ensure 'X-Ipfs-Root' header is valid #337

Merged

lidel previously requested changes Jun 7, 2023

View reviewed changes

gateway/handler_car.go Outdated Show resolved Hide resolved

gateway/handler_car_test.go Show resolved Hide resolved

gateway/blocks_backend.go Outdated Show resolved Hide resolved

hacdias force-pushed the feat/gateway-ipip-402 branch 3 times, most recently from df19eb1 to a93ce13 Compare June 8, 2023 08:59

hacdias force-pushed the feat/gateway-ipip-402 branch from a93ce13 to c5f0cad Compare June 8, 2023 09:39

aschmahmann and others added 3 commits June 8, 2023 16:48

feat(gateway): implement IPIP-402 extensions for gateway CAR requests

ac5c706

Co-authored-by: Henrique Dias <[email protected]>

feat: use last segment cid as root

7be4474

- this also allows to return the PathMetadata, which also allows to set the correct X-Ipfs-Root header.

hacdias force-pushed the feat/gateway-ipip-402 branch from c5f0cad to 7be4474 Compare June 8, 2023 14:48

hacdias approved these changes Jun 8, 2023

View reviewed changes

hacdias merged commit 475d576 into main Jun 8, 2023

hacdias deleted the feat/gateway-ipip-402 branch June 8, 2023 15:18

aschmahmann mentioned this pull request Aug 14, 2023

feat: IPIP-402 based backpressure ipfs-inactive/bifrost-gateway#160

Merged

Conversation

aschmahmann commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hacdias commented Jun 8, 2023

Uh oh!

BigLep commented Jun 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aschmahmann commented May 16, 2023 •

edited

Loading

codecov bot commented May 16, 2023 •

edited

Loading