Skip to content

Partial-Content - Transmitting file chunks over nostr#112

Closed
Egge21M wants to merge 6 commits intonostr-protocol:masterfrom
Egge21M:master
Closed

Partial-Content - Transmitting file chunks over nostr#112
Egge21M wants to merge 6 commits intonostr-protocol:masterfrom
Egge21M:master

Conversation

@Egge21M
Copy link
Copy Markdown
Contributor

@Egge21M Egge21M commented Dec 22, 2022

This draft introduces

  • Two new kinds (11,12)
  • Three new Tags (Content-Type, Content-Length, Content-Range)
  • A general flow for transmitting files over nostr that is based on HTTP's 206 Partial Content

So far there are no clients or relays implementing these. I have written an article about a proof-of-concept that I did: https://starbackr.com/sending-images-and-files-on-nostr/

@fiatjaf
Copy link
Copy Markdown
Member

fiatjaf commented Dec 22, 2022

Please don't do this.

@Egge21M
Copy link
Copy Markdown
Contributor Author

Egge21M commented Dec 22, 2022

@fiatjaf care to elaborate what problems you see if it is optional and Relays can decide wether to offer it or not?

Copy link
Copy Markdown
Contributor

@Semisol Semisol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue with this is that nostr is not designed for large files, this is just a hack to store files on Nostr. You get the overhead of:

  • database queries for each chunk
  • wasted bytes due to hex and event data
  • relays were not meant for this

instead, file servers can exist, storing large files or chunks of files (at most ~5 chunks per file), which are suited for storing large files and not small JSON events that need indexing and have different uses

@Egge21M
Copy link
Copy Markdown
Contributor Author

Egge21M commented Dec 22, 2022

We could avoid query-hell by introducing a file-id tag that would be consistent over all chunks. Relays could index file-ids to allow a fast retrieval of all required events from database and would not need to look up all event-ids.

Entry points would no longer need to specify all ids and it would allow transmitting live content, as chunks of data could be added while transmission has already begun.

@fernandolguevara
Copy link
Copy Markdown
Contributor

agreed with @fiatjaf this is a bad idea
@Egge7 u can use http request to upload files to the server

@Egge21M
Copy link
Copy Markdown
Contributor Author

Egge21M commented Dec 22, 2022

@fernandolguevara of course you can, but how would you share it with other clients without having a single point of failure? File servers and URLs in notes is what people use today, but it is not censorship resistant.

Maybe we could start to mirror content on different servers and include URLs to all mirrors in a note...

@fiatjaf
Copy link
Copy Markdown
Member

fiatjaf commented Dec 22, 2022

IPFS was created to solve this problem specifically.

If it works, you can use that, if it doesn't work then we must make a new protocol to solve that problem. The problem is too big in itself to be solved with an unsustainable hack like this.

@Egge21M
Copy link
Copy Markdown
Contributor Author

Egge21M commented Dec 23, 2022

@fiatjaf I am mostly aligned when it comes to long term storage of files, especially because of the inefficiencies with the approach proposed here. We will continue working on it.

Now where Partial-Content shines IMO (especially if we replace the event-is-list proposal with a static file-id as proposed here: #112 (comment)) is live content that is ephemeral.

We could have nostr spaces where audio is streamed to a relay and then to clients listening to the file-id. Of course not every Relay will be able to handle the load, so Relays can choose wether to support it or pay-gate it.

@fiatjaf
Copy link
Copy Markdown
Member

fiatjaf commented Dec 23, 2022

But this is so inefficient.

Considering that clients will already have to implement a very custom way of transmitting audio or files and the same for relays, wouldn't it be better to spec a more efficient protocol that used binary streams over WebSocket directly? Then relays that wanted to support it would have an easier time adding support, and client integration would be simpler too.

@jb55
Copy link
Copy Markdown
Contributor

jb55 commented Dec 24, 2022

NACK

@adamritter
Copy link
Copy Markdown

I have thought about implementing it as well, and I think the best way would be to send an epheremal DM with file metadata and hash (or magnet hash) and a websocket address that the other party can connect to.

After establishing a direct connection, it's rasy to send the file.

@adamritter
Copy link
Copy Markdown

Just a bit of a followup: nostr already has deletion, so whenever the websocket address changes, the client could just delete the older dm.

@ghost
Copy link
Copy Markdown

ghost commented Dec 27, 2022

After establishing a direct connection, it's rasy to send the file.

But that won't be using nostr to save the file on relays. So does it require a NIP?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants