feat(pass-style,marshal): ByteArray, a new binary Passable type by erights · Pull Request #1538 · endojs/endo

erights · 2023-04-04T03:44:22Z

Closes: #1331
Refs: ocapn/ocapn#5 (comment) #2414

Description

Recognize Immutable ArrayBuffers, when also frozen, as the new pass-style byteArray. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders.

Security Considerations

The underlying shim implementation, if we're still using that, uses a normal mutable ArrayBuffer internally, but safely encapsulates it so nothing outside the shim can mutate it.

Scaling Considerations

The underlying shim implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner.

Documentation Considerations

Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array.

Testing Considerations

TODO tests. For each stubbed out case, there instead should be a test.failing test.

Compatibility Considerations

Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays.

The shim cannot reasonably support TypedArrays of DataViews backed by Immutable ArrayBuffers.

Upgrade Considerations

TODO BREAKING.md
TODO NEWS.md

mhofman

I think we should try to align copyBytes with the prior art of the Web platform's Blob.

gibson042

This looks mostly good, but I have reservations about treating ByteArray as a Container [of Passables].

erights · 2025-05-01T02:46:11Z

Wow, @gibson042 , those suggestions were extremely helpful, thanks!

PTAL

gibson042

Thanks! LGTM modulo updating packages/patterns/README.md.

gibson042

Oh, I think this PR should also update arb-passable.js, although accommodations will be necessary to exclude ByteArrays in encodePassable.test.js until they have a defined encoding, e.g.:

-/** @param {typeof import('@fast-check/ava').fc} fc */
-export const makeArbitraries = fc => {
+/**
+ * @param {typeof import('@fast-check/ava').fc} fc
+ * @param {Array<'byteArray'>} [exclusions]
+ */
+export const makeArbitraries = (fc, exclusions = []) => {
   const arbString = fc.oneof(fc.string(), fc.fullUnicodeString());
 
   const keyableLeaves = [
     fc.constantFrom(null, undefined, false, true),
     arbString,
     arbString.map(s => Symbol.for(s)),
     // primordial symbols and registered lookalikes
     fc.constantFrom(
       ...Object.getOwnPropertyNames(Symbol).flatMap(k => {
         const v = Symbol[k];
         if (typeof v !== 'symbol') return [];
         return [v, Symbol.for(k), Symbol.for(`@@${k}`)];
       }),
     ),
     fc.bigInt(),
     fc.integer(),
+    ...[fc.uint8Array().map(arr => arr.buffer.transferToImmutable())].filter(
+      () => !exclusions.includes('byteArray'),
+    ),
     fc.constantFrom(-0, NaN, Infinity, -Infinity),
     fc.record({}),
     fc.constantFrom(exampleAlice, exampleBob, exampleCarol),
   ];

gibson042 · 2025-05-01T19:01:16Z

And to capture my thoughts on the encoding itself: given that we want both shortlex ordering and (to abuse vocabulary somewhat) isomorphic rank-order comparisons of encodePassable output, that encoding should be something like a length prefix (using the same Elias delta variant as for bigints) followed by hexadecimal:

buffer data	encoding
`[]`	`a1:0:`
`[0]`	`a1:1:00`
`[255]`	`a1:1:ff`
`[0 255]`	`a1:2:00ff`
`[0 … 0]` (10**9 + 2 zeros)	`a~10:1000000002:00…00`

erights · 2025-05-01T21:13:40Z

Oh, I think this PR should also update arb-passable.js, although accommodations will be necessary to exclude ByteArrays in encodePassable.test.js until they have a defined encoding,

Done

erights · 2025-05-01T22:16:57Z

Thanks! LGTM modulo updating packages/patterns/README.md.

I just read patterns/README.md carefully, and did not spot the flaw you have in mind. Is this indeed the document you have in mind? What needs updating?

gibson042

I just read patterns/README.md carefully, and did not spot the flaw you have in mind. Is this indeed the document you have in mind? What needs updating?

Ah, no change is needed now that we acknowledge ByteArray as primitive.

erights requested review from dckc and gibson042 April 4, 2023 03:44

erights self-assigned this Apr 4, 2023

erights mentioned this pull request Apr 4, 2023

What are the core "data" types? ocapn/ocapn#5

Closed

dckc reviewed Apr 4, 2023

View reviewed changes

Comment thread packages/marshal/src/rankOrder.js Outdated

dckc reviewed Apr 4, 2023

View reviewed changes

Comment thread packages/pass-style/src/copyBytes.js Outdated

mhofman reviewed Apr 4, 2023

View reviewed changes

erights force-pushed the markm-binary branch from 268a47f to dfd0450 Compare April 6, 2023 20:45

erights mentioned this pull request Apr 13, 2023

Support passable immutable ArrayBuffer as copy-data #1331

Closed

erights force-pushed the markm-binary branch from dfd0450 to 3a8a10d Compare May 20, 2023 20:50

erights force-pushed the markm-binary branch from 89456e0 to b500d00 Compare June 6, 2023 03:17

erights mentioned this pull request Jun 25, 2023

Changes to @endo/pass-style and up likely needed for ocapn conformance. #1587

Open

6 tasks

erights force-pushed the markm-binary branch 2 times, most recently from 80cc7f2 to fca467e Compare September 16, 2023 03:35

erights force-pushed the markm-binary branch 4 times, most recently from 15cbf4f to a6a6ca0 Compare September 26, 2023 05:51

erights force-pushed the markm-binary branch from a6a6ca0 to c7ea4d9 Compare October 3, 2023 20:27

erights force-pushed the markm-binary branch from c7ea4d9 to e7c0a09 Compare October 13, 2023 22:34

erights force-pushed the markm-binary branch from e7c0a09 to 0547292 Compare October 24, 2023 23:34

erights force-pushed the markm-binary branch 2 times, most recently from 4bf3d8c to ce5f3bf Compare November 7, 2023 19:22

erights force-pushed the markm-binary branch from ce5f3bf to 947e166 Compare December 21, 2023 02:59

erights force-pushed the markm-binary branch from e24eced to ac041b2 Compare December 31, 2023 22:41

erights force-pushed the markm-binary branch from ac041b2 to 221be50 Compare January 11, 2024 02:31

erights changed the title ~~WIP fix: CopyBytes new binary Passable type~~ WIP feat(pass-style,marshal)!: ByteArray, a new binary Passable type Jan 11, 2024

erights force-pushed the markm-binary branch 3 times, most recently from b25c593 to fe561d2 Compare January 11, 2024 06:24

erights force-pushed the markm-binary branch from a815a51 to 1281d0c Compare October 29, 2024 00:02

erights force-pushed the markm-to-immutable-3 branch from b32ab45 to 544a213 Compare November 17, 2024 00:50

erights force-pushed the markm-binary branch from 1281d0c to 42075f3 Compare November 17, 2024 00:51

erights force-pushed the markm-to-immutable-3 branch 2 times, most recently from d0f1cd5 to bc99d9e Compare January 15, 2025 23:18

erights force-pushed the markm-to-immutable-3 branch from bc99d9e to 0509bcc Compare February 17, 2025 03:22

erights force-pushed the markm-binary branch from 42075f3 to 134bd68 Compare February 17, 2025 03:27

erights mentioned this pull request Feb 17, 2025

Path to Stage 4! tc39/proposal-immutable-arraybuffer#1

Open

45 tasks

erights mentioned this pull request Apr 29, 2025

feat(ocapn): Minimum viable OCapN #2781

Closed

erights requested a review from kumavis April 30, 2025 02:30

erights force-pushed the markm-to-immutable-3 branch from 0509bcc to 07017b0 Compare April 30, 2025 02:51

erights force-pushed the markm-binary branch 2 times, most recently from 4502570 to fa09e05 Compare April 30, 2025 03:29

erights marked this pull request as ready for review April 30, 2025 03:30

gibson042 requested changes Apr 30, 2025

View reviewed changes

erights force-pushed the markm-binary branch from fa09e05 to 98f0140 Compare May 1, 2025 00:05

erights commented May 1, 2025

View reviewed changes

erights requested a review from gibson042 May 1, 2025 02:46

gibson042 approved these changes May 1, 2025

View reviewed changes

Comment thread packages/pass-style/src/passStyleOf.js Outdated

Comment thread packages/patterns/src/keys/checkKey.js

gibson042 reviewed May 1, 2025

View reviewed changes

erights force-pushed the markm-to-immutable-3 branch from e2fd1ff to 75ea396 Compare May 1, 2025 21:30

gibson042 approved these changes May 1, 2025

View reviewed changes

gibson042 mentioned this pull request May 1, 2025

feat(ses): ArrayBuffer.prototype.transferToImmutable #2400

Merged

3 tasks

Base automatically changed from markm-to-immutable-3 to master May 1, 2025 22:36

feat(pass-style,marshal): ByteArray, a new binary Passable type

0c0a80a

erights mentioned this pull request Jul 17, 2025

chore: Sync Endo 2025-07-11 Agoric/agoric-sdk#11594

Merged

Uh oh!

Conversation

erights commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

Uh oh!

Uh oh!

Uh oh!

mhofman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gibson042 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erights commented May 1, 2025

Uh oh!

gibson042 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gibson042 left a comment

Choose a reason for hiding this comment

Uh oh!

gibson042 commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erights commented May 1, 2025

Uh oh!

erights commented May 1, 2025

Uh oh!

gibson042 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

erights commented Apr 4, 2023 •

edited

Loading

gibson042 commented May 1, 2025 •

edited

Loading