feat(pass-style,marshal): ByteArray, a binary Passable#2414
Closed
erights wants to merge 3 commits into
Closed
Conversation
erights
force-pushed
the
markm-binary-direct
branch
from
August 16, 2024 22:21
eed07e2 to
1df60e9
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
August 19, 2024 22:47
835d70e to
74007b3
Compare
erights
changed the base branch from
master
to
markm-repair-arraybuffer-transfer
August 19, 2024 22:48
erights
marked this pull request as ready for review
August 19, 2024 23:13
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
from
August 22, 2024 01:08
4dd9bf8 to
d19e509
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
August 22, 2024 01:09
74007b3 to
6af60d7
Compare
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
from
August 26, 2024 20:48
d19e509 to
585042f
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
August 26, 2024 20:49
6af60d7 to
affa6b3
Compare
erights
marked this pull request as draft
August 26, 2024 20:52
Contributor
Author
|
Converting to draft until more of the stubbed out cases are implemented, and until more of both are tested. |
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
3 times, most recently
from
September 2, 2024 21:30
8dce401 to
06e15fe
Compare
erights
force-pushed
the
markm-binary-direct
branch
2 times, most recently
from
September 4, 2024 00:11
53ec91b to
ce0fd14
Compare
erights
added a commit
that referenced
this pull request
Sep 4, 2024
Staged on #2419 Closes: #XXXX Refs: #2414 #2418 #2419 ## Description #2414 by itself does not work on Node 18 and Node 20 because - those platforms do not have `Array.prototype.transfer`, so #2414 must use `structuredClone` instead - `structuredClone` does exist on Node >= 18, so it should be on supported platforms (though see #2418 ). However, `structuredClone` itself is dangerous and so must not be added to the shared intrinsics. As a result, in #2414 , when `@endo/pass-style` is initialized in a created compartment, it fails to find either `Array.prototype.transfer` and `structuredClone To solve this, @kriskowal suggested that we also shim `Array.prototype.transfer` if needed during `lockdown`, along with other repairs. We are avoiding similarly shimming `Array.prototype.transferToImmutable` because it is not yet standard. But `Array.prototype.transfer` is standard, and so `lockdown` can globally shim it before hardening the shared intrinsics. This PR implements @kriskowal 's suggestion. ### Security Considerations none ### Scaling Considerations by itself, none ### Documentation Considerations nothing signicant. ### Testing Considerations See #2418 . Aside from that, none ### Compatibility and Upgrade Considerations On platforms with neither `Array.prototype.transfer` nor a global `structuredClone`, the ses-shim will *currently* not install an emulation of `Array.prototype.transfer`. However, once we verify that endo is not intended to support platforms without both, we may change lockdown to throw, failing to lock down.
erights
force-pushed
the
markm-binary-direct
branch
from
September 7, 2024 20:42
ce0fd14 to
066637e
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
October 14, 2024 19:49
066637e to
e696f45
Compare
erights
commented
Oct 14, 2024
| // then according to their lengths. | ||
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| return comparator(left.byteLength, right.byteLength); |
Contributor
Author
There was a problem hiding this comment.
Should switch to shortlex. See https://en.wikipedia.org/wiki/Shortlex_order
That way, encodePassable could encode as prefix followed by the unescaped content.
erights
commented
Oct 14, 2024
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| // @ts-expect-error narrowed | ||
| return compareRank(left.byteLength, right.byteLength); |
Contributor
Author
There was a problem hiding this comment.
Should delegate the entire compare to rankOrder, which, btw, will switch to shortlex order.
erights
force-pushed
the
markm-binary-direct
branch
from
October 29, 2024 00:03
e696f45 to
6312dd7
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
November 17, 2024 00:52
6312dd7 to
566c4c8
Compare
erights
added a commit
that referenced
this pull request
Jan 1, 2025
Closes: #XXXX Refs: - ocapn/ocapn#125 - https://github.com/ocapn/ocapn/blob/28af626441da888c4a520309222e18266dd2f1f2/draft-specifications/Model.md - #2452 - Agoric/agoric-sdk#10084 - https://github.com/tc39/proposal-immutable-arraybuffer - #2414 ## Description Revise Smallcaps cheatsheet to track OCapn as of ocapn/ocapn#125 ### Security Considerations none ### Scaling Considerations none ### Documentation Considerations the point. The cheatsheet was becoming too stale. ### Testing Considerations none ### Compatibility Considerations none ### Upgrade Considerations none
erights
force-pushed
the
markm-binary-direct
branch
from
February 17, 2025 03:33
566c4c8 to
a89c4e2
Compare
Contributor
Author
|
Closing in favor of #1538 , since we no longer need to use the ponyfill and avoid the shim. |
3 tasks
erights
added a commit
that referenced
this pull request
May 1, 2025
Closes: #1331 Refs: ocapn/ocapn#5 (comment) #2414 ## Description Recognize Immutable ArrayBuffers, when also frozen, as the new pass-style `byteArray`. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders. ### Security Considerations The underlying shim implementation, if we're still using that, uses a normal mutable ArrayBuffer internally, but safely encapsulates it so nothing outside the shim can mutate it. ### Scaling Considerations The underlying shim implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner. ### Documentation Considerations Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array. ### Testing Considerations - [ ] TODO tests. For each stubbed out case, there instead should be a `test.failing` test. ### Compatibility Considerations Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays. The shim cannot reasonably support TypedArrays of DataViews backed by Immutable ArrayBuffers. ### Upgrade Considerations - [ ] TODO BREAKING.md - [ ] TODO NEWS.md
boneskull
pushed a commit
that referenced
this pull request
Jun 4, 2025
Closes: #1331 Refs: ocapn/ocapn#5 (comment) #2414 ## Description Recognize Immutable ArrayBuffers, when also frozen, as the new pass-style `byteArray`. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders. ### Security Considerations The underlying shim implementation, if we're still using that, uses a normal mutable ArrayBuffer internally, but safely encapsulates it so nothing outside the shim can mutate it. ### Scaling Considerations The underlying shim implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner. ### Documentation Considerations Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array. ### Testing Considerations - [ ] TODO tests. For each stubbed out case, there instead should be a `test.failing` test. ### Compatibility Considerations Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays. The shim cannot reasonably support TypedArrays of DataViews backed by Immutable ArrayBuffers. ### Upgrade Considerations - [ ] TODO BREAKING.md - [ ] TODO NEWS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #1331
Refs: #1538 ocapn/ocapn#5 (comment)
Description
Use the
@endo/immutable-arraybufferponyfill directly, rather than the shim@endo/immutable-arraybuffer/shim.js, to provide objects that act like the immutable ArrayBuffers that would result from thetransferToImmutableproposal.Recognize these objects, when also frozen, as the new pass-style
byteArray. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders.Security Considerations
Does correctly enforce immutability. But by depending only on the ponyfill rather than the shim, this PR inherits the eval twin problem of the ponyfill. Each instantiation of
passStyleOfwill only recognize the byte-arrays from the instantiation of@endo/pass-stylethat made that byte-array. For environment that, for example, use thepassStyleOfendowed to them by liveslots, liveslots will also need to endow them with the corresponding byte-array maker.In theory, this is not much worse than the need to co-endow
passStyleOfandProxy, so thatpassStyleOfcan reject apparent copy structures that are also proxies.Scaling Considerations
The underlying ponyfill implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner.
Documentation Considerations
Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array.
Testing Considerations
test.failingtest.Compatibility Considerations
Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays.
Upgrade Considerations