feat(ocapn): Minimum viable OCapN by kumavis · Pull Request #2781 · endojs/endo

kumavis · 2025-04-28T22:19:54Z

This PR is intended to land a work-in-progress implementation of the client, and is not ready to be published.

The implementation is a single vat single process (no comms vat).

Implements Passables
Remote calls + Pipelining
Third Party Handoffs
Eager promise subscription (lazy promise subscription to be explored later)
A tcp-testing-only netlayer (no tor netlayer yet)

This does not yet implement GC messages.

this was mostly built against this test suite https://github.com/ocapn/ocapn-test-suite
which lags behind the spec a bit. I expect dramatic spec changes to come, so I don't mind being behind spec at this point.

There is some implementation weirdness around use of pass-style due to a need for an ImmutableArrayBuffer. This implementation currently still uses mutable Uint8Arrays and thus some homespun passStyleOf checks. The intention is to replace these with exclusively real pass-style before publishing.

rebased on #2772

kumavis · 2025-04-29T22:44:48Z

When attempting to use @endo/pass-style's passStyleOf with selectors reified as registered symbols, I encountered an additional issue unrelated to symbols/selectors:

OCapN reifies ByteArrays as Uint8Arrays, but passStyleOf complains they are not hardened, even after a call to harden().

@erights i think you're working on support for something like an ImmutableArrayBuffer ? If you can point me at the code I can investigate using that. Until then I'm setting aside use of passStyleOf, temporarily working just with @endo/pass-style's PASS_STYLE. I have full intent to migrate to passStyleOf proper before marking this PR as ready for review.

erights · 2025-04-29T23:04:18Z

OCapN reifies ByteArrays as Uint8Arrays, but passStyleOf complains they are not hardened, even after a call to harden().

Doh! I think I know what the problem is. Will investigate. Attn @kriskowal

@erights i think you're working on support for something like an ImmutableArrayBuffer

https://github.com/endojs/endo/blob/master/packages/immutable-arraybuffer/shim.js
#2400
#1538
#2414

I think one of #1538 or #2414 is redundant and should be closed in favor of the other one. I plan to look at these soon anyway, doing maintenance and bringing them up to date.

Would love any feedback!

erights · 2025-04-29T23:07:51Z

I plan to look at these soon anyway, doing maintenance and bringing them up to date.

Especially since the proposal has reached stage 2.7 with no significant impediments to stage 3.0 (just further work on the test262 tests). We agreed at a recent Endo meeting that this was close enough for these dependent PRs to move ahead towards merging into endo master. (IIRC, you were there.)

kriskowal

Looks like progress to me. I’m recommending that we capture issues for all the TODOs especially debts accrued by deferring alignment with @endo/pass-style.

Given that this doesn’t line up with the past, I would like it to line up with our best understanding of the anticipated future, so

please use ArrayBuffer for the representation of ByteArray (not Uint8Array) and also
use the name and representation of Symbol instead of Selector. I’ve made a note about what we want to do with methodNameForSymbol.

I am not sure what we’re going to do about your OCapNFar functions to reconcile with Far, but we must do something in pass-style and not bind that solution to OCapN. I don’t know what that solution is yet. But, makeExo and Far need to be sufficiently generic that they produce objects that work equally well with both CapTP and OCapN. I would be fine, for now, if you simply renamed your routine Far and moved it into your pass-style utility.

kriskowal · 2025-05-24T00:42:02Z

+  const otherDir = slot[1] === '+' ? '-' : '+';
+  const revslot = `${slot[0]}${otherDir}${slot.slice(2)}`;
+  return revslot;


I have a fantasy where this function is simply:

const [table, pos] = slot; return [table, -pos];

kriskowal · 2025-05-24T01:04:20Z

+export const getSelectorName = selector => {
+  if (selector[PASS_STYLE] !== 'selector') {
+    throw Error(`Selector expected, got ${typeof selector}`);
+  }
+  return selector[Symbol.toStringTag];
+};


I think we’re converging on (attn @erights). For context, the notion is to merge this OCapN implementation before pass-style is ready and to leave extensive notes with linked issues. The issue here is that this module will eventually be replaced with @endo/pass-style and so needs to converge on the same end state.

Suggested change

export const getSelectorName = selector => {

if (selector[PASS_STYLE] !== 'selector') {

throw Error(`Selector expected, got ${typeof selector}`);

}

return selector[Symbol.toStringTag];

};

export const methodNameForSymbol = symbol => {

if (symbol[PASS_STYLE] !== 'symbol') {

throw Error(`Selector expected, got ${typeof symbol}`);

}

tag = symbol[Symbol.toStringTag];

if (tag === '@@asyncIterator') {

return Symbol.asyncIterator;

} else if (tag === 'then') {

return Symbol.for('then'); // import { THEN } from "@endo/pass-style";

} else if (tag === 'constructor') {

return Symbol.for('constructor'); // import { CONSTRUCTOR } from "@endo/pass-style";

} else {

return tag;

}

};

I might punt adding special behavior for asyncIterator etc until I understand it better. also relates to how we handle OCapNFar.

kriskowal · 2025-05-24T01:07:10Z

+import { PASS_STYLE } from '@endo/pass-style';
+
+const { freeze, prototype: ObjectPrototype, create } = Object;
+


Should capture a makeByteArray or byteArrayFrom here so we have a single place to change when lining up the representations.

i tried implementing this and it started snaking out all over the code base. unsure what should be a ByteArray and what is fine being eg a Uint8Array. I'd rather revisit this with the ImmutableArrayBuffer shim.

Let’s think about what it will take to make ocapn usable in a shim-free environment. @erights

Let’s think about what it will take to make ocapn usable in a shim-free environment.

IMO, too hard, unless you want to wait for native implementations of the Immutable ArrayBuffer proposal. passStyleOf should only succeed on passables, which must therefore be hardened. The proposal would make either hardened Immutable ArrayBuffers or hardened frozen Uint8Arrays plausible choices for binding to ocapn ByteArray.

However, IMO, waiting for native implementations of Immutable ArrayBuffer would take too long. So the only reasonable choice IMO is to use the shim. The Shim's emulated Immutable ArrayBuffers can be made passable. But the shim cannot make Uint8Arrays passable, and has no reasonable way to emulate a frozen Uint8Array. So, for ocapn to proceed prior to native implementations, we gotta use the shim and bind Immutable ArrayBuffers to ocapn ByteArrays.

kriskowal · 2025-05-24T01:09:28Z

This might be model.js or passable.js.

this is used for testing and debugging. this is an otherwise unused direct reification of syrup to javascript. it is used by the ocapn client to log extra error info when it fails to parse an ocapn message.

so i left it alone for now, but yeah "naming things"

…ass-style selectors

kriskowal

I’ve made some notes. This meets my bar for a package that is not yet published and based on a still-wet specification and I’m eager to move forward quickly.

kriskowal · 2025-06-10T21:37:44Z

+  const makeSlotForValue = val => {
+    /** @type {CapTPSlot} */
+    let slot;
+    if (isPromise(val)) {


passStyleOf(valu) === 'promise' will be more resilient to future changes.

kriskowal · 2025-06-10T21:38:50Z

+      }
+      seen.clear();
+    },
+    abort() {


Maybe just clear for this name.

kriskowal · 2025-06-10T21:46:08Z

+  swissnumTable = new Map(),
+  giftTable = new Map(),


Given that the caller can retain these options, is there an expectation that they can update them or observe changes to them after calling makeClient? If there is no such expectation, we should capture a snapshot and discard the option. If there is such an expectation, I’m weary about the semantics and the observability story.

is there an expectation that they can update them or observe changes to them after calling makeClient

no. i've introduced them for tests, but they could also be part of a persistence system

we should capture a snapshot and discard the option.

seems fine. though we may inspect giftTable in tests

kriskowal · 2025-06-10T21:49:32Z

+  return `${location.transport}:${location.address}`;
+};
+
+const swissnumDecoder = new TextDecoder('ascii', { fatal: true });


I don’t believe that this usage is portable. Consider trivially hand-rolling this routine. I think it’s just something like String.fromCode(...array) as long as you assert that the values of the array are covered by the 0x7f mask.

kriskowal · 2025-06-10T21:53:15Z

+import { PASS_STYLE } from '@endo/pass-style';
+
+const { freeze, prototype: ObjectPrototype, create } = Object;
+


Let’s think about what it will take to make ocapn usable in a shim-free environment. @erights

kriskowal · 2025-06-10T21:55:32Z

+    swissnumTable: makeDefaultSwissnumTable && makeDefaultSwissnumTable(),
+  });
+  const tcpNetlayer = await makeTcpNetLayer({ client });
+  client.registerNetlayer(tcpNetlayer);


I’m sure you’ve got an eye on how this integrates with the pet dæmon. We can add and remove netlayers at runtime? We probably ought to synchronously disconnect all living sessions for a netlayer when it’s unregistered.

kumavis · 2025-09-18T00:47:42Z

replaced by #2962

kumavis changed the base branch from master to kumavis/ocapn-improvements April 28, 2025 22:20

kumavis force-pushed the kumavis/ocapn-python-test-suite branch 3 times, most recently from c7275c8 to 8cc0aec Compare April 28, 2025 22:36

kumavis force-pushed the kumavis/ocapn-improvements branch from fe3ebc4 to 41f429e Compare April 29, 2025 17:43

kumavis changed the base branch from kumavis/ocapn-improvements to master April 29, 2025 19:42

kumavis mentioned this pull request Apr 29, 2025

OCapN: improved error messages and testing #2772

Closed

kumavis force-pushed the kumavis/ocapn-python-test-suite branch 2 times, most recently from 14236e1 to db7bd65 Compare May 9, 2025 02:17

kumavis force-pushed the kumavis/ocapn-python-test-suite branch 3 times, most recently from 50ad13e to 0708a36 Compare May 19, 2025 21:36

kumavis changed the title ~~Feat(ocapn): Compatibility with OCapN Python test server~~ Feat(ocapn): OCapN mvp May 23, 2025

kumavis requested a review from kriskowal May 23, 2025 20:15

kumavis force-pushed the kumavis/ocapn-python-test-suite branch from 933f2c4 to e1b3226 Compare May 23, 2025 20:15

kumavis marked this pull request as ready for review May 23, 2025 20:16

kriskowal changed the title ~~Feat(ocapn): OCapN mvp~~ feat(ocapn): Minimum viable OCapN May 23, 2025

kriskowal requested changes May 24, 2025

View reviewed changes

kumavis added 9 commits May 28, 2025 09:27

fix(ocapn): always decode utf-8 text conservatively

b831b28

feat(ocapn): chained errors on codecs

918a09d

fix(ocapn): Passable OCapNStructCodec wraps errors

157f797

test(ocapn): add float64 infinities to syrup test table

982a78f

fix(ocapn): ocapn selector should be an object with passStyle "selector"

edfee2a

refactor(ocapn): SyrupRecordDefinition as object

d0e2f6e

fix(ocapn): prefer unabbreviated names

050314c

refactor(ocapn): use pass-style helpers to ease migration to future p…

82aeff0

…ass-style selectors

fix(ocapn): fix ordering of actual/expected test arguments

fdd7b7d

kumavis added 12 commits May 28, 2025 09:40

test(ocapn): check OCapN types against passStyleOf support

09592b4

feat(ocapn): mitigate crossed hellos

d22364c

fix(ocapn): desc:handoff-give codec

0c94458

feat(ocapn): add captp-engine

ad433ce

refactor(ocapn): codecs integrate directly with the export table

4112ab2

refactor(ocapn): make session manager

56f4328

fix(ocapn): end session on connection close

b5d82eb

fix(ocapn): fix captp import/export/importCollected hooks

1dab1f3

feat(ocapn): third party handoff with sturdy-refs

41f31d2

feat(ocapn): third party handoff receiving signed gift

d85f29d

feat(ocapn): third party handoffs

cc3eca7

fix(ocapn): removed unused session tables

f481aa8

kumavis force-pushed the kumavis/ocapn-python-test-suite branch from 24ebd47 to f481aa8 Compare May 28, 2025 20:00

kumavis added 4 commits May 28, 2025 10:10

doc(ocapn): fix readme links

0471dfe

test(ocapn): test compareByteArrays

18e119e

fix(ocapn): normalize codec creation and naming

45cd300

test(ocapn): fix test makeTableKit args

c7bf43a

kumavis force-pushed the kumavis/ocapn-python-test-suite branch 2 times, most recently from 59a0d27 to 88f56ff Compare May 29, 2025 01:34

kumavis added 3 commits May 28, 2025 15:36

test(ocapn): fix types in tests

95a26a1

refactor(ocapn): uniform de/code of swissnums

4575e65

fix(ocapn): prefer Ocapn casing to OCapN in code

770fa30

kumavis force-pushed the kumavis/ocapn-python-test-suite branch from 88f56ff to 770fa30 Compare May 29, 2025 02:22

kumavis added 2 commits May 28, 2025 16:59

test(ocapn): test handoff receive in descriptors

5fb2843

chore(ocapn): format package.json dependencies

dc2272b

kriskowal approved these changes Jun 10, 2025

View reviewed changes

kumavis mentioned this pull request Sep 18, 2025

feat(ocapn): Minimum viable OCapN #2962

Merged

kumavis closed this Sep 18, 2025

kumavis deleted the kumavis/ocapn-python-test-suite branch May 1, 2026 06:48

-export const getSelectorName = selector => {
-  if (selector[PASS_STYLE] !== 'selector') {
-    throw Error(`Selector expected, got ${typeof selector}`);
-  }
-  return selector[Symbol.toStringTag];
-};
+export const methodNameForSymbol = symbol => {
+  if (symbol[PASS_STYLE] !== 'symbol') {
+    throw Error(`Selector expected, got ${typeof symbol}`);
+  }
+  tag = symbol[Symbol.toStringTag];
+  if (tag === '@@asyncIterator') {
+    return Symbol.asyncIterator;
+  } else if (tag === 'then') {
+     return Symbol.for('then'); //  import { THEN } from "@endo/pass-style";
+  } else if (tag === 'constructor') {
+     return Symbol.for('constructor'); //  import { CONSTRUCTOR } from "@endo/pass-style";
+  } else {
+    return tag;
+  }
+};

		import { PASS_STYLE } from '@endo/pass-style';

		const { freeze, prototype: ObjectPrototype, create } = Object;

Uh oh!

Conversation

kumavis commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumavis commented Apr 29, 2025

Uh oh!

erights commented Apr 29, 2025

Uh oh!

erights commented Apr 29, 2025

Uh oh!

kriskowal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erights Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kriskowal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumavis Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumavis commented Sep 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kumavis commented Apr 28, 2025 •

edited

Loading

erights Jun 10, 2025 •

edited

Loading

kumavis Sep 18, 2025 •

edited

Loading