Add polykey agent by robert-cronin · Pull Request #53 · MatrixAI/Polykey

robert-cronin · 2020-07-03T09:54:40Z

This PR will add the pk agent and IPC communications between the agent and any clients. It will also introduce lockfiles (not done) for preventing two pk agents from operating on the same keynode at once.

Fixes #50
Fixes #54
Fixes #59

robert-cronin · 2020-07-07T07:01:07Z

branch has been rebased onto master

robert-cronin · 2020-07-09T11:57:41Z

It seems like there is a lot of redundancy in defining the communication protocol between cli and agent. At first I tried to simply define the cli on the agent and then pass the arguments via the unix domain socket but I ran into some issues especially with controlling the UX on the cli so I went with defining a messaging protocol between the agent and any potential front end. I ended up having what seems to be a lot of redundancy so if you have any tips on how to reduce this, it would be greatly appreciated, unless this is the best way to do it?

CMCDragonkai · 2020-07-09T14:57:05Z

What do you mean by redundancy here? I suppose grpc has protoc files? Or do you mean you want to "generate" the marshalling code. There are libraries to do this.

LICENSE.sig

CMCDragonkai · 2020-07-09T14:58:38Z

This MR is too big for me to review. I mean that literally, my browser doesn't load it properly. Won't be able to review from this interface. Are you able to either split it up into separate MRs, if not, we can instead perhaps do this by just merging and it and going through the pieces.

robert-cronin · 2020-07-10T02:54:51Z

What do you mean by redundancy here? I suppose grpc has protoc files? Or do you mean you want to "generate" the marshalling code. There are libraries to do this.

Yes I just mean there is a lot of code to convert proto files into javascript and back again. I am going to take a look into pbts which is the protobufjs cli for generating bundled type definition and javascript files that describe the proto files directly.

robert-cronin · 2020-07-10T02:56:30Z

This MR is too big for me to review. I mean that literally, my browser doesn't load it properly. Won't be able to review from this interface. Are you able to either split it up into separate MRs, if not, we can instead perhaps do this by just merging and it and going through the pieces.

I didn't realize it was this big! there are a few things I can do to make it smaller.
Using pbts to replace all the code for converting the proto files into javascript should reduce it significantly.

CMCDragonkai · 2020-07-13T09:56:21Z

There's a conflict in package.json. This commit will need to be rebased.

.env.example

CMCDragonkai · 2020-07-13T09:59:38Z

README.md

+All `.proto` files are stored in the the `proto` directory. JavaScript and type definition files are build using the following command:
+
+```
+npm run build:proto


Should mention how this is integrated into the webpack build system?

This is a work in progress since grpc_tools_node_protoc doesn't seem to work on nixos, might need your help on that one, it should install with grpc-tools but grpc_tools_node_protoc doesn't seem to be added to the path or installed properly

Does it require native compilation? Is the failure occurring during npm install? If so, there are ways around it...

Some npm installs require compilation. This can be done in a nix-shell if the nix-shell has all the tooling required. There's quite a bit of stuff like node-pre-gyp or whatever. And python2. I did this before when I was trying to put dat and other binary npm stuff into Nixpkgs: svanderburg/node2nix#71 and NixOS/nixpkgs#32940.

However that's for using the packages with CLI executables in Nixpkgs itself. That may be the right idea to do if we can use the protoc compilers from shell.nix and that means pushing those packages to upstream (but first in our overlay).

Of course bringing all the tools needed to do compilation for Node is a bit of a mess.

BTW have you seen this: https://github.com/agreatfool/grpc_tools_node_protoc_ts? It builds on top of the grpc_tools_node_protoc to generate typescript code instead of just JS code.

The node-pre-gyp would be best used as part of creating a new nodePackage in nixpkgs upstream. See the PR I created for dat. It's a similar process.

ENOENT means something doesn't exist. Perhaps the tool is trying to execute something that doesn't exist. This can happen due to absolute paths or paths that don't expect a NixOS environment. I had a similar problem occur when trying to package dvc for Nixpkgs: treeverse/dvc#1509

You might want to chase down that error, and find out exaxtly what the path is failing.

Does this path exist? /home/robbie/Documents/github/js-polykey/node_modules/grpc-tools/bin/protoc.

Recommend reading: https://github.com/svanderburg/node2nix

And: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/node.section.md

I suggest putting up an issue in the grpc-node repo. I did this before for the dvc repo regarding "Packaging for NixOS". And put down your woes there. Someone there might know something about it and help. The dvc guys were helpful before too.

src/lib/peers/PeerManager.ts

webpack.config.ts

CMCDragonkai · 2020-07-14T11:11:47Z

I'm guessing there's only 1 thing blocking this merge atm:

The protoc issue and native compilation of the dependency

This problem is really about understanding how to use nix.

Once this is done, we merge and then start reviewing each part in detail... in preparation for the UI work and the demo integration into Ouroboros and/or Matrix Relay VPN. These 2 will require you to talk to @nzhang-zh and @DrFacepalm

CMCDragonkai · 2020-07-14T11:12:55Z

In the mean time, can we also get a demo of the pk command line.

If you can record that, with https://asciinema.org/ and show that to the team for the all the common use-cases: generating secrets, passing secrets around, renaming stuff, sharing stuff... etc etc.

robert-cronin added the development Standard development label Jul 3, 2020

robert-cronin requested a review from CMCDragonkai July 3, 2020 09:54

robert-cronin self-assigned this Jul 3, 2020

robert-cronin linked an issue Jul 7, 2020 that may be closed by this pull request

Prevent webpack from compiling lib code thrice for agent/cli/lib #54

Closed

robert-cronin force-pushed the ipc branch 2 times, most recently from 5da4fd7 to 1218646 Compare July 7, 2020 07:00

CMCDragonkai reviewed Jul 9, 2020

View reviewed changes

LICENSE.sig Outdated Show resolved Hide resolved

robert-cronin force-pushed the ipc branch 3 times, most recently from bafea7f to bd93958 Compare July 13, 2020 05:09

CMCDragonkai reviewed Jul 13, 2020

View reviewed changes

.env.example Show resolved Hide resolved

CMCDragonkai reviewed Jul 13, 2020

View reviewed changes

src/lib/peers/PeerManager.ts Outdated Show resolved Hide resolved

CMCDragonkai reviewed Jul 13, 2020

View reviewed changes

webpack.config.ts Outdated Show resolved Hide resolved

robert-cronin force-pushed the ipc branch 2 times, most recently from 47099bf to c98a48b Compare July 16, 2020 06:11

robert-cronin added 4 commits July 16, 2020 14:16

Add Polykey agent - src files

5565933

Add Polykey agent - test files

19973d8

Add Polykey agent - dist & docs files

e6caeb6

Add Polykey agent - misc. files

a1e2353

robert-cronin force-pushed the ipc branch from c98a48b to a1e2353 Compare July 16, 2020 06:18

robert-cronin merged commit e216269 into master Jul 16, 2020

robert-cronin deleted the ipc branch July 16, 2020 06:29

amydevs mentioned this pull request Nov 8, 2023

CLI Beta Launch MatrixAI/Polykey-CLI#40

Closed

Conversation

robert-cronin commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robert-cronin commented Jul 7, 2020

Uh oh!

robert-cronin commented Jul 9, 2020

Uh oh!

CMCDragonkai commented Jul 9, 2020

Uh oh!

Uh oh!

CMCDragonkai commented Jul 9, 2020

Uh oh!

robert-cronin commented Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robert-cronin commented Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CMCDragonkai commented Jul 13, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CMCDragonkai commented Jul 14, 2020

Uh oh!

CMCDragonkai commented Jul 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

robert-cronin commented Jul 3, 2020 •

edited

Loading

robert-cronin commented Jul 10, 2020 •

edited

Loading

robert-cronin commented Jul 10, 2020 •

edited

Loading