Add PeerTube package (v3.1.0), without module by stevenroose · Pull Request #119319 · NixOS/nixpkgs

stevenroose · 2021-04-13T11:26:24Z

Split off of #106492 with only the pkg addition, not the module. Will open a separate MR for the module. Changed since #106492 is that I bumped version to v3.1.0.

SuperSandro2000 · 2021-04-13T12:02:36Z

pkgs/servers/peertube/server.nix

Node is used in scripts/build/server.sh so it should be in nativeBuildInputs.

Oh cool thanks. I pushed a change already that makes sure it's using the same nodejs version as default.nix.

SuperSandro2000 · 2021-04-13T12:02:50Z

pkgs/servers/peertube/default.nix

Suggested change

description = "A free software to take back control of your videos";

longDescription = ''

description = "A free software to take back control of your videos";

longDescription = ''

SuperSandro2000 · 2021-04-13T12:02:58Z

pkgs/servers/peertube/default.nix

Suggested change

'';

license = lib.licenses.agpl3Plus;

homepage = "https://joinpeertube.org/";

platforms = lib.platforms.unix;

maintainers = with lib.maintainers; [ immae stevenroose ];

'';

license = lib.licenses.agpl3Plus;

homepage = "https://joinpeertube.org/";

platforms = lib.platforms.unix;

maintainers = with lib.maintainers; [ immae stevenroose ];

SuperSandro2000 · 2021-04-13T12:03:15Z

pkgs/servers/peertube/default.nix

Suggested change

rm -rf dist && cp -a ${server.dist}/dist dist

rm -rf client/dist && cp -a ${client.dist}/dist client/

rm -r dist

cp -a ${server.dist}/dist dist

rm -r client/dist

cp -a ${client.dist}/dist client/

SuperSandro2000 · 2021-04-13T12:04:04Z

pkgs/servers/peertube/default.nix

Suggested change

, nodejs ? pkgs.nodejs-14_x

, nodejs

What's wrong with this line? It's currently the default, but I wouldn't expect the team to keep track of which pkgs are expecting which defaults. Or is it bad that we're also passing it in in all-packages?

I think you misunderstood what the ? operator does. name ? e provides a default value in the attribute set argument which is used in the case the name is missing.

nix-repl> f = { n ? "default" }: n nix-repl> f { n = "non default"; } "non default" nix-repl> f { } "default"

That's it. In your case, the attribute set argument is generated by the callPackage mechanism, not manually. If callPackage knows a package named nodejs it will provide it to your function. This package name will always be around so this ? construct does not what you intended to do.
If you really want 14.x version, just use nodejs-14_x in your attribute set or override nodejs in the callPackage call like you did :)
https://nixos.org/manual/nix/stable/#ss-functions

Then you can remove light ? null, pkgs.

Izorkin · 2021-04-13T12:32:13Z

@stevenroose error build:

> [email protected] install /build/node_modules/bcrypt
> node-pre-gyp install --fallback-to-build

sh: /build/node_modules/bcrypt/node_modules/.bin/node-pre-gyp: /usr/bin/env: bad interpreter: No such file or directory
npm ERR! code ELIFECYCLE
npm ERR! errno 126
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 126
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/yarn_home/.npm/_logs/2021-04-13T12_30_32_587Z-debug.log

builder for '/nix/store/q9w24n8rvhwwvqd2p9ifg8s1b5rpfily-peertube-server-yarn-modules-3.1.0.drv' failed with exit code 126

Synthetica9 · 2021-04-13T12:46:20Z

Commit messages should be [pkg]: init at [version]. So, in this case: peertube: init at 3.1.0. See here: https://nixos.org/manual/nixpkgs/stable/#submitting-changes-making-patches

stevenroose · 2021-04-13T13:08:12Z

@stevenroose error build:

> [email protected] install /build/node_modules/bcrypt
> node-pre-gyp install --fallback-to-build

sh: /build/node_modules/bcrypt/node_modules/.bin/node-pre-gyp: /usr/bin/env: bad interpreter: No such file or directory
npm ERR! code ELIFECYCLE
npm ERR! errno 126
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 126
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/yarn_home/.npm/_logs/2021-04-13T12_30_32_587Z-debug.log

builder for '/nix/store/q9w24n8rvhwwvqd2p9ifg8s1b5rpfily-peertube-server-yarn-modules-3.1.0.drv' failed with exit code 126

Strange, that works for me. If I change the buildInputs with nativeBuildInputs, it stops working for me..

stevenroose · 2021-04-13T13:11:33Z

Pushed some changes:

add nodejs export so that module can use the same version
inherit nodejs from default.nix into server.nix and client.nix
@SuperSandro2000 's comments (I don't understand the difference between buildInputs and nativeBuildInputs)
fixed commit message as per @Synthetica9 's suggestion

Izorkin · 2021-04-13T13:18:27Z

@stevenroose error build with sandbox true.

stevenroose · 2021-04-13T14:17:27Z

@stevenroose error build with sandbox true.

How do you do that? I'm trying nix-build --option build-use-sandbox true . -A peertube --verbose but that seems to work..

Izorkin · 2021-04-13T14:20:58Z

Used this configuration:

  imports = [
...
    <nixpkgs/nixos/modules/profiles/hardened.nix>
  ];

  nix = {
...
    useSandbox = true;

stevenroose · 2021-04-13T14:21:39Z

I'm just running nix-build on a non-NixOS system, so I have no idea where I would put that.. In /etc/nix/nix.conf or so?

stevenroose · 2021-04-13T14:22:13Z

I do have this there:

build-sandbox-paths = /bin/sh=/nix/store/ily14d68xl11cnbbkf9svwnzwsrrnzah-bash-4.4-p23/bin/bash /usr/bin/bash=/nix/store/ily14d68xl11cnbbkf9svwnzwsrrnzah-bash-4.4-p23/bin/bash /usr/bin/bzip2=/nix/store/mymkz6qhf46j6j7nanilyd15cgj5p9br-bzip2-1.0.6.0.1-bin/bin/bzip2 /usr/bin/env=/nix/store/8h3bgs39dhwkbyb8ii1fbb6zrcp4kd1p-coreutils-8.31/bin/coreutils /usr/bin/gzip=/nix/store/rvybssv5lxm6rfdbqmkiapv91hrl5cmx-gzip-1.10/bin/gzip /usr/bin/mkdir=/nix/store/8h3bgs39dhwkbyb8ii1fbb6zrcp4kd1p-coreutils-8.31/bin/coreutils /usr/bin/mv=/nix/store/8h3bgs39dhwkbyb8ii1fbb6zrcp4kd1p-coreutils-8.31/bin/coreutils /usr/bin/tar=/nix/store/znsa81k72j5aaclg1crv5575n4w726fc-gnutar-1.32/bin/tar /usr/bin/tr=/nix/store/8h3bgs39dhwkbyb8ii1fbb6zrcp4kd1p-coreutils-8.31/bin/coreutils /usr/bin/xz=/nix/store/ch63h3ysgnsyjgw0inqplfh5af2bqhxb-xz-5.2.4-bin/bin/xz

Izorkin · 2021-04-13T14:30:47Z

I'm just running nix-build on a non-NixOS system, so I have no idea where I would put that.. In /etc/nix/nix.conf or so?

Yes, you need to write there - useSandbox = true;
I don't know how to reproduce the error on non-NixOS system.

ThibautMarty · 2021-04-14T10:27:41Z

Thanks for the effort into packaging PeerTube.
However, I don't understand the package's complexity around yarn. It seems unnecessary to me.

I ran in the PeerTube source directory on commit e81af30 (I admit v3.1.0 seems to have a somehow broken yarn.lock file, but that could be fixed):

nix-shell -p yarn yarn2nix nodejs python3
yarn install --production
yarn2nix > yarn.nix

(Be sure to have a recent yarn2nix with nix-community/yarn2nix#130, I think it solves the problem with http-node dependency but I didn't checked).

Then I copied yarn.nix to the nixpkgs package's directory, plus this default.nix file:

{ stdenv, lib, fetchFromGitHub, mkYarnPackage, yarn2nix-moretea }:

mkYarnPackage rec {
  pname = "peertube";
  version = "3.1.0-e81af30";

  src = fetchFromGitHub {
    owner = "Chocobozzz";
    repo = "PeerTube";
    rev = "e81af3000f1881ec5041678ae54b402b0314bd42";
    sha256 = "0cwgla4w27cbygf8rkjpr0bqgqwbb332yvla1jbhrgd86rkn47x9";
  };

  yarnNix = ./yarn.nix;
  yarnFlags = yarn2nix-moretea.defaultYarnFlags ++ [ "--production" ];
}

This builds correctly on top of your commit.
However, the package is somewhat broken: the $out/bin/peertube symlink is broken. This is probably something related to the dist phase? But I think it can be fixed without the yarn-related scaffolding.
The dependencies (youtube-dl?) can be added if needed (or maybe they should just be in the future NixOS' module $PATH).
Please tell me if I'm wrong or if I missed something (client? dist?)

stevenroose · 2021-04-14T17:56:21Z

Thanks for the effort into packaging PeerTube.
However, I don't understand the package's complexity around yarn. It seems unnecessary to me.

I ran in the PeerTube source directory on commit e81af30 (I admit v3.1.0 seems to have a somehow broken yarn.lock file, but that could be fixed):
nix-shell -p yarn yarn2nix nodejs python3
yarn install --production
yarn2nix > yarn.nix
(Be sure to have a recent yarn2nix with nix-community/yarn2nix#130, I think it solves the problem with http-node dependency but I didn't checked).

Then I copied yarn.nix to the nixpkgs package's directory, plus this default.nix file:
{ stdenv, lib, fetchFromGitHub, mkYarnPackage, yarn2nix-moretea }:

mkYarnPackage rec {
  pname = "peertube";
  version = "3.1.0-e81af30";

  src = fetchFromGitHub {
    owner = "Chocobozzz";
    repo = "PeerTube";
    rev = "e81af3000f1881ec5041678ae54b402b0314bd42";
    sha256 = "0cwgla4w27cbygf8rkjpr0bqgqwbb332yvla1jbhrgd86rkn47x9";
  };

  yarnNix = ./yarn.nix;
  yarnFlags = yarn2nix-moretea.defaultYarnFlags ++ [ "--production" ];
}
This builds correctly on top of your commit.
However, the package is somewhat broken: the $out/bin/peertube symlink is broken. This is probably something related to the dist phase? But I think it can be fixed without the yarn-related scaffolding.
The dependencies (youtube-dl?) can be added if needed (or maybe they should just be in the future NixOS' module $PATH).
Please tell me if I'm wrong or if I missed something (client? dist?)

Tbh I don't understand too much about yarn. But I tried for several weeks to get this to work from scratch. There's some dependencies in the package.json that are forced to use a certain version and that didn't seem to work with yarn install --offline. The approach in this package wsa done by @immae after I struggled for multiple weeks.

There's indeed also the patch for the yarn.lock and the bcrypt special need.

Also, in your example, you don't have the server and the client dists built both.

ThibautMarty · 2021-04-19T10:13:33Z

I see. I took a look and PeerTube seems to need two (or three) steps to be installed:

yarn install
(npm install)
npm run build

Step 1 creates the node_modules directories in the root directory and the client directory.

My strategy would be to keep a single mkYarnPackage with a postInstall for the extra steps:

  pkgConfig.peertube = {
    postInstall = ''
      patchShebangs --build ../..
      npm install
      npm run build
    '';
  };

It's possible to use the Nix code generated node2nix to generate Nix files allowing to run the npm commands in a sandboxed environment. (see https://github.com/svanderburg/node2nix#using-the-nodejs-environment-in-other-nix-derivations)
Indeed, some packages needs fixes (bcrypt) and I failed at this stage…

One thing is certain: you need to test this package with sandboxing enabled, or you won't catch 100% of errors related to Yarn/NPM doing (non fixed-output) network requests.

Izorkin · 2021-05-11T14:52:58Z

@stevenroose version 3.2.0-rc.1 released

mohe2015 · 2021-05-12T19:42:07Z

@stevenroose https://github.com/mohe2015/nixpkgs/tree/my-add-peertube-pkg is a working version with sandboxing (though without the suggestions here). But it should be pretty simple to adjust your current code to this as I only changed one or two lines.

Izorkin · 2021-05-27T18:42:31Z

Version 3.2.0 released!

Izorkin · 2021-05-30T06:36:48Z

cc @stevenroose

Izorkin · 2021-05-31T14:24:27Z

How to build cli tools - https://docs.joinpeertube.org/maintain-tools?id=installation

happysalada · 2021-09-10T04:35:11Z

Just a small update on this.
I tried to update this PR (on my local) and build it with version 3.4.0
However I encountered 2 errors both on the server and the client.
I posted those errors upstream, I don't think they are related to nix.
Chocobozzz/PeerTube#4393
Chocobozzz/PeerTube#4394

mohe2015 · 2021-09-10T10:17:04Z

We should really close this as #119110 contains the current state and nothing happened here in the last few months

maintainers: Add Steven Roose

e34913d

stevenroose mentioned this pull request Apr 13, 2021

Second attempt at adding PeerTube package #106492

Closed

SuperSandro2000 reviewed Apr 13, 2021

View reviewed changes

stevenroose force-pushed the peertube-pkg branch from 8095117 to 71b22ee Compare April 13, 2021 12:55

peertube: init at 3.1.0

db529c2

stevenroose force-pushed the peertube-pkg branch from 71b22ee to db529c2 Compare April 13, 2021 13:09

stevenroose mentioned this pull request Apr 13, 2021

nixos/peertube: init service #119110

Merged

10 tasks

mohe2015 closed this Sep 10, 2021

-    description = "A free software to take back control of your videos";
-    longDescription = ''
+    description = "A free software to take back control of your videos";
+    longDescription = ''

-    '';
-    license = lib.licenses.agpl3Plus;
-    homepage = "https://joinpeertube.org/";
-    platforms = lib.platforms.unix;
-    maintainers = with lib.maintainers; [ immae stevenroose ];
+    '';
+    license = lib.licenses.agpl3Plus;
+    homepage = "https://joinpeertube.org/";
+    platforms = lib.platforms.unix;
+    maintainers = with lib.maintainers; [ immae stevenroose ];

-    rm -rf dist && cp -a ${server.dist}/dist dist
-    rm -rf client/dist && cp -a ${client.dist}/dist client/
+    rm -r dist
+    cp -a ${server.dist}/dist dist
+    rm -r client/dist
+    cp -a ${client.dist}/dist client/

Uh oh!

Comments

Conversation

stevenroose commented Apr 13, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThibautMarty Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Izorkin commented Apr 13, 2021

Uh oh!

Synthetica9 commented Apr 13, 2021

Uh oh!

stevenroose commented Apr 13, 2021

Uh oh!

stevenroose commented Apr 13, 2021

Uh oh!

Izorkin commented Apr 13, 2021

Uh oh!

stevenroose commented Apr 13, 2021

Uh oh!

Izorkin commented Apr 13, 2021

Uh oh!

stevenroose commented Apr 13, 2021

Uh oh!

stevenroose commented Apr 13, 2021

Uh oh!

Izorkin commented Apr 13, 2021

Uh oh!

ThibautMarty commented Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevenroose commented Apr 14, 2021

Uh oh!

ThibautMarty commented Apr 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Izorkin commented May 11, 2021

Uh oh!

mohe2015 commented May 12, 2021

Uh oh!

Izorkin commented May 27, 2021

Uh oh!

Izorkin commented May 30, 2021

Uh oh!

Izorkin commented May 31, 2021

Uh oh!

happysalada commented Sep 10, 2021

Uh oh!

mohe2015 commented Sep 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

ThibautMarty Apr 14, 2021 •

edited

Loading

ThibautMarty commented Apr 14, 2021 •

edited

Loading

ThibautMarty commented Apr 19, 2021 •

edited

Loading