android-tools: init at 31.0.0p1 by primeos · Pull Request #124992 · NixOS/nixpkgs

primeos · 2021-05-30T20:25:43Z

TODO:
[ 52%] Built target fipsmodule
[ 52%] Generating chacha/chacha-x86_64.S
[ 52%] Generating test/trampoline-x86_64.S
[ 52%] Generating cipher_extra/aes128gcmsiv-x86_64.S
[ 52%] Generating cipher_extra/chacha20_poly1305_x86_64.S
[ 52%] Generating err_data.c
go: golang.org/x/[email protected]: Get "https://proxy.golang.org/golang.org/x/crypto/@v/v0.0.0-20200622213623-75b288015ac9.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:56418->[::1]:53: read: connection refused
make[2]: *** [vendor/boringssl/crypto/CMakeFiles/crypto.dir/build.make:104: vendor/boringssl/crypto/err_data.c] Error 1
make[2]: *** Deleting file 'vendor/boringssl/crypto/err_data.c'
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:1351: vendor/boringssl/crypto/CMakeFiles/crypto.dir/all] Error 2
make: *** [Makefile:171: all] Error 2
builder for '/nix/store/yib15nh8kzgrf94ijbgsjmbppcs59jij-android-tools-31.0.0p1.drv' failed with exit code 2
error: build of '/nix/store/yib15nh8kzgrf94ijbgsjmbppcs59jij-android-tools-31.0.0p1.drv' failed

Motivation for this change

See #75603 and #75603 (comment). This let's us finally build the Android tools from source.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Added a release notes entry if the change is major or breaking
Fits CONTRIBUTING.md.

primeos · 2021-05-30T20:34:33Z

The good news is that only the vendored BoringSSL seems to use Go modules (./vendor/boringssl/go.sum) but unfortunately we seem to lack an equivalent of #112500 for Go. We could use a workaround but since this'll likely affect more and more Nixpkgs packages it would be nice if we could implement a general abstraction. I'll open a dedicated issue for that. Edit: #124993.

musfay · 2021-05-30T21:08:05Z

@primeos simg2img, img2simg and append2simg binaries are already provided by simg2img package.

primeos · 2021-05-30T21:17:06Z

@musfay good to know. We should probably still use the bundled ones to avoid version conflicts (if they're used by the other tools) but then we should at least use lowPrio to avoid conflicts with the simg2img package or move those binaries out of bin/.

IvarWithoutBones · 2021-05-31T00:12:44Z

pkgs/tools/misc/android-tools/default.nix

+  pname = "android-tools";
+  version = "31.0.0p1";
+
+  src = fetchurl {


Could this be using fetchFromGitHub?

We need to use release tarball as the repo only have build-system and documentation. Upstream doesn't vendorize platform-tools source code in repo. They use CPack to create tarball from used source files.
https://github.com/nmeum/android-tools/tree/master/vendor

How many sources would we need to declare in Nix to achieve the same? I'd prefer if all sources were declared transparently instead of relying on an opaque tarball.

Well in that case we need to checkout some of AOSP repositories. Those repositories are actually HUGE and Google doesn't provide snapshots from tags. Checking out repositories takes nearly 1 hour on my computer.

The whole tree is obviously huge but we shouldn't need all of it. That's why I was asking.

We can declare the source files but there is 13-14 repos and lots of files all aroud the tree. Leaving the tracking job to CPack would be much easier. I don't agree tarballs are opaque as you can reproduce them. We can create our tarballs or vendorize CPack output in a repository.

Is it legally OK if we mirror some of Google's repositories on other platforms like this one?

there is 13-14 repos

Yeah, that's a bit much.

I don't agree tarballs are opaque as you can reproduce them.

The problem with that is that the reproduction is not transparent and in this case probably not even reproducible at a different point in time.
You don't know anything about how CPack was made nor which version of the sources it fetched.
Google's repo doesn't have a single "revision" or an equivalent; it's an aggregation of whatever happens to be at the top of a few hundred repos at a certain point in time.
It'd only become transparent if the generation was declared in Nix because then you'd have to lock the exact revision and hash of every included repo.

As I said though, too many repos. I was hoping for, like, 5 which would've been a feasible amount to maintain ourselves.

Is it legally OK if we mirror some of Google's repositories

The APL2 allows you to redistribute source code in any way you like but, why?

I don't agree tarballs are opaque as you can reproduce them.

The problem with that is that the reproduction is not transparent and in this case probably not even reproducible at a different point in time.
You don't know anything about how CPack was made nor which version of the sources it fetched.
Google's repo doesn't have a single "revision" or an equivalent; it's an aggregation of whatever happens to be at the top of a few hundred repos at a certain point in time.
It'd only become transparent if the generation was declared in Nix because then you'd have to lock the exact revision and hash of every included repo.

It makes sense now. Thanks for the definition!

As I said though, too many repos. I was hoping for, like, 5 which would've been a feasible amount to maintain ourselves.

Yep there is lots of repos.

Is it legally OK if we mirror some of Google's repositories

The APL2 allows you to redistribute source code in any way you like but, why?

Just asked, offtopic.

primeos · 2021-05-31T20:13:41Z

I haven't done any extensive testing yet but while implementing a workaround for fetching the Go modules I've noticed that setting GOFLAGS = [ "-mod=vendor" ]; is enough and that the Go modules are apparently not required (at least the build succeeds without them).

lib/licenses.nix

musfay · 2021-06-01T06:17:31Z

failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied
make[2]: *** [vendor/boringssl/crypto/CMakeFiles/crypto.dir/build.make:104: vendor/boringssl/crypto/err_data.c] Error 1
make[2]: *** Deleting file 'vendor/boringssl/crypto/err_data.c'
make[1]: *** [CMakeFiles/Makefile2:1351: vendor/boringssl/crypto/CMakeFiles/crypto.dir/all] Error 2
make: *** [Makefile:171: all] Error 2
builder for '/nix/store/7zfz1im7k03y87hb051mg7b67hr5irrn-android-tools-31.0.0p1.drv' failed with exit code 2
error: build of '/nix/store/7zfz1im7k03y87hb051mg7b67hr5irrn-android-tools-31.0.0p1.drv' failed

Oops.

Atemu · 2021-06-01T06:49:30Z

You'll probably have to set some Go environment variable to use /build for the build instead of $HOME.

musfay · 2021-06-01T06:53:28Z

Ofborg fails too. We can create adb-boringssl package with required revision of boringssl as it's only required by adb. Using nixpkgs's boringssl also fails.

primeos · 2021-06-01T11:17:46Z

Thanks, it looks like the Nix build sandboxing on my non-NixOS build host isn't properly working (anymore?).
I've built it on a NixOS host now.

Edit: Force pushed again to fix the AArch64 build:

[ 50%] Building C object vendor/CMakeFiles/libext2fs.dir/e2fsprogs/misc/create_inode.c.o
In file included from /build/android-tools-31.0.0p1/vendor/adb/client/file_sync_client.cpp:46:
/build/android-tools-31.0.0p1/vendor/adb/compression_utils.h:21:10: fatal error: span: No such file or directory
   21 | #include <span>
      |          ^~~~~~
compilation terminated.
make[2]: *** [vendor/CMakeFiles/libadb.dir/build.make:205: vendor/CMakeFiles/libadb.dir/adb/client/file_sync_client.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....

musfay · 2021-06-02T05:53:57Z

I can confirm that everything works as expected. Flashed some images via fastboot. Tested adb through WiFi, pushed some files, had no issues.

lowPrio is used to avoid collisions with the simg2img package. Licensing information is in share/licenses/android-tools/AOSP_LICENSE.

primeos · 2021-06-02T19:05:43Z

@musfay thanks for the testing! I've also used adb today and didn't notice any issues.

I'll merge this now (it works, we finally have a FLOSS package for adb and fastboot, and we can still improve the Nixpkgs expressions later - e.g. to generate the tarballs in a reproducible way).

Edit: The Darwin build failure (just for future reference):
https://logs.nix.ci/?key=nixos/nixpkgs.124992&attempt_id=796eb353-3873-4198-a1b7-1763192f1153

[ 20%] Building CXX object vendor/CMakeFiles/liblog.dir/logging/liblog/logger_write.cpp.o
/tmp/nix-build-android-tools-31.0.0p1.drv-0/android-tools-31.0.0p1/vendor/logging/liblog/logger_write.cpp:140:36: error: use of undeclared identifier 'getprogname'
  static std::string default_tag = getprogname();
                                   ^
/tmp/nix-build-android-tools-31.0.0p1.drv-0/android-tools-31.0.0p1/vendor/logging/liblog/logger_write.cpp:235:3: error: use of undeclared identifier 'pthread_threadid_np'
  pthread_threadid_np(NULL, &tid);
  ^
2 errors generated.
make[2]: *** [vendor/CMakeFiles/liblog.dir/build.make:134: vendor/CMakeFiles/liblog.dir/logging/liblog/logger_write.cpp.o] Error 1

Atemu · 2021-06-02T20:24:24Z

Should we update programs.adb to use adb from here instead?

primeos · 2021-06-02T20:50:51Z

Sounds like a good idea to me (cc @Mic92).
Interestingly androidenv.androidPkgs_9_0.platform-tools doesn't have meta.license set which seems like a bug...

Atemu · 2022-04-22T07:37:28Z

pkgs/top-level/all-packages.nix

  analog = callPackage ../tools/admin/analog {};

+  android-tools = lowPrio (callPackage ../tools/misc/android-tools {
+    stdenv = if stdenv.targetPlatform.isAarch64 then gcc10Stdenv else stdenv;


Why is this needed?

Was likely required for #124992 (comment) ;)

This was referenced May 30, 2021

go: Add an abstraction to fetch Go modules #124993

Open

Build Android tools from source (adb, SDK, Android Studio, etc.) #75603

Open

primeos force-pushed the android-tools branch from c8a6ee5 to 6c89a3a Compare May 30, 2021 21:06

primeos force-pushed the android-tools branch from 6c89a3a to e17a879 Compare May 30, 2021 21:19

IvarWithoutBones suggested changes May 31, 2021

View reviewed changes

primeos force-pushed the android-tools branch from e17a879 to c96adfe Compare May 31, 2021 20:10

primeos marked this pull request as ready for review May 31, 2021 20:13

primeos force-pushed the android-tools branch from c96adfe to 9e8b9ff Compare May 31, 2021 20:31

primeos requested a review from alyssais as a code owner May 31, 2021 20:31

primeos commented May 31, 2021

View reviewed changes

lib/licenses.nix Outdated Show resolved Hide resolved

primeos force-pushed the android-tools branch from 9e8b9ff to 043f16e Compare June 1, 2021 11:16

primeos force-pushed the android-tools branch from 043f16e to 8a6c8ab Compare June 1, 2021 12:22

android-tools: init at 31.0.0p1

c57b309

lowPrio is used to avoid collisions with the simg2img package. Licensing information is in share/licenses/android-tools/AOSP_LICENSE.

primeos force-pushed the android-tools branch from 8a6c8ab to c57b309 Compare June 2, 2021 10:08

ofborg bot added the 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. label Jun 2, 2021

ofborg bot removed 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. labels Jun 2, 2021

primeos merged commit ed8115e into NixOS:master Jun 2, 2021

primeos mentioned this pull request Jun 5, 2021

Get adb and fastboot builds from source (via robotnix) #101425

Closed

Atemu mentioned this pull request Jun 5, 2021

nixos/anbox: use mainline drivers when available #102341

Closed

10 tasks

samueldr mentioned this pull request Jun 6, 2021

anbox: fixes and upgrades #125600

Closed

14 tasks

xaverdh mentioned this pull request Aug 28, 2021

nixos/adb: switch to android-tools built from source #136040

Merged

12 tasks

Atemu reviewed Apr 22, 2022

View reviewed changes

Uh oh!

Conversation

primeos commented May 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation for this change

Things done

Uh oh!

primeos commented May 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

musfay commented May 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

primeos commented May 30, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musfay May 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musfay May 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

musfay Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

primeos commented May 31, 2021

Uh oh!

Uh oh!

musfay commented Jun 1, 2021

Uh oh!

Atemu commented Jun 1, 2021

Uh oh!

musfay commented Jun 1, 2021

Uh oh!

primeos commented Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

musfay commented Jun 2, 2021

Uh oh!

primeos commented Jun 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Atemu commented Jun 2, 2021

Uh oh!

primeos commented Jun 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

primeos commented May 30, 2021 •

edited

Loading

primeos commented May 30, 2021 •

edited

Loading

musfay commented May 30, 2021 •

edited

Loading

musfay May 31, 2021 •

edited

Loading

musfay May 31, 2021 •

edited

Loading

musfay Jun 1, 2021 •

edited

Loading

primeos commented Jun 1, 2021 •

edited

Loading

primeos commented Jun 2, 2021 •

edited

Loading