madbfs

madbfs (Modern adb Filesystem, formerly adbfsm) is a userspace filesystem for Android via adb built using libfuse.

Preview

out.mp4

What's happening above:

• viewing a video on Waydroid from host and copying it to host, and
• copying a picture from host to Waydroid.

Motivation

I want to manage my Android phone storage from my computer without using MTP.

This project is inspired by the adbfs-rootless project. While adbfs-rootless works most of the time under light load it has reliability issues. Under high load (such as when a thumbnailer service is running) adbfs-rootless tends to crash, making it unusable until it is forcefully unmounted and its cache cleaned.

The file I/O approach of adbfs-rootless is also similar to that of MTP, in which each file is first pulled from device (using adb pull) before any operations like reading and writing performed. If a write operation is performed, the file is then pushed back into the device (using adb push). This approach limits the size of the file that can be handled before the cache is filled (it uses /tmp).

These limitations lead me to decide to rebuild the project from ground up in order to create a more stable, modern, and possibly faster solution.

Features

TLDR: full file and directory traversal with concurrent file streaming approach, partial read/write, and active caching.

• No root access required

  Non-rooted device will have standard file and directory access as regular user. Rooted device will have less constraints on file and directory access.

• Full file and directory traversal

  Browse the entire file tree of your Android device (subject to permission constraints).

• Read and write support

  Open, read, and write files (subject to permission constraints).

• Create and delete files and directories

  Support creating new files and directories as well as deleting them (subject to permission constraints).

• Rename and move

  Rename or move files and directory seamlessly (subject to permission constraints).

• Modify file timestamps

  Update file access and modification times.

• Automatic in-memory caching

  Recently accessed files are cached in memory using an LRU paging mechanism, allowing faster repeated access.

• Streamed file access (partial read/write)

  Read files on-demand without pulling the entire file first. Unlike MTP’s "pull-whole-file" approach, this filesystem streams data over adb, enabling efficient access to large files or specific file segments.

• Efficient resource use

  Loads only what you access, conserving memory usage and bandwidth. You can also control how much the cache stores file data.

• Concurrent file access

  Allows for concurrent access to files and directories without blocking. This ability comes out-of-the-box by virtue of using FUSE and adb.

• Flexible connection method

  madbfs offers two kinds of transport:

  • adb transport

    The simplest transport. It executes all FUSE operations by running adb shell commands like dd, stat, and touch. No additional component is required.

  • Proxy transport

    Communicates with a lightweight TCP server running natively on the Android device via a custom RPC protocol. It requires a server binary compiled for the phone architecture but has better I/O throughput than the adb transport.

  Both of the transport can work wired via USB or wireless in your local network. madbfs will automatically fall back to adb transport if the proxy transport is not available.

• Resilient to disconnections

  Stays mounted even when the device disconnects. Cached files and directories remain accessible, and full functionality resumes automatically when the connection is restored.

• Built with modern C++

  Developed in C++23 using coroutine-style asynchronous programming for clean, lightweight, high-performance I/O.

Dependencies

• madbfs and madbfs-msg

  Runtime dependencies

  • adb

  Build dependencies

  • GCC or Clang (support C++23 or above, tested on GCC 15 and GCC 16)
  • CMake (version 3.22+)
  • Ninja
  • Conan

  Library dependencies

  • Boost (Asio, Process, and JSON component)
  • libfuse
  • rapidhash
  • spdlog

• madbfs-server

  Build dependencies

  • Android NDK (support C++23 or above, tested on NDK version 29 (beta 1))
  • CMake (version 3.22+)
  • Ninja
  • Conan

  Library dependencies

  • Asio (non-Boost variant)
  • spdlog

Building

if you just want a prebuilt binaries, jump to installation

Since this project dependencies are managed by Conan, you need to make sure it's installed and configured correctly on your system (consult the documentation on how to do it).

There are two CMake project that produces executables:

• madbfs: produces madbfs and madbfs-msg
• madbfs-server: produces multiple madbfs-server for each architecture Android supports

madbfs project is compiled once for the host machine, while madbfs-server is compiled multiple times for different architectures using Android NDK. madbfs embeds the produced madbfs-server in order to simplify the software usage. This requirement makes it so that the compilation step of the project is quite strict. Because of that, I have written a handy script to automate this process.

Using script

./build.sh    # try adding --help to this script

In order to use this script, you are required have ANDROID_NDK_HOME variable defined and point to valid Android NDK.

The script above will produce build directories at this specified stage:

• madbfs: build/<Build_type>/
• madbfs-server: madbfs-server/build/android-<arch>-<build_type>/ and madbfs-server/build/android-all-<build_type>/

• madbfs executable can be obtained at build/<Build_type>/madbfs/madbfs
• madbfs-msg executable can be obtained at build/<Build_type>/madbfs-msg/madbfs-msg

When the script is invoked with --package option, it will produce a tar.gz file containing madbfs and madbfs-msg and its hash at:

• build/package-<madbfs_Build_type>-<madbfs-server_Build_type>

Manually

• madbfs-server

  While Conan is a package manager, it can also be used as toolchain for cross-compiling via profile. A profile for build environment and host environment is required. I used profiles at ./.github/workflows/ both for development and CI. If you read profile-server.ini you might notice that settings.arch and conf.tools.android:ndk_path values are not set concretely, instead it uses environment variable. This is because the profile needs to be reused for multiple architecture and not hardcode Android NDK path on the profile respectively.

  Conan uses build-host naming convention instead of host-target for cross-compilation

  For this instruction, I will use the profiles above.

  The first step is moving to madbfs-server directory from the root. Then you need to get build dependencies using Conan. At this point ANDROID_NDK_HOME and MADBFS_SERVER_ARCH (for conf.tools.android:ndk_path and settings.arch respectively) need to be define already.

  Assume the ${variables} point to valid arch, profiles, and build type (capitalized)

  export ANDROID_NDK_HOME=path/to/android/ndk
  export MADBFS_SERVER_ARCH=${arch}
  
  conan install . --build missing -pr:b "${profile_client}" -pr:h "${profile_server}" -s build_type=${build_type}

  you can also use -c:h tools.android:ndk_path=path/to/android/ndk and -s:h arch=${arch} Conan options to specify the NDK path and arch respectively instead of environment variable

  Then you can generate and compile them:

  cmake --preset conan-android-${arch}-${build_type,,}
  cmake --build --preset conan-android-${arch}-${build_type,,}

  ,, is bash parameter expansion for lowering string case

  This will produce madbfs-server executable at ./build/android-<arch>-<build_type>/madbfs-server

  Do this repeatedly for all supported arch.

• madbfs (and madbfs-msg)

  This instruction will only focus on madbfs since madbfs-msg compilation is simple and doesn't need additional configuration.

  First, move to the root of the project, then install the dependencies.

  Assume the ${variables} point to valid arch, profiles, and build type (capitalized)

  # notice that both build and host profiles point to ${profile_client} because it will be used on the build machine
  conan install . --build missing -pr:b "${profile_client}" -pr:h "${profile_client}" "&:build_type=${build_type}" -s build_type=Release

  Then you need to prepare a directory that contains the madbfs-server for all architecture. The executables should be renamed to have each of their ABI (not arch): madbfs-server-abi. This step is required because madbfs expects madbfs-server files to have specified type but on which directory.

  mkdkir -p ${servers_path}
  
  # ...
  cp ./build/android-${arch}-${build_type,,} ${servers_path}/madbfs-server-${abi}   # for each arch

  To tell where madbfs can find the server files, you need to pass MADBF_SERVER_BINARY_DIR to CMake, which is done at generation time

  cmake --preset conan-${build_type,,} -D MADBFS_SERVER_BINARY_DIR="${servers_path}"

  Building madbfs is then just this:

  cmake --build --preset conan-${build_type,,}

  The executable can be found at ./build/<Build_type>/madbfs/madbfs (and ./build/<Build_type>/madbfs-msg/madbfs-msg)

Installation

You can download madbfs from the GitHub releases page.

There is no installation step required, as the application is built statically. You can place the binaries wherever you prefer.

Installing from the AUR

madbfs can be installed using your prefered AUR helper, as with every other AUR package. Using yay, you'd run the following command as superuser:

yay -S madbfs-bin

Usage

The help message can help you start using this program

usage: madbfs [options] <mountpoint>

Options for madbfs:
    --serial=<str>         serial number of the device to mount
                             (you can omit this [detection is similar to adb])
                             (will prompt if more than one device exists)
    --root=<path>          directory to mount on device as root of the filesystem
                             (by default madbfs mounts the root path of the device)
                             (the path must be absolute and points to an existing path)
                             (if the path is a symlink, it will be resolved first)
    --log-level=<enum>     log level to use
                             (default: "warning")
                             (enum: "trace", "debug", "info", "warning", "error", "critical", "off")
    --log-file=<path>      log file to write to
                             (default: "-" for stdout)
    --cache-size=<int>     maximum size of the cache in MiB
                             (default: 256)
                             (minimum: 128)
                             (value will be rounded up to the next power of 2)
                             (ignored if 'no-cache' is provided)
    --page-size=<int>      page size for cache & transfer in KiB
                             (default: 128)
                             (minimum: 64)
                             (maximum: 4096)
                             (value will be rounded up to the next power of 2)
                             (ignored if 'no-cache' is provided)
    --ttl=<int>            set the TTL of the stat cache of the filesystem in seconds
                             (default: 60)
                             (set to 0 to disable it)
    --timeout=<int>        set the timeout of every remote operation
                             (default: 2)
                             (set to 0 to disable it)
    --port=<int>           set the port number the server will listen on
                             (default: 23237)
    --no-server            don't launch server
                             (will still attempt to connect to specified port)
                             (fall back to adb shell calls if connection failed)
                             (useful for debugging the server)
    --adb-only             don't launch server and don't try to connect
    --no-cache             don't use data caching

Options for libfuse:
    -h   --help            print help
    -V   --version         print version
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation
    -o clone_fd            use separate fuse device fd for each thread
                           (may improve performance)
    -o max_idle_threads    the maximum number of idle worker threads
                           allowed (default: -1)
    -o max_threads         the maximum number of worker threads
                           allowed (default: 10)
    -o allow_other         allow access by all users
    -o allow_root          allow access by root
    -o auto_unmount        auto unmount on process termination

Unmounting

To unmount the filesystem, you run this command like for any other FUSE filesystem:

fusermount -u <mountpoint>

You can also use the IPC operation unmount for unmounting. This will unmount the filesystem on the next operation immediately.

Selecting device

You don't need to specify the device if there is only one device connected via adb on your computer. If there are more than one device then you can specify the serial using --serial option. If you omit the --serial option when there are multiple device connected to the computer, you will be prompted to specify the device you want to mount.

$ madbfs <mountpoint>
[madbfs] checking adb availability...
[madbfs] multiple devices detected,
         - 1: 068832516O101622
         - 2: 192.168.240.112:5555
[madbfs] please specify which one you would like to use: _

madbfs respects the env variable ANDROID_SERIAL (mimicking adb behavior) so you can alternately use it to specify the device.

$ ANDROID_SERIAL=068832516O101622 ./madbfs
[madbfs] checking adb availability...
[madbfs] using serial '068832516O101622' from env variable 'ANDROID_SERIAL'

Mounting subdirectory

madbfs supports mounting subdirectory. Use --root option to specify it.

$ madbfs --root=<root> <mountpoint>

The <root> is the subdirectory on your device you want to treat as the root of the filesystem (effectively mounting a subdirectory). The option only accepts absolute path and if the directory is a symlink, it will resolve it first (/sdcard -> /storage/emulated/0).

Do note that mounting subdirectory might break symbolic links since it is possible that a link may contain components that are unreachable from the specified root. For example if you mount /sdcard/ and have a link in /sdcard/link that points to /storage/ the filesystem can't reach it, because /storage/ is outside of /sdcard/.

Specifying server and port number

If you want the filesystem to use only use adb transport, use --adb-only flag. This flag prevents madbfs from pushing the server into your phone and running it. If you rather want to manually run the server yourself (for debugging purpose for example), use --no-server flag instead.

The proxy communicates with madbfs over TCP enabled by port forwarding and by default it will listen on port 23237 (adbfs on dial pad). If you find this port to be not suitable for your use you can always specify it with --port option.

$ madbfs --server=<path/to/server-with-abi> --port=23237 <mountpoint>

To cache or not to cache

By default madbfs caches file stat and file content (data) of files operated by the filesystem. While file stat caching is always active, file content caching can be disabled using --no-cache program option. This option will ignore other cache related options.

$ madbfs --no-cache <mountpoint>

The no-cache mode is basically a direct I/O, file content will always be read/written to device immediately via the connection method (proxy or adb transport).

Cache size

madbfs caches all the read/write operations on the files on the device. This cache is stored in memory. You can control the size of this cache using --cache-size option (in MiB). The default value is 256 (256 MiB).

$ madbfs --cache-size=256 <mountpoint>    # 256 MiB of memory will be used as file cache

Page size

In the cache, each file is divided into pages. The --page-size option dictates the size of this page (in KiB). Page size also dictates the size of the buffer used to read/write into the file on the device. You can adjust this value according to your use.

$ madbfs --page-size=128 <mountpoint>    # read/write operations are communicated in 128 KiB chunks

Logging

The default log file is stdout (specified by "-"; which goes to nowhere when not run in foreground mode). You can manually set the log file using --log-file option and set the log level using --log-level.

$ madbfs --log-file=madbfs.log --log-level=debug <mountpoint>

You can always watch the logs of the filesystem at runtime even if you don't specify a log-file beforehand by using IPC logcat operation (see below).

IPC (and madbfs-msg)

Filesystem parameters can be reconfigured and queried during runtime though IPC using unix socket. The supported operations are:

• help,
• version,
• info,
• invalidate cache,
• set page size,
• set cache size,
• set ttl,
• set timeout,
• set log level, and
• unmount (on next FUSE operation)
• logcat (read madbfs log in real-time, similar to adb logcat).

The address of the socket in which you can connect to as client is composed of the name of the filesystem and the serial of the device. The socket itself is created in directory defined by XDG_RUNTIME_DIR environment variable (it's usually set to /run/user/<uid>). If the XDG_RUNTIME_DIR is not defined, as fallback, the directory is set to /tmp. The socket will be created when the filesystem initializes.

For example, the socket path for a device with serial 192.168.240.112:5555:

/run/user/1000/[email protected]:5555.sock

If at initialization this socket file exists, the IPC won't start. This may happen if the filesystem is terminated unexpectedly (crash or kill signal). You need to remove this file manually if that happens.

For the specification of the message protocol used on the IPC and how to use it read IPC.md file. To make it easier for user to use the IPC without having to write their own socket code, I have created another executable: madbfs-msg. The possible operations are explained in IPC.md file as well.

Debug mode

As part of debugging functionality libfuse has provided debug mode through -d flag. You can use this to monitor madbfs operations (if you don't want to use log file or want to see the log in real-time). If the debugging information is too verbose, you can use -f instead to make madbfs run in foreground mode without printing fuse debug information.

$ madbfs --log-file=- --log-level=debug -f <mountpoint>                     # runs in foreground (not daemonized)
$ madbfs --log-file=- --log-level=debug -d <mountpoint>                     # this will print the libfuse debug messages and madbfs log messages
$ madbfs --log-file=- --log-level=debug -d <mountpoint> 2> /dev/null        # this will print only madbfs log messages since libfuse debug messages are printed to stderr

Benchmark

Benchmark is done by writing a 64 MiB file using dd and then reading it back. The statistics printed by dd is used for the speed value so is for adb push and adb pull. The test is done on an Android 11 phone (armv8) using USB cable with proxy transport. As baseline, the speed on which an adb push (write) and an adb pull (read) operation is done on a file with the same size is measured. madbfs is launched using its default parameters (cache size = 256 MiB, page size = 128 KiB).

The write command is as follows:

  dd if=/dev/random of=random bs=128K count=1024

The read command is as follows:

  dd if=random of=/dev/null bs=128K count=1024

Proxy transport

• Write

  operation	speed (MB/s)	relative speed
  adb push	15.3	1.00
  write	13.1	0.86

• Read

  operation	speed (MB/s)	relative speed
  adb pull	11.7	1.00
  read (no cache)	10.8	0.93
  read (cache)	2100.0	179.49

adb transport

• Write

  operation	speed (MB/s)	relative speed
  adb push	15.3	1.00
  write	12.4	0.81

• Read

  operation	speed (MB/s)	relative speed
  adb pull	11.7	1.00
  read (no cache)	3.6	0.24
  read (cache)	2100.0	179.49