Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"):
  Relax-and-Recover 2.4 / Git

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
  RedHatEnterpriseServer 7

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
  not available

• Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
  PC

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
  x86_64

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  Any/GRUB

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  any

• Description of the issue (ideally so that others can reproduce it):
  With a huge number of users, groups and users in each group (tens of thousands), ReaR may abort with this in the log:

  Including rescue/default/900_clone_users_and_groups.sh
  Cloning users:  daemon rpc usbmuxd usbmux vcsa nobody dbus
  /usr/share/rear/rescue/default/900_clone_users_and_groups.sh: xrealloc: cannot allocate 18446744071562067968 bytes (6635520 bytes allocated)
  

  The underlying problem is:

  • /usr/share/rear/conf/GNU/Linux.conf ends up setting ${CLONE_USERS[0]} to "":

    CLONE_USERS=( "${CLONE_USERS[@]:-}" daemon rpc usbmuxd usbmux vcsa nobody dbus )

  • This then results in the for loop in

    rear/usr/share/rear/rescue/default/900_clone_users_and_groups.sh

    Line 42 in b7a75c3

    for user in "${CLONE_USERS[@]}" ; do

    doing 'getent passwd' without additional parameters in the first iteration, thus dumping ALL users ($user is set to "")

    for user in "${CLONE_USERS[@]}" ; do
        # Skip if the user exists already in the ReaR recovery system:
        grep -q "^$user:" $ROOTFS_DIR/etc/passwd && continue
        # Skip if the user does not exist in the current system:
        if ! passwd_entry="$( getent passwd $user )" ; then
            Debug "Cannot clone user $user because it does not exist"
            continue
        fi

  • The script then ends up doing a 'getent group $groupID' where groupID which is every users GID

    groupID="$( cut -d ':' -f '4' <<<"$passwd_entry" )"
    if ! group_entry="$( getent group $groupID )" ; then

  • With such a large number of users, groups and users in each groups, the output of the last command is so huge that the assignment to group_entry overflows some internal limit of bash.

  The problematic case is hard to reproduce as it requires a really large amount of users, groups and users per group. However, the underlying problem (which seems mostly harmless in more limited settings) manifests itself easily:

  Cloning users:  daemon rpc usbmuxd usbmux vcsa nobody dbus
  

  Note the additional space after :. This space is printed in the first iteration of the loop with $user empty.

  The root cause is the introduction of empty array elements in PR Added separated debugscripts option and first steps so that 'set -eu' works #699 to pacify set -ue in bash 3, which can not cope with empty arrays - so those were replaced by arrays with one empty member. (Since there are more arrays which were changed this way in this commit, it is possible that similar errors are more widespread.) See our discussion in c19647f#r32912187, c19647f#r32911995, c19647f#r32912160

• Workaround, if any:
  Not sure about workaround, but there are several possible fixes:

  • The cleanest would be to revert the parts of PR Added separated debugscripts option and first steps so that 'set -eu' works #699 that introduce those empty array elements. The resulting problem in bash 3 and set -ue could be dealt with in the following ways:

    • by ignoring it, i.e desupport bash 3 together with set -ue.
      This does not seem to be a big problem, as set -ue in general (even bash 4) is apparently broken without anybody noticing: https://github.com/rear/rear/pull/1720/files#r317556097 and there are further places that probably broke it more for bash 3 (non-exhaustive list, and some of them may be false positives, i.e. the arrays become non-empty at some point):
      • BACKUP_PROG_INCLUDE:
        eb89244#r34830311

        rear/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh

        Line 3 in b7a75c3

        for backup_include_item in "${BACKUP_PROG_INCLUDE[@]}" ; do

        rear/usr/share/rear/layout/save/default/310_include_exclude.sh

        Line 27 in d8f1571

        if ! IsInArray "$mountpoint" "${BACKUP_PROG_INCLUDE[@]}" ; then

      • BACKUP_PROG_EXCLUDE:

        rear/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh

        Line 19 in 950d14a

        for k in "${BACKUP_PROG_EXCLUDE[@]}" ; do

      • EXCLUDE_MOUNTPOINTS: 7c3b34d#r34830344

        rear/usr/share/rear/backup/NETFS/default/400_create_include_exclude_files.sh

        Line 29 in b7a75c3

        for excluded_mountpoint in "${EXCLUDE_MOUNTPOINTS[@]}" ; do

      • EXCLUDE_MD:

        rear/usr/share/rear/layout/save/default/310_include_exclude.sh

        Line 35 in d8f1571

        for md in "${EXCLUDE_MD[@]}" ; do

      • EXCLUDE_VG:

        rear/usr/share/rear/layout/save/default/310_include_exclude.sh

        Line 41 in d8f1571

        for vg in "${EXCLUDE_VG[@]}" ; do

      • EXCLUDE_COMPONENTS:

        rear/usr/share/rear/layout/save/default/310_include_exclude.sh

        Line 57 in d8f1571

        for component in "${EXCLUDE_COMPONENTS[@]}" ; do

      • EXCLUDE_RECREATE:

        rear/usr/share/rear/layout/save/default/310_include_exclude.sh

        Line 63 in d8f1571

        for component in "${EXCLUDE_RECREATE[@]}" ; do

      • RmInArray: e51bbef#r34792188
    • by protecting the array expansion in a bash 3 and set -ue compatible way: i.e. instead of for i in "${A[@]}" write for i in ${A+"${A[@]:-}"}. To test for an empty array, write (( ${#A[@]} )) or [ ${#A[@]} -gt 0 ]. This is possible, but would require lots of changes which would make the code a bit less readable.

  • In the present case, a less invasive change would be to replace the appending to the array

       CLONE_USERS=( "${CLONE_USERS[@]:-}" daemon rpc usbmuxd usbmux vcsa nobody dbus )

    by +=, i.e.

       CLONE_USERS+=( daemon rpc usbmuxd usbmux vcsa nobody dbus )

    (I've verified that += works properly even in bash 3 and set -ue.) This would fix the present case and other cases where the problem lies in appending to arrays, but not those where the array is already initialized to ("").

  • The most basic fix is to compensate for the fake empty element:

    for user in "${CLONE_USERS[@]}" ; do
        [[ -z ${user:-} ]] && continue
        # Skip if the user exists already in the ReaR recovery system:

    and if similar problems are found or newly introduced elsewhere, fix them in a similar way.

    It is also the least desirable fix IMHO, because it pollutes the code to work around an artificial element that should not be there in the first place, and one will have to remember to work around it in every new loop that is introduced.

  I would to like to discuss which path to take and I will then submit a PR.

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  See above for relevant log snippets.