Support for Borg #1037

gozora · 2016-10-15T17:17:52Z

Hello all,

This pull request is for issue #1030 (add support for Borg Backup).
It was tested on following distros, with latest stable Borg release 1.0.7:

OS	Release	Kernel	Notes
SLES 11	SP3	3.0.101-0.40-default	LEGACY BOOT
SLES 11	SP3	3.0.76-0.11-default	UEFI BOOT
SLES 12	SP1	3.12.49-11-default	UEFI BOOT, btrfs
CentOS	7.2.1511 (Core)	3.10.0-327.36.2.el7.x86_64	LEGACY BOOT
CentOS	7.2.1511 (Core)	3.10.0-327.18.2.el7.x86_64	UEFI BOOT
Debian GNU/Linux	8.6 (jessie)	3.16.0-4-amd64	UEFI BOOT

I'll update documentation soon, for now here are some hits on setup:

Borg installation manual recommends to install Borg to /usr/local/bin. I'd however recommend to use /usr/sbin instead, due reasons mentioned in point one in comment.
Before using ReaR with Borg, setup ssh keys for user that will be running backup. In other words, issuing following command must work without any password prompts or remote host identity confirmation.
ssh <borg_user>@<borg_server>
I've used following configuration to make Borg work with ReaR:

BACKUP=BORG
OUTPUT=ISO

OUTPUT_URL=nfs://<server_to_store_backups>/mnt/rear/iso
BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} /mnt /test/exclude )

BORG_HOST=<server_to_store_backups>
BORG_USERNAME=root
BORG_REPO=/mnt/rear/borg/$HOSTNAME
BORG_PRUNE_WEEKLY=2

GRUB_RESCUE=n

#BTRFS stuff
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

jsmeix · 2016-10-17T08:30:33Z

In usr/share/rear/restore/BORG/default/40_restore_backup.sh
I found hardcoded 'mnt/local'.
Nowadays $TARGET_FS_ROOT should be used, cf.
#708 (comment)
unless there is a good reason why hardcoded 'mnt/local'
must be used in your particular case.

In your documentation you sometimes write "borg"
and sometimes "Borg",
cf. https://github.com/borgbackup/borg that reads:

BorgBackup (short: Borg) is a deduplicating backup program.

gozora · 2016-10-17T08:41:10Z

Didn't know about TARGET_FS_ROOT... I'll correct it.
Maybe I could update 09-design-concepts.adoc as well?

I've try to used "borg" (small 'b') when referencing command and Borg (capital 'B') everywhere else. I'll review documentation one more time.

Thanks for your inputs.

Small documentation improvements

jsmeix · 2016-10-17T10:08:26Z

If @gdha does not object, I will merge it tomorrow.

FYI: I missed
https://github.com/rear/rear/blob/master/doc/user-guide/09-design-concepts.adoc#inter-module-communication
because 'mnt/local' was not mentioned at that place and I only
checked all places where 'mnt/local' was already used.

gozora · 2016-10-17T10:55:09Z

@jsmeix , @gdha
Did you noticed my change in usr/share/rear/build/GNU/Linux/10_copy_as_is.sh ?

> cp $v -r $CONFIG_DIR/. $ROOTFS_DIR/etc/rear/ >&2

< cp $v -r $CONFIG_DIR/* $ROOTFS_DIR/etc/rear/ >&2
< cp $v -r $CONFIG_DIR/.[a-z]* $ROOTFS_DIR/etc/rear/ >&2

Do you think it is safe?

gdha · 2016-10-17T10:57:40Z

@gozora sure no problem with copying dot files (already did the same for CIFS in 1.19) to initrd rootfs

gdha · 2016-10-17T11:00:29Z

@gozora Now that you got a fresh integration in your fingertips why don't you update the https://github.com/rear/rear/blob/master/doc/user-guide/10-integrating-external-backup.adoc document?

gdha · 2016-10-17T11:04:49Z

usr/share/rear/restore/BORG/default/30_load_archives.sh

+echo ""
+
+# Display archive_cache file content and prompt user for archive to restore
+while(true); do


@gozora I am always afraid when a see a while(true) that the code might be caught in an endless loop. Are you 100% sure that that will never happen? If possible I would avoid this construction...

I'm with you here, I don't like it either. It is however unavoidable sometimes.
This part of code asks user, which archive he wants to restore, and code loops until user enters valid archive or decides to exit restore. I think it is safe.

I think in such a case it could be even mandatory
to let "rear recover" operate on the safe side
and loop (possibly endlessly) until the user
provided a valid response.

I think it could be unsafe if "rear recover" would
restore something (e.g. a default or fallback)
that was not explicitly confirmed by the user.

I think the only exception is when there is
only one archive available to be restored.
Then I think "rear recover" could also for Borg assume
that one should be restored in the same way as it works
e.g. for 'tar' where the one backup.tar.gz is "just restored".

In other words:
"if [ $archive_cache_lines -eq 1 ]; then"
it could perhaps be skipped to ask the user
if he wants to restore from that one Borg archive?

I was thinking about auto start when only one archive is present.
But I personally prefer to be asked, It could happen that only present archive is not that I'd like to restore (for whatever reason), this would avoid loosing time with restore ...

As this is mostly matter of taste, we can vote :-)
My vote is for "always ask"

You are a Borg user and you implemented it,
so that your point of view matters most and
"always ask" is perfectly o.k. for me - in particular
now after you explained why it is this way (the latter
is a hint about a possibly missing explanatory comment
it the code so that others understand the reason behind ;-)

Thanks, ;-) I'm not an coding autocrat, so I have no problem change this behavior if someone requests it in the future ..
The point with comment is however relevant, I'll add it ...

gozora · 2016-10-17T11:35:35Z

10-integrating-external-backup.adoc looks fine for me. Unfortunately I've discovered it after coding was done (typical me ...).
Do you have some some specific part that needs to be updated on your mind? Because it looks to be quite accurate to me.

gdha · 2016-10-17T11:50:45Z

@gozora No, I was wondering if it could be improved then now was the right moment to update it so it makes more sense for (new) developers. However, if it is OK for you then it I'm glad to hear that.

ThomasWaldmann

lots of typos, maybe consider a spell checker or manually reading your stuff again.

BORG_* naming might get you into trouble, consider these reserved for usage by borg. You can use them, but then expect borg does something with them (now or in future) and use them according to borg docs.

ThomasWaldmann · 2016-10-18T14:53:13Z

doc/user-guide/10-integrating-external-backup.adoc

 Relax-and-Recover can be used only to restore the disk layout of your system and boot loader. However, that means you are responsible for taking backups. And, more important, to restore these before you reboot recovered system.

-However, we have sucessfully already integrated external backup programs within rear, such as Netbackup, EMC NetWorker, Tivoli Storage Manager, Data Protetctor to name a few commercial backup programs. Furthermore, open source external backup programs which are also working with rear are, Bacula, Bareos and Duplicity to name the most known ones.
+However, we have sucessfully already integrated external backup programs within rear, such as Netbackup, EMC NetWorker, Tivoli Storage Manager, Data Protetctor to name a few commercial backup programs. Furthermore, open source external backup programs which are also working with rear are, Bacula, Bareos, Duplicity and Borg to name the most known ones.


successfully
and i guess the comma after "are" should not be there?

ThomasWaldmann · 2016-10-18T14:53:42Z

usr/share/rear/backup/BORG/default/10_load_init_archives.sh

+# 10_load_archives.sh
+
+# Check if BORG_ARCHIVE_PREFIX is correctly set
+# Using '_' could result to some unpleasant site effects,


ThomasWaldmann · 2016-10-18T14:54:42Z

usr/share/rear/backup/BORG/default/10_load_init_archives.sh

+# Check if BORG_ARCHIVE_PREFIX is correctly set
+# Using '_' could result to some unpleasant site effects,
+# as this character is used as delimiter in latter `for' loop ...
+# Excluding other non aplhanum characeters is not really necessary,


alphanumeric
characters

ThomasWaldmann · 2016-10-18T14:55:34Z

usr/share/rear/backup/BORG/default/10_load_init_archives.sh

+# I'm sure archive handling can be done better, but no time for it now ...
+if [[ $BORG_ARCHIVE_PREFIX =~ [^a-zA-Z0-9] ]] \
+|| [[ -z $BORG_ARCHIVE_PREFIX ]]; then
+    Error "BORG_ARCHIVE_PREFIX must be alphanumeric non emply value only"


ThomasWaldmann · 2016-10-18T14:56:02Z

usr/share/rear/backup/BORG/default/10_load_init_archives.sh

+has_binary borg
+StopIfError "Could not find Borg binary"
+
+# Query Borg server for repoitory information and store it to archive_cache.


ThomasWaldmann · 2016-10-18T15:58:58Z

usr/share/rear/restore/BORG/default/30_load_archives.sh

+# Query Borg server for repoitory information and store it to archive_cache.
+# This should avoid repeatingly quering Borg server, which could be slow.
+archive_cache=$TMP_DIR/borg_archive
+borg list $BORG_USERNAME@$BORG_HOST:$BORG_REPO > $archive_cache


hmm, didn't I see that above already? consider --short option.

ThomasWaldmann · 2016-10-18T15:59:45Z

usr/share/rear/restore/BORG/default/30_load_archives.sh

+# This means empty repository
+if [ $archive_cache_lines -eq 0 ]; then
+    Error "Borg repository $BORG_REPO on $BORG_HOST is empty"
+fi


why is that an error?

@ThomasWaldmann That is in OS restore stage. Once you run restore and have nothing to restore from, there ain't much options left.

Just thought whether it could not just show an empty list.

@ThomasWaldmann
Let me roughly describe how ReaR restore works.

Boot small ISO image that contains some base system (including your configuration at the time backup was running)

Start restore with rear recover

ReaR loads and and creates your file system layout

ReaR starts restore of your data (this is stage when Borg comes to game)

ReaR does some exit tasks like boot loader installation, etc.

As you can see outputting empty list will you no good ...
In my opinion result must be failure as you can't correctly restore your OS. Of course after such failure you can reconfigure ReaR recovery system to use different settings (e.g. different Borg repository, different file system layout) and try to run restore again.

ThomasWaldmann · 2016-10-18T16:01:17Z

usr/share/rear/restore/BORG/default/30_load_archives.sh

+# This gives possibility to abort restore if repository doesn't contain
+# desired archive, hence saves some time.
+while(true); do
+    cat -n $archive_cache | awk '{print "["$1"]", $2,"\t"$3,$4,$5}'


ah, here you use more than just $1. so ignore my --short suggestion.

ThomasWaldmann · 2016-10-18T16:04:03Z

usr/share/rear/restore/BORG/default/40_restore_backup.sh

+LogPrint "Recovering from Borg archive $ARCHIVE"
+LC_ALL=rear.UTF-8 \
+borg extract --sparse $BORG_USERNAME@$BORG_HOST:$BORG_REPO::$ARCHIVE
+StopIfError "Could not successfully finish Borg restore"


this likely should consider rc=0 vs 1 vs 2.

ThomasWaldmann · 2016-10-18T16:04:20Z

usr/share/rear/restore/BORG/default/40_restore_backup.sh

+# Scope of LC_ALL is only within run of `borg extract'.
+# This avoids Borg problems with restoring UTF-8 encoded files names in archive
+# and should not interfere with remaining stages of rear recover.
+# This is still not the ideal sollution, but best I can think of so far :-/


gozora · 2016-10-18T19:20:14Z

@ThomasWaldmann And one more thanks for all the typos corrections. I'm obviously lost without spellchecker!

jsmeix · 2016-10-19T09:00:36Z

@gozora
don't worry about a spellchecker - I could read it well without
noticing all those typos - I'm obviously running some kind of
silently working full automated autocorrection in my mind ;-)

gozora · 2016-10-19T09:09:32Z

@jsmeix :-) I basically don't, but @ThomasWaldmann gave me wake up call, and I realized how much work is actually done by spell checker for me!
Before I didn't even think of how many mistakes I'm doing ...
I need to be more careful.

jsmeix · 2016-10-19T10:09:17Z

@gozora only a side note:
What I would like to have is an
actually useful "spell checker" for bash
(not only the simple "bash -n" syntax check)
that could detect bugs like

CLEANUP_COMMAND="..."
... || $CLEANUP_C0MMAND

Even with "set -eu" the bug would usually not appear
during runtime (assuming the cleanup is usually not needed).
I know that it is impossible to really validate a bash script
but I would be already happy if I had some basic testing
and get warnings for some usual things that "look suspicious".
Cf. #549

gozora · 2016-10-19T10:46:08Z

@jsmeix heh, this might be tricky one, I guess you are looking for similar functionality like gcc have, which reports unused variable and (static) functions ...

alxgu · 2016-10-19T11:13:44Z

@jsmeix Maybe have a look on https://github.com/koalaman/shellcheck / https://www.shellcheck.net/

jsmeix · 2016-10-19T12:30:58Z

@alxgu
many thanks for the hint!

Some time ago I had a look at ShellCheck
but as it is written in Haskell I did not try to
compile it on my own.
But now it seems there are ready-made packages
available for some recent openSUSE systems:

$ osc search ShellCheck | grep ^openSUSE
openSUSE:Backports:SLE-12         ShellCheck
openSUSE:Factory                  ShellCheck
openSUSE:Leap:42.2                ShellCheck
$ osc results -v devel:languages:haskell ShellCheck | grep succeeded
openSUSE_Leap_42.1   x86_64     succeeded
openSUSE_Factory_PowerPC_standard ppc64le    succeeded
openSUSE_Factory_PowerPC_standard ppc64      succeeded
openSUSE_13.2_Ports_ppc ppc64le    succeeded
openSUSE_13.2_Ports_ppc ppc64      succeeded
openSUSE_13.2        i586       succeeded
openSUSE_13.2        x86_64     succeeded
SLE_12_SP1_Backports ppc64le    succeeded
SLE_12_SP1_Backports x86_64     succeeded
SLE_12_Backports     ppc64le    succeeded
SLE_12_Backports     x86_64     succeeded

I will try it out soon...

gozora · 2016-10-19T20:25:29Z

@ThomasWaldmann maybe one more off topic question. Is there any difference in doing:
borg delete TARGET and rm -rf TARGET ?

ThomasWaldmann · 2016-10-19T23:33:30Z

yes, it also deletes the .cache/borg/id. see borg delete --help.

Applied corrections discussed in #1037 (about BORGBACKUP)

gozora added 3 commits October 15, 2016 18:46

Support for Borg

3cd93ce

Some code cleanup and small bug repair for commit 3cd93ce

136bc49

Documentation update for Borg backup

4f5ebb1

jsmeix self-assigned this Oct 17, 2016

jsmeix added the enhancement Adaptions and new features label Oct 17, 2016

jsmeix added this to the Rear v1.20 milestone Oct 17, 2016

TARGET_FS_ROOT is used instead "/mnt/local"

ee8ed28

Small documentation improvements

jsmeix assigned gdha Oct 17, 2016

gdha reviewed Oct 17, 2016

View reviewed changes

Minor update in comment

73b7be2

jsmeix merged commit 64ae0dc into rear:master Oct 18, 2016

jsmeix added the fixed / solved / done label Oct 18, 2016

This was referenced Oct 18, 2016

add support for Borg Backup #1030

Closed

Integration with Relax-and-Recover borgbackup/borg#1702

Closed

ThomasWaldmann reviewed Oct 18, 2016

View reviewed changes

jsmeix mentioned this pull request Oct 19, 2016

Use ShellCheck for ReaR scripts static analysis #1040

Closed

gdha added a commit that referenced this pull request Oct 20, 2016

Merge pull request #1041 from gozora/master

419d517

Applied corrections discussed in #1037 (about BORGBACKUP)

jsmeix mentioned this pull request Oct 20, 2016

Applied corrections discussed in #1037 #1041

Merged

gozora mentioned this pull request Oct 20, 2016

Borg as back end, now accepts options for repository encryption. #1045

Closed

Uh oh!

Support for Borg #1037

Support for Borg #1037

Uh oh!

Conversation

gozora commented Oct 15, 2016

Uh oh!

jsmeix commented Oct 17, 2016

Uh oh!

gozora commented Oct 17, 2016

Uh oh!

jsmeix commented Oct 17, 2016

Uh oh!

gozora commented Oct 17, 2016

Uh oh!

gdha commented Oct 17, 2016

Uh oh!

gdha commented Oct 17, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsmeix Oct 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gozora commented Oct 17, 2016

Uh oh!

gdha commented Oct 17, 2016

Uh oh!

ThomasWaldmann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gozora Oct 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gozora Oct 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gozora commented Oct 18, 2016

Uh oh!

jsmeix commented Oct 19, 2016

Uh oh!

jsmeix Oct 17, 2016 •

edited

Loading

gozora Oct 18, 2016 •

edited

Loading

gozora Oct 18, 2016 •

edited

Loading