cjones/restic-automatic-backup-scheduler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Multiple profile configuration (#45)

Browse Source 
Adds possibility to have multiple profiles with different backup paths, retention etc.

Co-authored-by: Matt Feifarek <matt.feifarek@gmail.com>
Co-authored-by: Erik Westrup <erik.westrup@gmail.com>
This commit is contained in:
Gerard Bosch
2022-02-01 10:09:36 +01:00
committed by GitHub
parent 16dbe699ab
commit d8f25cdf88
16 changed files with 221 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+106 -32
View File
@@ -7,7 +7,7 @@
[![Open issues](https://img.shields.io/github/issues/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/issues)
[![Closed issues](https://img.shields.io/github/issues-closed/erikw/restic-systemd-automatic-backup?color=success)](https://github.com/erikw/restic-systemd-automatic-backup/issues?q=is%3Aissue+is%3Aclosed)
[![Closed PRs](https://img.shields.io/github/issues-pr-closed/erikw/restic-systemd-automatic-backup?color=success)](https://github.com/erikw/restic-systemd-automatic-backup/pulls?q=is%3Apr+is%3Aclosed)
[![License](https://img.shields.io/badge/license-BSD--3-blue)](LICENSE.txt)
[![License](https://img.shields.io/badge/license-BSD--3-blue)](LICENSE)
[![OSS Lifecycle](https://img.shields.io/osslifecycle/erikw/restic-systemd-automatic-backup)](https://github.com/Netflix/osstracker)
[![Latest tag](https://img.shields.io/github/v/tag/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/tags)
<br>
@@ -25,7 +25,6 @@ Unfortunately restic does not come pre-configured with a way to run automated ba
Here follows a step-by step tutorial on how to set it up, with my sample script and configurations that you can modify to suit your needs.
Note, you can use any of the supported [storage backends](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html). The setup should be similar but you will have to use other configuration variables to match your backend of choice.
# Requirements
@@ -33,7 +32,51 @@ Note, you can use any of the supported [storage backends](https://restic.readthe
* (recommended) `make` if you want an automated install
* Arch: part of the `base-devel` meta package, Debian/Ubuntu: part of the `build-essential` meta package, macOS: preinstalled make works)
# Set up
# TL;DR Setup
1. Create B2 credentials as instructed [below](#1-create-backblaze-b2-account)
1. Install config and scripts:
```console
$ sudo make install
```
☝ **Note**: `sudo` is required here, as some files are installed into system directories (`/etc/`
and `/usr/sbin`). Have a look to the `Makefile` to know more.
1. Fill out configuration values (edit with sudo):
* `/etc/restic/pw.txt` - Contains the password (single line) to be used by restic to encrypt the repository files. Should be different than your B2 password!
* `/etc/restic/_global.env` - Global environment variables.
* `/etc/restic/default.env` - Profile specific environment variables (multiple profiles can be defined by copying to `/etc/restic/something.env`).
* `/etc/restic/backup_exclude` - List of file patterns to ignore. This will trim down your backup size and the speed of the backup a lot when done properly!
1. Initialize remote repo as described [below](#3-initialize-remote-repo)
1. Configure [how often](https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events) back up should be made.
* Edit if needed `OnCalendar` in `/etc/systemd/system/restic-check@.timer`.
1. Enable automated backup for starting with the system (`enable` creates symlinks):
```console
$ sudo systemctl start restic-backup@default.timer
$ sudo systemctl enable restic-backup@default.timer
```
1. And run an immediate backup if you want (if not, it will run on daily basis):
```console
$ sudo systemctl start restic-backup@default
```
1. Watch its progress with Systemd journal:
```console
$ journalctl -f --lines=50 -u restic-backup@default
```
1. Verify the backup
```console
$ sudo -i
$ source /etc/restic/default.env
$ restic snapshots
```
1. (optional) Define multiple profiles: just make a copy of the `default.env` and use the defined profile name in place of `default` to run backups or enable timers. Notice that the value after `@` works as a parameter.
1. (optional) Enable the check job that verifies that the backups for the profile are all intact.
```console
$ sudo systemctl start restic-check@default.timer
$ sudo systemctl enable restic-check@default.timer
````
1. (optional) Setup email on failure as described [here](#8-email-notification-on-failure)
# Step-by-step and manual setup
This is a more detailed explanation than the TL;DR section that will give you more understanding in the setup, and maybe inspire you to develop your own setup based on this one even!
Tip: The steps in this section will instruct you to copy files from this repo to system directories. If you don't want to do this manually, you can use the Makefile:
@@ -42,65 +85,89 @@ $ git clone https://github.com/erikw/restic-systemd-automatic-backup.git && cd $
$ sudo make install
````
Arch Linux users can install the aur package [restic-systemd-automatic-backup](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/) e.g. :
Arch Linux users can install the aur package [restic-systemd-automatic-backup](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/) e.g.:
```console
$ yaourt -S restic-systemd-automatic-backup
````
## 1. Create Backblaze B2 account
First, see this official Backblaze [tutorial](https://help.backblaze.com/hc/en-us/articles/4403944998811-Quickstart-Guide-for-Restic-and-Backblaze-B2-Cloud-Storage) on restic, and follow the instructions ("Create Backblaze account with B2 enabled") there on how to create a new B2 bucket. In general, you'd want a private bucket, without B2 encryption (restic does the encryption client side for us) and without the object lock feature.
First, see this official Backblaze [tutorial](https://help.backblaze.com/hc/en-us/articles/115002880514-How-to-configure-Backblaze-B2-with-Restic-on-Linux) on restic, and follow the instructions ("Create Backblaze account with B2 enabled") there on how to create a new B2 bucket.
Take note of the your account ID, application key and password for the next steps.
Take note of the your account ID and application key for the next steps. It's a good idea to create a separate application key that has access only to the newly created b2 bucket you created.
## 2. Configure your B2 account locally
> **Attention!** Going the manual way requires that most of the following commands are run as root.
Put these files in `/etc/restic/`:
* `b2_env.sh`: Fill this file out with your B2 bucket settings etc. The reason for putting these in a separate file is that it can be used also for you to simply source, when you want to issue some restic commands. For example:
```console
$ source /etc/restic/b2_env.sh
$ restic snapshots # You don't have to supply all parameters like --repo, as they are now in your environment!
````
* `b2_pw.txt`: This file should contain the restic repository password. This is a new password what soon will be used when initializing the new repository. It should be unique to this restic backup repository and is needed for restoring from it. Don't re-use your B2 login password, this should be different.
* `_global.env`: Fill this file out with your global settings including B2 accountID & accountKey. A global exclude list is set here (explained in section below).
* `default.env`: This is the default profile. Fill this out with bucket name, backup paths and retention policy. This file sources `_global.env` and is thus self-contained and can be sourced in the shell when you want to issue some manual restic commands. For example:
```console
$ source /etc/restic/default.env
$ restic snapshots # You don't have to supply all parameters like --repo, as they are now in your environment!
````
* `pw.txt`: This file should contain the restic password used to encrypt the repository. This is a new password what soon will be used when initializing the new repository. It should be unique to this restic backup repository and is needed for restoring from it. Don't re-use your B2 login password, this should be different. For example you can generate a 128 character password (must all be on one line) with:
```console
$ openssl rand -base64 128 | tr -d '\n' > /etc/restic/pw.txt
```
## 3. Initialize remote repo
Now we must initialize the repository on the remote end:
```console
$ source /etc/restic/b2_env.sh
$ sudo -i
$ source /etc/restic/default.env
$ restic init
```
## 4. Script for doing the backup
Put this file in `/usr/local/sbin`:
* `restic_backup.sh`: A script that defines how to run the backup. Edit this file to respect your needs in terms of backup which paths to backup, retention (number of backups to save), etc.
* `restic_backup.sh`: A script that defines how to run the backup. The intention is that you should not need to edit this script yourself, but be able to control everything from the `*.env` profiles.
Copy this file to `/etc/restic/backup_exclude` or `~/.backup_exclude`:
* `.backup_exclude`: A list of file pattern paths to exclude from you backups, files that just occupy storage space, backup-time, network and money.
Restic support exclude files. They list file pattern paths to exclude from you backups, files that just occupy storage space, backup-time, network and money. `restic_backup.sh` allows for a few different exclude files.
* `/etc/restic/backup_exclude` - global exclude list. You can use only this one if your setup is easy. This is set in `_global.env`. If you need a different file for another profile, you can override the envvar `RESTIC_BACKUP_EXCLUDE_FILE` in this profile.
* `.backup_exclude` per backup path. If you have e.g. an USB disk mounted at /mnt/media and this path is included in the `$BACKUP_PATHS`, you can place a file `/mnt/media/.backup_exclude` and it will automatically picked up. The nice thing about this is that the backup paths are self-contained in terms of what they shoud exclude!
## 5. Make first backup & verify
Now see if the backup itself works, by running
## 5. Make first backup
Now see if the backup itself works, by running as root
```console
$ sudo -i
$ source /etc/restic/default.env
$ /usr/local/sbin/restic_backup.sh
$ restic snapshots
````
## 6. Backup automatically; systemd service + timer
## 6. Verify the backup
As the `default.env` is already sourced in your root shell, you can now just list the snapshos
```console
$ sudo -i
$ source /etc/restic/default.env
$ restic snapshots
```
Alternatively you can mount the restic snapshots to a directory set `/mnt/restic`
```console
$ restic mount /mnt/restic
$ ls /mnt/restic
```
## 7. Backup automatically; systemd service + timer
Now we can do the modern version of a cron-job, a systemd service + timer, to run the backup every day!
Put these files in `/etc/systemd/system/`:
* `restic-backup.service`: A service that calls the backup script.
* `restic-backup.timer`: A timer that starts the backup every day.
* `restic-backup@.service`: A service that calls the backup script with the specified profile. The profile is specified
by the value after `@` when running it (see below).
* `restic-backup@.timer`: A timer that starts the former backup every day (same thing about profile here).
* If needed, edit this file to configure [how often](https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events) back up should be made. See the `OnCalendar` key in the file.
Now simply enable the timer with:
```console
$ systemctl start restic-backup.timer
$ systemctl enable restic-backup.timer
$ systemctl start restic-backup@default.timer
$ systemctl enable restic-backup@default.timer
````
☝ **Note**: You can run it with different values instead of `default` if you use multiple profiles.
You can see when your next backup is scheduled to run with
```console
$ systemctl list-timers | grep restic
@@ -115,20 +182,20 @@ $ systemctl status restic-backup
or start a backup manually
```console
$ systemctl start restic-backup
$ systemctl start restic-backup@default
```
You can follow the backup stdout output live as backup is running with:
```console
$ journalctl -f -u restic-backup.service
$ journalctl -f -u restic-backup@default.service
````
(skip `-f` to see all backups that has run)
## 7. Email notification on failure
## 8. Email notification on failure
We want to be aware when the automatic backup fails, so we can fix it. Since my laptop does not run a mail server, I went for a solution to set up my laptop to be able to send emails with [postfix via my Gmail](https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/). Follow the instructions over there.
Put this file in `/usr/local/sbin`:
@@ -140,10 +207,17 @@ Put this files in `/etc/systemd/system/`:
As you maybe noticed already before, `restic-backup.service` is configured to start `status-email-user.service` on failure.
## 8. Optional: automated backup checks
## 9. Optional: automated backup checks
Once in a while it can be good to do a health check of the remote repository, to make sure it's not getting corrupt. This can be done with `$ restic check`.
There are some `*-check*`-files in this git repo. Install these in the same way you installed the `*-backup*`-files.
There is companion scripts, service and timer (`*check*`) to restic-backup.sh that checks the restic backup for errors; look in the repo in `etc/systemd/system` and `usr/local/sbin` and copy what you need over to their corresponding locations.
```console
$ sudo -i
$ systemctl start restic-check@default.timer
$ systemctl enable restic-check@default.timer
````
# Cron?