Automatic restic backups using systemd services and timers

formerly named restic-systemd-automatic-backup

including these top contributors:

Intro

restic is a command-line tool for making backups, the right way. Check the official website for a feature explanation. As a storage backend, I recommend Backblaze B2 as restic works well with it, and it is (at the time of writing) very affordable for the hobbyist hacker! (anecdotal: I pay for my full-systems backups each month typically < 1 USD).

Unfortunately restic does not come pre-configured with a way to run automated backups, say every day. However it's possible to set this up yourself using built-in tools in your OS and some wrappers. For Linux with systemd, it's convenient to use systemd timers. For macOS systems, we can use built-in LaunchAgents. For Windows we can use ScheduledTasks. Any OS having something cron-like will also work!

Here follows a step-by step tutorial on how to set it up, with my sample script and configurations that you can modify to suit your needs.

Note, you can use any of the supported storage backends. The setup should be similar but you will have to use other configuration variables to match your backend of choice.

Project Scope

The scope for this is not to be a full-fledged super solution that solves all the problems and all possible setups. The aim is to be a hackable code base for you to start sewing up the perfect backup solution that fits your requirements!

Nevertheless the project should work out of the box, be minimal but still open the doors for configuration and extensions by users.

Navigate this README

Tip: use the Section icon in the top left of this document to navigate the sections.

Requirements

• restic >=v0.9.6
• bash >=v4.0.0
• (recommended) GNU make if you want an automated install
  • Arch: part of the base-devel meta package, Debian/Ubuntu: part of the build-essential meta package, macOS: use the preinstalled or a more recent with Homebrew)

Setup

Depending on your system, the setup will look different. Choose one of

• Linux + Systemd
• macOS + LaunchAgent
• Windows + ScheduledTask
• Cron - for any system having a cron daemon. Tested on FreeBSD and macOS.

Setup Linux Systemd

TL;DR Setup

1. Create B2 credentials as instructed below
2. Install config and scripts:

   $ sudo make install-systemd
   

   ☝ Note: sudo is required here, as some files are installed into system directories (/etc/ and /usr/bin). Have a look to the Makefile to know more.
3. Fill out configuration values (edit with sudo):
   • /etc/restic/pw.txt - Contains the password (single line) to be used by restic to encrypt the repository files. Should be different than your B2 password!
   • /etc/restic/_global.env.sh - Global environment variables.
   • /etc/restic/default.env.sh - Profile specific environment variables (multiple profiles can be defined by copying to /etc/restic/something.env.sh).
   • /etc/restic/backup_exclude.txt - List of file patterns to ignore. This will trim down your backup size and the speed of the backup a lot when done properly!
4. Initialize remote repo as described below
5. Configure how often back up should be made.
   • Edit if needed OnCalendar in /usr/lib/systemd/system/restic-backup@.timer.
6. Enable automated backup for starting with the system (enable creates symlinks):

   $ sudo systemctl enable --now restic-backup@default.timer
   

7. And run an immediate backup if you want (if not, it will run on daily basis):

   $ sudo systemctl start restic-backup@default
   

8. Watch its progress with Systemd journal:

   $ journalctl -f --lines=50 -u restic-backup@default
   

9. Verify the backup

   $ sudo -i
   $ source /etc/restic/default.env.sh
   $ restic snapshots
   

10. (optional) Define multiple profiles: just make a copy of the default.env.sh and use the defined profile name in place of default to run backups or enable timers. Notice that the value after @ works as a parameter.
11. (optional) Enable the check job that verifies that the backups for the profile are all intact.

    $ sudo systemctl enable --now restic-check@default.timer
    

12. (optional) Setup email on failure as described here

Step-by-step and manual setup

This is a more detailed explanation than the TL;DR section that will give you more understanding in the setup, and maybe inspire you to develop your own setup based on this one even!

Tip: The steps in this section will instruct you to copy files from this repo to system directories. If you don't want to do this manually, you can use the Makefile:

$ git clone https://github.com/erikw/restic-automatic-backup-scheduler.git && cd $(basename "$_" .git)
$ sudo make install-systemd

If you want to install everything manually, we will install files to /etc, /bin, and not use the $make install-systemd command, then you need to clean up a placeholder $INSTALL_PREFIX in the souce files first by running:

$ find etc bin -type f -exec sed -i.bak -e 's|$INSTALL_PREFIX||g' {} \; -exec rm {}.bak \;

and you should now see that all files have been changed like e.g.

-export RESTIC_PASSWORD_FILE="$INSTALL_PREFIX/etc/restic/pw.txt"
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"

This prefix is there so that make users can set a different $PREFIX when installing like PREFIX=/usr/local make install-systemd. So if we don't use the makefile, we need to remove this prefix with the command above just.

Arch Linux users can install the aur package restic-automatic-backup-scheduler e.g. with yay(1):

$ yay -S restic-automatic-backup-scheduler

1. Create Backblaze B2 Account, Bucket and keys

First, see this official Backblaze tutorial on restic, and follow the instructions ("Create Backblaze account with B2 enabled") there on how to create a new B2 bucket. In general, you'd want a private bucket, without B2 encryption (restic does the encryption client side for us) and without the object lock feature.

For restic to be able to connect to your bucket, you want to in the B2 settings create a pair of keyID and applicationKey. It's a good idea to create a separate pair of ID and Key with for each bucket that you will use, with limited read&write access to only that bucket.

2. Configure your B2 credentials locally

Attention! Going the manual way requires that most of the following commands are run as root.

Put these files in /etc/restic/:

• _global.env.sh: Fill this file out with your global settings including B2 keyID & applicationKey. A global exclude list is set here (explained in section below).
• default.env.sh: This is the default profile. Fill this out with bucket name, backup paths and retention policy. This file sources _global.env.sh and is thus self-contained and can be sourced in the shell when you want to issue some manual restic commands. For example:

  $ source /etc/restic/default.env.sh
  $ restic snapshots    # You don't have to supply all parameters like --repo, as they are now in your environment!
  

• pw.txt: This file should contain the restic password used to encrypt the repository. This is a new password what soon will be used when initializing the new repository. It should be unique to this restic backup repository and is needed for restoring from it. Don't re-use your B2 login password, this should be different. For example you can generate a 128 character password (must all be on one line) with:

  $ openssl rand -base64 128 | tr -d '\n' > /etc/restic/pw.txt
  

3. Initialize remote repo

Now we must initialize the repository on the remote end:

$ sudo -i
$ source /etc/restic/default.env.sh
$ restic init

4. Script for doing the backup

Put this file in /bin:

• restic_backup.sh: A script that defines how to run the backup. The intention is that you should not need to edit this script yourself, but be able to control everything from the *.env.sh profiles.

Restic support exclude files. They list file pattern paths to exclude from you backups, files that just occupy storage space, backup-time, network and money. restic_backup.sh allows for a few different exclude files.

• /etc/restic/backup_exclude.txt - global exclude list. You can use only this one if your setup is easy. This is set in _global.env.sh. If you need a different file for another profile, you can override the envvar RESTIC_BACKUP_EXCLUDE_FILE in this profile.
• .backup_exclude.txt per backup path. If you have e.g. an USB disk mounted at /mnt/media and this path is included in the $RESTIC_BACKUP_PATHS, you can place a file /mnt/media/.backup_exclude.txt and it will automatically picked up. The nice thing about this is that the backup paths are self-contained in terms of what they shoud exclude!

5. Make first backup

Now see if the backup itself works, by running as root

$ sudo -i
$ source /etc/restic/default.env.sh
$ /bin/restic_backup.sh

6. Verify the backup

As the default.env.sh is already sourced in your root shell, you can now just list the snapshos

$ sudo -i
$ source /etc/restic/default.env.sh
$ restic snapshots

Alternatively you can mount the restic snapshots to a directory set /mnt/restic

$ restic mount /mnt/restic
$ ls /mnt/restic

7. Backup automatically; systemd service + timer

Now we can do the modern version of a cron-job, a systemd service + timer, to run the backup every day!

Put these files in /etc/systemd/system (note that the Makefile installs as package to /usr/lib/systemd/system)

• restic-backup@.service: A service that calls the backup script with the specified profile. The profile is specified by the value after @ when running it (see below).
• restic-backup@.timer: A timer that starts the former backup every day (same thing about profile here).
  • If needed, edit this file to configure how often back up should be made. See the OnCalendar key in the file.

Now simply enable the timer with:

$ sudo systemctl enable --now restic-backup@default.timer

☝ Note: You can run it with different values instead of default if you use multiple profiles.

You can see when your next backup is scheduled to run with

$ systemctl list-timers | grep restic

and see the status of a currently running backup with

$ systemctl status restic-backup

or start a backup manually

$ systemctl start restic-backup@default

You can follow the backup stdout output live as backup is running with:

$ journalctl -f -u restic-backup@default.service

(skip -f to see all backups that has run)

8. Email notification on failure

We want to be aware when the automatic backup fails, so we can fix it. Since my laptop does not run a mail server, I went for a solution to set up my laptop to be able to send emails with postfix via my Gmail. Follow the instructions over there.

Put this file in /bin:

• systemd-email: Sends email using sendmail(1). This script also features time-out for not spamming Gmail servers and getting my account blocked.

Put this files in /etc/systemd/system/:

• status-email-user@.service: A service that can notify you via email when a systemd service fails. Edit the target email address in this file.

As you maybe noticed already before, restic-backup.service is configured to start status-email-user.service on failure.

9. Optional: automated backup checks

Once in a while it can be good to do a health check of the remote repository, to make sure it's not getting corrupt. This can be done with $ restic check.

There is companion scripts, service and timer (*check*) to restic-backup.sh that checks the restic backup for errors; look in the repo in usr/lib/systemd/system/ and bin/ and copy what you need over to their corresponding locations.

$ sudo systemctl enable --now restic-check@default.timer

10. Optional: 🏃 Restic wrapper

For convenience there's a restic wrapper script that makes loading profiles and running restic straightforward (it needs to run with sudo to read environment). Just run:

• sudo resticw WHATEVER (e.g. sudo resticw snapshots) to use the default profile.
• You can run the wrapper by passing a specific profile: resticw -p anotherprofile snapshots.

Useful commands

Command	Description
resticw snapshots	List backup snapshots
resticw diff <snapshot-id> latest	Show the changes from the latest backup
resticw stats / resticw stats snapshot-id ...	Show the statistics for the whole repo or the specified snapshots
resticw mount /mnt/restic	Mount your remote repository

Setup macOS LaunchAgent

Launchd is the modern built-in service scheduler in macOS. It has support for running services as root (Daemon) or as a normal user (Agent). Here we set up an LauchAgent to be run as your normal user for starting regular backups.

Homebrew

With Homebrew you can easily install this project from the erikw/homebrew-tap:

$ brew install erikw/tap/restic-automatic-backup-scheduler

Then control the service with homebrew:

$ brew services start restic-automatic-backup-scheduler
$ brew services restart restic-automatic-backup-scheduler
$ brew services stop restic-automatic-backup-scheduler

Manual

1. In general, follow the same setup as in (#setup-linux-systemd) except for:

• use make install-launchagent instead of make install-systemd
• install everything to /usr/local and run restic as your own user, not root
• Thus, install with

  $ PREFIX=/usr/local make install-launchagent
  

1. After installation with make , edit the installed LaunchAgent if you want to change the default schedule or profile used:

   $ vim ~/Library/LaunchAgents/com.github.erikw.restic-automatic-backup.plist
   

2. Now install, enable and start the first run!

   $ launchctl bootstrap gui/$UID ~/Library/LaunchAgents/com.github.erikw.restic-automatic-backup-scheduler.plist
   $ launchctl enable gui/$UID/com.github.erikw.restic-automatic-backup-scheduler
   $ launchctl kickstart -p gui/$UID/com.github.erikw.restic-automatic-backup-scheduler
   

   As a convenience, a shortcut for the above commands are $ make activate-launchagent.

Use the disable command to temporarily pause the agent, or bootout to uninstall it.

$ launchctl disable gui/$UID/com.github.erikw.restic-automatic-backup-scheduler
$ launchctl bootout gui/$UID/com.github.erikw.restic-automatic-backup-scheduler

If you updated the .plist file, you need to issue the bootout followed by bootrstrap and enable sub-commands of launchctl. This will guarantee that the file is properly reloaded.

Setup Windows ScheduledTask

This is one of may ways you can get restic and this backup script working on Windows:

1. Install scoop
2. Install dependencies from a PowerShell with administrator privileges:

    powershell> scoop install restic make git
   

3. In a non-privileged PowerShell, start git-bash and clone this repo

    powershell> git-bash
    git-bash$ mkdir ~/src && cd ~/src/
    git-bash$ git clone https://github.com/erikw/restic-automatic-backup-scheduler.git && cd $(basename "$_" .git)
   

4. Install scripts, conf and the ScheduledTask

    git-bash$ make install-schedtask
   

5. Edit configs and initialize repo according to TL;DR section above

    git-bash$ vim /etc/restic/*
    git-bash$ source /etc/restic/default.env.sh
    git-bash$ restic init
    git-bash$ restic_backup.sh
   

   Note that you should use cygwin/git-bash paths e.g. in default.env.sh you can have

    export RESTIC_BACKUP_PATHS='/c/Users/<username>/My Documents'
   

6. Inspect the installed tasks and make a test run
   1. Open the app "Task Scheduler" (taskschd.msc)
   2. Go to the local "Task Scheduler Library"
   3. Right click on one of the newly installed tasks e.g. restic_backup and click "run".
      • If the tasks are not there, maybe you opended it up before make install-schedtask: just close and start it again to refresh.
   4. Now a git-bash window should open running restic_backup.sh, and the next time the configured schedule hits!
7. With taskschd.msc you can easily start, stop, delete and configure the scheduled tasks to your liking.

After installing, you can control your backups through tasksched.msc:

Setup Cron

If you want to run an all-classic cron job instead, do like this:

1. Follow the main setup from Step-by-step and manual setup but skip the systemd parts.
2. etc/cron.d/restic: Depending on your system's cron, put this in /etc/cron.d/ or similar, or copy the contents to $(sudo crontab -e). The format of this file is tested under FreeBSD, and might need adaptions depending on your cron.

• You can use $ make install-cron to copy it over to /etc/cron.d.

1. (Optional) bin/cron_mail: A wrapper for running cron jobs, that sends output of the job as an email using the mail(1) command.

Uninstall

There is a make target to remove all files (scripts and configs) that were installed by sudo make install-*. Just run:

$ sudo make uninstall

Variations

A list of variations of this setup:

• Using --files-from #44

Development

• To not mess up your real installation when changing the Makefile simply install to a $PREFIX like

  $ PREFIX=/tmp/restic-test make install-systemd
  

• Updating the resticw parser: If you ever update the usage DOC, you will need to refresh the auto-generated parser:

  $ pip install doctopt.sh
  $ doctopt.sh usr/local/bin/resticw
  

Releasing

To make a new release: 1.

$ vi CHANGELOG.md && git commit -am "Update CHANGELOG.md"
$ git tag vX.Y.Z
$ git push && git push --tags

1. Update version in the AUR PKGBUILD.
2. Update version in the Homebrew Formula.