80 Commits

Author SHA1 Message Date
Erik Westrup
b838ebcb1a Modified: CHANGELOG.md 2022-02-11 22:08:10 +01:00
Erik Westrup
8f118b6ddf Allow make to override etc dir 2022-02-11 22:07:26 +01:00
Erik Westrup
fc01b56e1e Modified: CHANGELOG.md 2022-02-11 21:47:34 +01:00
Erik Westrup
289016f8ea No need to write perm on scripts 2022-02-11 21:46:56 +01:00
Erik Westrup
0c43f93901 Modified: CHANGELOG.md 2022-02-11 20:45:49 +01:00
Erik Westrup
2a287910cb Allow override INSTALL_PREFIX on make cli 2022-02-11 20:34:33 +01:00
Erik Westrup
5f3ebd5b9d resticw: fix install prefix 2022-02-11 10:27:35 +01:00
Erik Westrup
8111a5f663 Bash is dependency 2022-02-10 10:58:41 +01:00
Erik Westrup
ae900a3834 Merge pull request #75 from gerardbosch/patch-1 2022-02-09 18:47:21 +01:00
Gerard Bosch
1db2395353 Minor typos 2022-02-09 18:41:20 +01:00
Erik Westrup
59899f5683 Merge pull request #74 from erikw/launchagent
Add macOS launchagent support
2022-02-09 17:25:31 +01:00
Erik Westrup
4cae1ca2f0 Update CHANGELOG.md
Fixes #51
2022-02-09 17:23:09 +01:00
Erik Westrup
caf95ef393 Add make target to activate launchagent 2022-02-09 17:16:18 +01:00
Erik Westrup
99b0bf5ac5 Modified: README.md 2022-02-09 16:29:14 +01:00
Erik Westrup
d20954b9f4 Modified: README.md 2022-02-09 15:58:26 +01:00
Erik Westrup
991476715a install instructions 2022-02-09 15:56:37 +01:00
Erik Westrup
c4429681d8 Modified: Library/LaunchAgents/com.github.erikw.restic-automatic-backup.plist 2022-02-09 15:49:25 +01:00
Erik Westrup
f073f10ddf Modified: Library/LaunchAgents/com.github.erikw.restic-automatic-backup.plist 2022-02-09 15:47:04 +01:00
Erik Westrup
a28160371c Fix log path 2022-02-09 15:45:49 +01:00
Erik Westrup
1977b3c67f Add OS icons 2022-02-09 15:31:42 +01:00
Erik Westrup
9412a8750c trim 2022-02-09 15:08:57 +01:00
Erik Westrup
d38616d697 Path working 2022-02-09 14:58:31 +01:00
Erik Westrup
c8ab0b4421 iterate 2022-02-09 14:04:07 +01:00
Erik Westrup
c1c25d718c schedule 2022-02-09 13:32:25 +01:00
Erik Westrup
f7f69a0142 instructions 2022-02-09 13:32:19 +01:00
Erik Westrup
2f4f48a59b Modified: CHANGELOG.md 2022-02-09 13:09:10 +01:00
Erik Westrup
e9d77d0a31 Surpress mkdir 2022-02-09 13:08:12 +01:00
Erik Westrup
bd3123bef4 Modified: Makefile 2022-02-09 13:08:12 +01:00
Erik Westrup
ddd2233c58 Install LaunchAgent in user Library 2022-02-09 13:08:12 +01:00
Erik Westrup
7973eedae4 Modified: Makefile 2022-02-09 13:08:12 +01:00
Erik Westrup
db26c4e517 Launch instructions 2022-02-09 13:08:12 +01:00
Erik Westrup
193bf926cf PoC souce files 2022-02-09 13:08:12 +01:00
Erik Westrup
8ea1d70a15 Merge pull request #72 from gerardbosch/patch-1
Update comment
2022-02-09 13:07:38 +01:00
Gerard Bosch
8eb7d56fdd Update comment 2022-02-09 12:59:54 +01:00
Erik Westrup
03d9399686 Merge pull request #71 from gerardbosch/fix/resticw-arguments
Fix resticw arguments line
2022-02-08 18:57:42 +01:00
Gerard Bosch
59da5da0da Fix resticw arguments line
It needs to be treated as an array so that complex commands like
`stats latest` work.
2022-02-08 18:51:20 +01:00
Erik Westrup
7a1409ca03 Allow users to execute scripts 2022-02-08 18:20:49 +01:00
Erik Westrup
b5dcce5d2a Adjust comment 2022-02-08 17:58:46 +01:00
Erik Westrup
e0139ff9c8 Address linter issues 2022-02-08 17:33:24 +01:00
Erik Westrup
78f5cb7b62 Modified: CHANGELOG.md 2022-02-08 17:27:22 +01:00
Erik Westrup
ab928d0dfd Rename *.env files to *.env.sh
Fixes #66
2022-02-08 17:26:08 +01:00
Erik Westrup
27d757fc29 Merge pull request #65 from erikw/fix/49
Truly support custom `PREFIX=` install
2022-02-08 17:17:25 +01:00
Erik Westrup
6c1d182e21 Only sleep in service if other restic running 2022-02-08 17:13:26 +01:00
Erik Westrup
e688499838 Surpress mkdir in make install 2022-02-08 16:49:10 +01:00
Gerard Bosch
2dbd618344 Fix README: Enabling/starting systemd unit (#69)
systemctl start restic-backup@default.timer was failing with the following message:

```console
❯ sudo systemctl start restic-backup@default.timer
Failed to start restic-backup@default.timer: Unit nm-unmetered-connection.service failed to load properly, please adjust/correct and reload service manager: Device or resource busy
See system logs and 'systemctl status restic-backup@default.timer' for details.
```

and by doing `systemctl daemon-reload` before that command solved the issue.

But according to `man systemd` in order to avoid this, it seems that `enable` command must be issued before `start`, as it reloads the system manager configuration (in a way equivalent to daemon-reload). So, once properly sorted, it seems that both commands can be fused into a single `enable --now` command :)
2022-02-08 16:43:30 +01:00
Gerard Bosch
b4684d3cfb Fix README: Backup frequency (#68) 2022-02-08 13:01:32 +01:00
Erik Westrup
645df1a0d4 Move /sbin to /bin
As more users would have /bin than /sbin in their PATH
2022-02-07 18:22:36 +01:00
Erik Westrup
54e2d17e23 Modified: CHANGELOG.md Makefile 2022-02-06 15:28:56 +01:00
Erik Westrup
0fc7c29c2f Push image down 2022-02-06 15:26:43 +01:00
Erik Westrup
a57971e27e Add navigation tip 2022-02-06 15:25:45 +01:00
Erik Westrup
eb6e7b4a8d Begin structure README after system setup 2022-02-06 15:16:19 +01:00
Erik Westrup
c01bb5a00e Fix broken help target 2022-02-06 15:10:31 +01:00
Erik Westrup
8db3d27174 Rename top-level install targets
"make install" -> "make install-systemd"
so that I could now add a
""make install-cron"
target.

Soon there will be
"make install-macos"
"make install-windows"

Thus it does not make sense to have a general `make install` anymore.
2022-02-06 14:49:00 +01:00
Erik Westrup
1ee0e2c919 Fix cron with multiple profiles 2022-02-06 14:36:41 +01:00
Erik Westrup
21bb0535ee Note about make needing to be GNU make 2022-02-05 15:56:32 +01:00
Erik Westrup
0255ad03d8 Format comments 2022-02-05 15:52:36 +01:00
Erik Westrup
5942762514 Correct build source depdendency 2022-02-05 15:48:58 +01:00
Erik Westrup
6b315814d0 spell 2022-02-05 15:45:45 +01:00
Erik Westrup
dd092af6a1 Remove legacy .template 2022-02-05 15:43:57 +01:00
Erik Westrup
7f86dbd9f4 Consistent macro names in Makefile 2022-02-05 15:40:00 +01:00
Erik Westrup
494bcc29f0 s/SCRIPTS/SCRIPT/g 2022-02-05 15:40:00 +01:00
Erik Westrup
dc28ad2b52 Update README.md
Co-authored-by: Gerard Bosch <30733556+gerardbosch@users.noreply.github.com>
2022-02-05 15:28:14 +01:00
Erik Westrup
c43367fab3 Simpler sed expression 2022-02-05 15:27:43 +01:00
Erik Westrup
67cd6ee94b Add back dependency on builddir 2022-02-05 15:18:39 +01:00
Erik Westrup
a117c95508 Format Makefile 2022-02-05 15:13:01 +01:00
Erik Westrup
fc04da5a2d Fix comment 2022-02-05 15:04:44 +01:00
Erik Westrup
25666f17a0 Remove old out-commented solution 2022-02-05 15:02:53 +01:00
Erik Westrup
ccf39e822d Remove comment not relevant 2022-02-05 15:02:24 +01:00
Erik Westrup
1a890de7ac Remove target dependency that was not cleaned up before 2022-02-05 14:55:24 +01:00
Erik Westrup
4236c93b57 spell 2022-02-05 14:53:32 +01:00
Erik Westrup
861535a86e Check properly for GNU install 2022-02-05 14:48:38 +01:00
Erik Westrup
9760cd05ec Truly support custom PREFIX= install
* Before, doing `$ PREFIX=/usr/local make install` would install files to`/usr/local/usr/local..` which is wrong
* With this PR, files will be installed to the expected location e.g. `/usr/local/etc/restic`
* `Makefile` almost completely rewritten
   * As e.g. `default.env` would source `_global.env`, `default.env` must be edited to find the right location of `_global.env` depending on what `$PREFIX` was set to.
   * see documented build stages in the `Makefile` itself.
   *  Made sure that the rules are correct so that only modifed files are installed, not all at once unnecessarily like before.
* A sub-goal was that the [PKGBUILD](https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=restic-systemd-automatic-backup#n20) for Arch should not need to do any custom install configuration, to keep everything easier to maintain. `$ make install` should work out of the box for Arch.
* Additionally added the `-b` flag to `install(1)` that makes a backup of existing `etc/restic/*` files before installing a newer version.

Fixes #49
2022-02-03 20:52:00 +01:00
Gerard Bosch
3852e305b6 Add resticw (restic wrapper) utility (#60)
The script provides a convenient way to load environment config, deal
with profiles and act as a pass-through to restic. The overall thing is
to improve the UX when running restic, integrating the features this
project provides.

## Note

The script itself is a very simple thing.
The command line parser is auto-generated using docopt.sh driven from
the script's DOC. It can be refreshed upon DOC changes with:
`docopt.sh path/to/resticw`.

## How to use it

### Examples

```console
sudo resticw stats latest

sudo resticw -p profileA snapshots
```

### Help

```console
❯ resticw --help
A little wrapper over restic just to handle profiles and environment loading.

  It loads the backup profile/environment in a subshell to avoid any credential leak (Note: Run it with sudo so it can load the environment).

  Usage:
    resticw [options] <restic_arguments>

    The restic_arguments is just the regular unwrapped restic arguments, e.g. stats latest

  Options:
    -p --profile=<name>        Specify the profile to load or use default [default: default].

  Examples:
    sudo resticw --profile profileA snapshots
    sudo resticw stats latest  # this will use the profile: default
```



Co-authored-by: Erik Westrup <erik.westrup@gmail.com>
2022-02-03 20:40:45 +01:00
Erik Westrup
73bce43f7d s/TODO/*EDIT*/ to not confuse with development TODOs 2022-02-03 14:41:21 +01:00
Erik Westrup
7499dbefcd Modified: .github/workflows/linter.yml 2022-02-02 08:10:19 +01:00
Erik Westrup
10c46e8d54 Manually trigger linter 2022-02-02 07:57:02 +01:00
Erik Westrup
ce73193128 Trigger linter on all usr/local/sbin edits 2022-02-02 07:53:30 +01:00
Erik Westrup
90358c4f7a README: project scope 2022-02-01 19:51:35 +01:00
Erik Westrup
7f3647fb0c Rel should have been major as of breaking change 2022-02-01 19:41:02 +01:00
Erik Westrup
dd073dfc73 Fix arguments by using arrays to buildup 2022-02-01 19:24:38 +01:00
24 changed files with 632 additions and 169 deletions

17
.editorconfig Normal file
View File

@@ -0,0 +1,17 @@
root = true
[*]
indent_style = tab
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
#max_line_length = 120
[Makefile]
# Enforce tab (Makefiles require tabs)
indent_style = tab
[*.md]
trim_trailing_whitespace = false

View File

@@ -1,16 +1,19 @@
name: Lint Code Base
on:
workflow_dispatch:
push:
branches: master
paths:
- '**.sh'
- '.github/workflows/linter.yml'
- 'bin/**'
pull_request:
branches: master
paths:
- '**.sh'
- '.github/workflows/linter.yml'
- 'bin/**'
jobs:
build:
name: Lint Code Base

6
.gitignore vendored
View File

@@ -1,7 +1,5 @@
# Prevent check-in of these sensitive files. Instead they are generated from the corresponding *.template file.
etc/restic/pw.txt
etc/restic/_global.env
etc/restic/default.env
# make install
/build
# IntelliJ
.idea/

View File

@@ -6,6 +6,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [5.2.0] - 2022-02-11
### Added
- Make option to override destination dir for configuration files. Needed for Homebrew.
### Changed
- Write permissions on installed scripts removed (0755 -> 0555). Homebrew was complaining.
## [5.1.0] - 2022-02-11
### Added
- macos LaunchAgent support. Install with `make install-launchagent` and activate with `make activate-launchagent`. See [README.md](README.md) for details.
- make option INSTALL_PREFIX to make PKGBUILD and such easier to write.
## [5.0.0] - 2022-02-08
### Added
- `resticw` wrapper for working with different profiles without the need to source the profiles first.
- `$ make install-systemd` will now make a timestamped backup of any existing `/etc/restic/*` files before installing a newer version.
- `$ make install-cron` for installing the cron-job.
### Changed
- **BREAKING CHANGE** moved systemd installation with makefile from `/etc/systemd/system` to `/usr/lib/systemd/system` as this is what packages should do. This is to be able to simplify the arch [PKGBUILD](https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=restic-systemd-automatic-backup) so that it does not need to do anything else than `make install`.
- If you upgrade form an existing install, you should disable and then re-enable the timer, so that the symlink is pointing to the new location of the timer.
```console
# systemctl disable restic-backup@<profile>.timer
# systemctl enable restic-backup@<profile>.timer
```
- **BREAKING CHANGE** moved script installation with makefile from `/usr/local/sbin` to `/bin` to have a simpler interface to work with `$PREFIX`.
- **BREAKING CHANGE** renamed `etc/restic/*.env` files to `etc/restic/*.env.sh` to clearly communicate that it's a shell script that will be executed (source), and also hint at code editors what file this is to set corect syntax highligting etc. This also enables the shellcheck linter to work more easily on these files as well.
- Renamed top level make install targets. The old `$ make install` is now `$ make install-systemd`
### Fixed
- Installation with custom `PREFIX` now works properly with Make: `$ PREFIX=/usr/local make install` whill now install everything at the expected location. With this, it's easy to use this script as non-root user on e.g. an macOS system.
## [4.0.0] - 2022-02-01
### Fixed
- Use arrays to build up command lines. When fixing `shellcheck(1)` errors, quotes would disable expansion on e.g. $RESTIC_BACKUP_PATHS
- **BREAKING CHANGE** `RESTIC_BACKUP_PATHS` is now a string with `:` separated values
## [3.0.1] - 2022-02-01
### Fixed
- Environment variable assertion should allow empty values e.g. `RESTIC_BACKUP_EXTRA_ARGS`
@@ -39,6 +76,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- `restic_backup.sh` now finds `.backup_exclude` files on each backup path as intended.
- Install executeables to `$PREFIX/sbin` instead of `$PREFIX/user/local/sbin`, so that `$ PREFIX=/usr/local make install` does what is expected.
## [1.0.1] - 2021-12-03
### Fixed

View File

@@ -0,0 +1,39 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- See launchd.plist(5)for documentation on this file. -->
<!-- See https://www.launchd.info/ for a tutorial. -->
<!-- Debug with: $ tail -f /var/log/com.apple.xpc.launchd/launchd.log | grep erikw.restic -->
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.github.erikw.restic-automatic-backup</string>
<key>ProgramArguments</key>
<!-- exec in subshell to 1) source *.env.sh 2) expand $HOME to logpath (ref. https://apple.stackexchange.com/a/365880/197493) -->
<array>
<string>/bin/bash</string>
<string>-c</string>
<string>source /usr/local/etc/restic/erikw.env.sh &amp;&amp; /usr/local/bin/restic_backup.sh >>$HOME/$LOG_OUT 2>>$HOME/$LOG_ERR</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key>
<string>/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
<key>LOG_OUT</key>
<string>/Library/Logs/restic/restic_stdout.log</string>
<key>LOG_ERR</key>
<string>/Library/Logs/restic/restic_stderr.log</string>
</dict>
<key>RunAtLoad</key>
<true/>
<!-- Will schedule backup every day at 19:00 -->
<key>StartCalendarInterval</key>
<array>
<dict>
<key>Hour</key>
<integer>19</integer>
<key>Minute</key>
<integer>0</integer>
</dict>
</array>
</dict>
</plist>

222
Makefile
View File

@@ -1,59 +1,189 @@
# Not file targets.
.PHONY: help install install-scripts install-conf install-systemd uninstall
# TODO add install for launchagent completely, and unisntall target using bootstrap?
#### Notes ####################################################################
# This build process is done in three stages (out-of-source build):
# 1. copy source files to the local build directory.
# 2. build dir: replace the string "$INSTALL_PREFIX" with the value of $PREFIX
# 3. install files from the build directory to the target directory.
#
# Why this dance?
# * To fully support that a user can install this project to a custom path e.g.
# $(PREFIX=/usr/local make install), we need to modify the files that refer
# to other files on disk. We do this by having a placeholder
# "$INSTALL_PREFIX" that is substituted with the value of $PREFIX when
# installed.
# * We don't want to modify the files that are controlled by git, thus let's
# copy them to a build directory and then modify.
### Macros ###
SRCS_SCRIPTS = $(filter-out %cron_mail, $(wildcard usr/local/sbin/*))
# $(sort) remove duplicates that comes from running make install >1 times.
SRCS_CONF = $(sort $(patsubst %.template, %, $(wildcard etc/restic/*)))
SRCS_SYSTEMD = $(wildcard etc/systemd/system/*)
#### Non-file targets #########################################################
.PHONY: help clean uninstall \
install-systemd install-cron \
install-targets-script install-targets-conf install-targets-systemd \
install-targets-cron \
activate-launchagent deactivate-launchagent
# To change the installation root path, set the PREFIX variable in your shell's environment, like:
# $ PREFIX=/usr/local make install
# $ PREFIX=/tmp/test make install
DEST_SCRIPTS = $(PREFIX)/usr/local/sbin
DEST_CONF = $(PREFIX)/etc/restic
DEST_SYSTEMD = $(PREFIX)/etc/systemd/system
#### Macros ###################################################################
NOW := $(shell date +%Y-%m-%d_%H:%M:%S)
INSTALLED_FILES = $(addprefix $(PREFIX)/, $(SRCS_SCRIPTS) $(SRCS_CONF) $(SRCS_SYSTEMD))
# GNU and macOS install have incompatible command line arguments.
GNU_INSTALL := $(shell install --version 2>/dev/null | \
grep -q GNU && echo true || echo false)
ifeq ($(GNU_INSTALL),true)
BAK_SUFFIX = --suffix=.$(NOW).bak
else
BAK_SUFFIX = -B .$(NOW).bak
endif
### Targets ###
# target: all - Default target.
all: install
# Create parent directories of a file, if not existing.
# Reference: https://stackoverflow.com/a/25574592/265508
MKDIR_PARENTS=sh -c '\
dir=$$(dirname $$1); \
test -d $$dir || mkdir -p $$dir \
' MKDIR_PARENTS
# target: help - Display all targets.
# LaunchAgent names.
UID := $(shell id -u)
LAUNCHAGENT = com.github.erikw.restic-automatic-backup
LAUNCHAGENT_TARGET = gui/$(UID)/$(LAUNCHAGENT)
# What to substitute $INSTALL_PREFIX in sources to.
# This can be useful to set to empty on commandline when building e.g. an AUR
# package in a separate build directory (PREFIX).
INSTALL_PREFIX := $(PREFIX)
# Where to install persistent configuration files. Used by Homebrew.
SYSCONFDIR := $(PREFIX)
# Source directories.
DIR_SCRIPT = bin
DIR_CONF = etc/restic
DIR_SYSTEMD = usr/lib/systemd/system
DIR_CRON = etc/cron.d
DIR_LAUNCHAGENT = Library/LaunchAgents
# Source files.
SRCS_SCRIPT = $(filter-out %cron_mail, $(wildcard $(DIR_SCRIPT)/*))
SRCS_CONF = $(wildcard $(DIR_CONF)/*)
SRCS_SYSTEMD = $(wildcard $(DIR_SYSTEMD)/*)
SRCS_CRON = $(wildcard $(DIR_CRON)/*)
SRCS_LAUNCHAGENT= $(wildcard $(DIR_LAUNCHAGENT)/*)
# Local build directory. Sources will be copied here,
# modified and then installed from this directory.
BUILD_DIR := build
BUILD_DIR_SCRIPT = $(BUILD_DIR)/$(DIR_SCRIPT)
BUILD_DIR_CONF = $(BUILD_DIR)/$(DIR_CONF)
BUILD_DIR_SYSTEMD = $(BUILD_DIR)/$(DIR_SYSTEMD)
BUILD_DIR_CRON = $(BUILD_DIR)/$(DIR_CRON)
BUILD_DIR_LAUNCHAGENT = $(BUILD_DIR)/$(DIR_LAUNCHAGENT)
# Sources copied to build directory.
BUILD_SRCS_SCRIPT = $(addprefix $(BUILD_DIR)/, $(SRCS_SCRIPT))
BUILD_SRCS_CONF = $(addprefix $(BUILD_DIR)/, $(SRCS_CONF))
BUILD_SRCS_SYSTEMD = $(addprefix $(BUILD_DIR)/, $(SRCS_SYSTEMD))
BUILD_SRCS_CRON = $(addprefix $(BUILD_DIR)/, $(SRCS_CRON))
BUILD_SRCS_LAUNCHAGENT = $(addprefix $(BUILD_DIR)/, $(SRCS_LAUNCHAGENT))
# Destination directories
DEST_DIR_SCRIPT = $(PREFIX)/$(DIR_SCRIPT)
DEST_DIR_CONF = $(SYSCONFDIR)/$(DIR_CONF)
DEST_DIR_SYSTEMD = $(PREFIX)/$(DIR_SYSTEMD)
DEST_DIR_CRON = $(PREFIX)/$(DIR_CRON)
DEST_DIR_LAUNCHAGENT= $(HOME)/$(DIR_LAUNCHAGENT)
DEST_DIR_MAC_LOG = $(HOME)/Library/Logs/restic
# Destination file targets.
DEST_TARGS_SCRIPT = $(addprefix $(PREFIX)/, $(SRCS_SCRIPT))
DEST_TARGS_CONF = $(addprefix $(SYSCONFDIR)/, $(SRCS_CONF))
DEST_TARGS_SYSTEMD = $(addprefix $(PREFIX)/, $(SRCS_SYSTEMD))
DEST_TARGS_CRON = $(addprefix $(PREFIX)/, $(SRCS_CRON))
DEST_TARGS_LAUNCHAGENT = $(addprefix $(HOME)/, $(SRCS_LAUNCHAGENT))
INSTALLED_FILES = $(DEST_TARGS_SCRIPT) $(DEST_TARGS_CONF) \
$(DEST_TARGS_SYSTEMD) $(DEST_TARGS_CRON) \
$(DEST_TARGS_LAUNCHAGENT)
#### Targets ##################################################################
# target: help - Default target; displays all targets.
help:
@egrep "#\starget:" [Mm]akefile | sed 's/\s-\s/\t\t\t/' | cut -d " " -f3- | sort -d
@egrep "#\starget:" [Mm]akefile | cut -d " " -f3- | sort -d
# target: install - Install all files
install: install-scripts install-conf install-systemd
# target: clean - Remove build files.
clean:
$(RM) -r $(BUILD_DIR)
# target: install-scripts - Install executables.
install-scripts:
install -d $(DEST_SCRIPTS)
install -m 0744 $(SRCS_SCRIPTS) $(DEST_SCRIPTS)
# Copy templates to new files with restricted permissions.
# Why? Because the non-template files are git-ignored to prevent that someone who clones or forks this repo checks in their sensitive data like the B2 password!
etc/restic/_global.env etc/restic/default.env etc/restic/pw.txt:
install -m 0600 $@.template $@
# target: install-conf - Install restic configuration files.
# will create these files locally only if they don't already exist
# `|` means that dependencies are order-only, i.e. only created if they don't already exist.
install-conf: | $(SRCS_CONF)
install -d $(DEST_CONF)
install -b -m 0600 $(SRCS_CONF) $(DEST_CONF)
$(RM) etc/restic/_global.env etc/restic/default.env etc/restic/pw.txt
# target: install-systemd - Install systemd timer and service files.
install-systemd:
install -d $(DEST_SYSTEMD)
install -m 0644 $(SRCS_SYSTEMD) $(DEST_SYSTEMD)
# target: uninstall - Uninstall ALL files from the install targets.
# target: uninstall - Uninstall ALL installed (including config) files.
uninstall:
@for file in $(INSTALLED_FILES); do \
echo $(RM) $$file; \
$(RM) $$file; \
done
# To change the installation root path,
# set the PREFIX variable in your shell's environment, like:
# $ PREFIX=/usr/local make install-systemd
# $ PREFIX=/tmp/test make install-systemd
# target: install-systemd - Install systemd setup.
install-systemd: install-targets-script install-targets-conf \
install-targets-systemd
# target: install-cron - Install cron setup.
install-cron: install-targets-script install-targets-conf install-targets-cron
# target: install-launchagent - Install LaunchAgent setup.
install-launchagent: install-targets-script install-targets-conf \
install-targets-launchagent
# Install targets. Prereq build sources as well,
# so that build dir is re-created if deleted.
install-targets-script: $(DEST_TARGS_SCRIPT) $(BUILD_SRCS_SCRIPT)
install-targets-conf: $(DEST_TARGS_CONF) $(BUILD_SRCS_CONF)
install-targets-systemd: $(DEST_TARGS_SYSTEMD) $(BUILD_SRCS_SYSTEMD)
install-targets-cron: $(DEST_TARGS_CRON) $(BUILD_SRCS_CRON)
install-targets-launchagent: $(DEST_TARGS_LAUNCHAGENT) \
$(BUILD_SRCS_LAUNCHAGENT) $(DEST_DIR_MAC_LOG)
# Copies sources to build directory & replace "$INSTALL_PREFIX".
$(BUILD_DIR)/% : %
@${MKDIR_PARENTS} $@
cp $< $@
sed -i.bak -e 's|$$INSTALL_PREFIX|$(INSTALL_PREFIX)|g' $@; rm $@.bak
# Install destination script files.
$(DEST_DIR_SCRIPT)/%: $(BUILD_DIR_SCRIPT)/%
@${MKDIR_PARENTS} $@
install -m 0555 $< $@
# Install destination conf files. Additionally backup existing files.
$(DEST_DIR_CONF)/%: $(BUILD_DIR_CONF)/%
@${MKDIR_PARENTS} $@
install -m 0600 -b $(BAK_SUFFIX) $< $@
# Install destination systemd files.
$(DEST_DIR_SYSTEMD)/%: $(BUILD_DIR_SYSTEMD)/%
@${MKDIR_PARENTS} $@
install -m 0644 $< $@
# Install destination cron files.
$(DEST_DIR_CRON)/%: $(BUILD_DIR_CRON)/%
@${MKDIR_PARENTS} $@
install -m 0644 $< $@
# Install destination launchagent files.
$(DEST_DIR_LAUNCHAGENT)/%: $(BUILD_DIR_LAUNCHAGENT)/%
@${MKDIR_PARENTS} $@
install -m 0444 $< $@
# Install destination mac log dir.
$(DEST_DIR_MAC_LOG):
mkdir -p $@
# target: activate-launchagent - Activate the LaunchAgent.
activate-launchagent:
launchctl bootstrap gui/$(UID) $(DEST_TARGS_LAUNCHAGENT)
launchctl enable $(LAUNCHAGENT_TARGET)
launchctl kickstart -p $(LAUNCHAGENT_TARGET)
# target: deactivate-launchagent - Deactivate and remove the LaunchAgent.
deactivate-launchagent:
launchctl bootout $(LAUNCHAGENT_TARGET)

187
README.md
View File

@@ -27,31 +27,50 @@ Here follows a step-by step tutorial on how to set it up, with my sample script
Note, you can use any of the supported [storage backends](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html). The setup should be similar but you will have to use other configuration variables to match your backend of choice.
## Project Scope
The scope for this is not to be a full-fledged super solution that solves all the problems and all possible setups. The aim is to be a hackable code base for you to start sewing up the perfect backup solution that fits your requirements!
Nevertheless the project should work out of the box, be minimal but still open the doors for configuration and extensions by users.
## Navigate this README
Tip: use the Section icon in the top left of this document to navigate the sections.
![README Sections](img/readme_sections.png)
# Requirements
* `restic >=v0.9.6`
* (recommended) `make` if you want an automated install
* Arch: part of the `base-devel` meta package, Debian/Ubuntu: part of the `build-essential` meta package, macOS: preinstalled make works)
* `bash >=v4.0.0`
* (recommended) GNU `make` if you want an automated install
* Arch: part of the `base-devel` meta package, Debian/Ubuntu: part of the `build-essential` meta package, macOS: use the preinstalled or a more recent with Homebrew)
# TL;DR Setup
# Setup
Depending on your system, the setup will look different. Choose one of
* <img height="16" width="16" src="https://unpkg.com/simple-icons@v6/icons/linux.svg" /> [Linux + Systemd](#setup-linux-systemd)
* <img height="16" width="16" src="https://unpkg.com/simple-icons@v6/icons/apple.svg" /> [macOS + LaunchAgent](#setup-macos-launchagent)
* <img height="16" width="16" src="https://unpkg.com/simple-icons@v6/icons/clockify.svg" /> [Cron](#setup-cron) - for any system having a cron daemon. Tested on FreeBSD and macOS.
## Setup Linux Systemd
### TL;DR Setup
1. Create B2 credentials as instructed [below](#1-create-backblaze-b2-account)
1. Install config and scripts:
```console
$ sudo make install
$ sudo make install-systemd
```
☝ **Note**: `sudo` is required here, as some files are installed into system directories (`/etc/`
and `/usr/sbin`). Have a look to the `Makefile` to know more.
and `/usr/bin`). Have a look to the `Makefile` to know more.
1. Fill out configuration values (edit with sudo):
* `/etc/restic/pw.txt` - Contains the password (single line) to be used by restic to encrypt the repository files. Should be different than your B2 password!
* `/etc/restic/_global.env` - Global environment variables.
* `/etc/restic/default.env` - Profile specific environment variables (multiple profiles can be defined by copying to `/etc/restic/something.env`).
* `/etc/restic/_global.env.sh` - Global environment variables.
* `/etc/restic/default.env.sh` - Profile specific environment variables (multiple profiles can be defined by copying to `/etc/restic/something.env.sh`).
* `/etc/restic/backup_exclude.txt` - List of file patterns to ignore. This will trim down your backup size and the speed of the backup a lot when done properly!
1. Initialize remote repo as described [below](#3-initialize-remote-repo)
1. Configure [how often](https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events) back up should be made.
* Edit if needed `OnCalendar` in `/etc/systemd/system/restic-check@.timer`.
* Edit if needed `OnCalendar` in `/usr/lib/systemd/system/restic-backup@.timer`.
1. Enable automated backup for starting with the system (`enable` creates symlinks):
```console
$ sudo systemctl start restic-backup@default.timer
$ sudo systemctl enable restic-backup@default.timer
$ sudo systemctl enable --now restic-backup@default.timer
```
1. And run an immediate backup if you want (if not, it will run on daily basis):
```console
@@ -64,46 +83,58 @@ Note, you can use any of the supported [storage backends](https://restic.readthe
1. Verify the backup
```console
$ sudo -i
$ source /etc/restic/default.env
$ source /etc/restic/default.env.sh
$ restic snapshots
```
1. (optional) Define multiple profiles: just make a copy of the `default.env` and use the defined profile name in place of `default` to run backups or enable timers. Notice that the value after `@` works as a parameter.
1. (optional) Define multiple profiles: just make a copy of the `default.env.sh` and use the defined profile name in place of `default` to run backups or enable timers. Notice that the value after `@` works as a parameter.
1. (optional) Enable the check job that verifies that the backups for the profile are all intact.
```console
$ sudo systemctl start restic-check@default.timer
$ sudo systemctl enable restic-check@default.timer
$ sudo systemctl enable --now restic-check@default.timer
````
1. (optional) Setup email on failure as described [here](#8-email-notification-on-failure)
# Step-by-step and manual setup
### Step-by-step and manual setup
This is a more detailed explanation than the TL;DR section that will give you more understanding in the setup, and maybe inspire you to develop your own setup based on this one even!
Tip: The steps in this section will instruct you to copy files from this repo to system directories. If you don't want to do this manually, you can use the Makefile:
```console
$ git clone https://github.com/erikw/restic-systemd-automatic-backup.git && cd $(basename "$_" .git)
$ sudo make install
$ sudo make install-systemd
````
If you want to install everything manually, we will install files to `/etc`, `/bin`, and not use the `$make install-systemd` command, then you need to clean up a placeholder `$INSTALL_PREFIX` in the souce files first by running:
```console
$ find etc bin -type f -exec sed -i.bak -e 's|$INSTALL_PREFIX||g' {} \; -exec rm {}.bak \;
```
and you should now see that all files have been changed like e.g.
```diff
-export RESTIC_PASSWORD_FILE="$INSTALL_PREFIX/etc/restic/pw.txt"
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
```
This prefix is there so that `make` users can set a different `$PREFIX` when installing like `PREFIX=/usr/local make install-systemd`. So if we don't use the makefile, we need to remove this prefix with the command above just.
Arch Linux users can install the aur package [restic-systemd-automatic-backup](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/) e.g.:
```console
$ yaourt -S restic-systemd-automatic-backup
````
## 1. Create Backblaze B2 Account, Bucket and keys
#### 1. Create Backblaze B2 Account, Bucket and keys
First, see this official Backblaze [tutorial](https://help.backblaze.com/hc/en-us/articles/4403944998811-Quickstart-Guide-for-Restic-and-Backblaze-B2-Cloud-Storage) on restic, and follow the instructions ("Create Backblaze account with B2 enabled") there on how to create a new B2 bucket. In general, you'd want a private bucket, without B2 encryption (restic does the encryption client side for us) and without the object lock feature.
For restic to be able to connect to your bucket, you want to in the B2 settings create a pair of keyID and applicationKey. It's a good idea to create a separate pair of ID and Key with for each bucket that you will use, with limited read&write access to only that bucket.
## 2. Configure your B2 credentials locally
#### 2. Configure your B2 credentials locally
> **Attention!** Going the manual way requires that most of the following commands are run as root.
Put these files in `/etc/restic/`:
* `_global.env`: Fill this file out with your global settings including B2 keyID & applicationKey. A global exclude list is set here (explained in section below).
* `default.env`: This is the default profile. Fill this out with bucket name, backup paths and retention policy. This file sources `_global.env` and is thus self-contained and can be sourced in the shell when you want to issue some manual restic commands. For example:
* `_global.env.sh`: Fill this file out with your global settings including B2 keyID & applicationKey. A global exclude list is set here (explained in section below).
* `default.env.sh`: This is the default profile. Fill this out with bucket name, backup paths and retention policy. This file sources `_global.env.sh` and is thus self-contained and can be sourced in the shell when you want to issue some manual restic commands. For example:
```console
$ source /etc/restic/default.env
$ source /etc/restic/default.env.sh
$ restic snapshots # You don't have to supply all parameters like --repo, as they are now in your environment!
````
* `pw.txt`: This file should contain the restic password used to encrypt the repository. This is a new password what soon will be used when initializing the new repository. It should be unique to this restic backup repository and is needed for restoring from it. Don't re-use your B2 login password, this should be different. For example you can generate a 128 character password (must all be on one line) with:
@@ -111,36 +142,36 @@ Put these files in `/etc/restic/`:
$ openssl rand -base64 128 | tr -d '\n' > /etc/restic/pw.txt
```
## 3. Initialize remote repo
#### 3. Initialize remote repo
Now we must initialize the repository on the remote end:
```console
$ sudo -i
$ source /etc/restic/default.env
$ source /etc/restic/default.env.sh
$ restic init
```
## 4. Script for doing the backup
Put this file in `/usr/local/sbin`:
* `restic_backup.sh`: A script that defines how to run the backup. The intention is that you should not need to edit this script yourself, but be able to control everything from the `*.env` profiles.
#### 4. Script for doing the backup
Put this file in `/bin`:
* `restic_backup.sh`: A script that defines how to run the backup. The intention is that you should not need to edit this script yourself, but be able to control everything from the `*.env.sh` profiles.
Restic support exclude files. They list file pattern paths to exclude from you backups, files that just occupy storage space, backup-time, network and money. `restic_backup.sh` allows for a few different exclude files.
* `/etc/restic/backup_exclude.txt` - global exclude list. You can use only this one if your setup is easy. This is set in `_global.env`. If you need a different file for another profile, you can override the envvar `RESTIC_BACKUP_EXCLUDE_FILE` in this profile.
* `/etc/restic/backup_exclude.txt` - global exclude list. You can use only this one if your setup is easy. This is set in `_global.env.sh`. If you need a different file for another profile, you can override the envvar `RESTIC_BACKUP_EXCLUDE_FILE` in this profile.
* `.backup_exclude.txt` per backup path. If you have e.g. an USB disk mounted at /mnt/media and this path is included in the `$RESTIC_BACKUP_PATHS`, you can place a file `/mnt/media/.backup_exclude.txt` and it will automatically picked up. The nice thing about this is that the backup paths are self-contained in terms of what they shoud exclude!
## 5. Make first backup
#### 5. Make first backup
Now see if the backup itself works, by running as root
```console
$ sudo -i
$ source /etc/restic/default.env
$ /usr/local/sbin/restic_backup.sh
$ source /etc/restic/default.env.sh
$ /bin/restic_backup.sh
````
## 6. Verify the backup
As the `default.env` is already sourced in your root shell, you can now just list the snapshos
#### 6. Verify the backup
As the `default.env.sh` is already sourced in your root shell, you can now just list the snapshos
```console
$ sudo -i
$ source /etc/restic/default.env
$ source /etc/restic/default.env.sh
$ restic snapshots
```
@@ -150,11 +181,10 @@ $ restic mount /mnt/restic
$ ls /mnt/restic
```
## 7. Backup automatically; systemd service + timer
#### 7. Backup automatically; systemd service + timer
Now we can do the modern version of a cron-job, a systemd service + timer, to run the backup every day!
Put these files in `/etc/systemd/system/`:
Put these files in `/etc/systemd/system` (note that the Makefile installs as package to `/usr/lib/systemd/system`)
* `restic-backup@.service`: A service that calls the backup script with the specified profile. The profile is specified
by the value after `@` when running it (see below).
* `restic-backup@.timer`: A timer that starts the former backup every day (same thing about profile here).
@@ -162,8 +192,7 @@ Put these files in `/etc/systemd/system/`:
Now simply enable the timer with:
```console
$ systemctl start restic-backup@default.timer
$ systemctl enable restic-backup@default.timer
$ sudo systemctl enable --now restic-backup@default.timer
````
☝ **Note**: You can run it with different values instead of `default` if you use multiple profiles.
@@ -195,10 +224,10 @@ $ journalctl -f -u restic-backup@default.service
## 8. Email notification on failure
#### 8. Email notification on failure
We want to be aware when the automatic backup fails, so we can fix it. Since my laptop does not run a mail server, I went for a solution to set up my laptop to be able to send emails with [postfix via my Gmail](https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/). Follow the instructions over there.
Put this file in `/usr/local/sbin`:
Put this file in `/bin`:
* `systemd-email`: Sends email using sendmail(1). This script also features time-out for not spamming Gmail servers and getting my account blocked.
Put this files in `/etc/systemd/system/`:
@@ -207,28 +236,73 @@ Put this files in `/etc/systemd/system/`:
As you maybe noticed already before, `restic-backup.service` is configured to start `status-email-user.service` on failure.
## 9. Optional: automated backup checks
#### 9. Optional: automated backup checks
Once in a while it can be good to do a health check of the remote repository, to make sure it's not getting corrupt. This can be done with `$ restic check`.
There is companion scripts, service and timer (`*check*`) to restic-backup.sh that checks the restic backup for errors; look in the repo in `etc/systemd/system` and `usr/local/sbin` and copy what you need over to their corresponding locations.
There is companion scripts, service and timer (`*check*`) to restic-backup.sh that checks the restic backup for errors; look in the repo in `usr/lib/systemd/system/` and `bin/` and copy what you need over to their corresponding locations.
```console
$ sudo -i
$ systemctl start restic-check@default.timer
$ systemctl enable restic-check@default.timer
$ sudo systemctl enable --now restic-check@default.timer
````
#### 10. Optional: 🏃 Restic wrapper
For convenience there's a `restic` wrapper script that makes loading profiles and **running restic**
straightforward (it needs to run with sudo to read environment). Just run:
- `sudo resticw WHATEVER` (e.g. `sudo resticw snapshots`) to use the default profile.
- You can run the wrapper by passing a specific profile: `resticw -p anotherprofile snapshots`.
# Cron?
##### Useful commands
| Command | Description |
|---------------------------------------------------|-------------------------------------------------------------------|
| `resticw snapshots` | List backup snapshots |
| `resticw diff <snapshot-id> latest` | Show the changes from the latest backup |
| `resticw stats` / `resticw stats snapshot-id ...` | Show the statistics for the whole repo or the specified snapshots |
| `resticw mount /mnt/restic` | Mount your remote repository |
## Setup macOS LaunchAgent
LaunchAgent is the modern service scheduler in in macOS that uses [Launchd](https://www.launchd.info/).
[Launchd](https://www.launchd.info/) is the modern built-in service scheduler in macOS. It has support for running services as root (Daemon) or as a normal user (Agent). Here we set up an LauchAgent to be run as your normal user for starting regular backups.
1. In general, follow the same setup as in (#setup-linux-systemd) except for:
* use `make install-launchagent` instead of `make install-systemd`
* install everything to `/usr/local` and run restic as your own user, not root
* Thus, install with
```console
$ PREFIX=/usr/local make install-launchagent
```
1. After installation with `make` , edit the installed LaunchAgent if you want to change the default schedule or profile used:
```console
$ vim ~/Library/LaunchAgents/com.github.erikw.restic-automatic-backup.plist
```
1. Now install, enable and start the first run!
```console
$ launchctl bootstrap gui/$UID ~/Library/LaunchAgents/com.github.erikw.restic-automatic-backup.plist
$ launchctl enable gui/$UID/com.github.erikw.restic-automatic-backup
$ launchctl kickstart -p gui/$UID/com.github.erikw.restic-automatic-backup
```
As a convenience, a shortcut for the above commands are `$ make activate-launchagent`.
Use the `disable` command to temporarily pause the agent, or `bootout` to uninstall it.
```
$ launchctl disable gui/$UID/com.github.erikw.restic-automatic-backup
$ launchctl bootout gui/$UID/com.github.erikw.restic-automatic-backup
```
If you updated the `.plist` file, you need to issue the `bootout` followed by `bootrstrap` and `enable` sub-commands of `launchctl`. This will guarantee that the file is properly reloaded.
## Setup Cron
If you want to run an all-classic cron job instead, do like this:
* `etc/cron.d/restic`: Depending on your system's cron, put this in `/etc/cron.d/` or similar, or copy the contents to $(sudo crontab -e). The format of this file is tested under FreeBSD, and might need adaptions depending on your cron.
* `usr/local/sbin/cron_mail`: A wrapper for running cron jobs, that sends output of the job as an email using the mail(1) command.
1. Follow the main setup from [Step-by-step and manual setup](#step-by-step-and-manual-setup) but skip the systemd parts.
1. `etc/cron.d/restic`: Depending on your system's cron, put this in `/etc/cron.d/` or similar, or copy the contents to $(sudo crontab -e). The format of this file is tested under FreeBSD, and might need adaptions depending on your cron.
* You can use `$ make install-cron` to copy it over to `/etc/cron.d`.
1. (Optional) `bin/cron_mail`: A wrapper for running cron jobs, that sends output of the job as an email using the mail(1) command.
# Uninstall
There is a make target to remove all files (scripts and configs) that were installed by `sudo make install`. Just run:
There is a make target to remove all files (scripts and configs) that were installed by `sudo make install-*`. Just run:
```console
$ sudo make uninstall
@@ -239,10 +313,15 @@ A list of variations of this setup:
* Using `--files-from` [#44](https://github.com/erikw/restic-systemd-automatic-backup/issues/44)
# Development
To not mess up your real installation when changing the `Makefile` simply install to a `$PREFIX` like
```console
$ PREFIX=/tmp/restic-test make install
```
* To not mess up your real installation when changing the `Makefile` simply install to a `$PREFIX` like
```console
$ PREFIX=/tmp/restic-test make install-systemd
```
* **Updating the `resticw` parser:** If you ever update the usage `DOC`, you will need to refresh the auto-generated parser:
```console
$ pip install doctopt.sh
$ doctopt.sh usr/local/bin/resticw
```
# Releasing
To make a new release:

View File

@@ -5,7 +5,7 @@
# Why? Because of FreeBSD the system cron uses sendmail, and I want to use ssmtp.
# Make your crontab files like:
#SHELL=/bin/sh
#PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin
#PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:$INSTALL_PREFIX/bin
#@daily root cron_mail freebsd-update cron
mail_target=root

View File

@@ -5,30 +5,12 @@
# - from restic service/timer: $PREFIX/etc/systemd/system/restic-backup.{service,timer}
# - from a cronjob: $PREFIX/etc/cron.d/restic
# - manually by a user. For it to work, the environment variables must be set in the shell where this script is executed
# $ source $PREFIX/etc/default.env
# $ source $PREFIX/etc/default.env.sh
# $ restic_backup.sh
# Exit on error, unset var, pipe failure
set -euo pipefail
# Assert that all needed environment variables are set.
# TODO in future if this grows, move this to a restic_lib.sh
assert_envvars() {
local varnames=("$@")
for varname in "${varnames[@]}"; do
if [ -z ${!varname+x} ]; then
printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env profile in the current shell before executing this script?\n" "$varname" >&2
exit 1
fi
done
}
assert_envvars \
B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
RESTIC_BACKUP_PATHS RESTIC_BACKUP_TAG \
RESTIC_BACKUP_EXCLUDE_FILE RESTIC_BACKUP_EXTRA_ARGS RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL \
RESTIC_RETENTION_DAYS RESTIC_RETENTION_MONTHS RESTIC_RETENTION_WEEKS RESTIC_RETENTION_YEARS
# Clean up lock if we are killed.
# If killed by systemd, like $(systemctl stop restic), then it kills the whole cgroup and all it's subprocesses.
# However if we kill this script ourselves, we need this trap that kills all subprocesses manually.
@@ -39,15 +21,39 @@ exit_hook() {
}
trap exit_hook INT TERM
# Assert that all needed environment variables are set.
# TODO in future if this grows, move this to a restic_lib.sh
assert_envvars() {
local varnames=("$@")
for varname in "${varnames[@]}"; do
if [ -z ${!varname+x} ]; then
printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
exit 1
fi
done
}
assert_envvars \
B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
RESTIC_BACKUP_PATHS RESTIC_BACKUP_TAG \
RESTIC_BACKUP_EXCLUDE_FILE RESTIC_BACKUP_EXTRA_ARGS RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL \
RESTIC_RETENTION_DAYS RESTIC_RETENTION_MONTHS RESTIC_RETENTION_WEEKS RESTIC_RETENTION_YEARS
# Convert to arrays, as arrays should be used to build command lines. See https://github.com/koalaman/shellcheck/wiki/SC2086
IFS=':' read -ra backup_paths <<< "$RESTIC_BACKUP_PATHS"
IFS=' ' read -ra extra_args <<< "$RESTIC_BACKUP_EXTRA_ARGS"
# Set up exclude files: global + path-specific ones
# NOTE that restic will fail the backup if not all listed --exclude-files exist. Thus we should only list them if they are really all available.
## Global backup configuration.
exclusion_args="--exclude-file ${RESTIC_BACKUP_EXCLUDE_FILE}"
## Self-contained backup files per backup path. E.g. having an USB disk at /mnt/media in RESTIC_BACKUP_PATHS,
# a file /mnt/media/.backup_exclude.txt will automatically be detected and used:
for backup_path in "${RESTIC_BACKUP_PATHS[@]}"; do
exclusion_args=(--exclude-file "$RESTIC_BACKUP_EXCLUDE_FILE")
## Self-contained backup exclusion files per backup path. E.g. having an USB disk at /mnt/media in RESTIC_BACKUP_PATHS,
# then a file /mnt/media/.backup_exclude.txt will automatically be detected and used:
for backup_path in "${backup_paths[@]}"; do
if [ -f "$backup_path/.backup_exclude.txt" ]; then
exclusion_args+=" --exclude-file $backup_path/.backup_exclude.txt"
exclusion_args=("${exclusion_args[@]}" --exclude-file "$backup_path/.backup_exclude.txt")
fi
done
@@ -69,9 +75,9 @@ restic backup \
--one-file-system \
--tag "$RESTIC_BACKUP_TAG" \
--option b2.connections="$B2_CONNECTIONS" \
"$exclusion_args" \
"$RESTIC_BACKUP_EXTRA_ARGS" \
"$RESTIC_BACKUP_PATHS" &
"${exclusion_args[@]}" \
"${extra_args[@]}" \
"${backup_paths[@]}" &
wait $!
# Dereference and delete/prune old backups.

View File

@@ -5,20 +5,6 @@
# Exit on error, unset var, pipe failure
set -euo pipefail
# Assert that all needed environment variables are set.
assert_envvars() {
local varnames=("$@")
for varname in "${varnames[@]}"; do
if [ -z ${!varname+x} ]; then
printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env profile in the current shell before executing this script?\n" "$varname" >&2
exit 1
fi
done
}
assert_envvars \
B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL
# Clean up lock if we are killed.
# If killed by systemd, like $(systemctl stop restic), then it kills the whole cgroup and all it's subprocesses.
# However if we kill this script ourselves, we need this trap that kills all subprocesses manually.
@@ -29,6 +15,21 @@ exit_hook() {
}
trap exit_hook INT TERM
# Assert that all needed environment variables are set.
assert_envvars() {
local varnames=("$@")
for varname in "${varnames[@]}"; do
if [ -z ${!varname+x} ]; then
printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
exit 1
fi
done
}
assert_envvars \
B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL
# Remove locks from other stale processes to keep the automated backup running.
# NOTE nope, don't unlock like restic_backup.sh. restic_backup.sh should take precedence over this script.
#restic unlock &

146
bin/resticw Executable file
View File

@@ -0,0 +1,146 @@
#!/usr/bin/env bash
DOC="A little wrapper over restic just to handle profiles and environment loading.
Usage:
resticw [options] <restic_arguments_line>...
The <restic_arguments_line> is just the regular unwrapped restic command arguments, e.g. stats latest
Options:
-p --profile=<name> Specify the profile to load or use default [default: default].
Examples:
resticw --profile profileA snapshots
resticw stats latest # this will use the profile: default
"
# The following argument parser is generated with docopt.sh from the above docstring.
# See https://github.com/andsens/docopt.sh. If the DOC is updated or new options are added, refresh the parser!
# docopt parser below, refresh this parser with `docopt.sh resticw`
# shellcheck disable=2016,1075,2154
docopt() { parse() { if ${DOCOPT_DOC_CHECK:-true}; then local doc_hash
if doc_hash=$(printf "%s" "$DOC" | (sha256sum 2>/dev/null || shasum -a 256)); then
if [[ ${doc_hash:0:5} != "$digest" ]]; then
stderr "The current usage doc (${doc_hash:0:5}) does not match \
what the parser was generated with (${digest})
Run \`docopt.sh\` to refresh the parser."; _return 70; fi; fi; fi
local root_idx=$1; shift; argv=("$@"); parsed_params=(); parsed_values=()
left=(); testdepth=0; local arg; while [[ ${#argv[@]} -gt 0 ]]; do
if [[ ${argv[0]} = "--" ]]; then for arg in "${argv[@]}"; do
parsed_params+=('a'); parsed_values+=("$arg"); done; break
elif [[ ${argv[0]} = --* ]]; then parse_long
elif [[ ${argv[0]} = -* && ${argv[0]} != "-" ]]; then parse_shorts
elif ${DOCOPT_OPTIONS_FIRST:-false}; then for arg in "${argv[@]}"; do
parsed_params+=('a'); parsed_values+=("$arg"); done; break; else
parsed_params+=('a'); parsed_values+=("${argv[0]}"); argv=("${argv[@]:1}"); fi
done; local idx; if ${DOCOPT_ADD_HELP:-true}; then
for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
if [[ ${shorts[$idx]} = "-h" || ${longs[$idx]} = "--help" ]]; then
stdout "$trimmed_doc"; _return 0; fi; done; fi
if [[ ${DOCOPT_PROGRAM_VERSION:-false} != 'false' ]]; then
for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
if [[ ${longs[$idx]} = "--version" ]]; then stdout "$DOCOPT_PROGRAM_VERSION"
_return 0; fi; done; fi; local i=0; while [[ $i -lt ${#parsed_params[@]} ]]; do
left+=("$i"); ((i++)) || true; done
if ! required "$root_idx" || [ ${#left[@]} -gt 0 ]; then error; fi; return 0; }
parse_shorts() { local token=${argv[0]}; local value; argv=("${argv[@]:1}")
[[ $token = -* && $token != --* ]] || _return 88; local remaining=${token#-}
while [[ -n $remaining ]]; do local short="-${remaining:0:1}"
remaining="${remaining:1}"; local i=0; local similar=(); local match=false
for o in "${shorts[@]}"; do if [[ $o = "$short" ]]; then similar+=("$short")
[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
if [[ ${#similar[@]} -gt 1 ]]; then
error "${short} is specified ambiguously ${#similar[@]} times"
elif [[ ${#similar[@]} -lt 1 ]]; then match=${#shorts[@]}; value=true
shorts+=("$short"); longs+=(''); argcounts+=(0); else value=false
if [[ ${argcounts[$match]} -ne 0 ]]; then if [[ $remaining = '' ]]; then
if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
error "${short} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
else value=$remaining; remaining=''; fi; fi; if [[ $value = false ]]; then
value=true; fi; fi; parsed_params+=("$match"); parsed_values+=("$value"); done
}; parse_long() { local token=${argv[0]}; local long=${token%%=*}
local value=${token#*=}; local argcount; argv=("${argv[@]:1}")
[[ $token = --* ]] || _return 88; if [[ $token = *=* ]]; then eq='='; else eq=''
value=false; fi; local i=0; local similar=(); local match=false
for o in "${longs[@]}"; do if [[ $o = "$long" ]]; then similar+=("$long")
[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
if [[ $match = false ]]; then i=0; for o in "${longs[@]}"; do
if [[ $o = $long* ]]; then similar+=("$long"); [[ $match = false ]] && match=$i
fi; ((i++)) || true; done; fi; if [[ ${#similar[@]} -gt 1 ]]; then
error "${long} is not a unique prefix: ${similar[*]}?"
elif [[ ${#similar[@]} -lt 1 ]]; then
[[ $eq = '=' ]] && argcount=1 || argcount=0; match=${#shorts[@]}
[[ $argcount -eq 0 ]] && value=true; shorts+=(''); longs+=("$long")
argcounts+=("$argcount"); else if [[ ${argcounts[$match]} -eq 0 ]]; then
if [[ $value != false ]]; then
error "${longs[$match]} must not have an argument"; fi
elif [[ $value = false ]]; then
if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
error "${long} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
fi; if [[ $value = false ]]; then value=true; fi; fi; parsed_params+=("$match")
parsed_values+=("$value"); }; required() { local initial_left=("${left[@]}")
local node_idx; ((testdepth++)) || true; for node_idx in "$@"; do
if ! "node_$node_idx"; then left=("${initial_left[@]}"); ((testdepth--)) || true
return 1; fi; done; if [[ $((--testdepth)) -eq 0 ]]; then
left=("${initial_left[@]}"); for node_idx in "$@"; do "node_$node_idx"; done; fi
return 0; }; optional() { local node_idx; for node_idx in "$@"; do
"node_$node_idx"; done; return 0; }; oneormore() { local i=0
local prev=${#left[@]}; while "node_$1"; do ((i++)) || true
[[ $prev -eq ${#left[@]} ]] && break; prev=${#left[@]}; done
if [[ $i -ge 1 ]]; then return 0; fi; return 1; }; value() { local i
for i in "${!left[@]}"; do local l=${left[$i]}
if [[ ${parsed_params[$l]} = "$2" ]]; then
left=("${left[@]:0:$i}" "${left[@]:((i+1))}")
[[ $testdepth -gt 0 ]] && return 0; local value
value=$(printf -- "%q" "${parsed_values[$l]}"); if [[ $3 = true ]]; then
eval "var_$1+=($value)"; else eval "var_$1=$value"; fi; return 0; fi; done
return 1; }; stdout() { printf -- "cat <<'EOM'\n%s\nEOM\n" "$1"; }; stderr() {
printf -- "cat <<'EOM' >&2\n%s\nEOM\n" "$1"; }; error() {
[[ -n $1 ]] && stderr "$1"; stderr "$usage"; _return 1; }; _return() {
printf -- "exit %d\n" "$1"; exit "$1"; }; set -e; trimmed_doc=${DOC:0:450}
usage=${DOC:79:53}; digest=dc31d; shorts=(-p); longs=(--profile); argcounts=(1)
node_0(){ value __profile 0; }; node_1(){ value _restic_arguments_line_ a true
}; node_2(){ optional 0; }; node_3(){ optional 2; }; node_4(){ oneormore 1; }
node_5(){ required 3 4; }; node_6(){ required 5; }; cat <<<' docopt_exit() {
[[ -n $1 ]] && printf "%s\n" "$1" >&2; printf "%s\n" "${DOC:79:53}" >&2; exit 1
}'; unset var___profile var__restic_arguments_line_; parse 6 "$@"
local prefix=${DOCOPT_PREFIX:-''}; unset "${prefix}__profile" \
"${prefix}_restic_arguments_line_"
eval "${prefix}"'__profile=${var___profile:-default}'
if declare -p var__restic_arguments_line_ >/dev/null 2>&1; then
eval "${prefix}"'_restic_arguments_line_=("${var__restic_arguments_line_[@]}")'
else eval "${prefix}"'_restic_arguments_line_=()'; fi; local docopt_i=1
[[ $BASH_VERSION =~ ^4.3 ]] && docopt_i=2; for ((;docopt_i>0;docopt_i--)); do
declare -p "${prefix}__profile" "${prefix}_restic_arguments_line_"; done; }
# docopt parser above, complete command for generating this parser is `docopt.sh resticw`
# Parse arguments
eval "$(docopt "$@")" # See https://github.com/andsens/docopt.sh for the magic :)
# --^^^-- END OF GENERATED COMMAND LINE PARSING STUFF --^^^--
#
# --vvv-- ACTUAL SCRIPT BELOW --vvv--
# Exit on error, unbound variable, pipe error
set -euo pipefail
ENV_DIR=$INSTALL_PREFIX/etc/restic
ERR_NO_SUCH_PROFILE=2
ERR_PROFILE_NO_READ_PERM=3
# shellcheck disable=SC2154
profile_file="${ENV_DIR}/${__profile}.env.sh"
[[ ! -f "$profile_file" ]] && echo "Invalid profile: No such environment file ${profile_file}" && exit "$ERR_NO_SUCH_PROFILE"
if [[ ! -r "$profile_file" ]]; then
echo "Error: Could not read the environment file ${profile_file}. Are you running this script as the correct user? Maybe try sudo with the right user."
exit "$ERR_PROFILE_NO_READ_PERM"
fi
echo -e "‣ Using profile: ${__profile} -- (${profile_file})\n"
# shellcheck disable=SC2154,SC1090
source "$profile_file" && restic "${_restic_arguments_line_[@]}"

View File

@@ -1,8 +1,8 @@
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/:/usr/local/sbin/
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/:$INSTALL_PREFIX/bin/
# Order of crontab fields
# minute hour mday month wday command
# Reference: https://www.freebsd.org/doc/handbook/configtuning-cron.html
# Reference: crontab(5).
@midnight root cron_mail restic_backup.sh
@monthly root cron_mail restic_check.sh
@midnight root . $INSTALL_PREFIX/etc/restic/default.sh && cron_mail restic_backup.sh
@monthly root . $INSTALL_PREFIX/etc/restic/default.sh && cron_mail restic_check.sh

View File

@@ -1,5 +1,7 @@
# shellcheck shell=sh
# Global envionment variables
# These variables are sourced FIRST, and any values inside of *.env files for
# These variables are sourced FIRST, and any values inside of *.env.sh files for
# specific configurations will override if also defined there.
@@ -8,19 +10,20 @@
# The restic repository encryption key
export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
export RESTIC_PASSWORD_FILE="$INSTALL_PREFIX/etc/restic/pw.txt"
# The global restic exclude file
export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
export RESTIC_BACKUP_EXCLUDE_FILE="$INSTALL_PREFIX/etc/restic/backup_exclude.txt"
# Backblaze B2 credentials keyID & applicationKey pair.
# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
export B2_ACCOUNT_ID="<b2-key-id>" # TODO fill with your keyID
export B2_ACCOUNT_KEY="<b2-application-key>" # TODO fill with your applicationKey
export B2_ACCOUNT_ID="<b2-key-id>" # *EDIT* fill with your keyID
export B2_ACCOUNT_KEY="<b2-application-key>" # *EDIT* fill with your applicationKey
# How many network connections to set up to B2. Default is 5.
export B2_CONNECTIONS=10
# Extra args to restic-backup. This is empty here and profiles can override this after sourcing this file.
# Optional extra space-separated args to restic-backup.
# This is empty here and profiles can override this after sourcing this file.
export RESTIC_BACKUP_EXTRA_ARGS=
# Verbosity level from 0-3. 0 means no --verbose.

View File

@@ -1,23 +1,27 @@
# shellcheck shell=sh
# This is the default profile. Fill it with your desired configuration.
# Additionally, you can create and use more profiles by copying this file.
# This file (and other .env files) has two purposes:
# This file (and other .env.sh files) has two purposes:
# - being sourced by systemd timers to setup the backup before running restic_backup.sh
# - being sourced in a user's shell to work directly with restic commands e.g.
# $ source /etc/restic/default.env
# $ source /etc/restic/default.env.sh
# $ restic snapshots
# Thus you don't have to provide all the arguments like
# $ restic --repo ... --password-file ...
source /etc/restic/_global.env
# shellcheck source=etc/restic/_global.env.sh
. "$INSTALL_PREFIX/etc/restic/_global.env.sh"
# Below envvar will override those in _global.env
# Envvars below will override those in _global.env.sh if present.
export RESTIC_REPOSITORY="b2:<b2-repo-name>" # TODO fill with your repo name
export RESTIC_REPOSITORY="b2:<b2-repo-name>" # *EDIT* fill with your repo name
# What to backup (paths our mountpoints) e.g. "/ /boot /home /mnt/media".
# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
# To backup only your home directory, set "/home/your-user"
export RESTIC_BACKUP_PATHS="" # TODO fill conveniently with one or multiple paths
export RESTIC_BACKUP_PATHS="" # *EDIT* fill conveniently with one or multiple paths
# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
@@ -33,8 +37,8 @@ export RESTIC_RETENTION_WEEKS=16
export RESTIC_RETENTION_MONTHS=18
export RESTIC_RETENTION_YEARS=3
# Optional extra arguments to restic-backup.
# Optional extra space-separated arguments to restic-backup.
# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a /path/to/extra/exclude/file/b"
#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
# Example: exclude all directories that have a .git/ directory inside it.
#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

BIN
img/readme_sections.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

View File

@@ -3,5 +3,4 @@ Description=Check if the current NetworkManager connection is metered
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/nm-unmetered-connection.sh
ExecStart=$INSTALL_PREFIX/bin/nm-unmetered-connection.sh

View File

@@ -10,4 +10,4 @@ Nice=10
Environment="HOME=/root"
# The random sleep (in seconds) is in the case of multiple backup profiles. Many restic instances started at the same time could case high load or network bandwith usage.
# `systemd-cat` allows showing the restic output to the systemd journal
ExecStart=bash -c 'sleep $(shuf -i 0-300 -n 1) && source /etc/restic/%I.env && /usr/local/sbin/restic_backup.sh | systemd-cat'
ExecStart=bash -c 'ps cax | grep -q restic && sleep $(shuf -i 0-300 -n 1); source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_backup.sh | systemd-cat'

View File

@@ -8,4 +8,4 @@ Requires=nm-unmetered-connection.service
Type=simple
Nice=10
# `systemd-cat` allows showing the restic output to the systemd journal
ExecStart=bash -c 'source /etc/restic/%I.env && /usr/local/sbin/restic_check.sh | systemd-cat'
ExecStart=bash -c 'source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_check.sh | systemd-cat'

View File

@@ -6,6 +6,6 @@ Description=Send status email for %i to user
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/systemd-email abc@gmail.com %i
ExecStart=$INSTALL_PREFIX/bin/systemd-email abc@gmail.com %i
User=root
Group=systemd-journal