Allow set but empty envvars

This reverts commit 082089a203.

.github/workflows/linter.yml

@@ -0,0 +1,30 @@
+name: Lint Code Base
+
+on:
+  push:
+    branches: master
+    paths:
+      - '**.sh'
+      - '.github/workflows/linter.yml'
+  pull_request:
+    branches: master
+    paths:
+      - '**.sh'
+      - '.github/workflows/linter.yml'
+jobs:
+  build:
+    name: Lint Code Base
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v2
+        with:
+          # Full git history is needed to get a proper list of changed files within `super-linter`
+          fetch-depth: 0
+      - name: Lint Code Base
+        uses: github/super-linter@v4
+        env:
+          VALIDATE_ALL_CODEBASE: true
+          VALIDATE_BASH: true
+          DEFAULT_BRANCH: master
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CHANGELOG.md

@@ -6,6 +6,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [3.0.1] - 2022-02-01
+### Fixed
+- Environment variable assertion should allow empty values e.g. `RESTIC_BACKUP_EXTRA_ARGS`
+
+## [3.0.0] - 2022-02-01
+### Added
+- Allow extra arguments to restic-backup with `$RESTIC_BACKUP_EXTRA_ARGS`.
+- Add `$RESTIC_VERBOSITY_LEVEL` for debugging.
+- Assertion on all needed environment variables in the backup and check scripts.
+- Added linter (`shellcheck(1)`) that is run on push and PRs.
+
+### Changed
+- **BREAKING CHANGE** renamed
+   - `/etc/restic/backup_exclude` to `/etc/restic/backup_exclude.txt`
+   - `<backup-dest>/.backup_exclude` to `<backup-dest>/.backup_exclude.txt`.
+- **BREAKING CHANGE** renamed envvars for consistency
+   - `BACKUP_PATHS` -> `RESTIC_BACKUP_PATHS`
+   - `BACKUP_TAG` -> `RESTIC_BACKUP_TAG`
+   - `RETENTION_DAYS` -> `RESTIC_RETENTION_DAYS`
+   - `RETENTION_WEEKS` -> `RESTIC_RETENTION_WEEKS`
+   - `RETENTION_MONTHS` -> `RESTIC_RETENTION_MONTHS`
+   - `RETENTION_YEARS` -> `RESTIC_RETENTION_YEARS`
+- Align terminology used in README with the one used by B2 for credentials (keyId + applicationKey pair).
+
 ## [2.0.0] - 2022-02-01
 ### Changed
 - **BREAKING CHANGE** [#45](https://github.com/erikw/restic-systemd-automatic-backup/pull/45): multiple backup profiles are now supported. Please backup your configuration before upgrading. The setup of configuration files are now laied out differently. See the [README.md](README.md) TL;DR setup section.

LICENSE

@@ -1,7 +1,7 @@
 restic-systemd-automatic-backup -  My restic backup solution using Backblaze B2 storage, systemd timers (or cron) and email notifications on failure.
 
 
-Copyright (c) 2018, see commit log for auhtors
+Copyright (c) 2022, Erik Westrup + see commit log for auhtors
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

README.md

@@ -2,6 +2,8 @@
 [![GitHub Stars](https://img.shields.io/github/stars/erikw/restic-systemd-automatic-backup?style=social)](#)
 [![GitHub Forks](https://img.shields.io/github/forks/erikw/restic-systemd-automatic-backup?style=social)](#)
 <br>
+[![Lint Code Base](https://github.com/erikw/restic-systemd-automatic-backup/actions/workflows/linter.yml/badge.svg)](https://github.com/erikw/restic-systemd-automatic-backup/actions/workflows/linter.yml)
+[![Latest tag](https://img.shields.io/github/v/tag/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/tags)
 [![AUR version](https://img.shields.io/aur/version/restic-systemd-automatic-backup)](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/)
 [![AUR maintainer](https://img.shields.io/aur/maintainer/restic-systemd-automatic-backup?label=AUR%20maintainer)](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/)
 [![Open issues](https://img.shields.io/github/issues/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/issues)
@@ -9,7 +11,6 @@
 [![Closed PRs](https://img.shields.io/github/issues-pr-closed/erikw/restic-systemd-automatic-backup?color=success)](https://github.com/erikw/restic-systemd-automatic-backup/pulls?q=is%3Apr+is%3Aclosed)
 [![License](https://img.shields.io/badge/license-BSD--3-blue)](LICENSE)
 [![OSS Lifecycle](https://img.shields.io/osslifecycle/erikw/restic-systemd-automatic-backup)](https://github.com/Netflix/osstracker)
-[![Latest tag](https://img.shields.io/github/v/tag/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/tags)
 <br>
 
 [![Contributors](https://img.shields.io/github/contributors/erikw/restic-systemd-automatic-backup)](https://github.com/erikw/restic-systemd-automatic-backup/graphs/contributors) including these top contributors:
@@ -18,7 +19,6 @@
 </a>
 
 # Intro
-
 [restic](https://restic.net/) is a command-line tool for making backups, the right way. Check the official website for a feature explanation. As a storage backend, I recommend [Backblaze B2](https://www.backblaze.com/b2/cloud-storage.html) as restic works well with it, and it is (at the time of writing) very affordable for the hobbyist hacker! (anecdotal: I pay for my full-systems backups each month typically < 1 USD).
 
 Unfortunately restic does not come pre-configured with a way to run automated backups, say every day. However it's possible to set this up yourself using systemd/cron and some wrappers. This example also features email notifications when a backup fails to complete.
@@ -44,7 +44,7 @@ Note, you can use any of the supported [storage backends](https://restic.readthe
    * `/etc/restic/pw.txt` - Contains the password (single line) to be used by restic to encrypt the repository files. Should be different than your B2 password!
    * `/etc/restic/_global.env` - Global environment variables.
    * `/etc/restic/default.env` - Profile specific environment variables (multiple profiles can be defined by copying to `/etc/restic/something.env`).
-   * `/etc/restic/backup_exclude` - List of file patterns to ignore. This will trim down your backup size and the speed of the backup a lot when done properly!
+   * `/etc/restic/backup_exclude.txt` - List of file patterns to ignore. This will trim down your backup size and the speed of the backup a lot when done properly!
 1. Initialize remote repo as described [below](#3-initialize-remote-repo)
 1. Configure [how often](https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events) back up should be made.
    * Edit if needed `OnCalendar` in `/etc/systemd/system/restic-check@.timer`.
@@ -90,17 +90,17 @@ Arch Linux users can install the aur package [restic-systemd-automatic-backup](h
 $ yaourt -S restic-systemd-automatic-backup
 ````
 
-## 1. Create Backblaze B2 account
+## 1. Create Backblaze B2 Account, Bucket and keys
 First, see this official Backblaze [tutorial](https://help.backblaze.com/hc/en-us/articles/4403944998811-Quickstart-Guide-for-Restic-and-Backblaze-B2-Cloud-Storage) on restic, and follow the instructions ("Create Backblaze account with B2 enabled") there on how to create a new B2 bucket. In general, you'd want a private bucket, without B2 encryption (restic does the encryption client side for us) and without the object lock feature.
 
-Take note of the your account ID and application key for the next steps. It's a good idea to create a separate application key that has access only to the newly created b2 bucket you created.
+For restic to be able to connect to your bucket, you want to in the B2 settings create a pair of keyID and applicationKey. It's a good idea to create a separate pair of ID and Key with for each bucket that you will use, with limited read&write access to only that bucket.
 
 
-## 2. Configure your B2 account locally
+## 2. Configure your B2 credentials locally
 > **Attention!** Going the manual way requires that most of the following commands are run as root.
 
 Put these files in `/etc/restic/`:
-* `_global.env`: Fill this file out with your global settings including B2 accountID & accountKey. A global exclude list is set here (explained in section below).
+* `_global.env`: Fill this file out with your global settings including B2 keyID & applicationKey. A global exclude list is set here (explained in section below).
 * `default.env`: This is the default profile. Fill this out with bucket name, backup paths and retention policy. This file sources `_global.env` and is thus self-contained and can be sourced in the shell when you want to issue some manual restic commands. For example:
    ```console
    $ source /etc/restic/default.env
@@ -124,8 +124,8 @@ Put this file in `/usr/local/sbin`:
 * `restic_backup.sh`: A script that defines how to run the backup. The intention is that you should not need to edit this script yourself, but be able to control everything from the `*.env` profiles.
 
 Restic support exclude files. They list file pattern paths to exclude from you backups, files that just occupy storage space, backup-time, network and money. `restic_backup.sh` allows for a few different exclude files.
-* `/etc/restic/backup_exclude` - global exclude list. You can use only this one if your setup is easy. This is set in `_global.env`. If you need a different file for another profile, you can override the envvar `RESTIC_BACKUP_EXCLUDE_FILE` in this profile.
-* `.backup_exclude` per backup path. If you have e.g. an USB disk mounted at /mnt/media and this path is included in the `$BACKUP_PATHS`, you can place a file `/mnt/media/.backup_exclude` and it will automatically picked up. The nice thing about this is that the backup paths are self-contained in terms of what they shoud exclude!
+* `/etc/restic/backup_exclude.txt` - global exclude list. You can use only this one if your setup is easy. This is set in `_global.env`. If you need a different file for another profile, you can override the envvar `RESTIC_BACKUP_EXCLUDE_FILE` in this profile.
+* `.backup_exclude.txt` per backup path. If you have e.g. an USB disk mounted at /mnt/media and this path is included in the `$RESTIC_BACKUP_PATHS`, you can place a file `/mnt/media/.backup_exclude.txt` and it will automatically picked up. The nice thing about this is that the backup paths are self-contained in terms of what they shoud exclude!
 
 ## 5. Make first backup
 Now see if the backup itself works, by running as root
@@ -243,3 +243,13 @@ To not mess up your real installation when changing the `Makefile` simply instal
 ```console
 $ PREFIX=/tmp/restic-test make install
 ```
+
+# Releasing
+To make a new release:
+1.
+   ```console
+   $ vi CHANGELOG.md && git commit -am "Update CHANGELOG.md"
+   $ git tag vX.Y.Z
+   $ git push && git push --tags
+   ```
+1. Test and update the AUR [PKGBUILD](https://aur.archlinux.org/packages/restic-systemd-automatic-backup/) if needed.

etc/restic/_global.env.template

@@ -10,11 +10,19 @@
 # The restic repository encryption key
 export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
 # The global restic exclude file
-export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude"
+export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
 
-# Backblaze B2 credentials
-export B2_ACCOUNT_ID="<b2-account-id>"   # TODO fill with your account info
-export B2_ACCOUNT_KEY="<b2-account-key>" # TODO fill with your account info
+# Backblaze B2 credentials keyID & applicationKey pair.
+# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
+export B2_ACCOUNT_ID="<b2-key-id>"   # TODO fill with your keyID
+export B2_ACCOUNT_KEY="<b2-application-key>" # TODO fill with your applicationKey
 
 # How many network connections to set up to B2. Default is 5.
 export B2_CONNECTIONS=10
+
+# Extra args to restic-backup. This is empty here and profiles can override this after sourcing this file.
+export RESTIC_BACKUP_EXTRA_ARGS=
+
+# Verbosity level from 0-3. 0 means no --verbose.
+# Override this value in a profile if needed.
+export RESTIC_VERBOSITY_LEVEL=0

etc/restic/default.env.template

@@ -17,18 +17,24 @@ export RESTIC_REPOSITORY="b2:<b2-repo-name>"   # TODO fill with your repo name
 
 # What to backup (paths our mountpoints) e.g. "/ /boot /home /mnt/media".
 # To backup only your home directory, set "/home/your-user"
-export BACKUP_PATHS=""  # TODO fill conveniently with one or multiple paths
+export RESTIC_BACKUP_PATHS=""  # TODO fill conveniently with one or multiple paths
 
 # Example below of how to dynamically add a path that is mounted e.g. external USB disk.
 # restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
-#test -d /mnt/media && BACKUP_PATHS+=" /mnt/media"
+#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"
 
 # A tag to identify backup snapshots.
-export BACKUP_TAG=systemd.timer
+export RESTIC_BACKUP_TAG=systemd.timer
 
 # Retention policy - How many backups to keep.
 # See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
-export RETENTION_DAYS=14
-export RETENTION_WEEKS=16
-export RETENTION_MONTHS=18
-export RETENTION_YEARS=3
+export RESTIC_RETENTION_DAYS=14
+export RESTIC_RETENTION_WEEKS=16
+export RESTIC_RETENTION_MONTHS=18
+export RESTIC_RETENTION_YEARS=3
+
+# Optional extra arguments to restic-backup.
+# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a /path/to/extra/exclude/file/b"
+# Example: exclude all directories that have a .git/ directory inside it.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

usr/local/sbin/cron_mail

@@ -15,7 +15,7 @@ if [ $# -eq 0 ]; then
 	echo "No program to run given!" >&2
 	exit 1
 fi
-cmd="$@"
+cmd="$*"
 
 body=$(eval "$cmd" 2>&1)
 

usr/local/sbin/restic_backup.sh

@@ -11,6 +11,24 @@
 # Exit on error, unset var, pipe failure
 set -euo pipefail
 
+# Assert that all needed environment variables are set.
+# TODO in future if this grows, move this to a restic_lib.sh
+assert_envvars() {
+	local varnames=("$@")
+	for varname in "${varnames[@]}"; do
+		if [ -z ${!varname+x} ]; then
+			printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env profile in the current shell before executing this script?\n" "$varname" >&2
+			exit 1
+		fi
+	done
+}
+assert_envvars \
+	B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
+	RESTIC_BACKUP_PATHS RESTIC_BACKUP_TAG \
+	RESTIC_BACKUP_EXCLUDE_FILE RESTIC_BACKUP_EXTRA_ARGS RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL \
+	RESTIC_RETENTION_DAYS RESTIC_RETENTION_MONTHS RESTIC_RETENTION_WEEKS RESTIC_RETENTION_YEARS
+
+
 # Clean up lock if we are killed.
 # If killed by systemd, like $(systemctl stop restic), then it kills the whole cgroup and all it's subprocesses.
 # However if we kill this script ourselves, we need this trap that kills all subprocesses manually.
@@ -25,11 +43,11 @@ trap exit_hook INT TERM
 # NOTE that restic will fail the backup if not all listed --exclude-files exist. Thus we should only list them if they are really all available.
 ##  Global backup configuration.
 exclusion_args="--exclude-file ${RESTIC_BACKUP_EXCLUDE_FILE}"
-## Self-contained backup files per backup path. E.g. having an USB disk at /mnt/media in BACKUP_PATHS,
-# a file /mnt/media/.backup_exclude will automatically be detected and used:
-for backup_path in ${BACKUP_PATHS[@]}; do
-	if [ -f "$backup_path/.backup_exclude" ]; then
-		exclusion_args+=" --exclude-file $backup_path/.backup_exclude"
+## Self-contained backup files per backup path. E.g. having an USB disk at /mnt/media in RESTIC_BACKUP_PATHS,
+# a file /mnt/media/.backup_exclude.txt will automatically be detected and used:
+for backup_path in "${RESTIC_BACKUP_PATHS[@]}"; do
+	if [ -f "$backup_path/.backup_exclude.txt" ]; then
+		exclusion_args+=" --exclude-file $backup_path/.backup_exclude.txt"
 	fi
 done
 
@@ -44,30 +62,31 @@ wait $!
 
 # Do the backup!
 # See restic-backup(1) or http://restic.readthedocs.io/en/latest/040_backup.html
-# --one-file-system makes sure we only backup exactly those mounted file systems specified in $BACKUP_PATHS, and thus not directories like /dev, /sys etc.
+# --one-file-system makes sure we only backup exactly those mounted file systems specified in $RESTIC_BACKUP_PATHS, and thus not directories like /dev, /sys etc.
 # --tag lets us reference these backups later when doing restic-forget.
 restic backup \
-	--verbose \
+	--verbose="$RESTIC_VERBOSITY_LEVEL" \
 	--one-file-system \
-	--tag $BACKUP_TAG \
-	--option b2.connections=$B2_CONNECTIONS \
-	$exclusion_args \
-	$BACKUP_PATHS &
+	--tag "$RESTIC_BACKUP_TAG" \
+	--option b2.connections="$B2_CONNECTIONS" \
+	"$exclusion_args" \
+	"$RESTIC_BACKUP_EXTRA_ARGS" \
+	"$RESTIC_BACKUP_PATHS" &
 wait $!
 
 # Dereference and delete/prune old backups.
 # See restic-forget(1) or http://restic.readthedocs.io/en/latest/060_forget.html
 # --group-by only the tag and path, and not by hostname. This is because I create a B2 Bucket per host, and if this hostname accidentially change some time, there would now be multiple backup sets.
 restic forget \
-	--verbose \
-	--tag $BACKUP_TAG \
-	--option b2.connections=$B2_CONNECTIONS \
+	--verbose="$RESTIC_VERBOSITY_LEVEL" \
+	--tag "$RESTIC_BACKUP_TAG" \
+	--option b2.connections="$B2_CONNECTIONS" \
 	--prune \
 	--group-by "paths,tags" \
-	--keep-daily $RETENTION_DAYS \
-	--keep-weekly $RETENTION_WEEKS \
-	--keep-monthly $RETENTION_MONTHS \
-	--keep-yearly $RETENTION_YEARS &
+	--keep-daily "$RESTIC_RETENTION_DAYS" \
+	--keep-weekly "$RESTIC_RETENTION_WEEKS" \
+	--keep-monthly "$RESTIC_RETENTION_MONTHS" \
+	--keep-yearly "$RESTIC_RETENTION_YEARS" &
 wait $!
 
 # Check repository for errors.

usr/local/sbin/restic_check.sh

@@ -2,8 +2,22 @@
 # Check the backups made with restic to Backblaze B2 for errors.
 # See restic_backup.sh on how this script is run (as it's analogous for this script).
 
-# Exit on failure, pipe failure
-set -e -o pipefail
+# Exit on error, unset var, pipe failure
+set -euo pipefail
+
+# Assert that all needed environment variables are set.
+assert_envvars() {
+	local varnames=("$@")
+	for varname in "${varnames[@]}"; do
+		if [ -z ${!varname+x} ]; then
+			printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env profile in the current shell before executing this script?\n" "$varname" >&2
+			exit 1
+		fi
+	done
+}
+assert_envvars \
+	B2_ACCOUNT_ID B2_ACCOUNT_KEY B2_CONNECTIONS \
+	RESTIC_PASSWORD_FILE RESTIC_REPOSITORY RESTIC_VERBOSITY_LEVEL
 
 # Clean up lock if we are killed.
 # If killed by systemd, like $(systemctl stop restic), then it kills the whole cgroup and all it's subprocesses.
@@ -22,6 +36,6 @@ trap exit_hook INT TERM
 
 # Check repository for errors.
 restic check \
-	--option b2.connections=$B2_CONNECTIONS \
-	--verbose &
+	--option b2.connections="$B2_CONNECTIONS" \
+	--verbose="$RESTIC_VERBOSITY_LEVEL" &
 wait $!

usr/local/sbin/systemd-email

@@ -1,4 +1,4 @@
-#!/usr/bin/env sh
+#!/usr/bin/env bash
 # Send email notification from systemd.
 # Source: https://serverfault.com/questions/876233/how-to-send-an-email-if-a-systemd-service-is-restarted
 # Source: https://wiki.archlinux.org/index.php/Systemd/Timers#MAILTO
@@ -15,19 +15,19 @@
 # (24 * 60 * 60) / 100 = 864 second
 # One option that I used with my old Axis cameras it to use my gmx.com accunt for sending emails instead, as there are (no?) higher limits there.
 MIN_WAIT_TIME_S=900
-SCRIPT_NAME=$(basename $0)
+SCRIPT_NAME=$(basename "$0")
 LAST_RUN_FILE="/tmp/${SCRIPT_NAME}_last_run.txt"
 
 last_touch() {
-	stat -c %Y $1
+	stat -c %Y "$1"
 }
 
 waited_long_enough() {
 	retval=1
-	if [ -e $LAST_RUN_FILE ]; then
+	if [ -e "$LAST_RUN_FILE" ]; then
 		now=$(date +%s)
-		last=$(last_touch $LAST_RUN_FILE)
-		wait_s=$(expr $now - $last)
+		last=$(last_touch "$LAST_RUN_FILE")
+		wait_s=$((now - last))
 		if [ "$wait_s" -gt "$MIN_WAIT_TIME_S" ]; then
 			retval=0
 		fi
@@ -35,7 +35,7 @@ waited_long_enough() {
 		retval=0
 	fi
 
-	[ $retval -eq 0 ] && touch $LAST_RUN_FILE
+	[ $retval -eq 0 ] && touch "$LAST_RUN_FILE"
 	return $retval
 }