Fix notification stats bug with more than 2 latest snapshots

Fixes #86

.github/workflows/linter.yml

@@ -3,13 +3,13 @@ name: Lint Code Base
 on:
   workflow_dispatch:
   push:
-    branches: master
+    branches: main
     paths:
       - '**.sh'
       - '.github/workflows/linter.yml'
       - 'bin/**'
   pull_request:
-    branches: master
+    branches: main
     paths:
       - '**.sh'
       - '.github/workflows/linter.yml'
@@ -29,5 +29,5 @@ jobs:
         env:
           VALIDATE_ALL_CODEBASE: true
           VALIDATE_BASH: true
-          DEFAULT_BRANCH: master
+          DEFAULT_BRANCH: main
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CHANGELOG.md

@@ -6,9 +6,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [7.3.4] - 2022-04-29
+### Fixed
+- Backup stats notifications: fix issue where `restic snapshots --latest 2` will show more than two snapshots due to different backup paths used.
+
+## [7.3.3] - 2022-04-14
+### Fixed
+- Trying to fix broken Homebrew bottles due to GitHub API issues.
+
+## [7.3.2] - 2022-04-11
+### Fixed
+- Trying to fix broken Homebrew bottles
+
+## [7.3.1] - 2022-04-11
+### Fixed
+- `resticw` is now a true wrapper in that it support `--` args to restic.
+- OnFailure no longer masked by the stderr redirect to systemd-cat. [#86](https://github.com/erikw/restic-automatic-backup-scheduler/pull/86)
+
+## [7.3.0] - 2022-02-15
+### Added
+- optional user-controlled notification. See `RESTIC_NOTIFY_BACKUP_STATS` and in `backup.sh`.
+
 ## [7.2.0] - 2022-02-15
 ### Added
-- restic-check launchagent.
+- restic-check LaunchAgent.
 
 ### Changed
 - [README.md](README.md) is restructured with easier TL;DR for each OS and a more general detailed section for the interested.

README.md

@@ -76,7 +76,7 @@ Many Linux distributions nowadays use [Systemd](https://en.wikipedia.org/wiki/Sy
    ```
 1. Fill out [configuration values](#2-configure-b2-credentials-locally) in `/etc/restic`.
 1. [Initialize](#3-initialize-remote-repo) the remote repo.
-	Source the profile to make all needed configuration available to `restic`. All commands after this assumes the profile is sourced in the current shell.
+	Source the profile to make all needed configuration available to `restic(1)`. All commands after this assumes the profile is sourced in the current shell.
    ```console
 	# source /etc/restic/default.env.sh
 	# restic init
@@ -126,7 +126,7 @@ Many Linux distributions nowadays use [Systemd](https://en.wikipedia.org/wiki/Sy
    ```
 1. Fill out [configuration values](#2-configure-b2-credentials-locally) in `/usr/local/etc/restic`.
 1. [Initialize](#3-initialize-remote-repo) the remote repo.
-	Source the profile to make all needed configuration available to `restic`. All commands after this assumes the profile is sourced in the current shell.
+	Source the profile to make all needed configuration available to `restic(1)`. All commands after this assumes the profile is sourced in the current shell.
    ```console
 	$ source /usr/local/etc/restic/default.env.sh
 	$ restic init
@@ -226,7 +226,7 @@ I describe here one of may ways you can get restic and this backup script workin
 	export RESTIC_BACKUP_PATHS='/c/Users/<username>/My Documents'
    ```
 1. [Initialize](#3-initialize-remote-repo) the remote repo.
-	Source the profile to make all needed configuration available to `restic`. All commands after this assumes the profile is sourced in the current shell.
+	Source the profile to make all needed configuration available to `restic(1)`. All commands after this assumes the profile is sourced in the current shell.
    ```console
     git-bash$ source /etc/restic/default.env.sh
     git-bash$ restic init
@@ -268,9 +268,12 @@ Any system that has a cron-like system can easily setup restic backups as well.
    $ sudo make install-cron
    ```
    * This assumes that your cron supports dropping files into `/etc/cron.d/`. If that is not the case, simply copy the relevant contents of the installed `/etc/cron.d/restic` in to your `/etc/crontab`.
+   ```console
+   # grep "^@.*restic_" /etc/cron.d/restic >> /etc/crontab
+   ```
 1. Fill out [configuration values](#2-configure-b2-credentials-locally) in `/etc/restic`.
 1. [Initialize](#3-initialize-remote-repo) the remote repo.
-	Source the profile to make all needed configuration available to `restic`. All commands after this assumes the profile is sourced in the current shell.
+	Source the profile to make all needed configuration available to `restic(1)`. All commands after this assumes the profile is sourced in the current shell.
    ```console
 	# source /etc/restic/default.env.sh
 	# restic init
@@ -435,6 +438,33 @@ To create a different backup and use you can do:
 # restic_backup.sh
 ```
 
+### Optional: Desktop Notifications
+<img src="img/macos_notification.png" align="right" />
+
+It's a good idea to be on top of your backups to make sure that they don't increase a lot in size and incur high costs. However it's notoriously tricky to make GUI notifications correctly from a non-user process (e.g. root).
+
+Therefore this project provides a lightweight solution for desktop notifications that works like this: Basically `restic_backup.sh` will append a summary line of the last backup to a user-owned file (the user running your OS's desktop environment) in a fire-and-forget fashion. Then the user has a process that reads this and forward each line as a new message to the desktop environment in use.
+
+To set desktop notifications up:
+1. Create a special FIFO file as your desktop user:
+	```console
+	$ mkfifo /home/user/.cache/notification-queue
+	```
+1. In your profile, e.g. `/etc/restic/default.sh`, set:
+	```bash
+	RESTIC_NOTIFY_BACKUP_STATS=true
+	RESTIC_BACKUP_NOTIFICATION_FILE=/home/user/.cache/notification-queue
+	```
+1. Create a listener on the notification queue file that forwards to desktop notifications
+   * Linux auto start + cross-platform notifier / notify-send
+      * [notification-queue-notifier](https://github.com/gerardbosch/dotfiles/blob/ddc1491056822eab45dedd131f1946308ef62135/home/bin/notification-queue-notifier)
+      * [notification-queue.desktop](https://github.com/gerardbosch/dotfiles-linux/blob/ea0f75bfd7a356945544ecaa42a2fc35c9fab3a1/home/.config/autostart/notification-queue.desktop)
+   * macOS auto start + [terminal-notifier](https://github.com/julienXX/terminal-notifier)
+      * [notification-queue-notifier.sh](https://github.com/erikw/dotfiles/blob/8a942defe268292200b614951cdf433ddccf7170/bin/notification-queue-notifier.sh)
+      * [com.user.notificationqueue.plist](https://github.com/erikw/dotfiles/blob/8a942defe268292200b614951cdf433ddccf7170/.config/LaunchAgents/com.user.notificationqueue.plist)
+
+
+
 ### Optional: Email Notification on Failure
 #### Systemd
 We want to be aware when the automatic backup fails, so we can fix it. Since my laptop does not run a mail server, I went for a solution to set up my laptop to be able to send emails with [postfix via my Gmail](https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/). Follow the instructions over there.
@@ -445,7 +475,7 @@ Put this file in `/bin`:
 Put this files in `/etc/systemd/system/`:
 * `status-email-user@.service`: A service that can notify you via email when a systemd service fails. Edit the target email address in this file.
 
-Now edit `restic-backup.service` and `status-email-user.service` to call this service failure.
+Now edit `restic-backup@.service` and `status-email-user@.service` to call this service failure.
 ```
 OnFailure=status-email-user@%n.service
 ```
@@ -459,18 +489,31 @@ To use this, wrap the restic script command with it in your cron file like:
 ```
 
 
-### Optional: No Backup on Metered Connections
+### Optional: No Backup on Metered Connections (Linux/systemd only)
 For a laptop, it can make sense to not do heavy backups when your on a metered connection like a shared connection from you mobile phone. To solve this we can set up a systemd service that is in success state only when a connection is unmetered. Then we can tell our backup service to depend on this service simply! When the unmetered service detects an unmetered connection it will go to failed state. Then our backup service will not run as it requires this other service to be in success state.
 
-Put this file in `/bin`:
-* `nm-unmetered-connection.sh`: Detects metered connections and returns will error code if one is detected. This scripts requires the Gnome [NetworkManager](https://wiki.gnome.org/Projects/NetworkManager) to be installed. Modify this script if your system has a different network manager.
+1. Edit `restic-backup@.service` and `restic-check@.service` to require the new service to be in success state:
+   ```
+   Requires=nm-unmetered-connection.service
+   ```
+1. Copy and paste the command below, it will install the following files and refresh systemd daemon:
+1. Put this file in `/etc/systemd/system/`:
+   * `nm-unmetered-connection.service`: A service that is in success state only if the connection is unmetered.
+1. Install this file in `/bin`:
+   * `nm-unmetered-connection.sh`: Detects metered connections and returns an error code if one is detected. This scripts requires the Gnome [NetworkManager](https://wiki.gnome.org/Projects/NetworkManager) to be installed (modify this script if your system has a different network manager).
+1. Reload systemd with
+	```console
+	# systemctl daemon-reload
+	```
 
-Put this files in `/etc/systemd/system/`:
-* `nm-unmetered-connection.service`: A service that is in success state if the connection is unmetered only.
-
-Now edit `restic-backup.service` and `status-email-user.service` to require the new service to be in success state:
-```
-Requires=nm-unmetered-connection.service
+☝ **Tip**: All steps but the first can be done in one go if you use the Makefile. Set `$PREFIX` as needed or leave empty for install to `/`.
+```bash
+sudo bash -c 'export PREFIX=
+ make build/usr/lib/systemd/system/nm-unmetered-connection.service
+ install -m 0644 build/usr/lib/systemd/system/nm-unmetered-connection.service $PREFIX/etc/systemd/system
+ install -m 0555 bin/nm-unmetered-connection.sh /bin
+ systemctl daemon-reload
+'
 ```
 
 ### Optional: Restic Wrapper Script
@@ -484,8 +527,8 @@ Useful commands:
 | Command                                           | Description                                                       |
 |---------------------------------------------------|-------------------------------------------------------------------|
 | `resticw snapshots`                               | List backup snapshots                                             |
-| `resticw diff <snapshot-id> latest`               | Show the changes from the latest backup                           |
-| `resticw stats` / `resticw stats snapshot-id ...` | Show the statistics for the whole repo or the specified snapshots |
+| `resticw diff <snapshotId-1> <snapshotId-2>`      | Show the changes between backup snapshots                         |
+| `resticw stats` / `resticw stats snapshotId ...`  | Show the statistics for the whole repo or the specified snapshots |
 | `resticw mount /mnt/restic`                       | Mount your remote repository                                      |
 
 
@@ -516,5 +559,7 @@ To make a new release:
    $ git tag vX.Y.Z
    $ git push && git push --tags
    ```
-1. Update version in the AUR [PKGBUILD](https://aur.archlinux.org/packages/restic-automatic-backup-scheduler/).
-1. Update version in the Homebrew [Formula](https://github.com/erikw/homebrew-tap/blob/main/Formula/restic-automatic-backup-scheduler.rb).
+1. Update version in the AUR [PKGBUILD](https://aur.archlinux.org/packages/restic-automatic-backup-scheduler/)
+1. Update version in the Homebrew Formulas (see the repo README):
+   * [restic-automatic-backup-scheduler](https://github.com/erikw/homebrew-tap/blob/main/Formula/restic-automatic-backup-scheduler.rb)
+   * [restic-automatic-backup-scheduler-check](https://github.com/erikw/homebrew-tap/blob/main/Formula/restic-automatic-backup-scheduler-check.rb)

bin/restic_backup.sh

@@ -28,7 +28,7 @@ assert_envvars() {
 	local varnames=("$@")
 	for varname in "${varnames[@]}"; do
 		if [ -z ${!varname+x} ]; then
-			printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
+			printf "%s must be set for this script to work.\n\nDid you forget to source a $INSTALL_PREFIX/etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
 			exit 1
 		fi
 	done
@@ -105,3 +105,24 @@ wait $!
 #wait $!
 
 echo "Backup & cleaning is done."
+
+# (optionally) Notify about backup summary stats.
+if [ "$RESTIC_NOTIFY_BACKUP_STATS" = true ]; then
+	if [ -w "$RESTIC_BACKUP_NOTIFICATION_FILE" ]; then
+		echo 'Notifications are enabled: Silently computing backup summary stats...'
+
+		snapshot_size=$(restic stats latest --tag "$RESTIC_BACKUP_TAG" | grep -i 'total size:' | cut -d ':' -f2 | xargs)  # xargs acts as trim
+		latest_snapshot_diff=$(restic snapshots --tag "$RESTIC_BACKUP_TAG" --latest 2 --compact \
+			| grep -Ei "^[abcdef0-9]{8} " \
+			| awk '{print $1}' \
+			| tail -2 \
+			| tr '\n' ' ' \
+			| xargs restic diff)
+        added=$(echo "$latest_snapshot_diff" | grep -i 'added:' | awk '{print $2 " " $3}')
+        removed=$(echo "$latest_snapshot_diff" | grep -i 'removed:' | awk '{print $2 " " $3}')
+
+		echo "Added: ${added}. Removed: ${removed}. Snap size: ${snapshot_size}" >> "$RESTIC_BACKUP_NOTIFICATION_FILE"
+	else
+		echo "[WARN] Couldn't write the backup summary stats. File not found or not writable: ${RESTIC_BACKUP_NOTIFICATION_FILE}"
+	fi
+fi

bin/restic_check.sh

@@ -20,7 +20,7 @@ assert_envvars() {
 	local varnames=("$@")
 	for varname in "${varnames[@]}"; do
 		if [ -z ${!varname+x} ]; then
-			printf "%s must be set for this script to work.\n\nDid you forget to source a /etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
+			printf "%s must be set for this script to work.\n\nDid you forget to source a $INSTALL_PREFIX/etc/restic/*.env.sh profile in the current shell before executing this script?\n" "$varname" >&2
 			exit 1
 		fi
 	done

bin/resticw

@@ -116,8 +116,9 @@ else eval "${prefix}"'_restic_arguments_line_=()'; fi; local docopt_i=1
 declare -p "${prefix}__profile" "${prefix}_restic_arguments_line_"; done; }
 # docopt parser above, complete command for generating this parser is `docopt.sh resticw`
 
-# Parse arguments
-eval "$(docopt "$@")" # See https://github.com/andsens/docopt.sh for the magic :)
+# Parse arguments - See https://github.com/andsens/docopt.sh for the magic :)
+DOCOPT_OPTIONS_FIRST=true  # treat everything after the first non-option as commands/arguments
+eval "$(docopt "$@")"
 
 # --^^^-- END OF GENERATED COMMAND LINE PARSING STUFF --^^^--
 #

etc/cron.d/restic

@@ -5,9 +5,9 @@ PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/:$INSTALL_PREFIX/bin/
 # Reference: https://www.freebsd.org/doc/handbook/configtuning-cron.html
 # Reference: crontab(5).
 
-@midnight	root	. $INSTALL_PREFIX/etc/restic/default.sh && restic_backup.sh
-@monthly	root	. $INSTALL_PREFIX/etc/restic/default.sh && restic_check.sh
+@midnight	root	. $INSTALL_PREFIX/etc/restic/default.env.sh && restic_backup.sh
+@monthly	root	. $INSTALL_PREFIX/etc/restic/default.env.sh && restic_check.sh
 
 # Email notification version. Make sure bin/cron_mail is in the above $PATH
-#@midnight	root	. $INSTALL_PREFIX/etc/restic/default.sh && cron_mail restic_backup.sh
-#@monthly	root	. $INSTALL_PREFIX/etc/restic/default.sh && cron_mail restic_check.sh
+#@midnight	root	. $INSTALL_PREFIX/etc/restic/default.env.sh && cron_mail restic_backup.sh
+#@monthly	root	. $INSTALL_PREFIX/etc/restic/default.env.sh && cron_mail restic_check.sh

etc/restic/_global.env.sh

@@ -29,3 +29,7 @@ export RESTIC_BACKUP_EXTRA_ARGS=
 # Verbosity level from 0-3. 0 means no --verbose.
 # Override this value in a profile if needed.
 export RESTIC_VERBOSITY_LEVEL=0
+
+# (optional) Desktop notifications. See restic_backup.sh for details on how to set this up.
+export RESTIC_NOTIFY_BACKUP_STATS=false
+export RESTIC_BACKUP_NOTIFICATION_FILE=

usr/lib/systemd/system/restic-backup@.service

@@ -10,6 +10,7 @@ Type=simple
 Nice=10
 # $HOME or $XDG_CACHE_HOME must be set for restic to find /root/.cache/restic/
 Environment="HOME=/root"
-# The random sleep (in seconds) is in the case of multiple backup profiles. Many restic instances started at the same time could case high load or network bandwith usage.
+# pipefail: so that redirecting stderr from the script to systemd-cat does not hide the failed command from OnFailure above.
+# Random sleep (in seconds): in the case of multiple backup profiles. Many restic instances started at the same time could case high load or network bandwith usage.
 # `systemd-cat` allows showing the restic output to the systemd journal
-ExecStart=bash -c 'ps cax | grep -q restic && sleep $(shuf -i 0-300 -n 1); source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_backup.sh | systemd-cat'
+ExecStart=bash -c 'set -o pipefail; ps cax | grep -q restic && sleep $(shuf -i 0-300 -n 1); source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_backup.sh 2>&1 | systemd-cat'

usr/lib/systemd/system/restic-check@.service

@@ -9,5 +9,6 @@ Conflicts=restic-backup.service
 [Service]
 Type=simple
 Nice=10
-# `systemd-cat` allows showing the restic output to the systemd journal
-ExecStart=bash -c 'source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_check.sh | systemd-cat'
+# pipefail: so that redirecting stderr from the script to systemd-cat does not hide the failed command from OnFailure above.
+# `systemd-cat`: allows showing the restic output to the systemd journal
+ExecStart=bash -c 'set -o pipefail; source $INSTALL_PREFIX/etc/restic/%I.env.sh && $INSTALL_PREFIX/bin/restic_check.sh 2>&1 | systemd-cat'