Document backup strategy and implementation.

design/BackupStrategy.md

@@ -1,17 +1,18 @@
-Backup Strategy
+# Requirements
 
-# Backup Strategy
-## Requirements
 - All data that are difficult or impossible to reconstruct must be backed up and able to be restored from backup if lost.
 - Backups must be automated and must occur without manual interaction with any user.
 - Backups must be monitored and tested on a regular basis, to ensure that
   1. Backups actually occur when they are scheduled
-	2. Backed-up data can be restored and that the restored data is correct.
+  2. Backed-up data can be restored and the restored data are correct.
 - Backups are encrypted for privacy and security.
 - All data exist in at least three places:  on the device (client or server) where it is used; on a backup device on our home network; and on an off-site device.
+
 ## Strategy
+
 - On-site backup for client devices will be provided by the Teal server.  The backup tool is Restic, accessible from client devices via SFTP.
 - On-site backup for the Teal server itself will be provided by the Cygnus server (Synology).  Backup is by Restic over SFTP.
+- Note that the choice of the SFTP transport requires that the user account under which Restic is executed on each backup source machine must have passwordless SSH / SFTP access to the backup target machine (where the Restic repository resides).  This means that you must log on to the source machine as the user which will be running Restic, execute ssh-keygen to generate a key (with no passphrase), and execute ssh-copy-id sftpuser@target-host to install the key on the target machine.
 - Off-site backup is not currently implemented.  Two different strategies are being considered:
   1. Build a custom ZFS-based NAS and deploy it at 28 Carlisle Rd.  This server and the Teal server would provide off-site backup for each other via ZFS send / receive.
   2. Contract with a cloud storage provider for off-backup, probably either Backblaze using Restic over B2 or rsync.net using ZFS send / receive.

design/ClientBackup.md

@@ -0,0 +1,26 @@
+# Client Device Backup Implementation
+
+## Target Repository
+
+Client devices are backed up to the client-backup Restic repository on Teal.  The transport protocol is SFTP.
+
+## Windows Clients
+
+Windows clients are backed up using the restic-windows-backup PowerShell scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-windows-backup.git (cloned from Kevin Woley's Github repository at https://github.com/kmwoley/restic-windows-backup.git).
+The restic-windows-backup scripts are installed at C:\restic on each Windows client system.  The script code is tailored to the requirements of the client system to specify directories to be backed up, needed passwords, etc.  The process for configuring backup on a Windows system, in brief, is:
+
+1. Install the scripts at C:\restic
+2. Configure the location and password of the target Restic repository and the email information to be used for notifications in the file secrets.ps1.  Use the supplied secrets_sample.ps1 as a guide.
+3. Configure the directories to be backed up and snapshot retention and pruning policies in the file config.ps1.  Use the supplied config_sample.ps1 as a guide.
+
+Consult the README.md in the root of the restic-windows-backup repository for full documentation of the install and configuration process.
+
+Scheduled backups are handled by creating a task in Windows Task Scheduler.  In order to use the Windows VSS facility to back up files that are in use or whose permissions would not allow reading for backup, the Scheduler task must run as the SYSTEM user.  That, in turn, means that SSH keys must be generated (and copied to the target server) for the SYSTEM user.  Use PsExec.exe from the PsTools suite to open a command window logged on as SYSTEM in order to generate and install these SSH keys.  See the discussion of "Backup Over SFTP" in the README file in the Git repository at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler/src/branch/main/README.md.
+
+## Linux Clients
+
+Linux clients are backed up using the restic-automatic-backup-scheduler scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler.git (cloned from Erik Westrup's Github repository at https://github.com/erikw/restic-automatic-backup-scheduler.git).
+The restic-automatic-backup-scheduler scripts are installed as a systemd service on each Linux client system (driven by a systemd timer).  Configuration is done by customizing two scripts at /etc/restic:  _global.env.sh and default.env.sh.  At present, _global.env.sh is used unchanged (as supplied from the Git repo).  default.env.sh is customized to specify the Restic repository and the directories to be backed up:
+
+* export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"
+* export RESTIC_BACKUP_PATHS="/etc:/root:/home"

devices/fedora/config/etc/restic/_global.env.sh

@@ -0,0 +1,37 @@
+# shellcheck shell=sh
+
+# Global environment variables
+# These variables are sourced FIRST, and any values inside of *.env.sh files for
+# specific configurations will override if also defined there.
+
+
+# Official instructions on how to setup the restic variables for Backblaze B2 can be found at
+# https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2
+
+
+# The restic repository encryption key
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
+# The global restic exclude file
+export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
+
+# Backblaze B2 credentials keyID & applicationKey pair.
+# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
+export B2_ACCOUNT_ID="<b2-key-id>"   # *EDIT* fill with your keyID
+export B2_ACCOUNT_KEY="<b2-application-key>" # *EDIT* fill with your applicationKey
+
+# How many network connections to set up to B2. Default is 5.
+export B2_CONNECTIONS=10
+
+# Optional extra space-separated args to restic-backup.
+# This is empty here and profiles can override this after sourcing this file.
+export RESTIC_BACKUP_EXTRA_ARGS=
+
+# Verbosity level from 0-3. 0 means no --verbose.
+# Override this value in a profile if needed.
+export RESTIC_VERBOSITY_LEVEL=0
+
+# (optional, uncomment to enable) Backup summary stats log: snapshot size, etc. (empty/unset won't log)
+#export RESTIC_BACKUP_STATS_DIR="/var/log/restic-automatic-backup-scheduler"
+
+# (optional) Desktop notifications. See README and restic_backup.sh for details on how to set this up (empty/unset means disabled)
+export RESTIC_BACKUP_NOTIFICATION_FILE=

devices/fedora/config/etc/restic/backup_exclude.txt

@@ -0,0 +1,10 @@
+/.snapshots/
+/opt
+/root/.cache/
+/usr/share/**/*.html
+/usr/share/help/
+/usr/share/licenses/
+/usr/share/man/
+/usr/src/
+/var/cache/
+/var/log/

devices/fedora/config/etc/restic/default.env.sh

@@ -0,0 +1,45 @@
+# shellcheck shell=sh
+
+# This is the default profile. Fill it with your desired configuration.
+# Additionally, you can create and use more profiles by copying this file.
+
+# This file (and other .env.sh files) has two purposes:
+# - being sourced by systemd timers to setup the backup before running restic_backup.sh
+# - being sourced in a user's shell to work directly with restic commands e.g.
+#  $ source /etc/restic/default.env.sh
+#  $ restic snapshots
+#  Thus you don't have to provide all the arguments like
+#  $ restic --repo ... --password-file ...
+
+# shellcheck source=etc/restic/_global.env.sh
+. "/etc/restic/_global.env.sh"
+
+# Envvars below will override those in _global.env.sh if present.
+
+export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"   # *EDIT* fill with your repo name
+
+# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
+# To backup only your home directory, set "/home/your-user"
+export RESTIC_BACKUP_PATHS="/etc:/root:/home"  # *EDIT* fill conveniently with one or multiple paths
+
+
+# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
+# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
+#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"
+
+# A tag to identify backup snapshots.
+export RESTIC_BACKUP_TAG=systemd.timer
+
+# Retention policy - How many backups to keep.
+# See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
+export RESTIC_RETENTION_HOURS=1
+export RESTIC_RETENTION_DAYS=14
+export RESTIC_RETENTION_WEEKS=16
+export RESTIC_RETENTION_MONTHS=18
+export RESTIC_RETENTION_YEARS=3
+
+# Optional extra space-separated arguments to restic-backup.
+# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
+# Example: exclude all directories that have a .git/ directory inside it.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

devices/fedora/config/etc/restic/pw.txt

@@ -0,0 +1 @@
+fJbPEg78Dpx5UM