Merge branch 'master' of https://gitea.objectbrokers.com/cjones/sysconfig

design/BackupStrategy.md

@@ -1,17 +1,18 @@
-Backup Strategy
+# Requirements
 
-# Backup Strategy
-## Requirements
 - All data that are difficult or impossible to reconstruct must be backed up and able to be restored from backup if lost.
 - Backups must be automated and must occur without manual interaction with any user.
 - Backups must be monitored and tested on a regular basis, to ensure that
   1. Backups actually occur when they are scheduled
-	2. Backed-up data can be restored and that the restored data is correct.
+  2. Backed-up data can be restored and the restored data are correct.
 - Backups are encrypted for privacy and security.
 - All data exist in at least three places:  on the device (client or server) where it is used; on a backup device on our home network; and on an off-site device.
+
 ## Strategy
+
 - On-site backup for client devices will be provided by the Teal server.  The backup tool is Restic, accessible from client devices via SFTP.
 - On-site backup for the Teal server itself will be provided by the Cygnus server (Synology).  Backup is by Restic over SFTP.
+- Note that the choice of the SFTP transport requires that the user account under which Restic is executed on each backup source machine must have passwordless SSH / SFTP access to the backup target machine (where the Restic repository resides).  This means that you must log on to the source machine as the user which will be running Restic, execute ssh-keygen to generate a key (with no passphrase), and execute ssh-copy-id sftpuser@target-host to install the key on the target machine.
 - Off-site backup is not currently implemented.  Two different strategies are being considered:
   1. Build a custom ZFS-based NAS and deploy it at 28 Carlisle Rd.  This server and the Teal server would provide off-site backup for each other via ZFS send / receive.
   2. Contract with a cloud storage provider for off-backup, probably either Backblaze using Restic over B2 or rsync.net using ZFS send / receive.

design/ClientBackup.md

@@ -0,0 +1,26 @@
+# Client Device Backup Implementation
+
+## Target Repository
+
+Client devices are backed up to the client-backup Restic repository on Teal.  The transport protocol is SFTP.
+
+## Windows Clients
+
+Windows clients are backed up using the restic-windows-backup PowerShell scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-windows-backup.git (cloned from Kevin Woley's Github repository at https://github.com/kmwoley/restic-windows-backup.git).
+The restic-windows-backup scripts are installed at C:\restic on each Windows client system.  The script code is tailored to the requirements of the client system to specify directories to be backed up, needed passwords, etc.  The process for configuring backup on a Windows system, in brief, is:
+
+1. Install the scripts at C:\restic
+2. Configure the location and password of the target Restic repository and the email information to be used for notifications in the file secrets.ps1.  Use the supplied secrets_sample.ps1 as a guide.
+3. Configure the directories to be backed up and snapshot retention and pruning policies in the file config.ps1.  Use the supplied config_sample.ps1 as a guide.
+
+Consult the README.md in the root of the restic-windows-backup repository for full documentation of the install and configuration process.
+
+Scheduled backups are handled by creating a task in Windows Task Scheduler.  In order to use the Windows VSS facility to back up files that are in use or whose permissions would not allow reading for backup, the Scheduler task must run as the SYSTEM user.  That, in turn, means that SSH keys must be generated (and copied to the target server) for the SYSTEM user.  Use PsExec.exe from the PsTools suite to open a command window logged on as SYSTEM in order to generate and install these SSH keys.  See the discussion of "Backup Over SFTP" in the README file in the Git repository at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler/src/branch/main/README.md.
+
+## Linux Clients
+
+Linux clients are backed up using the restic-automatic-backup-scheduler scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler.git (cloned from Erik Westrup's Github repository at https://github.com/erikw/restic-automatic-backup-scheduler.git).
+The restic-automatic-backup-scheduler scripts are installed as a systemd service on each Linux client system (driven by a systemd timer).  Configuration is done by customizing two scripts at /etc/restic:  _global.env.sh and default.env.sh.  At present, _global.env.sh is used unchanged (as supplied from the Git repo).  default.env.sh is customized to specify the Restic repository and the directories to be backed up:
+
+* export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"
+* export RESTIC_BACKUP_PATHS="/etc:/root:/home"

devices/cranberrypi/config/usr/local/etc/unbound/unbound.conf

@@ -49,23 +49,29 @@ view:
   name: "lan"
   view-first: yes
   local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 192.168.88.231"
+  local-data: "cygnus.objectbrokers.com. A 192.168.88.75"
   local-data: "nextcloud.objectbrokers.com. A 192.168.88.231"
   local-data: "photo.objectbrokers.com. A 192.168.88.231"
   local-data: "gitea.objectbrokers.com. A 192.168.88.231"
   local-data: "portainer.objectbrokers.com. A 192.168.88.231"
   local-data: "jellyfin.objectbrokers.com. A 192.168.88.231"
   local-data: "vaultwarden.objectbrokers.com. A 192.168.88.231"
+  local-data: "lambdesktop.objectbrokers.com. A 192.168.88.187"
 
 view:
   name: "tailnet"
   view-first: yes
   local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 100.81.165.11"
+  local-data: "cygnus.objectbrokers.com. A 100.99.151.65"
   local-data: "nextcloud.objectbrokers.com. A 100.81.165.11"
   local-data: "photo.objectbrokers.com. A 100.81.165.11"
   local-data: "gitea.objectbrokers.com. A 100.81.165.11"
   local-data: "portainer.objectbrokers.com. A 100.81.165.11"
   local-data: "jellyfin.objectbrokers.com. A 100.81.165.11"
   local-data: "vaultwarden.objectbrokers.com. A 100.81.165.11"
+  local-data: "lambdesktop.objectbrokers.com. A 100.95.22.24"
 
 remote-control:
   control-enable: yes

devices/fedora/config/etc/restic/_global.env.sh

@@ -0,0 +1,37 @@
+# shellcheck shell=sh
+
+# Global environment variables
+# These variables are sourced FIRST, and any values inside of *.env.sh files for
+# specific configurations will override if also defined there.
+
+
+# Official instructions on how to setup the restic variables for Backblaze B2 can be found at
+# https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2
+
+
+# The restic repository encryption key
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
+# The global restic exclude file
+export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
+
+# Backblaze B2 credentials keyID & applicationKey pair.
+# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
+export B2_ACCOUNT_ID="<b2-key-id>"   # *EDIT* fill with your keyID
+export B2_ACCOUNT_KEY="<b2-application-key>" # *EDIT* fill with your applicationKey
+
+# How many network connections to set up to B2. Default is 5.
+export B2_CONNECTIONS=10
+
+# Optional extra space-separated args to restic-backup.
+# This is empty here and profiles can override this after sourcing this file.
+export RESTIC_BACKUP_EXTRA_ARGS=
+
+# Verbosity level from 0-3. 0 means no --verbose.
+# Override this value in a profile if needed.
+export RESTIC_VERBOSITY_LEVEL=0
+
+# (optional, uncomment to enable) Backup summary stats log: snapshot size, etc. (empty/unset won't log)
+#export RESTIC_BACKUP_STATS_DIR="/var/log/restic-automatic-backup-scheduler"
+
+# (optional) Desktop notifications. See README and restic_backup.sh for details on how to set this up (empty/unset means disabled)
+export RESTIC_BACKUP_NOTIFICATION_FILE=

devices/fedora/config/etc/restic/backup_exclude.txt

@@ -0,0 +1,10 @@
+/.snapshots/
+/opt
+/root/.cache/
+/usr/share/**/*.html
+/usr/share/help/
+/usr/share/licenses/
+/usr/share/man/
+/usr/src/
+/var/cache/
+/var/log/

devices/fedora/config/etc/restic/default.env.sh

@@ -0,0 +1,45 @@
+# shellcheck shell=sh
+
+# This is the default profile. Fill it with your desired configuration.
+# Additionally, you can create and use more profiles by copying this file.
+
+# This file (and other .env.sh files) has two purposes:
+# - being sourced by systemd timers to setup the backup before running restic_backup.sh
+# - being sourced in a user's shell to work directly with restic commands e.g.
+#  $ source /etc/restic/default.env.sh
+#  $ restic snapshots
+#  Thus you don't have to provide all the arguments like
+#  $ restic --repo ... --password-file ...
+
+# shellcheck source=etc/restic/_global.env.sh
+. "/etc/restic/_global.env.sh"
+
+# Envvars below will override those in _global.env.sh if present.
+
+export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"   # *EDIT* fill with your repo name
+
+# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
+# To backup only your home directory, set "/home/your-user"
+export RESTIC_BACKUP_PATHS="/etc:/root:/home"  # *EDIT* fill conveniently with one or multiple paths
+
+
+# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
+# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
+#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"
+
+# A tag to identify backup snapshots.
+export RESTIC_BACKUP_TAG=systemd.timer
+
+# Retention policy - How many backups to keep.
+# See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
+export RESTIC_RETENTION_HOURS=1
+export RESTIC_RETENTION_DAYS=14
+export RESTIC_RETENTION_WEEKS=16
+export RESTIC_RETENTION_MONTHS=18
+export RESTIC_RETENTION_YEARS=3
+
+# Optional extra space-separated arguments to restic-backup.
+# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
+# Example: exclude all directories that have a .git/ directory inside it.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

devices/fedora/config/etc/restic/pw.txt

@@ -0,0 +1 @@
+fJbPEg78Dpx5UM

devices/teal/config/usr/local/etc/unbound/unbound.conf

@@ -50,24 +50,28 @@ view:
   view-first: yes
   local-zone: "objectbrokers.com." transparent
   local-data: "teal.objectbrokers.com. A 192.168.88.231"
+  local-data: "cygnus.objectbrokers.com. A 192.168.88.75"
   local-data: "nextcloud.objectbrokers.com. A 192.168.88.231"
   local-data: "photo.objectbrokers.com. A 192.168.88.231"
   local-data: "gitea.objectbrokers.com. A 192.168.88.231"
   local-data: "portainer.objectbrokers.com. A 192.168.88.231"
   local-data: "jellyfin.objectbrokers.com. A 192.168.88.231"
   local-data: "vaultwarden.objectbrokers.com. A 192.168.88.231"
+  local-data: "lambdesktop.objectbrokers.com. A 192.168.88.187"
 
 view:
   name: "tailnet"
   view-first: yes
   local-zone: "objectbrokers.com." transparent
   local-data: "teal.objectbrokers.com. A 100.81.165.11"
+  local-data: "cygnus.objectbrokers.com. A 100.99.151.65"
   local-data: "nextcloud.objectbrokers.com. A 100.81.165.11"
   local-data: "photo.objectbrokers.com. A 100.81.165.11"
   local-data: "gitea.objectbrokers.com. A 100.81.165.11"
   local-data: "portainer.objectbrokers.com. A 100.81.165.11"
   local-data: "jellyfin.objectbrokers.com. A 100.81.165.11"
   local-data: "vaultwarden.objectbrokers.com. A 100.81.165.11"
+  local-data: "lambdesktop.objectbrokers.com. A 100.95.22.24"
 
 remote-control:
   control-enable: yes

services/nextcloud/upgrading.md

@@ -1,16 +1,18 @@
-Steps to upgrade Nextcloud from one version to another.
+Steps to upgrade Nextcloud from one major version to the next.
 
-1.  Put the Nextcloud instance into maintenance mode
-    docker exec --user www-data nextcloud-app-1 php occ maintenance:mode --on
+1. Bring the Nextcloud instance down:
+   sudo docker compose down
+2. Take a ZFS snapshot of the tank/appdata/data/nextcloud-data and tank/appdata/databases/nextcloud-db dataset(s).
+3. Edit the compose.yaml file and change the tag on the Nextcloud image to the next major version number.
+4. Pull the next version of the Nextcloud image:
+   sudo docker compose pull
+5. Bring up the Nextcloud instance:
+   sudo docker compose up -d
+6. Wait a minute or so and bring up the Nextcloud web client in your browser.  Open the Administrative Settings Overview page and check for any issues.
+8. If the Overview page recommends any occ commands:
+
+   * Put the instance into maintenance mode : docker exec -u www-data nextcloud-app-1 php occ maintenance:mode --on
+   * Run the occ commands recommended (for example, docker exec -u www-data nextcloud-app-1 php occ db:add-missing-indices)
+   * Take the instance out of maintenance mode: docker exec -u www-data nextcloud-app-1 php occ maintenance:mode --off
    
    Note that 'docker exec' is used to run commands within the Nextcloud docker container; and '--user www-data' is used to run as the user that owns all of the Nextcloud files.
-
-2.  Back up the Nextcloud container's files to a location outside the container's mounted volume.  'rsync' is recommended:
-
-    rsync -Aavx /mnt/storage/appdata/nextcloud/www/ <backup-target-dir>/
-
-    rsync should be run as root.  rsync with these options should preserve file ownership & permissions
-
-3.  Back up MariaDB database:
-
-        docker exec nextcloud-db-container-name mysqldump --single-transaction -h localhost -u nextcloud_user -pnextcloud_password nextcloud_db_name > nextcloud-sqlbkp_$(date +"%Y%m%d").bak

services/upgrades.md

@@ -0,0 +1,30 @@
+# Upgrade Cadence For OS And Services
+
+The overall policy for software upgrades is to keep up with the most recent stable release of any operating system or service running on any of our devices.  When a new major version of a service is released, we can upgrade right away (especially if the project or vendor has earned a reputation for stability); or it is acceptable to wait for a point release (or simply the passage of a month or so) to avoid breakage.
+
+For operating systems, we rely on the update facility of the OS vendor to provide timely updates, especially for security vulnerabilities.
+
+## Notes for specific software
+
+### Operating Systems
+
+* **Microsoft Windows**.  Microsoft doesn't allow much leeway here.  To ensure security and full support on our Windows client devices, we let Windows Update handle this according to the normal Microsoft schedule.
+* **Ubuntu**.  All Unbuntu servers are currently on version 24.04 LTS.  When Canonical provides an update (particularly for security issues) it should be applied in a reasonably timely fashion.  When a major LTS version is available (e.g. 26.04), we will wait for a point release before upgrading to the next point release.
+* **Fedora**.  Fedora is currently running only on a client device.  When a major release is available, we will wait for a point release in the interest of stability, and then upgrade at our leisure.  We should not, however, allow Fedora to be more than one major version behind.  Check Fedora release status quarterly.
+* **Raspbian**.  Currently only one device (raspberrygrove) is running Raspbian (Buster).  Rather than upgrade to Raspberry Pi OS Trixie, we will install Ubuntu on this device, probably after Ubuntu 26.04 LTS is available has been installed on our other Ubuntu servers.  From then on, raspberrygrove will follow the same upgrade cadence as other Ubuntu servers.
+* **Android and iOS**.  Like Microsoft, Google and Apple keep a tight rein on this.  We will let the OS vendors drive the upgrade cadence for mobile clients.
+
+### Local Services
+
+For Docker-based services (and anything else that is installed to our ZFS pool),  take a ZFS snapshot of the relevant dataset(s) before doing the upgrade.  For Docker containers specifically, first bring the service down before taking the snapshot.  Don't rely on the regular snapshots scheduled by Sanoid; snapshots taken while a Docker instance is running may not be consistent, if a database is involved.
+* **caddy**.  Upgrade to major releases only.
+* **gitea**.  Upgrade to major releases only.
+* **immich**.  Immich have a history of rapid development and breaking changes.  Even though the project has now released a "stable" version, we will continue to be cautious.  Experience with Immich has shown that upgrading each minor release one by one, with a ZFS snapshot to roll back to, is the best practice.
+* **jellyfin**.  Upgrade to major releases only.
+* **nextcloud**.  Upgrade to major releases only.
+* **portainer**.  Upgrade to major releases only.
+* **sanoid**.  Upgrade to major releases only.
+* **tailscale**.  As a key part of the "network operating system", it's worth keeping Tailscale continuously up to date.  Tailscale have earned a reputation for quality and stability.  Therefore we will monitor the Tailscale admin console and ensure that all devices are up to date at least monthly.
+* **unbound**.  Upgrade to major releases only.
+* **vaultwarden**.  Upgrade for both major and minor releases.  Password managers are a high-value target, so being up to date with security fixes is particularly important.
+* **zfs**.  ZFS is provided by Canonical and is effectively part of the OS.  We will upgrade ZFS as part of the OS upgrade process for Ubuntu.  (Subject to change, of course, if ZFS is installed on a non-Ubuntu device.)