Add all local devices to DNS

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "editor.formatOnSave": true
+}

design/BackupStrategy.md

@@ -0,0 +1,18 @@
+# Requirements
+
+- All data that are difficult or impossible to reconstruct must be backed up and able to be restored from backup if lost.
+- Backups must be automated and must occur without manual interaction with any user.
+- Backups must be monitored and tested on a regular basis, to ensure that
+  1. Backups actually occur when they are scheduled
+  2. Backed-up data can be restored and the restored data are correct.
+- Backups are encrypted for privacy and security.
+- All data exist in at least three places:  on the device (client or server) where it is used; on a backup device on our home network; and on an off-site device.
+
+## Strategy
+
+- On-site backup for client devices will be provided by the Teal server.  The backup tool is Restic, accessible from client devices via SFTP.
+- On-site backup for the Teal server itself will be provided by the Cygnus server (Synology).  Backup is by Restic over SFTP.
+- Note that the choice of the SFTP transport requires that the user account under which Restic is executed on each backup source machine must have passwordless SSH / SFTP access to the backup target machine (where the Restic repository resides).  This means that you must log on to the source machine as the user which will be running Restic, execute ssh-keygen to generate a key (with no passphrase), and execute ssh-copy-id sftpuser@target-host to install the key on the target machine.
+- Off-site backup is not currently implemented.  Two different strategies are being considered:
+  1. Build a custom ZFS-based NAS and deploy it at 28 Carlisle Rd.  This server and the Teal server would provide off-site backup for each other via ZFS send / receive.
+  2. Contract with a cloud storage provider for off-backup, probably either Backblaze using Restic over B2 or rsync.net using ZFS send / receive.

design/ClientBackup.md

@@ -0,0 +1,26 @@
+# Client Device Backup Implementation
+
+## Target Repository
+
+Client devices are backed up to the client-backup Restic repository on Teal.  The transport protocol is SFTP.
+
+## Windows Clients
+
+Windows clients are backed up using the restic-windows-backup PowerShell scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-windows-backup.git (cloned from Kevin Woley's Github repository at https://github.com/kmwoley/restic-windows-backup.git).
+The restic-windows-backup scripts are installed at C:\restic on each Windows client system.  The script code is tailored to the requirements of the client system to specify directories to be backed up, needed passwords, etc.  The process for configuring backup on a Windows system, in brief, is:
+
+1. Install the scripts at C:\restic
+2. Configure the location and password of the target Restic repository and the email information to be used for notifications in the file secrets.ps1.  Use the supplied secrets_sample.ps1 as a guide.
+3. Configure the directories to be backed up and snapshot retention and pruning policies in the file config.ps1.  Use the supplied config_sample.ps1 as a guide.
+
+Consult the README.md in the root of the restic-windows-backup repository for full documentation of the install and configuration process.
+
+Scheduled backups are handled by creating a task in Windows Task Scheduler.  In order to use the Windows VSS facility to back up files that are in use or whose permissions would not allow reading for backup, the Scheduler task must run as the SYSTEM user.  That, in turn, means that SSH keys must be generated (and copied to the target server) for the SYSTEM user.  Use PsExec.exe from the PsTools suite to open a command window logged on as SYSTEM in order to generate and install these SSH keys.  See the discussion of "Backup Over SFTP" in the README file in the Git repository at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler/src/branch/main/README.md.
+
+## Linux Clients
+
+Linux clients are backed up using the restic-automatic-backup-scheduler scripts.  The source code and documentation for these scripts is in a repository on our Gitea server at https://gitea.objectbrokers.com/cjones/restic-automatic-backup-scheduler.git (cloned from Erik Westrup's Github repository at https://github.com/erikw/restic-automatic-backup-scheduler.git).
+The restic-automatic-backup-scheduler scripts are installed as a systemd service on each Linux client system (driven by a systemd timer).  Configuration is done by customizing two scripts at /etc/restic:  _global.env.sh and default.env.sh.  At present, _global.env.sh is used unchanged (as supplied from the Git repo).  default.env.sh is customized to specify the Restic repository and the directories to be backed up:
+
+* export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"
+* export RESTIC_BACKUP_PATHS="/etc:/root:/home"

design/network.md

@@ -0,0 +1,33 @@
+# Requirements
+
+* Internet access
+  * DHCP
+  * DNS
+* Robust backup of files, photos, media, etc
+* Secure, private access to data and services on the LAN from devices on the LAN and on the public internet
+* Audio and video media
+* File sharing (WebDAV)
+* System administration tools for remote access, monitoring, etc
+
+# Design
+
+## Network Infrastructure
+
+The family network is a standard home LAN over Ethernet and WiFi, connected to the public Internet through Verizon FiOS.
+
+Components of the network infrastructure:
+
+* Verizon Optical Network Terminal (ONT).  The ONT is owned and managed by Verizon; further technical detail is unavailable.  The ONT is connected to the main Ethernet router (RB5009) via Ethernet cable.
+* Mikrotik RB5009 Ethernet router
+* Mikrotik hAPax3 WiFi access point
+* Cisco/Linksys WiFi router configured in bridge mode to act as a WiFi access point (in Evan's room)
+* Category 6 Ethernet cabling providing the following Ethernet drops:
+  * Downstairs office
+  * Upstairs office
+  * Evan's room
+  * Living room drop 1:  entertainment stack under the main TV
+  * Living room drop 2:  corner near the fireplace
+* Ethernet switches as needed to provide wired Ethernet connectivity to additional devices
+  * On the main rack in the basement
+  * In Evan's room
+  * Behind the entertainment stack in the living room

devices/RB5009/RB5009Config.rsc

@@ -0,0 +1,108 @@
+# 2026-03-10 22:40:31 by RouterOS 7.19.2
+# software id = CIAZ-SUFT
+#
+# model = RB5009UG+S+
+# serial number = HEE08K82CQV
+/interface bridge
+add name=local port-cost-mode=short
+/interface wireless security-profiles
+set [ find default=yes ] supplicant-identity=MikroTik
+/ip pool
+add name=dhcp_pool0 ranges=192.168.88.2-192.168.88.254
+add name=dhcp_pool1 ranges=192.168.88.10-192.168.88.254
+/ip dhcp-server
+add address-pool=dhcp_pool1 interface=local lease-time=10m name=dhcp2
+/ip smb users
+add name=cjones
+add name=chris
+/ip smb
+set enabled=yes
+/interface bridge port
+add bridge=local interface=ether2 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether3 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether4 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether5 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether6 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether7 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether8 internal-path-cost=10 path-cost=10
+add bridge=local interface=ether1 internal-path-cost=10 path-cost=10
+/ip firewall connection tracking
+set udp-timeout=10s
+/interface ovpn-server server
+add mac-address=FE:73:F4:5A:2B:60 name=ovpn-server1
+/ip address
+add address=192.168.88.1/24 interface=local network=192.168.88.0
+/ip dhcp-client
+add interface=sfp-sfpplus1
+/ip dhcp-server lease
+add address=192.168.88.239 client-id=1:0:11:32:28:2:98 mac-address=\
+    00:11:32:28:02:98 server=dhcp2
+add address=192.168.88.47 client-id=1:48:a9:8a:c0:95:a mac-address=\
+    48:A9:8A:C0:95:0A server=dhcp2
+add address=192.168.88.232 client-id=1:dc:a6:32:67:1:16 mac-address=\
+    DC:A6:32:67:01:16 server=dhcp2
+add address=192.168.88.231 client-id=1:a8:a1:59:ae:a0:3e mac-address=\
+    A8:A1:59:AE:A0:3E server=dhcp2
+add address=192.168.88.15 client-id=1:dc:cd:2f:b:aa:b1 mac-address=\
+    DC:CD:2F:0B:AA:B1 server=dhcp2
+add address=192.168.88.87 client-id=1:5c:f9:dd:e5:41:eb mac-address=\
+    5C:F9:DD:E5:41:EB server=dhcp2
+add address=192.168.88.26 client-id=1:c8:b2:9b:db:b0:23 mac-address=\
+    C8:B2:9B:DB:B0:23 server=dhcp2
+add address=192.168.88.250 client-id=1:e0:2b:e9:cf:dc:d5 mac-address=\
+    E0:2B:E9:CF:DC:D5 server=dhcp2
+add address=192.168.88.20 client-id=1:dc:21:5c:84:3a:a5 mac-address=\
+    DC:21:5C:84:3A:A5 server=dhcp2
+add address=192.168.88.144 comment="Static IP for Clinitek engine" \
+    mac-address=3E:BE:90:50:0E:47
+add address=192.168.88.138 client-id=\
+    ff:f8:ce:1b:a1:0:2:0:0:ab:11:6f:15:1:e4:34:20:3c:8c mac-address=\
+    A2:53:3A:64:F4:DE server=dhcp2
+add address=192.168.88.25 client-id=1:bc:f8:7e:8f:32:ea mac-address=\
+    BC:F8:7E:8F:32:EA server=dhcp2
+add address=192.168.88.40 client-id=\
+    ff:e4:96:b0:28:0:2:0:0:ab:11:a:d3:57:3f:cd:69:67:6c mac-address=\
+    DC:A6:32:67:01:17 server=dhcp2
+/ip dhcp-server network
+add
+add address=192.168.88.0/24 dns-server=192.168.88.231,192.168.88.40 gateway=\
+    192.168.88.1 wins-server=0.0.0.0
+/ip dns
+set allow-remote-requests=yes servers=8.8.8.8
+/ip firewall filter
+add action=accept chain=input comment="accept established,related" \
+    connection-state=established,related
+add action=drop chain=input connection-state=invalid
+add action=accept chain=input comment="allow ICMP" in-interface=sfp-sfpplus1 \
+    protocol=icmp
+add action=accept chain=input comment="allow Winbox" in-interface=\
+    sfp-sfpplus1 port=8291 protocol=tcp
+add action=accept chain=input comment="allow SSH" in-interface=sfp-sfpplus1 \
+    port=22 protocol=tcp
+add action=drop chain=input comment="block everything else" in-interface=\
+    sfp-sfpplus1
+/ip firewall nat
+add action=masquerade chain=srcnat out-interface=sfp-sfpplus1
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=80 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=80
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=8080 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=8080
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=443 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=443
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=8070 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=8070
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=52199 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=52199
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=3389 \
+    protocol=tcp to-addresses=192.168.88.250 to-ports=3389
+add action=dst-nat chain=dstnat dst-address=173.48.126.187 dst-port=443 \
+    protocol=tcp to-addresses=192.168.88.231 to-ports=443
+/ip ipsec profile
+set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
+/ip smb shares
+set [ find default=yes ] directory=/pub disabled=no
+add directory=demoshare name=demoshare
+/system clock
+set time-zone-name=America/New_York
+/system identity
+set name=RB5009

devices/README.md

@@ -1 +1 @@
-This directory contains the system configuration of the devices connected to the network.  There is one directory per device, named with the host name of the device.  Each device directory corresponds to the root directory (/) of the device that it documents; the directory structure under it mirrors the actual directory structure on the device, as needed to represent the configuration files being documented.
+This directory contains the system configuration of the devices connected to the network.  There is one directory per device, named with the host name of the device.  Under the device directory is a 'config' directory, which corresponds to the root directory (/) of the device that it documents; the directory structure under it mirrors the actual directory structure on the device, as needed to represent the configuration files being documented.

devices/cranberrypi/config/etc/systemd/resolved.conf

@@ -0,0 +1,36 @@
+#  This file is part of systemd.
+#
+#  systemd is free software; you can redistribute it and/or modify it under the
+#  terms of the GNU Lesser General Public License as published by the Free
+#  Software Foundation; either version 2.1 of the License, or (at your option)
+#  any later version.
+#
+# Entries in this file show the compile time defaults. Local configuration
+# should be created by either modifying this file (or a copy of it placed in
+# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
+# the /etc/systemd/resolved.conf.d/ directory. The latter is generally
+# recommended. Defaults can be restored by simply deleting the main
+# configuration file and all drop-ins located in /etc/.
+#
+# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
+#
+# See resolved.conf(5) for details.
+
+[Resolve]
+# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
+# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
+# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
+# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
+DNS=192.168.88.231 192.168.88.40
+Domains=~objectbrokers.com
+#DNSSEC=no
+#DNSOverTLS=no
+#MulticastDNS=no
+#LLMNR=no
+#Cache=no-negative
+#CacheFromLocalhost=no
+DNSStubListener=no
+#DNSStubListenerExtra=
+#ReadEtcHosts=yes
+#ResolveUnicastSingleLabel=no
+#StaleRetentionSec=0

devices/cranberrypi/config/lib/systemd/system/unbound.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Unbound DNS server
+Documentation=man:unbound(8)
+After=network.target
+Before=nss-lookup.target
+Wants=nss-lookup.target
+
+[Service]
+Type=exec
+Restart=on-failure
+EnvironmentFile=-/usr/local/etc/unbound/unbound_env
+ExecStart=/usr/local/sbin/unbound -d -p $DAEMON_OPTS
+ExecReload=+/bin/kill -HUP $MAINPID
+
+[Install]
+WantedBy=multi-user.target
+

devices/cranberrypi/config/usr/local/etc/unbound/root.key

@@ -0,0 +1,10 @@
+; autotrust trust anchor file
+;;id: . 1
+;;last_queried: 1771250359 ;;Mon Feb 16 08:59:19 2026
+;;last_success: 1771250359 ;;Mon Feb 16 08:59:19 2026
+;;next_probe_time: 1771292919 ;;Mon Feb 16 20:48:39 2026
+;;query_failed: 0
+;;query_interval: 43200
+;;retry_time: 8640
+.	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1771031738 ;;Fri Feb 13 20:15:38 2026
+.	86400	IN	DNSKEY	257 3 8 AwEAAa96jeuknZlaeSrvyAJj6ZHv28hhOKkx3rLGXVaC6rXTsDc449/cidltpkyGwCJNnOAlFNKF2jBosZBU5eeHspaQWOmOElZsjICMQMC3aeHbGiShvZsx4wMYSjH8e7Vrhbu6irwCzVBApESjbUdpWWmEnhathWu1jo+siFUiRAAxm9qyJNg/wOZqqzL/dL/q8PkcRU5oUKEpUge71M3ej2/7CPqpdVwuMoTvoB+ZOT4YeGyxMvHmbrxlFzGOHOijtzN+u1TQNatX2XBuzZNQ1K+s2CXkPIZo7s6JgZyvaBevYtxPvYLw4z9mR7K2vaF18UYH9Z9GNUUeayffKC73PYc= ;{id = 38696 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1771031738 ;;Fri Feb 13 20:15:38 2026

devices/cranberrypi/config/usr/local/etc/unbound/unbound.conf

@@ -0,0 +1,78 @@
+# Unbound configuration file for Debian.
+#
+# See the unbound.conf(5) man page.
+#
+# See /usr/share/doc/unbound/objectbrokerss/unbound.conf for a commented
+# reference config file.
+#
+# The following line includes additional configuration files from the
+# /etc/unbound/unbound.conf.d directory.
+    server:
+    # location of the trust anchor file that enables DNSSEC
+    auto-trust-anchor-file: "/root.key"
+    # send minimal amount of information to upstream servers to enhance privacy
+    qname-minimisation: yes
+    prefetch: yes
+    serve-expired: yes
+    # the interface that is used to connect to the network (this will listen to all interfaces)
+    interface: 0.0.0.0
+    # interface: ::0
+    private-address: 192.168.0.0/16
+    private-address: 100.64.0.0/10
+
+    # addresses from the IP range that are allowed to connect to the resolver
+    access-control: 192.168.88.0/24 allow
+    # explicitly allow localhost access
+    access-control: 127.0.0.0/8 allow
+    # allow Tailnet
+    access-control: 100.64.0.0/10 allow
+    # uncomment the following line to allow Tailnet IPv6
+    # access-control: fd7a:115c:a1e0::/48 allow
+
+    access-control-view: 192.168.88.0/24 lan
+    access-control-view: 100.64.0.0/10 tailnet
+    
+    do-ip4: yes
+    do-ip6: no
+    do-udp: yes
+    do-tcp: yes
+
+    forward-zone:
+      name: "ts.net."
+      forward-addr: 100.100.100.100
+
+    forward-zone:
+      name: "100.in-addr.arpa."
+      forward-addr: 100.100.100.100
+
+view:
+  name: "lan"
+  view-first: yes
+  local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 192.168.88.231"
+  local-data: "cygnus.objectbrokers.com. A 192.168.88.75"
+  local-data: "nextcloud.objectbrokers.com. A 192.168.88.231"
+  local-data: "photo.objectbrokers.com. A 192.168.88.231"
+  local-data: "gitea.objectbrokers.com. A 192.168.88.231"
+  local-data: "portainer.objectbrokers.com. A 192.168.88.231"
+  local-data: "jellyfin.objectbrokers.com. A 192.168.88.231"
+  local-data: "vaultwarden.objectbrokers.com. A 192.168.88.231"
+
+view:
+  name: "tailnet"
+  view-first: yes
+  local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 100.81.165.11"
+  local-data: "cygnus.objectbrokers.com. A 100.99.151.65"
+  local-data: "nextcloud.objectbrokers.com. A 100.81.165.11"
+  local-data: "photo.objectbrokers.com. A 100.81.165.11"
+  local-data: "gitea.objectbrokers.com. A 100.81.165.11"
+  local-data: "portainer.objectbrokers.com. A 100.81.165.11"
+  local-data: "jellyfin.objectbrokers.com. A 100.81.165.11"
+  local-data: "vaultwarden.objectbrokers.com. A 100.81.165.11"
+
+remote-control:
+  control-enable: yes
+  control-interface: /run/unbound.ctl
+
+

devices/cranberrypi/config/usr/local/etc/unbound/unbound.conf.d/remote-control.conf

@@ -0,0 +1,5 @@
+remote-control:
+  control-enable: yes
+  # by default the control interface is is 127.0.0.1 and ::1 and port 8953
+  # it is possible to use a unix socket too
+  control-interface: /run/unbound.ctl

devices/cranberrypi/config/usr/local/etc/unbound/unbound.conf.txt

@@ -0,0 +1,62 @@
+# Unbound configuration file for Debian.
+#
+# See the unbound.conf(5) man page.
+#
+# See /usr/share/doc/unbound/examples/unbound.conf for a commented
+# reference config file.
+#
+# The following line includes additional configuration files from the
+# /etc/unbound/unbound.conf.d directory.
+    server:
+    # location of the trust anchor file that enables DNSSEC
+    auto-trust-anchor-file: "/root.key"
+    # send minimal amount of information to upstream servers to enhance privacy
+    qname-minimisation: yes
+    # the interface that is used to connect to the network (this will listen to all interfaces)
+    interface: 0.0.0.0
+    # interface: ::0
+    private-address: 192.168.0.0/16
+    private-address: 100.64.0.0/10
+
+    # addresses from the IP range that are allowed to connect to the resolver
+    access-control: 192.168.88.0/24 allow
+    # explicitly allow localhost access
+    access-control: 127.0.0.0/8 allow
+    # allow Tailnet
+    access-control: 100.64.0.0/10 allow
+    # uncomment the following line to allow Tailnet IPv6
+    # access-control: fd7a:115c:a1e0::/48 allow
+
+    access-control-view: 192.168.88.0/24 lan
+    access-control-view: 100.64.0.0/10 tailnet
+    
+    do-ip4: yes
+    do-ip6: no
+    do-udp: yes
+    do-tcp: yes
+
+view:
+  name: "lan"
+  view-first: yes
+  local-zone: "example.com." transparent
+  local-data: "nextcloud.example.com. A 192.168.88.231"
+  local-data: "photo.example.com. A 192.168.88.231"
+  local-data: "gitea.example.com. A 192.168.88.231"
+  local-data: "portainer.example.com. A 192.168.88.231"
+  local-data: "vaultwarden.example.com. A 192.168.88.231"
+
+view:
+  name: "tailnet"
+  view-first: yes
+  local-zone: "example.com." transparent
+  local-data: "nextcloud.example.com. A 100.81.165.11"
+  local-data: "photo.example.com. A 100.81.165.11"
+  local-data: "gitea.example.com. A 100.81.165.11"
+  local-data: "portainer.example.com. A 100.81.165.11"
+  local-data: "vaultwarden.example.com. A 100.81.165.11"
+
+remote-control:
+  control-enable: yes
+  control-interface: /run/unbound.ctl
+
+

devices/cranberrypi/readme.md

@@ -0,0 +1,2 @@
+System configuration files for host cranberrypi.  The directory hierarchy under this repo corresponds to the directory hierarchy under / (root) on the host cranberrypi.
+

devices/cranberrypi/unbound.md

@@ -0,0 +1,25 @@
+Unbound provides DNS resolution service for the local network.  Unbound was built from source and installed on cranberrypi, bare metal (configure, make, sudo make install).
+
+The configuration file for Unbound is at /usr/local/etc/unbound/unbound.conf, with included configuration files in the directory /usr/local/etc/unbound/unbound.conf.d.
+
+Notes on Unbound configuration
+
+Unbound is configured for Split DNS to provide a different address resolution for services running on the home LAN, depending on whether the requesting client is running on the home LAN, on our Tailnet, or on a system entirely outside our network, on the public Internet.  The Unbound view construct is used to implement this. 
+
+There are two Unbound views defined:  "lan" and "tailnet".  The "lan" view includes local-data records for the available services on our network (mostly, but not exclusively, running on Teal), for example:
+
+  local-data: "nextcloud.objectbrokers.com. A 192.168.88.231"
+
+Each local-data record in the "lan" view points to a physical IP address on the home LAN.
+
+The "tailnet" view includes local-data records for the same set of services on our network as the "lan" view, for example:
+
+  local-data: "nextcloud.objectbrokers.com. A 100.81.165.11"
+
+Each local-data record in the "tailnet" view points to a Tailscale IP address on our Tailnet.
+
+Maintenance
+
+The Unbound configuration must be carefully maintained to enable Unbound to resolve URLs for our services correctly.
+
+Both views must include local-data records for each published service; each view must include the same set of names to be resolved.  The view differ in the IP address referenced for each name, not in the names included in the view.  Thus when a new service is published, a local-data record for that service must be added to both views.  When a service is deleted from the network, its local-data records in both views ("lan" and "tailnet") must be deleted.

devices/fedora/config/etc/restic/_global.env.sh

@@ -0,0 +1,37 @@
+# shellcheck shell=sh
+
+# Global environment variables
+# These variables are sourced FIRST, and any values inside of *.env.sh files for
+# specific configurations will override if also defined there.
+
+
+# Official instructions on how to setup the restic variables for Backblaze B2 can be found at
+# https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2
+
+
+# The restic repository encryption key
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
+# The global restic exclude file
+export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
+
+# Backblaze B2 credentials keyID & applicationKey pair.
+# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
+export B2_ACCOUNT_ID="<b2-key-id>"   # *EDIT* fill with your keyID
+export B2_ACCOUNT_KEY="<b2-application-key>" # *EDIT* fill with your applicationKey
+
+# How many network connections to set up to B2. Default is 5.
+export B2_CONNECTIONS=10
+
+# Optional extra space-separated args to restic-backup.
+# This is empty here and profiles can override this after sourcing this file.
+export RESTIC_BACKUP_EXTRA_ARGS=
+
+# Verbosity level from 0-3. 0 means no --verbose.
+# Override this value in a profile if needed.
+export RESTIC_VERBOSITY_LEVEL=0
+
+# (optional, uncomment to enable) Backup summary stats log: snapshot size, etc. (empty/unset won't log)
+#export RESTIC_BACKUP_STATS_DIR="/var/log/restic-automatic-backup-scheduler"
+
+# (optional) Desktop notifications. See README and restic_backup.sh for details on how to set this up (empty/unset means disabled)
+export RESTIC_BACKUP_NOTIFICATION_FILE=

devices/fedora/config/etc/restic/backup_exclude.txt

@@ -0,0 +1,10 @@
+/.snapshots/
+/opt
+/root/.cache/
+/usr/share/**/*.html
+/usr/share/help/
+/usr/share/licenses/
+/usr/share/man/
+/usr/src/
+/var/cache/
+/var/log/

devices/fedora/config/etc/restic/default.env.sh

@@ -0,0 +1,45 @@
+# shellcheck shell=sh
+
+# This is the default profile. Fill it with your desired configuration.
+# Additionally, you can create and use more profiles by copying this file.
+
+# This file (and other .env.sh files) has two purposes:
+# - being sourced by systemd timers to setup the backup before running restic_backup.sh
+# - being sourced in a user's shell to work directly with restic commands e.g.
+#  $ source /etc/restic/default.env.sh
+#  $ restic snapshots
+#  Thus you don't have to provide all the arguments like
+#  $ restic --repo ... --password-file ...
+
+# shellcheck source=etc/restic/_global.env.sh
+. "/etc/restic/_global.env.sh"
+
+# Envvars below will override those in _global.env.sh if present.
+
+export RESTIC_REPOSITORY="sftp:sftpuser@teal.objectbrokers.com:/srv/restic/client-backup"   # *EDIT* fill with your repo name
+
+# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
+# To backup only your home directory, set "/home/your-user"
+export RESTIC_BACKUP_PATHS="/etc:/root:/home"  # *EDIT* fill conveniently with one or multiple paths
+
+
+# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
+# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
+#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"
+
+# A tag to identify backup snapshots.
+export RESTIC_BACKUP_TAG=systemd.timer
+
+# Retention policy - How many backups to keep.
+# See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
+export RESTIC_RETENTION_HOURS=1
+export RESTIC_RETENTION_DAYS=14
+export RESTIC_RETENTION_WEEKS=16
+export RESTIC_RETENTION_MONTHS=18
+export RESTIC_RETENTION_YEARS=3
+
+# Optional extra space-separated arguments to restic-backup.
+# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
+# Example: exclude all directories that have a .git/ directory inside it.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

devices/fedora/config/etc/restic/pw.txt

@@ -0,0 +1 @@
+fJbPEg78Dpx5UM

devices/merganser/config/C/restic/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Kevin Woley
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

devices/merganser/config/C/restic/README.md

@@ -0,0 +1,109 @@
+# restic-windows-backup
+Powershell scripts to run Restic backups on Windows.
+Simplifies the process of installation and running daily backups.
+
+# Features
+* **VSS (Volume Snapshot Service) support** - backup everything, don't worry about what files are open/in-use
+* **Removable, External Drives** - drives can be identified by their volume labels or serial numbers, making it easy to backup drives that occasionally aren't there or change drive letter.
+* **Easy Installation** - `install.ps1` script downloads Restic, initializes the restic repository, and setups up a Windows Task Scheduler task to run the backup daily
+* **Easy to update** - `update.ps1` script can be used to keep your scripts up to date with the latest release on GitHub
+* **Backup, Maintenance and Monitoring are Automated** - `backup.ps1` script handles
+  * Emailing the results of each execution, including log files when there are problems
+  * Runs routine maintenence (pruning and checking the repo for errors on a regular basis)
+  * And, of course backing up your files.
+  
+# Installation Instructions
+
+1. **Create your restic repository**
+   1. This is up to you to sort out where you want the data to go to. *Minio, B2, S3, etc.*. Refer to the restic documents about how to create your repository.
+1. **Install the scripts**
+   1. Create script directory: `C:\restic`
+   1. Download scripts using the `update.ps1` script.
+      1. Open PowerShell
+      1. Change your working directory to the installation directory
+         ```
+         cd c:\restic
+         ```
+      1. Run the `update.ps1` script:
+         ```
+         Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/kmwoley/restic-windows-backup/main/update.ps1" -UseBasicParsing).Content
+         ```
+      *Alternatively, you can download the scripts from this repository and and unzip them into `C:\restic`*
+   1. Launch PowerShell as Administrator
+   1. Change your working directory to `C:\restic`
+   1. If you haven't done so in the past, set your Powershell script [execution policy](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.1) to allow for scripts to run. For example, this is a good default:
+      ```
+      Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
+      ```
+   1. Depending on the policy you choose, may need to 'unblock' the execution of the scripts you download by running `Unblock-File *.ps1`
+1. Create `secrets.ps1` file. The secrets file contains location and passwords for your restic repository.
+   1. `secrets_sample.ps1` is an example of the `secrets.ps1` file. Copy or rename this file to `secrets.ps1` and edit.
+   1. Restic will pick up the repo destination from the environment variables you set in this file - see this doc for more information about configuring restic repos https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html
+   1. Email sending configuration is also contained with this file. The scripts are able to send email about the success/failure of each backup attempt.
+1. Create `config.ps1` file. The config file contains the settings that control how the script runs backups, forgets snapshots, and prunes the restic repository. It's important that you configure this file to meet your needs since it will be backing up and maintaining your repository.
+   1. `config_sample.ps1` contins an example configuration file. Copy or rename this file to `config.ps1` and edit to suit your needs.
+   1. Add your `$BackupSources` to `config.ps1`
+      1. By default, all of `C:\` will be backed up. You can add multiple root drives to be backed up. And you can define only specific folders you would like backed up.
+      1. External, removable disk drives (i.e. USB hard drives) can be identified by their Volume Label, Serial Number, or Device Name. For example, if you have an external device with the Volume Label "MY BOOK", you can define a backup source as `$BackupSources["MY BOOK"]=@()`. It is recommended to use the device serial number to identify external drives to backup, which you can find using the Powershell `get-disk` command. You may also want to set `$IgnoreMissingBackupSources=$true` to avoid seeing errors when the removable drive is not present.
+   1. Review all of the default settings in `config.ps1`. 
+      1. Most of the defaults are safe, but you should be sure restic is configured to meet your specifics needs.
+      1. **Warning** - if you're using a shared restic repository across multiple machines, pay close attention to the `$SnapshotRetentionPolicy` settings to be sure this script does not intentionally destroy backup data in your repository.
+1. Run `install.ps1` file
+   1. From the elevated (Run as Administrator) Powershell window, run `.\install.ps1`
+   1. This will initialize the repo, create your logfile directory, create a scheduled task in Windows Task Scheduler to run the task daily, and install Send-MailKitMessage module.
+1. Add files/paths not to backup to `local.exclude`
+   1. If you don't want to modify the included exclude file, you can add any files/paths you want to exclude from the backup to `local.exclude`
+1. Add `C:\restic\restic.exe` to the Windows Defender / Virus & Threat Detection Exclude list
+   1. Backups on Windows are really slow if you don't set the Antivirus to ignore restic.
+   1. Navigate from the Start menu to: *Virus & threat protection > Manage Settings > Exclusions (Add or remove exclusions) > Add an exclusion (Process) > Process Name: "C:\restic\restic.exe"*
+1. *(Recommended)* To a test backup triggered from Task Scheduler
+   1. It's recommended to open Windows Task Scheduler and trigger the task to run manually to test your first backup.
+      1. *Open Task Scheduler > Find "Restic Backup" > Right Click > Run*
+   1. The backup script will be executed as the SYSTEM user. Some of your files might not be accessible by this user. If you run into this, add the SYSTEM user to the files where you get "Access Denied" errors.
+      1. *Folder > Properties > Security > Advanced > Add ("SYSTEM" Principal/User) > Check "Replace all child object permission entries with inheritable permission entries from this object" > Apply > OK*
+1. *(Recommended)* Do a test restore
+   1. These scripts make it easy to work with Restic from the Powershell command line. If you run `. .\config.ps1; . .\secrets.ps1` you can then easily invoke restic commands like 
+      1. `& $ResticExe find -i "*filename*"`
+      1. `& $ResticExe restore ...`
+
+## Updating restic-windows-backup
+
+Use `update.ps1` to update the installed `restic-windows-backup` scripts to the latest release. 
+
+1. Open PowerShell (no need to be Administrator)
+1. Change directory to your installation directory (e.g. `c:\restic`)
+1. Run `update.ps1`
+
+### `update.ps1` Details
+
+Running `update.ps1` without any parameters will check for a new release from `kmwoley/restic-windows-backup`. If there is a newer release, the script will overwrite the local files in the script directory with the updated scripts. 
+
+* The script will not overwrite your local configuration files (i.e. `config.ps1` or `secrets.ps1`).
+* Any custom files created in the installation directory will not be deleted or modified (e.g. any custom action scripts, log files, etc.)
+* The script will warn before overwriting any files that have been changed since the last installation. 
+* When `update.ps1` is run the first time, it will prompt before overwriting (since it may not know the current version of the fiels installed).
+
+### `update.ps1` Options
+
+* `-Mode <release | branch> (Default: release)` - change if the script updates from the latest release or a branch of `kmwoley/restic-windows-backup`
+* `-Branch <branch> (Default: 'main')` - When in branch mode, this parameter controls which branch to install from. Defaults to the `main` branch.
+* `-InstallPath <directory>` - choose which directory to install the files into. Defaults to the directory that `update.ps1` is in.
+
+## Backup over SFTP
+
+You can use any restic repository type you like pretty easily. SFTP on Windows, however, can be particularly tricky given that these scripts execute as the SYSTEM user and need to have access to the .ssh keys. Here are some steps and tips to getting it working.
+
+1. Install as above. Your repository should be created properly. Tasked backups will fail for now though. This is because the `install.ps1` file is executed with your user, whereas the tasked backup will run as SYSTEM, which does not have any ssh config yet.
+1. Open Task Scheduler and make sure the restic task is not running anymore by checking the active tasks
+1. Edit `config.ps1` and turn off the internet connection test: `$InternetTestAttempts = 0` as the test does not recognize sftp addresses correctly
+1. Copy the .ssh directory content from `%USERPROFILE%\.ssh` to `%WINDIR%\System32\config\systemprofile\.ssh` (This is the ssh config the SYSTEM account uses)
+1. If you use a private key to access the sftp services it also needs to be in this directory. ssh checks the permissions though, so they need to be changed as well:
+	1. *Right click your key > Properties > Security > Advanced*
+		1. Change the owner to SYSTEM
+		1. *Disable inheritance* and keep the permissions
+		1. Remove all principals except SYSTEM and the Administrators group 
+
+This should get you up and running. If not, download [PsExec](https://docs.microsoft.com/en-us/sysinternals/downloads/psexec), start a powershell as admin user and run `.\PsExec.exe -s -i powershell.exe`. In this shell you will be the system user and you can try things out. See what `ssh user@server` says or try `cd C:\restic\; . .\config.ps1; . .\secrets.ps1; & $ResticExe check` (If you get lock errors, remember to check the Task Scheduler for any running restic instances in the background) 
+
+# Feedback?
+Feel free to open issues or create PRs!

devices/merganser/config/C/restic/backup.ps1

@@ -0,0 +1,711 @@
+#
+# Restic Windows Backup Script
+#
+
+# =========== start configuration =========== #
+
+# load restic configuration parameters (destination, passwords, etc.)
+$SecretsScript = Join-Path $PSScriptRoot "secrets.ps1"
+
+# load backup configuration variables
+$ConfigScript = Join-Path $PSScriptRoot "config.ps1"
+
+# =========== end configuration =========== #
+
+# make LASTEXITCODE global to enable error checking for Invoke-Expression commands
+$global:LASTEXITCODE=0
+
+# globals for state storage
+$Script:ResticStateRepositoryInitialized = $null
+$Script:ResticStateLastMaintenance = $null
+$Script:ResticStateLastDeepMaintenance = $null
+$Script:ResticStateMaintenanceCounter = $null
+$Script:ResticStateLastBackupSuccessful = $true
+$Script:ResticStateLastMaintenanceSuccessful = $true
+
+# Returns all drive letters which exactly match the serial number, drive label, or drive name of
+# the input parameter. Returns all drives if no input parameter is provided.
+# inspiration: https://stackoverflow.com/questions/31088930/combine-get-disk-info-and-logicaldisk-info-in-powershell
+function Get-Drives {
+    Param($ID)
+
+    foreach($disk in Get-CimInstance Win32_Diskdrive) {
+        $diskMetadata = Get-Disk | Where-Object { $_.Number -eq $disk.Index } | Select-Object -First 1
+        $partitions = Get-CimAssociatedInstance -ResultClassName Win32_DiskPartition -InputObject $disk
+
+        foreach($partition in $partitions) {
+
+            $drives = Get-CimAssociatedInstance -ResultClassName Win32_LogicalDisk -InputObject $partition
+
+            foreach($drive in $drives) {
+
+                $volume = Get-Volume |
+                          Where-Object { $_.DriveLetter -eq $drive.DeviceID.Trim(":") } |
+                          Select-Object -First 1
+
+                if(($diskMetadata.SerialNumber.trim() -eq $ID) -or
+                    ($disk.Caption -eq $ID) -or
+                    ($volume.FileSystemLabel  -eq $ID) -or
+                    ($null -eq $ID)) {
+
+                    [PSCustomObject] @{
+                        DriveLetter   = $drive.DeviceID
+                        Number        = $disk.Index
+                        Label         = $volume.FileSystemLabel
+                        Manufacturer  = $diskMetadata.Manufacturer
+                        Model         = $diskMetadata.Model
+                        SerialNumber  = $diskMetadata.SerialNumber.trim()
+                        Name          = $disk.Caption
+                        FileSystem    = $volume.FileSystem
+                        PartitionKind = $diskMetadata.PartitionStyle
+                        Drive         = $drive
+                        Partition     = $partition
+                        Disk          = $disk
+                    }
+                }
+            }
+        }
+    }
+}
+
+# test the path's storage media for VSS support
+#  returns $true if VSS is supported at the provided path
+function Test-VSSSupport {
+    Param($test_path)
+
+    $drive_letter = Split-Path $test_path -Qualifier
+    $volume = Get-WmiObject -Query "SELECT * FROM Win32_Volume WHERE DriveLetter = '$drive_letter'" 
+    $deviceID = ($volume.DeviceID -replace '.*(\{.*\}).*', '$1')
+    ### https://learn.microsoft.com/en-us/previous-versions/windows/desktop/vsswmi/win32-shadowvolumesupport
+    $supportedVolumes = Get-WmiObject -Query "SELECT * FROM Win32_ShadowVolumeSupport WHERE __PATH LIKE '%$deviceID%'"
+
+    return ($null -ne $supportedVolumes)
+}
+
+# restore backup state from disk
+function Get-BackupState {
+    if(Test-Path $Script:StateFile) {
+        Import-Clixml $Script:StateFile | ForEach-Object{ Set-Variable -Scope Script $_.Name $_.Value }
+    }
+}
+function Set-BackupState {
+    Get-Variable ResticState* | Export-Clixml $Script:StateFile
+}
+
+# unlock the repository if need be
+function Invoke-Unlock {
+    Param($SuccessLog, $ErrorLog)
+
+    $locks = Invoke-Expression "$Script:ResticExe list locks --no-lock -q 3>&1 2>> $ErrorLog"
+    if($LASTEXITCODE) {
+        "[[Unlock]] Warning: unable to list locks." | Tee-Object -Append $ErrorLog
+    }
+    if($locks.Length -gt 0) {
+        # unlock the repository (assumes this machine is the only one that will ever use it)
+        Invoke-Expression "$Script:ResticExe unlock 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+        if($LASTEXITCODE) {
+            "[[Unlock]] Error - unable to unlock repository." | Tee-Object -Append $ErrorLog
+        }
+        "[[Unlock]] Repository was locked. Unlocking." | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
+        Start-Sleep 120
+    }
+}
+
+# test if maintenance on the backup set is needed. Return $true if maintenance is needed
+function Test-Maintenance {
+    Param($SuccessLog, $ErrorLog)
+
+    # skip maintenance if disabled
+    if($SnapshotMaintenanceEnabled -eq $false) {
+        "[[Maintenance]] Skipping - maintenance disabled" | Out-File -Append $SuccessLog
+        return $false
+    }
+
+    # skip maintenance if it's been done recently
+    if(($null -ne $ResticStateLastMaintenance) -and ($null -ne $ResticStateMaintenanceCounter)) {
+        $Script:ResticStateMaintenanceCounter += 1
+        $delta = New-TimeSpan -Start $ResticStateLastMaintenance -End $(Get-Date)
+        if(($delta.Days -lt $SnapshotMaintenanceDays) -and ($ResticStateMaintenanceCounter -lt $SnapshotMaintenanceInterval)) {
+            "[[Maintenance]] Skipping - last maintenance $ResticStateLastMaintenance ($($delta.Days) days, $ResticStateMaintenanceCounter backups ago)" | Out-File -Append $SuccessLog
+            return $false
+        }
+        else {
+            "[[Maintenance]] Running - last maintenance $ResticStateLastMaintenance ($($delta.Days) days, $ResticStateMaintenanceCounter backups ago)" | Out-File -Append $SuccessLog
+            return $true
+        }
+    }
+    else {
+        "[[Maintenance]] Running - no past maintenance history known." | Out-File -Append $SuccessLog
+        return $true
+    }
+}
+
+# run maintenance on the backup set
+function Invoke-Maintenance {
+    Param($SuccessLog, $ErrorLog)
+
+    "[[Maintenance]] Start $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
+    $maintenance_success = $true
+    Start-Sleep 120
+
+    # forget snapshots based upon the retention policy
+    "[[Maintenance]] Start forgetting..." | Out-File -Append $SuccessLog
+    Invoke-Expression "$Script:ResticExe forget $SnapshotRetentionPolicy 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+    if($LASTEXITCODE) {
+        "[[Maintenance]] Forget operation completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
+        $maintenance_success = $false
+    }
+
+    # prune (remove) data from the backup step. Running this separate from `forget` because
+    #   `forget` only prunes when it detects removed snapshots upon invocation, not previously removed
+    "[[Maintenance]] Start pruning..." | Out-File -Append $SuccessLog
+    Invoke-Expression "$Script:ResticExe prune $SnapshotPrunePolicy 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+    if($LASTEXITCODE) {
+        "[[Maintenance]] Prune operation completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
+        $maintenance_success = $false
+    }
+
+    # check data to ensure consistency
+    "[[Maintenance]] Start checking..." | Out-File -Append $SuccessLog
+
+    # check to determine if we want to do a full data check or not
+    $data_check = @()
+    if($null -ne $ResticStateLastDeepMaintenance) {
+        $delta = New-TimeSpan -Start $ResticStateLastDeepMaintenance -End $(Get-Date)
+        if(($null -ne $SnapshotDeepMaintenanceDays) -and ($delta.Days -ge $SnapshotDeepMaintenanceDays)) {
+            "[[Maintenance]] Performing read data check. Last '--read-data' check ran $ResticStateLastDeepMaintenance ($($delta.Days) days ago)" | Out-File -Append $SuccessLog
+            $data_check = @("--read-data")
+            $Script:ResticStateLastDeepMaintenance = Get-Date
+        }
+        else {
+            "[[Maintenance]] Performing fast check. Last '--read-data' check ran $ResticStateLastDeepMaintenance ($($delta.Days) days ago)" | Out-File -Append $SuccessLog
+        }
+    }
+    else {
+        # set the date, but don't do a deep check if we've never done a full data read
+        $Script:ResticStateLastDeepMaintenance = Get-Date
+    }
+
+    Invoke-Expression "$Script:ResticExe check $data_check 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+    if($LASTEXITCODE) {
+        "[[Maintenance]] Data check completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
+        $maintenance_success = $false
+    }
+
+    # Invoke restic self-update to check for a newer version
+    # This is enabled by default unless configuration disables self-update
+    if ([String]::IsNullOrEmpty($SelfUpdateEnabled) -or ($SelfUpdateEnabled -eq $true)) {
+        # check for updated restic version
+        "[[Maintenance]] Checking for new version of restic..." | Out-File -Append $SuccessLog
+        Invoke-Expression "$Script:ResticExe self-update 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+        if($LASTEXITCODE) {
+            "[[Maintenance]] Self-update of restic.exe completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
+            $maintenance_success = $false
+        }
+    }
+
+    "[[Maintenance]] End $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
+
+    if($maintenance_success -eq $true) {
+        $Script:ResticStateLastMaintenance = Get-Date
+        $Script:ResticStateMaintenanceCounter = 0
+    }
+
+    return $maintenance_success
+}
+
+# Run restic backup
+function Invoke-Backup {
+    Param($SuccessLog, $ErrorLog)
+
+    "[[Backup]] Start $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
+    $return_value = $true
+    $starting_location = Get-Location
+    ForEach ($item in $BackupSources.GetEnumerator()) {
+
+        # Get the source drive letter or identifier and set as the root path
+        $root_path = $item.Key
+        $tag = $item.Key
+
+        # Test if root path is a valid path, if not assume it is an external drive identifier
+        if(-not (Test-Path $root_path)) {
+            # attempt to find a drive letter associated with the identifier provided
+            $drives = Get-Drives $root_path
+            if($drives.Count -gt 1) {
+                "[[Backup]] Fatal error - external drives with more than one partition are not currently supported." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+                $return_value = $false
+                continue
+            }
+            elseif ($drives.Count -eq 0) {
+                $ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
+                $warning_message = "[[Backup]] Warning - backup path $root_path not found."
+                if($ignore_error) {
+                    $warning_message | Out-File -Append $SuccessLog
+                }
+                else {
+                    $warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+                    $return_value = $false
+                }
+                continue
+            }
+
+            # there is exactly one drive
+            $root_path = Join-Path $drives[0].DriveLetter ""
+        }
+
+        # determine if VSS is supported by the drive
+        $vss_option = $null
+        if(Test-VSSSupport $root_path) {
+            $vss_option = "--use-fs-snapshot"
+        }
+        
+        "[[Backup]] Start $(Get-Date) [$tag]" | Out-File -Append $SuccessLog
+
+        # build the list of folders to backup
+        $folder_list = New-Object System.Collections.Generic.List[System.Object]
+        if ($item.Value.Count -eq 0) {
+            # backup everything in the root if no folders are provided
+            $folder_list.Add("`"$root_path`"")
+        }
+        else {
+            # Build the list of folders from settings
+            ForEach ($path in $item.Value) {
+                $p = '{0}' -f ((Join-Path $root_path $path) -replace "\\$")
+
+                if(Test-Path ($p -replace '"')) {
+                    # add the folder if it exists
+                    $folder_list.Add("`"$p`"")
+                }
+                else {
+                    # if the folder doesn't exist, log a warning/error
+                    $ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
+                    $warning_message = "[[Backup]] Warning - backup path $p not found."
+                    if($ignore_error) {
+                        $warning_message | Out-File -Append $SuccessLog
+                    }
+                    else {
+                        $warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+                        $return_value = $false
+                    }
+                }
+            }
+
+        }
+
+        if(-not $folder_list) {
+            # there are no folders to backup
+            $ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
+            $warning_message = "[[Backup]] Warning - no folders to back up!"
+            if($ignore_error) {
+                $warning_message | Out-File -Append $SuccessLog
+            }
+            else {
+                $warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+                $return_value = $false
+            }
+        }
+        else {
+            # Launch Restic
+            Invoke-Expression "$Script:ResticExe backup $folder_list $vss_option --tag $tag --exclude-file=$WindowsExcludeFile --exclude-file=$LocalExcludeFile $AdditionalBackupParameters 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
+            if($LASTEXITCODE) {
+                "[[Backup]] Completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
+                $return_value = $false
+            }
+        }
+
+        "[[Backup]] End $(Get-Date) [$tag]" | Out-File -Append $SuccessLog
+    }
+
+    Set-Location $starting_location
+    "[[Backup]] End $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
+
+    return $return_value
+}
+
+function Send-Email {
+    Param($SuccessLog, $ErrorLog, $Action)
+
+    Import-Module Send-MailKitMessage
+
+    # default the action string to "Backup"
+    if($null -eq $Action) {
+        $Action = "Backup"
+    }
+
+    # set email credentials if a username and passsword are provided in configuration
+    $credentials = @{}
+    if (-not [String]::IsNullOrEmpty($ResticEmailPassword) -and -not [String]::IsNullOrEmpty($ResticEmailUsername)) {
+        $password = ConvertTo-SecureString -String $ResticEmailPassword -AsPlainText -Force
+        $credentials = @{
+            "Credential" = [System.Management.Automation.PSCredential]::new($ResticEmailUsername, $password)
+        }
+    }
+
+    # Backwards compatibility for $ResticEmailConfig port definition:
+    # $ResticEmailConfig is obsolete and should be replaced with $ResticEmailPort
+    if ($null -ne $ResticEmailConfig -and $ResticEmailConfig.ContainsKey('Port')) {
+        if ($null -eq $ResticEmailPort) {
+            $ResticEmailPort = $ResticEmailConfig['Port']
+            '[[Email]] Warning - $ResticEmailConfig is deprecated. Define $ResticEmailPort in secrets.ps1 instead.' | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
+        }
+    }
+
+    # Backwards compatibility for $PSEmailServer rename to $ResticEmailServer
+    if (($null -ne $PSEmailServer) -and ($null -eq $ResticEmailServer)) {
+        $ResticEmailServer = $PSEmailServer
+        '[[Email]] Warning - $PSEmailServer is deprecated. Define $ResticEmailServer in secrets.ps1 instead.' | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
+    }
+
+    $status = "SUCCESS"
+    $past_failure = $false
+    $body = ""
+    if (($null -ne $SuccessLog) -and (Test-Path $SuccessLog) -and (Get-Item $SuccessLog).Length -gt 0) {
+        $body = $(Get-Content -Raw $SuccessLog)
+
+        # if previous run contained an error, send the success email confirming that the error has been resolved
+        if($Action -eq "Backup") {
+            $past_failure = -not $Script:ResticStateLastBackupSuccessful
+        }
+        else {
+            $past_failure = -not $Script:ResticStateLastMaintenanceSuccessful
+        }
+    }
+    else {
+        $body = "Critical Error! Restic $Action log is empty or missing. Check log file path."
+        $status = "ERROR"
+    }
+
+    $attachments = [System.Collections.Generic.List[string]]::new()
+    if (($null -ne $ErrorLog) -and (Test-Path $ErrorLog) -and (Get-Item $ErrorLog).Length -gt 0) {
+        $attachments.Add("$ErrorLog")
+        $status = "ERROR"
+    }
+
+    if((($status -eq "SUCCESS") -and ($SendEmailOnSuccess -ne $false)) -or ((($status -eq "ERROR") -or $past_failure) -and ($SendEmailOnError -ne $false))) {
+        $subject = "$env:COMPUTERNAME Restic $Action Report [$status]"
+
+        # create a temporary error log to log errors; can't write to the same file that Send-MailMessage is reading
+        $temp_error_log = $ErrorLog + "_temp"
+
+        $from = [MimeKit.MailboxAddress]$ResticEmailFrom;
+        $recipients = [MimeKit.InternetAddressList]::new();
+        $recipients.Add([MimeKit.InternetAddress]$ResticEmailTo);
+
+        Send-MailKitMessage -SMTPServer $ResticEmailServer -Port $ResticEmailPort -UseSecureConnectionIfAvailable @credentials -From $from -RecipientList $recipients -Subject $subject -TextBody $body -AttachmentList $attachments 3>&1 2>> $temp_error_log | Out-File -Append $SuccessLog
+
+        if(-not $?) {
+            "[[Email]] Sending email completed with errors" | Tee-Object -Append $temp_error_log | Tee-Object -Append $SuccessLog | Write-Host
+        }
+
+        # join error logs and remove the temporary
+        Get-Content $temp_error_log | Add-Content $ErrorLog
+        Remove-Item $temp_error_log
+    }
+}
+
+# check if on metered network,
+# returns $true the current connection is a metered network
+function Invoke-MeteredCheck {
+
+    $scriptBlock = {
+        # load NetworkInformation class from the Windows Runtime (WinRT) environment
+        [void][Windows.Networking.Connectivity.NetworkInformation, Windows, ContentType = WindowsRuntime]
+            
+        $cost = [Windows.Networking.Connectivity.NetworkInformation]::GetInternetConnectionProfile().GetConnectionCost()
+        return ($cost.ApproachingDataLimit -or $cost.OverDataLimit -or $cost.Roaming -or $cost.BackgroundDataUsageRestricted -or ($cost.NetworkCostType -ne 'Unrestricted'))
+    }
+    
+    # run this check in PowerShell 5.1
+    # this is a workaround for lack of WinRT support in PowerShell 7
+    $result = powershell.exe -Version 5.1 -Command "$scriptBlock"
+    return ($result -ieq "True")
+}
+
+# check network conditions, retrying a limited number of times until a connection is established
+# returns $true if the repository is accessible and the configuration allows us to use it
+function Invoke-ConnectivityCheck {
+    Param($SuccessLog, $ErrorLog)
+
+    $sleep_time = 30
+
+    if($InternetTestAttempts -le 0) {
+        "[[Internet]] Internet connectivity check disabled. Skipping." | Out-File -Append $SuccessLog
+        return $true
+    }
+
+    # skip the internet connectivity check for local repos
+    if(Test-Path $env:RESTIC_REPOSITORY) {
+        "[[Internet]] Local repository. Skipping internet connectivity check." | Out-File -Append $SuccessLog
+        return $true
+    }
+
+    $repository_host = ''
+
+    # use generic internet service for non-specific repo types (e.g. swift:, rclone:, etc. )
+    if(($env:RESTIC_REPOSITORY -match "^swift:") -or
+        ($env:RESTIC_REPOSITORY -match "^rclone:")) {
+        $repository_host = "cloudflare.com"
+    }
+    elseif($env:RESTIC_REPOSITORY -match "^b2:") {
+        $repository_host = "api.backblazeb2.com"
+    }
+    elseif($env:RESTIC_REPOSITORY -match "^azure:") {
+        $repository_host = "azure.microsoft.com"
+    }
+    elseif($env:RESTIC_REPOSITORY -match "^gs:") {
+        $repository_host = "storage.googleapis.com"
+    }
+    else {
+        # parse connection string for hostname
+        # Uri parser doesn't handle leading connection type info (s3:, sftp:, rest:)
+        $connection_string = $env:RESTIC_REPOSITORY -replace "^s3:" -replace "^sftp:" -replace "^rest:"
+        if(-not ($connection_string -match "://")) {
+            # Uri parser expects to have a protocol. Add 'https://' to make it parse correctly.
+            $connection_string = "https://" + $connection_string
+        }
+        $repository_host = ([System.Uri]$connection_string).DnsSafeHost
+    }
+
+    if([string]::IsNullOrEmpty($repository_host)) {
+        "[[Internet]] Repository string could not be parsed." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+        return $false
+    }
+
+    # test for internet connectivity
+    $connections = 0
+    $sleep_count = $InternetTestAttempts
+    $restricted_by_metered_network = $false
+    while($true) {
+        $connections = Get-NetRoute | Where-Object DestinationPrefix -eq '0.0.0.0/0' | Get-NetIPInterface | Where-Object ConnectionState -eq 'Connected' | Measure-Object | ForEach-Object{$_.Count}
+        if($sleep_count -le 0) {
+            if($restricted_by_metered_network) {
+                "[[Internet]] Connection to repository ($repository_host) is available but blocked by metered network." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+            }
+            else {
+                "[[Internet]] Connection to repository ($repository_host) could not be established." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
+            }
+            return $false
+        }
+        if(($null -eq $connections) -or ($connections -eq 0)) {
+            "[[Internet]] Waiting $sleep_time seconds for internet connectivity... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog
+            Start-Sleep $sleep_time
+        }
+        elseif(!(Test-Connection -ComputerName $repository_host -Quiet)) {
+            "[[Internet]] Waiting $sleep_time seconds for connection to repository ($repository_host)... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog
+            Start-Sleep $sleep_time
+        }
+        elseif((-not ([String]::IsNullOrEmpty($BackupOnMeteredNetwork) -or $BackupOnMeteredNetwork)) -and (Invoke-MeteredCheck)) {
+            "[[Internet]] Waiting $sleep_time seconds for an unmetered network connection... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog        
+            $restricted_by_metered_network = $true
+            Start-Sleep $sleep_time
+        }
+        else {
+            return $true
+        }
+        $sleep_count--
+    }
+}
+
+# check previous logs
+function Invoke-HistoryCheck {
+    Param($SuccessLog, $ErrorLog, $Action)
+
+    # default the action to "Backup"
+    if($null -eq $Action) {
+        $Action = "Backup"
+    }
+
+    $filter = "*$Action.err.txt".ToLower()
+    $logs = Get-ChildItem $Script:LogPath -Filter $filter | ForEach-Object{$_.Length -gt 0}
+    $logs_with_success = ($logs | Where-Object {($_ -eq $false)}).Count
+    if($logs.Count -gt 0) {
+        "[[History]] $Action success rate: $logs_with_success / $($logs.Count) ($(($logs_with_success / $logs.Count).tostring("P")))" | Tee-Object -Append $SuccessLog  | Write-Host
+    }
+}
+
+# main function
+function Invoke-Main {
+
+    # check for elevation, required for creation of shadow copy (VSS)
+    if (-not (New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator))
+    {
+        Write-Error "[[Backup]] Elevation required (run as administrator). Exiting."
+        exit 1
+    }
+
+    # initialize secrets
+    . $SecretsScript
+
+    # initialize config
+    . $ConfigScript
+
+    # apply global configuration
+    $Script:ResticExe = Join-Path $InstallPath $ExeName
+    if(-not [String]::IsNullOrEmpty($GlobalParameters)) {
+        $Script:ResticExe = "$Script:ResticExe $GlobalParameters"
+    }
+    $Script:StateFile = Join-Path $InstallPath "state.xml"
+    $Script:LogPath = Join-Path $InstallPath "logs"
+
+    Get-BackupState
+
+    if(!(Test-Path $Script:LogPath)) {
+        Write-Error "[[Backup]] Log file directory $Script:LogPath does not exist. Exiting."
+        Send-Email
+        exit 1
+    }
+
+    # custom start action
+    if($null -ne $CustomActionStart) {
+        Invoke-Expression $CustomActionStart
+    }
+
+    $error_count = 0
+    $backup_success = $false
+    $maintenance_success = $false
+    $maintenance_needed = $false
+
+    $attempt_count = $GlobalRetryAttempts
+    while ($attempt_count -gt 0) {
+        # setup logfiles
+        $timestamp = Get-Date -Format FileDateTime
+        $success_log = Join-Path $Script:LogPath ($timestamp + ".backup.log.txt")
+        $error_log = Join-Path $Script:LogPath ($timestamp + ".backup.err.txt")
+
+        $repository_available = Invoke-ConnectivityCheck $success_log $error_log
+        if($repository_available -eq $true) {
+            Invoke-Unlock $success_log $error_log
+            $backup_success = Invoke-Backup $success_log $error_log
+
+            # NOTE: a previously locked repository will cause errors in the log; but backup would be 'successful'
+            # Removing this overly-aggressive test for backup success and relying upon Invoke-Backup to report on success/failure
+            # $backup_success = ($backup_success -eq $true) -and (!(Test-Path $error_log) -or ((Get-Item $error_log).Length -eq 0))
+            $total_attempts = $GlobalRetryAttempts - $attempt_count + 1
+            if($backup_success -eq $true) {
+                # successful backup
+                "[[Backup]] Succeeded after $total_attempts attempt(s)" | Tee-Object -Append $success_log | Write-Host
+
+                # test to see if maintenance is needed if the backup was successful
+                $maintenance_needed = Test-Maintenance $success_log $error_log
+            }
+            else {
+                "[[Backup]] Ran with errors on attempt $total_attempts" | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
+                $error_count++
+            }
+        }
+        else {
+            "[[Backup]] Failed - cannot access repository." | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
+            $error_count++
+        }
+
+        $attempt_count--
+
+        # update logs prior to sending email
+        if($backup_success -eq $false) {
+            if($attempt_count -gt 0) {
+                "[[Backup]] Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log  | Write-Host
+            }
+            else {
+                "[[Backup]] Retry limit has been reached. No more attempts to backup will be made." | Tee-Object -Append $success_log | Write-Host
+            }
+        }
+
+        Invoke-HistoryCheck $success_log $error_log "Backup"
+        Send-Email $success_log $error_log "Backup"
+
+        # update the state of the last backup success or failure
+        $Script:ResticStateLastBackupSuccessful = $backup_success
+
+        # Save state to file
+        Set-BackupState
+
+        # loop exit/wait condition
+        if(($backup_success -eq $false) -and ($attempt_count -gt 0)) {
+            Start-Sleep (15*60)
+        }
+        else {
+            break
+        }
+    }
+
+    # only run maintenance if the backup was successful and maintenance is needed
+    $attempt_count = $GlobalRetryAttempts
+    while (($maintenance_needed -eq $true) -and ($attempt_count -gt 0)) {
+        # setup logfiles
+        $timestamp = Get-Date -Format FileDateTime
+        $success_log = Join-Path $Script:LogPath ($timestamp + ".maintenance.log.txt")
+        $error_log = Join-Path $Script:LogPath ($timestamp + ".maintenance.err.txt")
+
+        $repository_available = Invoke-ConnectivityCheck $success_log $error_log
+        if($repository_available -eq $true) {
+            $maintenance_success = Invoke-Maintenance $success_log $error_log
+
+            # $maintenance_success = ($maintenance_success -eq $true) -and (!(Test-Path $error_log) -or ((Get-Item $error_log).Length -eq 0))
+            $total_attempts = $GlobalRetryAttempts - $attempt_count + 1
+            if($maintenance_success -eq $true) {
+                "[[Maintenance]] Succeeded after $total_attempts attempt(s)" | Tee-Object -Append $success_log | Write-Host
+            }
+            else {
+                "[[Maintenance]] Ran with errors on attempt $total_attempts" | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
+                $error_count++
+            }
+        }
+        else {
+            "[[Maintenance]] Failed - cannot access repository." | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
+            $error_count++
+        }
+
+        $attempt_count--
+
+        # update logs prior to sending email
+        if($maintenance_success -eq $false) {
+            if($attempt_count -gt 0) {
+                "[[Maintenance]] Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log | Write-Host
+            }
+            else {
+                "[[Maintenance]] Retry limit has been reached. No more attempts to run maintenance will be made." | Tee-Object -Append $success_log | Write-Host
+            }
+        }
+
+        Invoke-HistoryCheck $success_log $error_log "Maintenance"
+        Send-Email $success_log $error_log "Maintenance"
+
+        # update the state of the last maintenance success or failure
+        $Script:ResticStateLastMaintenanceSuccessful = $maintenance_success
+
+        # Save state to file
+        Set-BackupState
+
+        # loop exit/wait condition
+        if(($maintenance_success -eq $false) -and ($attempt_count -gt 0)) {
+            Start-Sleep (15*60)
+        }
+        else {
+            break
+        }
+    }
+
+    # custom end actions
+    if((-not $backup_success) -or ($maintenance_needed -and -not $maintenance_success)) {
+        # call the custom error action if backup failed and/or maintenance was needed and failed
+        if($null -ne $CustomActionEndError) {
+            Invoke-Expression $CustomActionEndError
+        }
+    }
+    else {
+        # call custom success action if backup & maintenance were successful
+        if($null -ne $CustomActionEndSuccess) {
+            Invoke-Expression $CustomActionEndSuccess
+        }        
+    }
+
+    # Save state to file
+    Set-BackupState
+
+    # cleanup older log files
+    Get-ChildItem $Script:LogPath | Where-Object {$_.CreationTime -lt $(Get-Date).AddDays(-$LogRetentionDays)} | Remove-Item
+
+    exit $error_count
+}
+
+Invoke-Main

devices/merganser/config/C/restic/config.ps1

@@ -0,0 +1,68 @@
+# Sample configuration file
+# Update this file to control how the restic backup, forget, and purge operations are run
+# Rename to `config.ps1`
+
+# general configuration
+$InstallPath = "C:\restic"
+$ExeName = "restic.exe"
+$GlobalParameters = @()
+$LogRetentionDays = 30
+$BackupOnMeteredNetwork = $true
+$InternetTestAttempts = 10
+$GlobalRetryAttempts = 4
+
+# email configuration
+$SendEmailOnSuccess = $false
+$SendEmailOnError = $true
+
+# backup configuration
+$WindowsExcludeFile = Join-Path $InstallPath "windows.exclude"
+$LocalExcludeFile = Join-Path $InstallPath "local.exclude"
+$IgnoreMissingBackupSources = $false
+$AdditionalBackupParameters = @("--exclude-if-present", ".nobackup", "--no-scan")
+
+# Paths to backup
+$BackupSources = @{}
+# Paths to backup
+$BackupSources = @{}
+$BackupSources["C:\"] = @(
+    'Users\chris'
+)
+$BackupSources["D:\"] = @(
+    'workspace',
+    'Mikrotik'
+)
+#$BackupSources["DRIVE_LABEL_NAME_OR_SERIAL_NUMBER"] = @(
+#    "Example\FolderName"
+#)
+
+# maintenance configuration
+$SnapshotMaintenanceEnabled = $true
+$SnapshotRetentionPolicy = @("--host", $env:COMPUTERNAME, "--group-by", "host,tags", "--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
+# $SnapshotRetentionPolicy = @("--group-by", "host,tags", "--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
+$SnapshotPrunePolicy = @("--max-unused", "1%")
+$SnapshotMaintenanceInterval = 7
+$SnapshotMaintenanceDays = 30
+$SnapshotDeepMaintenanceDays = 90
+
+# restic.exe self update configuration
+$SelfUpdateEnabled = $true
+
+# (optional) custom actions 
+# Define commands to pass to Invoke-Expression at script start and script end
+# note: Errors will only be reported if the script does not eventually succeed. Errors 
+#       from unsuccessful attempts to backup or maintain the repository will not result
+#       in the custom error action being called unless all attempts to backup or maintain failed. 
+$CustomActionStart = $null
+$CustomActionEndError = $null
+$CustomActionEndSuccess = $null
+
+# Example: Calling a healthcheck remote service
+# $healthCheckURL = "https://healthcheckservice.com/etc/etc"
+# $CustomActionStart = "Invoke-RestMethod $healthCheckURL/start"
+# $CustomActionEndError = "Invoke-RestMethod $healthCheckURL/fail"
+# $CustomActionEndSuccess = "Invoke-RestMethod $healthCheckURL"
+
+# Example: Invoking a script
+# $successScript = Join-Path $InstallPath "mySuccessScript.ps1"
+# $CustomActionEndSuccess = "& $successScript"

devices/merganser/config/C/restic/install.ps1

@@ -0,0 +1,108 @@
+#
+# Restic Windows Backup - Installation Script
+#
+
+# =========== start configuration =========== #
+
+# load restic configuration parmeters (destination, passwords, etc.)
+$SecretsScript = Join-Path $PSScriptRoot "secrets.ps1"
+
+# load backup configuration variables
+$ConfigScript = Join-Path $PSScriptRoot "config.ps1"
+
+# initialize secrets
+. $SecretsScript
+
+# initialize config
+. $ConfigScript
+
+# apply global configuration
+$ResticExe = Join-Path $InstallPath $ExeName
+$LogPath = Join-Path $InstallPath "logs"
+
+# make LASTEXITCODE global to enable error checking for Invoke-Expression commands
+$global:LASTEXITCODE=0
+
+# =========== end configuration =========== #
+
+# download restic
+if(-not (Test-Path $ResticExe)) {
+    $url = $null
+    if([Environment]::Is64BitOperatingSystem){
+        $url = "https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_windows_amd64.zip"
+    }
+    else {
+        $url = "https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_windows_386.zip"
+    }
+    try {
+        $output = Join-Path $InstallPath "restic.zip"
+        Invoke-WebRequest -Uri $url -OutFile $output
+        Expand-Archive -LiteralPath $output $InstallPath
+        Remove-Item $output
+        Get-ChildItem *.exe | Rename-Item -NewName $ExeName
+    }
+    catch {
+        Write-Error "[[Install]] restic.exe download failed. Check errors and resolve: $_"
+        exit 1
+    }
+}
+
+# Apply global paramters to $ResticExe, after the $ResticExe has been downloaded/confirmed to exist
+if(-not [String]::IsNullOrEmpty($GlobalParameters)) {
+    $ResticExe = "$ResticExe $GlobalParameters"
+}
+
+# Invoke restic self-update to check for a newer version
+# This is enabled by default unless configuration disables self-update
+if ([String]::IsNullOrEmpty($SelfUpdateEnabled) -or ($SelfUpdateEnabled -eq $true)) {
+    Invoke-Expression "$ResticExe self-update"
+    if($LASTEXITCODE) {
+        Write-Warning "[[Update]] Restic self-update failed. Check errors and resolve."
+    }
+}
+
+# Create log directory if it doesn't exit
+if(-not (Test-Path $LogPath)) {
+    New-Item -ItemType Directory -Force -Path $LogPath | Out-Null
+    Write-Output "[[Init]] Created log directory: $LogPath"
+}
+
+# Create the local exclude file
+if(-not (Test-Path $LocalExcludeFile)) {
+    New-Item -Type File -Path $LocalExcludeFile | Out-Null
+}
+
+# Initialize the restic repository
+Invoke-Expression "$ResticExe --verbose init"
+if($LASTEXITCODE) {
+    Write-Warning "[[Init]] Repository initialization failed. Check errors and resolve."
+}
+else {
+    Write-Output "[[Init]] Repository successfully initialized."
+}
+
+# Scheduled Windows Task Scheduler to run the backup
+$backup_task_name = "Restic Backup"
+$backup_task = Get-ScheduledTask $backup_task_name -ErrorAction SilentlyContinue
+if($null -eq $backup_task) {
+    try {
+        $task_action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-ExecutionPolicy Bypass -NonInteractive -NoLogo -NoProfile -Command ".\backup.ps1; exit $LASTEXITCODE"' -WorkingDirectory $InstallPath
+        $task_user = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount -RunLevel Highest
+        $task_settings = New-ScheduledTaskSettingsSet -RestartCount 4 -RestartInterval (New-TimeSpan -Minutes 15) -ExecutionTimeLimit (New-TimeSpan -Days 3) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -DontStopOnIdleEnd -MultipleInstances IgnoreNew -IdleDuration 0 -IdleWaitTimeout 0 -StartWhenAvailable -RestartOnIdle
+        $task_trigger = New-ScheduledTaskTrigger -Daily -At 4:00am
+        Register-ScheduledTask $backup_task_name -Action $task_action -Principal $task_user -Settings $task_settings -Trigger $task_trigger | Out-Null
+        Write-Output "[[Scheduler]] Backup task scheduled."
+    }
+    catch {
+        Write-Error "[[Scheduler]] Setting up backup task schedule failed: $_"
+    }
+}
+else {
+    Write-Warning "[[Scheduler]] Backup task not scheduled: there is already a task with the name '$backup_task_name'."
+}
+
+# Install NuGet and Send-MailKitMessage module (by force)
+if ($PSVersionTable.PSVersion.Major -eq 5) {
+    Install-PackageProvider -Name NuGet -Force
+}
+Install-Module Send-MailKitMessage -Repository PSGallery -Scope AllUsers -Force

devices/merganser/config/C/restic/secrets.ps1

@@ -0,0 +1,17 @@
+# Template file for backup destination configuration and email passwords.
+# Update this file to point to your restic repository and email service.
+# Rename to `secrets.ps1`
+
+# restic backup repository configuration
+$Env:AWS_ACCESS_KEY_ID='<KEY>'
+$Env:AWS_SECRET_ACCESS_KEY='<KEY>'
+$Env:RESTIC_REPOSITORY='sftp:sftpuser@teal:/srv/restic/merganser-backup'
+$Env:RESTIC_PASSWORD='<LambDuck1977>'
+
+# email configuration
+$ResticEmailServer='smtp.gmail.com'
+$ResticEmailPort='465'
+$ResticEmailTo='jones.chrisk@gmail.com'
+$ResticEmailFrom='jones.chrisk@gmail.com'
+$ResticEmailUsername='jones.chrisk@gmail.com'
+$ResticEmailPassword='nvooxavrzeskrzlg'

devices/merganser/config/C/restic/update.ps1

@@ -0,0 +1,337 @@
+<#
+.SYNOPSIS
+    Updates the local installed restic backup scripts from GitHub,
+    either using the latest tagged release or by targeting a specific branch.
+
+.DESCRIPTION
+    This script supports two modes:
+
+    1. **Release mode (default):**
+       - Fetches the latest release info via GitHub’s API.
+       - Compares the release tag (after normalization) against a locally stored version (in state.xml).
+       - If the GitHub release is newer, downloads the release zip, extracts it, copies the files
+         over the local installation.
+
+    2. **Branch mode:**
+       - Targets a specific branch (default "main") by retrieving branch information from GitHub.
+       - Compares the latest commit SHA on that branch against a locally stored SHA (in state.xml).
+       - If the remote commit SHA differs, downloads the branch zip archive, extracts it,
+         copies the files over the local installation.
+
+.NOTES
+    Example 1 - update scripts to the latest tagged release
+    .\update.ps1
+
+    Example 2 - update scripts from a branch
+    .\update.ps1 -Mode branch -BranchName 'release_1.8'
+
+    Example 3 - download a new copy of the update scripts and run it
+    1. Change your directory to your installation directory (e.g. `cd c:\restic`)
+    2. Invoke-WebRequest "https://raw.githubusercontent.com/kmwoley/restic-windows-backup/main/update.ps1" -OutFile update.ps1
+    3. .\update.ps1
+#>
+
+[CmdletBinding()]
+param(
+    [Parameter(Mandatory = $false)]
+    [ValidateSet("release", "branch")]
+    [string]$Mode = "release",
+
+    [Parameter(Mandatory = $false)]
+    [string]$BranchName = "main",
+
+    [Parameter(Mandatory = $false)]
+    [string]$InstallPath = $null
+)
+
+# ====================================
+# Configuration and Setup
+# ====================================
+
+# GitHub repository details
+$repoOwner = "kmwoley"
+$repoName  = "restic-windows-backup"
+
+# User-Agent header (GitHub requires this)
+$headers = @{ "User-Agent" = "PowerShell" }
+
+# default the installation directory to the location of the running script
+if([string]::IsNullOrEmpty($InstallPath)) {
+    # default to the script's location, if running as a script
+    $InstallPath = $PSScriptRoot
+    if([string]::IsNullOrEmpty($InstallPath)) {
+        # default to the current working directory, if not running as a script
+        $InstallPath = Get-Location
+    }
+}
+
+# ====================================
+# Functions for state management
+# ====================================
+function Get-State {
+    if(Test-Path $Script:StateFile) {
+        Import-Clixml $Script:StateFile | ForEach-Object{ Set-Variable -Scope Script $_.Name $_.Value }
+    }
+}
+function Set-State {
+    Get-Variable ResticState* | Export-Clixml $Script:StateFile
+}
+
+# ===========================================
+# Functions for file management and download
+# ===========================================
+function Get-ModifiedFiles {
+    param(
+        [Parameter(Mandatory = $true)]
+        [string]$Source,
+
+        [Parameter(Mandatory = $true)]
+        [string]$Destination,
+
+        [Parameter(Mandatory = $true)]
+        [string]$DateTime
+    )
+
+    $modifiedFiles = New-Object System.Collections.Generic.List[System.Object]
+
+    if(-not (Test-Path $Source)) {
+        Write-Error "Source does not exist ($Source)"
+        exit 1
+    }
+    if(-not (Test-Path $Destination)) {
+        Write-Error "Destination does not exist ($Destination)"
+        exit 1
+    }
+
+    $sourceFiles = Get-ChildItem $Source
+
+    ForEach ($sourceFile in $sourceFiles) {
+        # find if there's a corrosponding file in the destination
+        $destFileName = Join-Path $Destination $sourceFile.Name
+        if(Test-Path $destFileName) {
+            $destFile = Get-ChildItem $destFileName
+            if($destFile.LastWriteTime -gt $DateTime) {
+                # destination file has been modified after $DateTime
+                $modifiedFiles.Add($destFile.FullName)
+            }
+        }
+    }
+    return $modifiedFiles
+}
+
+function Update-InstalledScripts {
+    param(
+        [Parameter(Mandatory=$true)][string]$ZipUrl,
+        [Parameter(Mandatory=$true)][string]$DestinationFolder
+    )
+
+    $timestamp = Get-Date -Format FileDateTime
+    $tempExtractDir = Join-Path $env:TEMP ("restic-windows-backup." + $timestamp)
+    $tempZipPath = Join-Path $env:TEMP ("restic-windows-backup." + $timestamp + ".zip")
+
+    # test temp location, fail if in use
+    if (Test-Path $tempExtractDir) {
+        Write-Error "Temporary directory already exists: $tempExtractDir"
+        exit 1
+    }
+    if (Test-Path $tempZipPath) {
+        Write-Error "Temporary directory already exists: $tempZipPath"
+        exit 1
+    }
+
+    # Create a temporary folder for extraction
+    New-Item -ItemType Directory -Path $tempExtractDir | Out-Null
+
+    Write-Host "Downloading from: $ZipUrl"
+    try {
+        Invoke-WebRequest -Uri $ZipUrl -OutFile $tempZipPath -Headers $headers
+    } catch {
+        Write-Error "Failed to download the file: $_"
+        exit 1
+    }
+
+    try {
+        Expand-Archive -LiteralPath $tempZipPath $tempExtractDir
+    } catch {
+        Write-Error "Error extracting zip file: $_"
+        exit 1 
+    }
+
+    # Determine the actual folder containing the repository files.
+    $extractedContent = Get-ChildItem -Path $tempExtractDir | Where-Object { $_.PSIsContainer }
+    if ($extractedContent.Count -eq 1) {
+        $extractedFolder = $extractedContent[0].FullName
+    } else {
+        $extractedFolder = $tempExtractDir
+    }
+
+    # Check to make sure not to overwrite modified files
+    $installedDate = $Script:ResticStateInstalledDate
+    if([string]::IsNullOrEmpty($installedDate)) {
+        # unkown install date; setting the date
+        $installedDate = [datetime]::MinValue
+    }
+    
+    $modifiedFiles = Get-ModifiedFiles -Source $extractedFolder -Destination $DestinationFolder -DateTime $installedDate
+    if($modifiedFiles) {
+        if([string]::IsNullOrEmpty($Script:ResticStateInstalledDate)) {
+            Write-Host "WARNING: The following files already exist in the target directory"
+        }        
+        else {
+            Write-Host "WARNING: The following files have been modified since they were installed on $installedDate"
+        }
+        ForEach ($fileName in $modifiedFiles) {
+            Write-Host " - " $fileName
+        }
+        
+        # TODO: add a "-Force" parameter to skip this check/question
+        Write-Host "Continuing will overwrite these files."
+        Write-host "Do you want to continue?"
+        $userInput = Read-Host "[Y] Yes  [N] No  (default is ""Y"")"
+        if ($userInput -ieq 'n') {
+            Write-Host "Operation cancelled."
+            exit 0
+        }
+    }
+
+    Write-Host "Updating files in installation directory ($DestinationFolder)..."
+    try {
+        # Recursively copy all content from the extracted folder to the local directory.
+        Copy-Item -Path (Join-Path $extractedFolder "*") -Destination $DestinationFolder -Recurse -Force
+    } catch {
+        Write-Error "Error copying files: $_"
+        exit 1
+    }
+
+    # Clean up temporary files
+    Remove-Item $tempZipPath -Force
+    Remove-Item $tempExtractDir -Recurse -Force
+}
+
+# ====================================
+# Main
+# ====================================
+
+# load restic state
+$Script:ResticStateInstalledVersion = $null
+$Script:ResticStateInstalledBranchSHA = $null
+$Script:ResticStateInstalledDate = $null
+$Script:StateFile = Join-Path $InstallPath "state.xml"
+Get-State
+
+# ====================================
+# Release mode
+# ====================================
+
+if ($Mode -eq "release") {
+
+    # Read the version of the scripts installed
+    $localVersion = $Script:ResticStateInstalledVersion
+    if ([string]::IsNullOrEmpty($localVersion)) {
+        # No version information stored locally
+        $localVersion = "0.0.0"
+    }
+  
+    # Get the Latest Release Info from GitHub
+    $releaseApiUrl = "https://api.github.com/repos/$repoOwner/$repoName/releases/latest"
+    try {
+        Write-Host "Checking GitHub for latest release of '$repoOwner/$repoName'..."
+        $release = Invoke-RestMethod -Uri $releaseApiUrl -Headers $headers
+    } catch {
+        Write-Error "Error fetching release information from GitHub: $_"
+        exit 1
+    }
+
+    $latestTagRaw = $release.tag_name
+    $latestTag    = $latestTagRaw.Trim()
+
+    # Normalize versions (remove leading "v" if present)
+    function Get-NormalizedVersion($versionString) {
+        if ($versionString.StartsWith("v", [System.StringComparison]::InvariantCultureIgnoreCase)) {
+            return $versionString.Substring(1)
+        }
+        return $versionString
+    }
+    $normalizedLocalVersion  = Get-NormalizedVersion $localVersion
+    $normalizedLatestVersion = Get-NormalizedVersion $latestTag
+
+    try {
+        $localVersionObj  = [Version]$normalizedLocalVersion
+        $latestVersionObj = [Version]$normalizedLatestVersion
+    } catch {
+        Write-Error "Error parsing version strings. Local: $normalizedLocalVersion, Latest: $normalizedLatestVersion. $_"
+        exit 1
+    }
+
+    if ($latestVersionObj -le $localVersionObj) {
+        Write-Host "Installed version ($localVersionObj) is up-to-date. No update needed."
+        exit 0
+    } else {
+        Write-Host "Newer release available: $latestVersionObj (installed: $localVersionObj). Proceeding with update..."
+    }
+
+    # get the zip URL from the release info
+    $zipUrl = $release.zipball_url
+
+    # Download and update the installed scripts
+    Update-InstalledScripts -ZipUrl $zipUrl -DestinationFolder $InstallPath
+
+    # Store the installed version number and time installed
+    $Script:ResticStateInstalledVersion = $normalizedLatestVersion
+    $Script:ResticStateInstalledDate = Get-Date
+    $Script:ResticStateInstalledBranchSHA = $null
+    Set-State
+
+    Write-Host "Update successful. Installed version is now $normalizedLatestVersion."
+}
+# ====================================
+# Branch mode
+# ====================================
+elseif ($Mode -eq "branch") {
+
+    # Read the SHA of the branch source installed
+    $localCommitSHA = $Script:ResticStateInstalledBranchSHA
+    if ([string]::IsNullOrEmpty($localCommitSHA)) {
+        # Write-Host "No branch information stored locally."
+        $localCommitSHA = "unknown"
+    }
+
+    # Retrieve branch information from GitHub
+    $branchApiUrl = "https://api.github.com/repos/$repoOwner/$repoName/branches/$BranchName"
+    try {
+        Write-Host "Checking GitHub for latest commit of '$repoOwner/$repoName' on branch '$BranchName'..."
+        $branchInfo = Invoke-RestMethod -Uri $branchApiUrl -Headers $headers
+    } catch {
+        Write-Error "Error fetching branch information from GitHub: $_"
+        exit 1
+    }
+
+    $latestCommitSHA = $branchInfo.commit.sha
+
+    if ($localCommitSHA -eq $latestCommitSHA) {
+        Write-Host "Installed commit ($latestCommitSHA) is up-to-date. No update needed."
+        exit 0
+    } else {
+        Write-Host "Latest commit: $latestCommitSHA (installed: $localCommitSHA). Proceeding with update..."
+    }
+
+    # Construct the zip URL for the branch.
+    # GitHub provides branch archives at:
+    # https://github.com/{owner}/{repo}/archive/refs/heads/{branch}.zip
+    $zipUrl = "https://github.com/$repoOwner/$repoName/archive/refs/heads/$BranchName.zip"
+
+    # Download and update the installed scripts
+    Update-InstalledScripts -ZipUrl $zipUrl -DestinationFolder $InstallPath
+
+    # Store the installed branch commit SHA and time installed
+    $Script:ResticStateInstalledVersion = $null
+    $Script:ResticStateInstalledDate = Get-Date
+    $Script:ResticStateInstalledBranchSHA = $latestCommitSHA
+    Set-State
+
+    Write-Host "Update successful. Local branch is now at commit $latestCommitSHA."
+}
+else {
+    Write-Error "Unsupported mode."
+    exit 1
+}

devices/merganser/config/C/restic/windows.exclude

@@ -0,0 +1,65 @@
+# default excludes
+# examples https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Utility/FilterGroups.cs
+C:\hiberfil.sys
+C:\pagefile.sys
+C:\swapfile.sys
+C:\$Recycle.Bin
+C:\autoexec.bat
+C:\Config.Msi
+C:\Documents and Settings
+C:\Recycled
+C:\Recycler
+C:\$$Recycle.Bin
+C:\System Volume Information
+C:\Recovery
+C:\Program Files
+C:\Program Files (x86)
+C:\ProgramData
+C:\PerfLogs
+C:\Windows
+C:\Windows.old
+C:\$$WINDOWS.~BT
+C:\$$WinREAgent
+Microsoft\Windows\Recent
+Microsoft\**\RecoveryStore*
+Microsoft\**\Windows\*.edb
+Microsoft\**\Windows\*.log
+Microsoft\**\Windows\Cookies*
+MSOCache
+NTUSER*
+ntuser*
+UsrClass.dat
+
+# cloud services
+Dropbox
+AppData\Local\Google\Drive
+Google Drive\.tmp.drivedownload
+C:\OneDriveTemp
+Users\**\Nextcloud
+
+# browsers
+Google\Chrome
+
+# AppData
+AppData\Local\Microsoft
+AppData\Local\Duplicati
+AppData\Local\D3DSCache
+AppData\Local\ConnectedDevicesPlatform
+AppData\Local\Packages
+AppData\Roaming\Signal
+AppData\Local\ElevatedDiagnostics
+AppData\Local\restic
+AppData\LocalLow\Microsoft\CryptnetUrlCache
+AppData\Local\IsolatedStorage
+AppData\Local\Spotify
+AppData\Local\Programs\signal-desktop
+AppData\Roaming\Code
+AppData\Roaming\Slack
+AppData\Roaming\Spotify
+AppData\Roaming\Zoom
+
+# misc. temporary files
+Temporary Internet Files
+Thumbs.db
+AppData\Local\Temp
+Users\Public\AccountPictures

devices/teal/README.md

@@ -1 +1,28 @@
 Principal storage server and host for most services.
+
+# Services
+
+## SystemD Services
+
+* caddy
+* restic-backup
+* rustdesk
+* sanoid
+* tailscale
+* unbound
+* zfs
+
+## Docker Services
+
+Most applications hosted on teal run in Docker containers.  For details of the Docker configuration of hosted applications, see the Git repository at [https://gitea.objectbrokers.com/cjones/containers.git](https://)
+
+Bound data volumes for Docker-hosted applications are generally found in the ZFS pool in subdirectories of /mnt/storage/appdata.
+
+* [Bookstack](../../services/bookstack/readme.md)
+* [Gitea](../../services/gitea/readme.md)
+* [Immich](../../services/immich/readme.md)
+* [Jellyfin](../../services/jellyfin/readme.md)
+* [JRiver Media Center](../../services/mc/readme.md)
+* [Nextcloud](../../services/nextcloud/readme.md)
+* [Portainer](../../services/portainer/readme.md)
+* [Vaultwarden](../../services/vaultwarden/readme.md)

devices/teal/config/etc/caddy/Caddyfile

@@ -0,0 +1,60 @@
+www.objectbrokers.com {
+        reverse_proxy localhost:8084
+}
+
+teal.objectbrokers.com {
+	respond "Caddy here."
+}
+
+cockpit.objectbrokers.com {
+	reverse_proxy :9090
+}
+
+portainer.objectbrokers.com {
+	reverse_proxy :9443 {
+		transport http {
+			tls
+			tls_insecure_skip_verify
+		}
+	}
+}
+
+jellyfin.objectbrokers.com {
+	reverse_proxy :8096
+}
+
+nextcloud.objectbrokers.com {
+	reverse_proxy :8080
+}
+
+photo.objectbrokers.com {
+	reverse_proxy :2283
+}
+
+photo.local {
+	reverse_proxy :2283
+}
+
+bw.objectbrokers.com {
+	reverse_proxy cygnus.objectbrokers.com:5555
+}
+
+vw.objectbrokers.com {
+	reverse_proxy raspberrygrove.tail7a910.ts.net:8030
+}
+
+jriver.objectbrokers.com {
+	reverse_proxy :52199
+}
+
+bookstack.objectbrokers.com {
+	reverse_proxy :6875
+}
+
+vaultwarden.objectbrokers.com {
+	reverse_proxy :8030
+}
+
+gitea.objectbrokers.com {
+	reverse_proxy :3000
+}

devices/teal/config/etc/restic/_global.env.sh

@@ -0,0 +1,37 @@
+# shellcheck shell=sh
+
+# Global environment variables
+# These variables are sourced FIRST, and any values inside of *.env.sh files for
+# specific configurations will override if also defined there.
+
+
+# Official instructions on how to setup the restic variables for Backblaze B2 can be found at
+# https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2
+
+
+# The restic repository encryption key
+export RESTIC_PASSWORD_FILE="/etc/restic/pw.txt"
+# The global restic exclude file
+export RESTIC_BACKUP_EXCLUDE_FILE="/etc/restic/backup_exclude.txt"
+
+# Backblaze B2 credentials keyID & applicationKey pair.
+# Restic environment variables are documented at https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables
+export B2_ACCOUNT_ID="<b2-key-id>"   # *EDIT* fill with your keyID
+export B2_ACCOUNT_KEY="<b2-application-key>" # *EDIT* fill with your applicationKey
+
+# How many network connections to set up to B2. Default is 5.
+export B2_CONNECTIONS=10
+
+# Optional extra space-separated args to restic-backup.
+# This is empty here and profiles can override this after sourcing this file.
+export RESTIC_BACKUP_EXTRA_ARGS=
+
+# Verbosity level from 0-3. 0 means no --verbose.
+# Override this value in a profile if needed.
+export RESTIC_VERBOSITY_LEVEL=0
+
+# (optional, uncomment to enable) Backup summary stats log: snapshot size, etc. (empty/unset won't log)
+#export RESTIC_BACKUP_STATS_DIR="/var/log/restic-automatic-backup-scheduler"
+
+# (optional) Desktop notifications. See README and restic_backup.sh for details on how to set this up (empty/unset means disabled)
+export RESTIC_BACKUP_NOTIFICATION_FILE=

devices/teal/config/etc/restic/backup_exclude.txt

@@ -0,0 +1,10 @@
+/.snapshots/
+/opt
+/root/.cache/
+/usr/share/**/*.html
+/usr/share/help/
+/usr/share/licenses/
+/usr/share/man/
+/usr/src/
+/var/cache/
+/var/log/

devices/teal/config/etc/restic/default.env.sh

@@ -0,0 +1,45 @@
+# shellcheck shell=sh
+
+# This is the default profile. Fill it with your desired configuration.
+# Additionally, you can create and use more profiles by copying this file.
+
+# This file (and other .env.sh files) has two purposes:
+# - being sourced by systemd timers to setup the backup before running restic_backup.sh
+# - being sourced in a user's shell to work directly with restic commands e.g.
+#  $ source /etc/restic/default.env.sh
+#  $ restic snapshots
+#  Thus you don't have to provide all the arguments like
+#  $ restic --repo ... --password-file ...
+
+# shellcheck source=etc/restic/_global.env.sh
+. "/etc/restic/_global.env.sh"
+
+# Envvars below will override those in _global.env.sh if present.
+
+export RESTIC_REPOSITORY="sftp:sftpuser@Cygnus:/Backup/restic/teal"   # *EDIT* fill with your repo name
+
+# What to backup. Colon-separated paths e.g. to different mountpoints "/home:/mnt/usb_disk".
+# To backup only your home directory, set "/home/your-user"
+export RESTIC_BACKUP_PATHS="/etc:/root:/home:/srv/restic:/mnt/storage/appdata"  # *EDIT* fill conveniently with one or multiple paths
+
+
+# Example below of how to dynamically add a path that is mounted e.g. external USB disk.
+# restic does not fail if a specified path is not mounted, but it's nicer to only add if they are available.
+#test -d /mnt/media && RESTIC_BACKUP_PATHS+=" /mnt/media"
+
+# A tag to identify backup snapshots.
+export RESTIC_BACKUP_TAG=systemd.timer
+
+# Retention policy - How many backups to keep.
+# See https://restic.readthedocs.io/en/stable/060_forget.html?highlight=month#removing-snapshots-according-to-a-policy
+export RESTIC_RETENTION_HOURS=1
+export RESTIC_RETENTION_DAYS=14
+export RESTIC_RETENTION_WEEKS=16
+export RESTIC_RETENTION_MONTHS=18
+export RESTIC_RETENTION_YEARS=3
+
+# Optional extra space-separated arguments to restic-backup.
+# Example: Add two additional exclude files to the global one in RESTIC_PASSWORD_FILE.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-file /path/to/extra/exclude/file/a --exclude-file /path/to/extra/exclude/file/b"
+# Example: exclude all directories that have a .git/ directory inside it.
+#RESTIC_BACKUP_EXTRA_ARGS="--exclude-if-present .git"

devices/teal/config/etc/restic/pw.txt

@@ -0,0 +1 @@
+LambDuck1977

devices/teal/config/etc/sanoid/sanoid.conf

@@ -0,0 +1,110 @@
+######################################
+# This is a sample sanoid.conf file. #
+# It should go in /etc/sanoid.       #
+######################################
+
+## name your backup modules with the path to their ZFS dataset - no leading slash.
+[tank/storage]
+#	# pick one or more templates - they're defined (and editable) below. Comma separated, processed in order.
+	use_template = production,demo
+
+## Per-service datasets for Docker based services
+[tank/appdata]
+        use_template = production
+        recursive = yes
+        process_children_only = yes
+        
+## you can also handle datasets recursively.
+#[zpoolname/parent]
+#	use_template = production
+#	recursive = yes
+#	# if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.
+#	process_children_only = yes
+#
+## you can selectively override settings for child datasets which already fall under a recursive definition.
+#[zpoolname/parent/child]
+#	# child datasets already initialized won't be wiped out, so if you use a new template, it will
+#	# only override the values already set by the parent template, not replace it completely.
+#	use_template = demo
+
+
+#############################
+# templates below this line #
+#############################
+
+# name your templates template_templatename. you can create your own, and use them in your module definitions above.
+
+[template_demo]
+	daily = 60
+
+[template_production]
+	frequently = 0
+	hourly = 36
+	daily = 30
+	monthly = 3
+	yearly = 0
+	autosnap = yes
+	autoprune = yes
+
+[template_backup]
+	autoprune = yes
+	frequently = 0
+	hourly = 30
+	daily = 90
+	monthly = 12
+	yearly = 0
+
+	### don't take new snapshots - snapshots on backup
+	### datasets are replicated in from source, not
+	### generated locally
+	autosnap = no
+
+	### monitor hourlies and dailies, but don't warn or
+	### crit until they're over 48h old, since replication
+	### is typically daily only
+	hourly_warn = 2880
+	hourly_crit = 3600
+	daily_warn = 48
+	daily_crit = 60
+
+[template_hotspare]
+	autoprune = yes
+	frequently = 0
+	hourly = 30
+	daily = 90
+	monthly = 3
+	yearly = 0
+
+	### don't take new snapshots - snapshots on backup
+	### datasets are replicated in from source, not
+	### generated locally
+	autosnap = no
+
+	### monitor hourlies and dailies, but don't warn or
+	### crit until they're over 4h old, since replication
+	### is typically hourly only
+	hourly_warn = 4h
+	hourly_crit = 6h
+	daily_warn = 2d
+	daily_crit = 4d
+
+[template_scripts]
+	### information about the snapshot will be supplied as environment variables,
+	### see the README.md file for details about what is passed when.
+	### run script before snapshot
+	pre_snapshot_script = /path/to/script.sh
+	### run script after snapshot
+	post_snapshot_script = /path/to/script.sh
+	### run script after pruning snapshot
+	pruning_script = /path/to/script.sh
+	### don't take an inconsistent snapshot (skip if pre script fails)
+	#no_inconsistent_snapshot = yes
+	### run post_snapshot_script when pre_snapshot_script is failing
+	#force_post_snapshot_script = yes
+	### limit allowed execution time of scripts before continuing (<= 0: infinite)
+	script_timeout = 5
+
+[template_ignore]
+	autoprune = no
+	autosnap = no
+	monitor = no

devices/teal/config/etc/systemd/resolved.conf

@@ -0,0 +1,36 @@
+#  This file is part of systemd.
+#
+#  systemd is free software; you can redistribute it and/or modify it under the
+#  terms of the GNU Lesser General Public License as published by the Free
+#  Software Foundation; either version 2.1 of the License, or (at your option)
+#  any later version.
+#
+# Entries in this file show the compile time defaults. Local configuration
+# should be created by either modifying this file (or a copy of it placed in
+# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
+# the /etc/systemd/resolved.conf.d/ directory. The latter is generally
+# recommended. Defaults can be restored by simply deleting the main
+# configuration file and all drop-ins located in /etc/.
+#
+# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
+#
+# See resolved.conf(5) for details.
+
+[Resolve]
+# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
+# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
+# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
+# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
+DNS=192.168.88.231 192.168.88.40
+Domains=~objectbrokers.com
+#DNSSEC=no
+#DNSOverTLS=no
+#MulticastDNS=no
+#LLMNR=no
+#Cache=no-negative
+#CacheFromLocalhost=no
+DNSStubListener=no
+#DNSStubListenerExtra=
+#ReadEtcHosts=yes
+#ResolveUnicastSingleLabel=no
+#StaleRetentionSec=0

devices/teal/config/lib/systemd/system/unbound.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Unbound DNS server
+Documentation=man:unbound(8)
+After=network.target
+Before=nss-lookup.target
+Wants=nss-lookup.target
+
+[Service]
+Type=exec
+Restart=on-failure
+EnvironmentFile=-/usr/local/etc/unbound/unbound_env
+ExecStart=/usr/local/sbin/unbound -d -p $DAEMON_OPTS
+ExecReload=+/bin/kill -HUP $MAINPID
+
+[Install]
+WantedBy=multi-user.target
+

devices/teal/config/usr/local/etc/unbound/root.key

@@ -0,0 +1,10 @@
+; autotrust trust anchor file
+;;id: . 1
+;;last_queried: 1773367002 ;;Thu Mar 12 21:56:42 2026
+;;last_success: 1773367002 ;;Thu Mar 12 21:56:42 2026
+;;next_probe_time: 1773409029 ;;Fri Mar 13 09:37:09 2026
+;;query_failed: 0
+;;query_interval: 43200
+;;retry_time: 8640
+.	86400	IN	DNSKEY	257 3 8 AwEAAa96jeuknZlaeSrvyAJj6ZHv28hhOKkx3rLGXVaC6rXTsDc449/cidltpkyGwCJNnOAlFNKF2jBosZBU5eeHspaQWOmOElZsjICMQMC3aeHbGiShvZsx4wMYSjH8e7Vrhbu6irwCzVBApESjbUdpWWmEnhathWu1jo+siFUiRAAxm9qyJNg/wOZqqzL/dL/q8PkcRU5oUKEpUge71M3ej2/7CPqpdVwuMoTvoB+ZOT4YeGyxMvHmbrxlFzGOHOijtzN+u1TQNatX2XBuzZNQ1K+s2CXkPIZo7s6JgZyvaBevYtxPvYLw4z9mR7K2vaF18UYH9Z9GNUUeayffKC73PYc= ;{id = 38696 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1771031738 ;;Fri Feb 13 20:15:38 2026
+.	86400	IN	DNSKEY	257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b} ;;state=2 [  VALID  ] ;;count=0 ;;lastchange=1771031738 ;;Fri Feb 13 20:15:38 2026

devices/teal/config/usr/local/etc/unbound/unbound.conf

@@ -0,0 +1,106 @@
+# Unbound configuration file for Debian.
+#
+# See the unbound.conf(5) man page.
+#
+# See /usr/share/doc/unbound/objectbrokerss/unbound.conf for a commented
+# reference config file.
+#
+# The following line includes additional configuration files from the
+# /etc/unbound/unbound.conf.d directory.
+    server:
+    # location of the trust anchor file that enables DNSSEC
+    auto-trust-anchor-file: "/root.key"
+    # send minimal amount of information to upstream servers to enhance privacy
+    qname-minimisation: yes
+    prefetch: yes
+    serve-expired: yes
+    # the interface that is used to connect to the network (this will listen to all interfaces)
+    interface: 0.0.0.0
+    # interface: ::0
+    private-address: 192.168.0.0/16
+    private-address: 100.64.0.0/10
+
+    # addresses from the IP range that are allowed to connect to the resolver
+    access-control: 192.168.88.0/24 allow
+    # explicitly allow localhost access
+    access-control: 127.0.0.0/8 allow
+    # allow Tailnet
+    access-control: 100.64.0.0/10 allow
+    # uncomment the following line to allow Tailnet IPv6
+    # access-control: fd7a:115c:a1e0::/48 allow
+
+    access-control-view: 192.168.88.0/24 lan
+    access-control-view: 100.64.0.0/10 tailnet
+    
+    do-ip4: yes
+    do-ip6: no
+    do-udp: yes
+    do-tcp: yes
+
+    forward-zone:
+      name: "ts.net."
+      forward-addr: 100.100.100.100
+
+    forward-zone:
+      name: "100.in-addr.arpa."
+      forward-addr: 100.100.100.100
+
+view:
+  name: "lan"
+  view-first: yes
+  local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 192.168.88.231"
+  local-data: "nextcloud.objectbrokers.com. A 192.168.88.231"
+  local-data: "photo.objectbrokers.com. A 192.168.88.231"
+  local-data: "gitea.objectbrokers.com. A 192.168.88.231"
+  local-data: "portainer.objectbrokers.com. A 192.168.88.231"
+  local-data: "jellyfin.objectbrokers.com. A 192.168.88.231"
+  local-data: "vaultwarden.objectbrokers.com. A 192.168.88.231"
+  local-data: "BOSCH-B36CL80ENS-68A40EB2F3BB.objectbrokers.com. A 192.168.88.64“
+  local-data: "bosch-dishwasher-014010536224152576.objectbrokers.com. A 192.168.88.13“
+  local-data: "chromecast.objectbrokers.com. A 192.168.88.156“
+  local-data: "cr1000b.objectbrokers.com. A 192.168.88.25“
+  local-data: "cranberrypi.objectbrokers.com. A 192.168.88.40“
+  local-data: "cygnus.objectbrokers.com. A 192.168.88.75“
+  local-data: "Denon-AVR-X3800H.objectbrokers.com. A 192.168.88.209“
+  local-data: "DESKTOP-V5OFVIA.objectbrokers.com. A 192.168.88.18“
+  local-data: "EPSON0BAAB1.objectbrokers.com. A 192.168.88.15“
+  local-data: "evan-s-S23.objectbrokers.com. A 192.168.88.29“
+  local-data: "evansroom.objectbrokers.com. A 192.168.88.82“
+  local-data: "hAPax3.objectbrokers.com. A 192.168.88.47“
+  local-data: "iris-s-S23-ultra.objectbrokers.com. A 192.168.88.86“
+  local-data: "lambdesktop.objectbrokers.com. A 192.168.88.187“
+  local-data: "lambtop.objectbrokers.com. A 192.168.88.20“
+  local-data: "LgwebOSTV.objectbrokers.com. A 192.168.88.14“
+  local-data: "mallard.objectbrokers.com. A 192.168.88.87“
+  local-data: "merganser.objectbrokers.com. A 192.168.88.26“
+  local-data: "pixel-9a.objectbrokers.com. A 192.168.88.142“
+  local-data: "rokuExpress4k.objectbrokers.com. A 192.168.88.178“
+  local-data: "samsung-washer.objectbrokers.com. A 192.168.88.220“
+  local-data: "tcl6.objectbrokers.com. A 192.168.88.23“
+
+view:
+  name: "tailnet"
+  view-first: yes
+  local-zone: "objectbrokers.com." transparent
+  local-data: "teal.objectbrokers.com. A 100.81.165.11"
+  local-data: "cygnus.objectbrokers.com. A 100.99.151.65"
+  local-data: "nextcloud.objectbrokers.com. A 100.81.165.11"
+  local-data: "photo.objectbrokers.com. A 100.81.165.11"
+  local-data: "gitea.objectbrokers.com. A 100.81.165.11"
+  local-data: "portainer.objectbrokers.com. A 100.81.165.11"
+  local-data: "jellyfin.objectbrokers.com. A 100.81.165.11"
+  local-data: "vaultwarden.objectbrokers.com. A 100.81.165.11"
+  local-data: "lambtop.objectbrokers.com. A 100.69.184.48"
+  local-data: "cranberrypi.objectbrokers.com. A 100.90.20.83"
+  local-data: "fedora.objectbrokers.com. A 100.99.193.122"
+  local-data: "lambdesktop.objectbrokers.com. A 100.95.22.24"
+  local-data: "mallard.objectbrokers.com. A 1100.116.60.98"
+  local-data: "merganser.objectbrokers.com. A 100.80.145.121"
+  local-data: "pixel-9a.objectbrokers.com. A 1100.76.129.107"
+
+remote-control:
+  control-enable: yes
+  control-interface: /run/unbound.ctl
+
+

devices/teal/config/usr/local/etc/unbound/unbound.conf.d/remote-control.conf

@@ -0,0 +1,5 @@
+remote-control:
+  control-enable: yes
+  # by default the control interface is is 127.0.0.1 and ::1 and port 8953
+  # it is possible to use a unix socket too
+  control-interface: /run/unbound.ctl

services/bookstack/readme.md

@@ -0,0 +1 @@
+Bookstack provides a self-hosted wiki.  For general information on Bookstack, see [bookstackapp.com](https://www.bookstackapp.com/)

services/caddy/readme.md

@@ -0,0 +1,9 @@
+caddy is a reverse proxy server providing secure access to https-based applications on teal.  Configuration
+
+of the reverse proxy is found in the file /etc/caddy/Caddyfile.
+
+For general information on caddy see [caddy reverse proxy quick start](https://caddyserver.com/docs/quick-starts/reverse-proxy).
+
+```
+
+```

services/gitea/readme.md

@@ -0,0 +1 @@
+Gitea is the Git source control server for the network.  For general information on Gitea, see [Gitea Official Website](https://about.gitea.com/)

services/immich/readme.md

@@ -0,0 +1 @@
+Immich is a photo backup solution.  For general information on Immich, see [Immich.app](https://immich.app//)

services/jellyfin/readme.md

@@ -0,0 +1 @@
+Jellyfin is the home network's media server.  For general information on Jellyfin, see [jellyfin.org](https://jellyfin.org/)

services/mc/readme.md

@@ -0,0 +1 @@
+mc provides a Docker-hosted implementation of the JRiver Media Center application.

services/nextcloud/readme.md

@@ -0,0 +1,4 @@
+Nextcloud provides file sharing, calendaring, contact management, and other services (depending on the Nextcloud apps installed and enabled).
+
+See https://nextcloud.com/ for general information on Nextcloud.
+

services/nextcloud/upgrading.md

@@ -0,0 +1,16 @@
+Steps to upgrade Nextcloud from one version to another.
+
+1.  Put the Nextcloud instance into maintenance mode
+    docker exec --user www-data nextcloud-app-1 php occ maintenance:mode --on
+
+    Note that 'docker exec' is used to run commands within the Nextcloud docker container; and '--user www-data' is used to run as the user that owns all of the Nextcloud files.
+
+2.  Back up the Nextcloud container's files to a location outside the container's mounted volume.  'rsync' is recommended:
+
+    rsync -Aavx /mnt/storage/appdata/nextcloud/www/ <backup-target-dir>/
+
+    rsync should be run as root.  rsync with these options should preserve file ownership & permissions
+
+3.  Back up MariaDB database:
+
+        docker exec nextcloud-db-container-name mysqldump --single-transaction -h localhost -u nextcloud_user -pnextcloud_password nextcloud_db_name > nextcloud-sqlbkp_$(date +"%Y%m%d").bak

services/portainer/readme.md

@@ -0,0 +1 @@
+Portainer is a web-based management application for Docker containers.  For general information on Portainer, see [Portainer](https://www.portainer.io/)

services/readme.md

@@ -0,0 +1,2 @@
+This directory documents the services provided by various devices on the network.  There is one subdirectory per service provided.
+

services/restic-backup/readme.md

@@ -0,0 +1,3 @@
+restic-backup is a systemd service to invoke restic to back up selected directories on teal to cygnus (our Synology NAS).  It is based on [restic-automic-backup-scheduler](https://github.com/erikw/restic-automatic-backup-scheduler).
+
+The systemd unit invokes the script /bin/restic_backup.sh.  The specifics of the backup source and target are defined in scripts at /etc/restic.

services/rustdesk/readme.md

@@ -0,0 +1 @@
+systemd service to enable RustDesk for remote access to teal's Gnome desktop.  Installed with the RustDesk package; configuration (if any) is done through the RustDesk UI.

services/sanoid/readme.md

@@ -0,0 +1,3 @@
+Sanoid is a policy-driven snapshot management tool for ZFS filesystems.  It is configured using the TOML file at /etc/sanoid/sanoid.conf.
+
+The sanoid service is currently configured to manage snapshots for the ZFS filesystem at /mnt/storage.

services/tailscale/readme.md

@@ -0,0 +1,3 @@
+The tailscaled service runs the Tailscale Node Agent, which enables the Tailscale VPN.
+
+Configuration of Tailscale is done either through the Tailscale Admin Console or the Tailscale CLI.