cjones/sysconfig
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' of https://gitea.objectbrokers.com/cjones/sysconfig

Browse Source
This commit is contained in:
cjones
2026-04-07 22:08:18 -04:00
parent e16c47fdc8 860f3bafb9
commit 06c1656cea
14 changed files with 1453 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
design/BackupStrategy.md Normal file
View File

@@ -0,0 +1,17 @@
Backup Strategy
# Backup Strategy
## Requirements
- All data that are difficult or impossible to reconstruct must be backed up and able to be restored from backup if lost.
- Backups must be automated and must occur without manual interaction with any user.
- Backups must be monitored and tested on a regular basis, to ensure that
1. Backups actually occur when they are scheduled
2. Backed-up data can be restored and that the restored data is correct.
- Backups are encrypted for privacy and security.
- All data exist in at least three places: on the device (client or server) where it is used; on a backup device on our home network; and on an off-site device.
## Strategy
- On-site backup for client devices will be provided by the Teal server. The backup tool is Restic, accessible from client devices via SFTP.
- On-site backup for the Teal server itself will be provided by the Cygnus server (Synology). Backup is by Restic over SFTP.
- Off-site backup is not currently implemented. Two different strategies are being considered:
1. Build a custom ZFS-based NAS and deploy it at 28 Carlisle Rd. This server and the Teal server would provide off-site backup for each other via ZFS send / receive.
2. Contract with a cloud storage provider for off-backup, probably either Backblaze using Restic over B2 or rsync.net using ZFS send / receive.

0
design/readme.md → design/network.md
View File

21
devices/merganser/config/C/restic/LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2020 Kevin Woley
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

109
devices/merganser/config/C/restic/README.md Normal file
View File

@@ -0,0 +1,109 @@
# restic-windows-backup
Powershell scripts to run Restic backups on Windows.
Simplifies the process of installation and running daily backups.
# Features
* **VSS (Volume Snapshot Service) support** - backup everything, don't worry about what files are open/in-use
* **Removable, External Drives** - drives can be identified by their volume labels or serial numbers, making it easy to backup drives that occasionally aren't there or change drive letter.
* **Easy Installation** - `install.ps1` script downloads Restic, initializes the restic repository, and setups up a Windows Task Scheduler task to run the backup daily
* **Easy to update** - `update.ps1` script can be used to keep your scripts up to date with the latest release on GitHub
* **Backup, Maintenance and Monitoring are Automated** - `backup.ps1` script handles
* Emailing the results of each execution, including log files when there are problems
* Runs routine maintenence (pruning and checking the repo for errors on a regular basis)
* And, of course backing up your files.
# Installation Instructions
1. **Create your restic repository**
1. This is up to you to sort out where you want the data to go to. *Minio, B2, S3, etc.*. Refer to the restic documents about how to create your repository.
1. **Install the scripts**
1. Create script directory: `C:\restic`
1. Download scripts using the `update.ps1` script.
1. Open PowerShell
1. Change your working directory to the installation directory
```
cd c:\restic
```
1. Run the `update.ps1` script:
```
Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/kmwoley/restic-windows-backup/main/update.ps1" -UseBasicParsing).Content
```
*Alternatively, you can download the scripts from this repository and and unzip them into `C:\restic`*
1. Launch PowerShell as Administrator
1. Change your working directory to `C:\restic`
1. If you haven't done so in the past, set your Powershell script [execution policy](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.1) to allow for scripts to run. For example, this is a good default:
```
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
```
1. Depending on the policy you choose, may need to 'unblock' the execution of the scripts you download by running `Unblock-File *.ps1`
1. Create `secrets.ps1` file. The secrets file contains location and passwords for your restic repository.
1. `secrets_sample.ps1` is an example of the `secrets.ps1` file. Copy or rename this file to `secrets.ps1` and edit.
1. Restic will pick up the repo destination from the environment variables you set in this file - see this doc for more information about configuring restic repos https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html
1. Email sending configuration is also contained with this file. The scripts are able to send email about the success/failure of each backup attempt.
1. Create `config.ps1` file. The config file contains the settings that control how the script runs backups, forgets snapshots, and prunes the restic repository. It's important that you configure this file to meet your needs since it will be backing up and maintaining your repository.
1. `config_sample.ps1` contins an example configuration file. Copy or rename this file to `config.ps1` and edit to suit your needs.
1. Add your `$BackupSources` to `config.ps1`
1. By default, all of `C:\` will be backed up. You can add multiple root drives to be backed up. And you can define only specific folders you would like backed up.
1. External, removable disk drives (i.e. USB hard drives) can be identified by their Volume Label, Serial Number, or Device Name. For example, if you have an external device with the Volume Label "MY BOOK", you can define a backup source as `$BackupSources["MY BOOK"]=@()`. It is recommended to use the device serial number to identify external drives to backup, which you can find using the Powershell `get-disk` command. You may also want to set `$IgnoreMissingBackupSources=$true` to avoid seeing errors when the removable drive is not present.
1. Review all of the default settings in `config.ps1`.
1. Most of the defaults are safe, but you should be sure restic is configured to meet your specifics needs.
1. **Warning** - if you're using a shared restic repository across multiple machines, pay close attention to the `$SnapshotRetentionPolicy` settings to be sure this script does not intentionally destroy backup data in your repository.
1. Run `install.ps1` file
1. From the elevated (Run as Administrator) Powershell window, run `.\install.ps1`
1. This will initialize the repo, create your logfile directory, create a scheduled task in Windows Task Scheduler to run the task daily, and install Send-MailKitMessage module.
1. Add files/paths not to backup to `local.exclude`
1. If you don't want to modify the included exclude file, you can add any files/paths you want to exclude from the backup to `local.exclude`
1. Add `C:\restic\restic.exe` to the Windows Defender / Virus & Threat Detection Exclude list
1. Backups on Windows are really slow if you don't set the Antivirus to ignore restic.
1. Navigate from the Start menu to: *Virus & threat protection > Manage Settings > Exclusions (Add or remove exclusions) > Add an exclusion (Process) > Process Name: "C:\restic\restic.exe"*
1. *(Recommended)* To a test backup triggered from Task Scheduler
1. It's recommended to open Windows Task Scheduler and trigger the task to run manually to test your first backup.
1. *Open Task Scheduler > Find "Restic Backup" > Right Click > Run*
1. The backup script will be executed as the SYSTEM user. Some of your files might not be accessible by this user. If you run into this, add the SYSTEM user to the files where you get "Access Denied" errors.
1. *Folder > Properties > Security > Advanced > Add ("SYSTEM" Principal/User) > Check "Replace all child object permission entries with inheritable permission entries from this object" > Apply > OK*
1. *(Recommended)* Do a test restore
1. These scripts make it easy to work with Restic from the Powershell command line. If you run `. .\config.ps1; . .\secrets.ps1` you can then easily invoke restic commands like
1. `& $ResticExe find -i "*filename*"`
1. `& $ResticExe restore ...`
## Updating restic-windows-backup
Use `update.ps1` to update the installed `restic-windows-backup` scripts to the latest release.
1. Open PowerShell (no need to be Administrator)
1. Change directory to your installation directory (e.g. `c:\restic`)
1. Run `update.ps1`
### `update.ps1` Details
Running `update.ps1` without any parameters will check for a new release from `kmwoley/restic-windows-backup`. If there is a newer release, the script will overwrite the local files in the script directory with the updated scripts.
* The script will not overwrite your local configuration files (i.e. `config.ps1` or `secrets.ps1`).
* Any custom files created in the installation directory will not be deleted or modified (e.g. any custom action scripts, log files, etc.)
* The script will warn before overwriting any files that have been changed since the last installation.
* When `update.ps1` is run the first time, it will prompt before overwriting (since it may not know the current version of the fiels installed).
### `update.ps1` Options
* `-Mode <release | branch> (Default: release)` - change if the script updates from the latest release or a branch of `kmwoley/restic-windows-backup`
* `-Branch <branch> (Default: 'main')` - When in branch mode, this parameter controls which branch to install from. Defaults to the `main` branch.
* `-InstallPath <directory>` - choose which directory to install the files into. Defaults to the directory that `update.ps1` is in.
## Backup over SFTP
You can use any restic repository type you like pretty easily. SFTP on Windows, however, can be particularly tricky given that these scripts execute as the SYSTEM user and need to have access to the .ssh keys. Here are some steps and tips to getting it working.
1. Install as above. Your repository should be created properly. Tasked backups will fail for now though. This is because the `install.ps1` file is executed with your user, whereas the tasked backup will run as SYSTEM, which does not have any ssh config yet.
1. Open Task Scheduler and make sure the restic task is not running anymore by checking the active tasks
1. Edit `config.ps1` and turn off the internet connection test: `$InternetTestAttempts = 0` as the test does not recognize sftp addresses correctly
1. Copy the .ssh directory content from `%USERPROFILE%\.ssh` to `%WINDIR%\System32\config\systemprofile\.ssh` (This is the ssh config the SYSTEM account uses)
1. If you use a private key to access the sftp services it also needs to be in this directory. ssh checks the permissions though, so they need to be changed as well:
1. *Right click your key > Properties > Security > Advanced*
1. Change the owner to SYSTEM
1. *Disable inheritance* and keep the permissions
1. Remove all principals except SYSTEM and the Administrators group
This should get you up and running. If not, download [PsExec](https://docs.microsoft.com/en-us/sysinternals/downloads/psexec), start a powershell as admin user and run `.\PsExec.exe -s -i powershell.exe`. In this shell you will be the system user and you can try things out. See what `ssh user@server` says or try `cd C:\restic\; . .\config.ps1; . .\secrets.ps1; & $ResticExe check` (If you get lock errors, remember to check the Task Scheduler for any running restic instances in the background)
# Feedback?
Feel free to open issues or create PRs!

711
devices/merganser/config/C/restic/backup.ps1 Normal file
View File

@@ -0,0 +1,711 @@
#
# Restic Windows Backup Script
#
# =========== start configuration =========== #
# load restic configuration parameters (destination, passwords, etc.)
$SecretsScript = Join-Path $PSScriptRoot "secrets.ps1"
# load backup configuration variables
$ConfigScript = Join-Path $PSScriptRoot "config.ps1"
# =========== end configuration =========== #
# make LASTEXITCODE global to enable error checking for Invoke-Expression commands
$global:LASTEXITCODE=0
# globals for state storage
$Script:ResticStateRepositoryInitialized = $null
$Script:ResticStateLastMaintenance = $null
$Script:ResticStateLastDeepMaintenance = $null
$Script:ResticStateMaintenanceCounter = $null
$Script:ResticStateLastBackupSuccessful = $true
$Script:ResticStateLastMaintenanceSuccessful = $true
# Returns all drive letters which exactly match the serial number, drive label, or drive name of
# the input parameter. Returns all drives if no input parameter is provided.
# inspiration: https://stackoverflow.com/questions/31088930/combine-get-disk-info-and-logicaldisk-info-in-powershell
function Get-Drives {
Param($ID)
foreach($disk in Get-CimInstance Win32_Diskdrive) {
$diskMetadata = Get-Disk | Where-Object { $_.Number -eq $disk.Index } | Select-Object -First 1
$partitions = Get-CimAssociatedInstance -ResultClassName Win32_DiskPartition -InputObject $disk
foreach($partition in $partitions) {
$drives = Get-CimAssociatedInstance -ResultClassName Win32_LogicalDisk -InputObject $partition
foreach($drive in $drives) {
$volume = Get-Volume |
Where-Object { $_.DriveLetter -eq $drive.DeviceID.Trim(":") } |
Select-Object -First 1
if(($diskMetadata.SerialNumber.trim() -eq $ID) -or
($disk.Caption -eq $ID) -or
($volume.FileSystemLabel -eq $ID) -or
($null -eq $ID)) {
[PSCustomObject] @{
DriveLetter = $drive.DeviceID
Number = $disk.Index
Label = $volume.FileSystemLabel
Manufacturer = $diskMetadata.Manufacturer
Model = $diskMetadata.Model
SerialNumber = $diskMetadata.SerialNumber.trim()
Name = $disk.Caption
FileSystem = $volume.FileSystem
PartitionKind = $diskMetadata.PartitionStyle
Drive = $drive
Partition = $partition
Disk = $disk
}
}
}
}
}
}
# test the path's storage media for VSS support
# returns $true if VSS is supported at the provided path
function Test-VSSSupport {
Param($test_path)
$drive_letter = Split-Path $test_path -Qualifier
$volume = Get-WmiObject -Query "SELECT * FROM Win32_Volume WHERE DriveLetter = '$drive_letter'"
$deviceID = ($volume.DeviceID -replace '.*(\{.*\}).*', '$1')
### https://learn.microsoft.com/en-us/previous-versions/windows/desktop/vsswmi/win32-shadowvolumesupport
$supportedVolumes = Get-WmiObject -Query "SELECT * FROM Win32_ShadowVolumeSupport WHERE __PATH LIKE '%$deviceID%'"
return ($null -ne $supportedVolumes)
}
# restore backup state from disk
function Get-BackupState {
if(Test-Path $Script:StateFile) {
Import-Clixml $Script:StateFile | ForEach-Object{ Set-Variable -Scope Script $_.Name $_.Value }
}
}
function Set-BackupState {
Get-Variable ResticState* | Export-Clixml $Script:StateFile
}
# unlock the repository if need be
function Invoke-Unlock {
Param($SuccessLog, $ErrorLog)
$locks = Invoke-Expression "$Script:ResticExe list locks --no-lock -q 3>&1 2>> $ErrorLog"
if($LASTEXITCODE) {
"[[Unlock]] Warning: unable to list locks." | Tee-Object -Append $ErrorLog
}
if($locks.Length -gt 0) {
# unlock the repository (assumes this machine is the only one that will ever use it)
Invoke-Expression "$Script:ResticExe unlock 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Unlock]] Error - unable to unlock repository." | Tee-Object -Append $ErrorLog
}
"[[Unlock]] Repository was locked. Unlocking." | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
Start-Sleep 120
}
}
# test if maintenance on the backup set is needed. Return $true if maintenance is needed
function Test-Maintenance {
Param($SuccessLog, $ErrorLog)
# skip maintenance if disabled
if($SnapshotMaintenanceEnabled -eq $false) {
"[[Maintenance]] Skipping - maintenance disabled" | Out-File -Append $SuccessLog
return $false
}
# skip maintenance if it's been done recently
if(($null -ne $ResticStateLastMaintenance) -and ($null -ne $ResticStateMaintenanceCounter)) {
$Script:ResticStateMaintenanceCounter += 1
$delta = New-TimeSpan -Start $ResticStateLastMaintenance -End $(Get-Date)
if(($delta.Days -lt $SnapshotMaintenanceDays) -and ($ResticStateMaintenanceCounter -lt $SnapshotMaintenanceInterval)) {
"[[Maintenance]] Skipping - last maintenance $ResticStateLastMaintenance ($($delta.Days) days, $ResticStateMaintenanceCounter backups ago)" | Out-File -Append $SuccessLog
return $false
}
else {
"[[Maintenance]] Running - last maintenance $ResticStateLastMaintenance ($($delta.Days) days, $ResticStateMaintenanceCounter backups ago)" | Out-File -Append $SuccessLog
return $true
}
}
else {
"[[Maintenance]] Running - no past maintenance history known." | Out-File -Append $SuccessLog
return $true
}
}
# run maintenance on the backup set
function Invoke-Maintenance {
Param($SuccessLog, $ErrorLog)
"[[Maintenance]] Start $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
$maintenance_success = $true
Start-Sleep 120
# forget snapshots based upon the retention policy
"[[Maintenance]] Start forgetting..." | Out-File -Append $SuccessLog
Invoke-Expression "$Script:ResticExe forget $SnapshotRetentionPolicy 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Maintenance]] Forget operation completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
$maintenance_success = $false
}
# prune (remove) data from the backup step. Running this separate from `forget` because
# `forget` only prunes when it detects removed snapshots upon invocation, not previously removed
"[[Maintenance]] Start pruning..." | Out-File -Append $SuccessLog
Invoke-Expression "$Script:ResticExe prune $SnapshotPrunePolicy 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Maintenance]] Prune operation completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
$maintenance_success = $false
}
# check data to ensure consistency
"[[Maintenance]] Start checking..." | Out-File -Append $SuccessLog
# check to determine if we want to do a full data check or not
$data_check = @()
if($null -ne $ResticStateLastDeepMaintenance) {
$delta = New-TimeSpan -Start $ResticStateLastDeepMaintenance -End $(Get-Date)
if(($null -ne $SnapshotDeepMaintenanceDays) -and ($delta.Days -ge $SnapshotDeepMaintenanceDays)) {
"[[Maintenance]] Performing read data check. Last '--read-data' check ran $ResticStateLastDeepMaintenance ($($delta.Days) days ago)" | Out-File -Append $SuccessLog
$data_check = @("--read-data")
$Script:ResticStateLastDeepMaintenance = Get-Date
}
else {
"[[Maintenance]] Performing fast check. Last '--read-data' check ran $ResticStateLastDeepMaintenance ($($delta.Days) days ago)" | Out-File -Append $SuccessLog
}
}
else {
# set the date, but don't do a deep check if we've never done a full data read
$Script:ResticStateLastDeepMaintenance = Get-Date
}
Invoke-Expression "$Script:ResticExe check $data_check 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Maintenance]] Data check completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
$maintenance_success = $false
}
# Invoke restic self-update to check for a newer version
# This is enabled by default unless configuration disables self-update
if ([String]::IsNullOrEmpty($SelfUpdateEnabled) -or ($SelfUpdateEnabled -eq $true)) {
# check for updated restic version
"[[Maintenance]] Checking for new version of restic..." | Out-File -Append $SuccessLog
Invoke-Expression "$Script:ResticExe self-update 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Maintenance]] Self-update of restic.exe completed with errors" | Tee-Object -Append $ErrorLog | Out-File -Append $SuccessLog
$maintenance_success = $false
}
}
"[[Maintenance]] End $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
if($maintenance_success -eq $true) {
$Script:ResticStateLastMaintenance = Get-Date
$Script:ResticStateMaintenanceCounter = 0
}
return $maintenance_success
}
# Run restic backup
function Invoke-Backup {
Param($SuccessLog, $ErrorLog)
"[[Backup]] Start $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
$return_value = $true
$starting_location = Get-Location
ForEach ($item in $BackupSources.GetEnumerator()) {
# Get the source drive letter or identifier and set as the root path
$root_path = $item.Key
$tag = $item.Key
# Test if root path is a valid path, if not assume it is an external drive identifier
if(-not (Test-Path $root_path)) {
# attempt to find a drive letter associated with the identifier provided
$drives = Get-Drives $root_path
if($drives.Count -gt 1) {
"[[Backup]] Fatal error - external drives with more than one partition are not currently supported." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
$return_value = $false
continue
}
elseif ($drives.Count -eq 0) {
$ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
$warning_message = "[[Backup]] Warning - backup path $root_path not found."
if($ignore_error) {
$warning_message | Out-File -Append $SuccessLog
}
else {
$warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
$return_value = $false
}
continue
}
# there is exactly one drive
$root_path = Join-Path $drives[0].DriveLetter ""
}
# determine if VSS is supported by the drive
$vss_option = $null
if(Test-VSSSupport $root_path) {
$vss_option = "--use-fs-snapshot"
}
"[[Backup]] Start $(Get-Date) [$tag]" | Out-File -Append $SuccessLog
# build the list of folders to backup
$folder_list = New-Object System.Collections.Generic.List[System.Object]
if ($item.Value.Count -eq 0) {
# backup everything in the root if no folders are provided
$folder_list.Add("`"$root_path`"")
}
else {
# Build the list of folders from settings
ForEach ($path in $item.Value) {
$p = '{0}' -f ((Join-Path $root_path $path) -replace "\\$")
if(Test-Path ($p -replace '"')) {
# add the folder if it exists
$folder_list.Add("`"$p`"")
}
else {
# if the folder doesn't exist, log a warning/error
$ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
$warning_message = "[[Backup]] Warning - backup path $p not found."
if($ignore_error) {
$warning_message | Out-File -Append $SuccessLog
}
else {
$warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
$return_value = $false
}
}
}
}
if(-not $folder_list) {
# there are no folders to backup
$ignore_error = ($null -ne $IgnoreMissingBackupSources) -and $IgnoreMissingBackupSources
$warning_message = "[[Backup]] Warning - no folders to back up!"
if($ignore_error) {
$warning_message | Out-File -Append $SuccessLog
}
else {
$warning_message | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
$return_value = $false
}
}
else {
# Launch Restic
Invoke-Expression "$Script:ResticExe backup $folder_list $vss_option --tag $tag --exclude-file=$WindowsExcludeFile --exclude-file=$LocalExcludeFile $AdditionalBackupParameters 3>&1 2>> $ErrorLog | Out-File -Append $SuccessLog"
if($LASTEXITCODE) {
"[[Backup]] Completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
$return_value = $false
}
}
"[[Backup]] End $(Get-Date) [$tag]" | Out-File -Append $SuccessLog
}
Set-Location $starting_location
"[[Backup]] End $(Get-Date)" | Tee-Object -Append $SuccessLog | Write-Host
return $return_value
}
function Send-Email {
Param($SuccessLog, $ErrorLog, $Action)
Import-Module Send-MailKitMessage
# default the action string to "Backup"
if($null -eq $Action) {
$Action = "Backup"
}
# set email credentials if a username and passsword are provided in configuration
$credentials = @{}
if (-not [String]::IsNullOrEmpty($ResticEmailPassword) -and -not [String]::IsNullOrEmpty($ResticEmailUsername)) {
$password = ConvertTo-SecureString -String $ResticEmailPassword -AsPlainText -Force
$credentials = @{
"Credential" = [System.Management.Automation.PSCredential]::new($ResticEmailUsername, $password)
}
}
# Backwards compatibility for $ResticEmailConfig port definition:
# $ResticEmailConfig is obsolete and should be replaced with $ResticEmailPort
if ($null -ne $ResticEmailConfig -and $ResticEmailConfig.ContainsKey('Port')) {
if ($null -eq $ResticEmailPort) {
$ResticEmailPort = $ResticEmailConfig['Port']
'[[Email]] Warning - $ResticEmailConfig is deprecated. Define $ResticEmailPort in secrets.ps1 instead.' | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
}
}
# Backwards compatibility for $PSEmailServer rename to $ResticEmailServer
if (($null -ne $PSEmailServer) -and ($null -eq $ResticEmailServer)) {
$ResticEmailServer = $PSEmailServer
'[[Email]] Warning - $PSEmailServer is deprecated. Define $ResticEmailServer in secrets.ps1 instead.' | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog | Write-Host
}
$status = "SUCCESS"
$past_failure = $false
$body = ""
if (($null -ne $SuccessLog) -and (Test-Path $SuccessLog) -and (Get-Item $SuccessLog).Length -gt 0) {
$body = $(Get-Content -Raw $SuccessLog)
# if previous run contained an error, send the success email confirming that the error has been resolved
if($Action -eq "Backup") {
$past_failure = -not $Script:ResticStateLastBackupSuccessful
}
else {
$past_failure = -not $Script:ResticStateLastMaintenanceSuccessful
}
}
else {
$body = "Critical Error! Restic $Action log is empty or missing. Check log file path."
$status = "ERROR"
}
$attachments = [System.Collections.Generic.List[string]]::new()
if (($null -ne $ErrorLog) -and (Test-Path $ErrorLog) -and (Get-Item $ErrorLog).Length -gt 0) {
$attachments.Add("$ErrorLog")
$status = "ERROR"
}
if((($status -eq "SUCCESS") -and ($SendEmailOnSuccess -ne $false)) -or ((($status -eq "ERROR") -or $past_failure) -and ($SendEmailOnError -ne $false))) {
$subject = "$env:COMPUTERNAME Restic $Action Report [$status]"
# create a temporary error log to log errors; can't write to the same file that Send-MailMessage is reading
$temp_error_log = $ErrorLog + "_temp"
$from = [MimeKit.MailboxAddress]$ResticEmailFrom;
$recipients = [MimeKit.InternetAddressList]::new();
$recipients.Add([MimeKit.InternetAddress]$ResticEmailTo);
Send-MailKitMessage -SMTPServer $ResticEmailServer -Port $ResticEmailPort -UseSecureConnectionIfAvailable @credentials -From $from -RecipientList $recipients -Subject $subject -TextBody $body -AttachmentList $attachments 3>&1 2>> $temp_error_log | Out-File -Append $SuccessLog
if(-not $?) {
"[[Email]] Sending email completed with errors" | Tee-Object -Append $temp_error_log | Tee-Object -Append $SuccessLog | Write-Host
}
# join error logs and remove the temporary
Get-Content $temp_error_log | Add-Content $ErrorLog
Remove-Item $temp_error_log
}
}
# check if on metered network,
# returns $true the current connection is a metered network
function Invoke-MeteredCheck {
$scriptBlock = {
# load NetworkInformation class from the Windows Runtime (WinRT) environment
[void][Windows.Networking.Connectivity.NetworkInformation, Windows, ContentType = WindowsRuntime]
$cost = [Windows.Networking.Connectivity.NetworkInformation]::GetInternetConnectionProfile().GetConnectionCost()
return ($cost.ApproachingDataLimit -or $cost.OverDataLimit -or $cost.Roaming -or $cost.BackgroundDataUsageRestricted -or ($cost.NetworkCostType -ne 'Unrestricted'))
}
# run this check in PowerShell 5.1
# this is a workaround for lack of WinRT support in PowerShell 7
$result = powershell.exe -Version 5.1 -Command "$scriptBlock"
return ($result -ieq "True")
}
# check network conditions, retrying a limited number of times until a connection is established
# returns $true if the repository is accessible and the configuration allows us to use it
function Invoke-ConnectivityCheck {
Param($SuccessLog, $ErrorLog)
$sleep_time = 30
if($InternetTestAttempts -le 0) {
"[[Internet]] Internet connectivity check disabled. Skipping." | Out-File -Append $SuccessLog
return $true
}
# skip the internet connectivity check for local repos
if(Test-Path $env:RESTIC_REPOSITORY) {
"[[Internet]] Local repository. Skipping internet connectivity check." | Out-File -Append $SuccessLog
return $true
}
$repository_host = ''
# use generic internet service for non-specific repo types (e.g. swift:, rclone:, etc. )
if(($env:RESTIC_REPOSITORY -match "^swift:") -or
($env:RESTIC_REPOSITORY -match "^rclone:")) {
$repository_host = "cloudflare.com"
}
elseif($env:RESTIC_REPOSITORY -match "^b2:") {
$repository_host = "api.backblazeb2.com"
}
elseif($env:RESTIC_REPOSITORY -match "^azure:") {
$repository_host = "azure.microsoft.com"
}
elseif($env:RESTIC_REPOSITORY -match "^gs:") {
$repository_host = "storage.googleapis.com"
}
else {
# parse connection string for hostname
# Uri parser doesn't handle leading connection type info (s3:, sftp:, rest:)
$connection_string = $env:RESTIC_REPOSITORY -replace "^s3:" -replace "^sftp:" -replace "^rest:"
if(-not ($connection_string -match "://")) {
# Uri parser expects to have a protocol. Add 'https://' to make it parse correctly.
$connection_string = "https://" + $connection_string
}
$repository_host = ([System.Uri]$connection_string).DnsSafeHost
}
if([string]::IsNullOrEmpty($repository_host)) {
"[[Internet]] Repository string could not be parsed." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
return $false
}
# test for internet connectivity
$connections = 0
$sleep_count = $InternetTestAttempts
$restricted_by_metered_network = $false
while($true) {
$connections = Get-NetRoute | Where-Object DestinationPrefix -eq '0.0.0.0/0' | Get-NetIPInterface | Where-Object ConnectionState -eq 'Connected' | Measure-Object | ForEach-Object{$_.Count}
if($sleep_count -le 0) {
if($restricted_by_metered_network) {
"[[Internet]] Connection to repository ($repository_host) is available but blocked by metered network." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
}
else {
"[[Internet]] Connection to repository ($repository_host) could not be established." | Tee-Object -Append $SuccessLog | Out-File -Append $ErrorLog
}
return $false
}
if(($null -eq $connections) -or ($connections -eq 0)) {
"[[Internet]] Waiting $sleep_time seconds for internet connectivity... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog
Start-Sleep $sleep_time
}
elseif(!(Test-Connection -ComputerName $repository_host -Quiet)) {
"[[Internet]] Waiting $sleep_time seconds for connection to repository ($repository_host)... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog
Start-Sleep $sleep_time
}
elseif((-not ([String]::IsNullOrEmpty($BackupOnMeteredNetwork) -or $BackupOnMeteredNetwork)) -and (Invoke-MeteredCheck)) {
"[[Internet]] Waiting $sleep_time seconds for an unmetered network connection... ($sleep_count/$InternetTestAttempts)" | Out-File -Append $SuccessLog
$restricted_by_metered_network = $true
Start-Sleep $sleep_time
}
else {
return $true
}
$sleep_count--
}
}
# check previous logs
function Invoke-HistoryCheck {
Param($SuccessLog, $ErrorLog, $Action)
# default the action to "Backup"
if($null -eq $Action) {
$Action = "Backup"
}
$filter = "*$Action.err.txt".ToLower()
$logs = Get-ChildItem $Script:LogPath -Filter $filter | ForEach-Object{$_.Length -gt 0}
$logs_with_success = ($logs | Where-Object {($_ -eq $false)}).Count
if($logs.Count -gt 0) {
"[[History]] $Action success rate: $logs_with_success / $($logs.Count) ($(($logs_with_success / $logs.Count).tostring("P")))" | Tee-Object -Append $SuccessLog | Write-Host
}
}
# main function
function Invoke-Main {
# check for elevation, required for creation of shadow copy (VSS)
if (-not (New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator))
{
Write-Error "[[Backup]] Elevation required (run as administrator). Exiting."
exit 1
}
# initialize secrets
. $SecretsScript
# initialize config
. $ConfigScript
# apply global configuration
$Script:ResticExe = Join-Path $InstallPath $ExeName
if(-not [String]::IsNullOrEmpty($GlobalParameters)) {
$Script:ResticExe = "$Script:ResticExe $GlobalParameters"
}
$Script:StateFile = Join-Path $InstallPath "state.xml"
$Script:LogPath = Join-Path $InstallPath "logs"
Get-BackupState
if(!(Test-Path $Script:LogPath)) {
Write-Error "[[Backup]] Log file directory $Script:LogPath does not exist. Exiting."
Send-Email
exit 1
}
# custom start action
if($null -ne $CustomActionStart) {
Invoke-Expression $CustomActionStart
}
$error_count = 0
$backup_success = $false
$maintenance_success = $false
$maintenance_needed = $false
$attempt_count = $GlobalRetryAttempts
while ($attempt_count -gt 0) {
# setup logfiles
$timestamp = Get-Date -Format FileDateTime
$success_log = Join-Path $Script:LogPath ($timestamp + ".backup.log.txt")
$error_log = Join-Path $Script:LogPath ($timestamp + ".backup.err.txt")
$repository_available = Invoke-ConnectivityCheck $success_log $error_log
if($repository_available -eq $true) {
Invoke-Unlock $success_log $error_log
$backup_success = Invoke-Backup $success_log $error_log
# NOTE: a previously locked repository will cause errors in the log; but backup would be 'successful'
# Removing this overly-aggressive test for backup success and relying upon Invoke-Backup to report on success/failure
# $backup_success = ($backup_success -eq $true) -and (!(Test-Path $error_log) -or ((Get-Item $error_log).Length -eq 0))
$total_attempts = $GlobalRetryAttempts - $attempt_count + 1
if($backup_success -eq $true) {
# successful backup
"[[Backup]] Succeeded after $total_attempts attempt(s)" | Tee-Object -Append $success_log | Write-Host
# test to see if maintenance is needed if the backup was successful
$maintenance_needed = Test-Maintenance $success_log $error_log
}
else {
"[[Backup]] Ran with errors on attempt $total_attempts" | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
$error_count++
}
}
else {
"[[Backup]] Failed - cannot access repository." | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
$error_count++
}
$attempt_count--
# update logs prior to sending email
if($backup_success -eq $false) {
if($attempt_count -gt 0) {
"[[Backup]] Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log | Write-Host
}
else {
"[[Backup]] Retry limit has been reached. No more attempts to backup will be made." | Tee-Object -Append $success_log | Write-Host
}
}
Invoke-HistoryCheck $success_log $error_log "Backup"
Send-Email $success_log $error_log "Backup"
# update the state of the last backup success or failure
$Script:ResticStateLastBackupSuccessful = $backup_success
# Save state to file
Set-BackupState
# loop exit/wait condition
if(($backup_success -eq $false) -and ($attempt_count -gt 0)) {
Start-Sleep (15*60)
}
else {
break
}
}
# only run maintenance if the backup was successful and maintenance is needed
$attempt_count = $GlobalRetryAttempts
while (($maintenance_needed -eq $true) -and ($attempt_count -gt 0)) {
# setup logfiles
$timestamp = Get-Date -Format FileDateTime
$success_log = Join-Path $Script:LogPath ($timestamp + ".maintenance.log.txt")
$error_log = Join-Path $Script:LogPath ($timestamp + ".maintenance.err.txt")
$repository_available = Invoke-ConnectivityCheck $success_log $error_log
if($repository_available -eq $true) {
$maintenance_success = Invoke-Maintenance $success_log $error_log
# $maintenance_success = ($maintenance_success -eq $true) -and (!(Test-Path $error_log) -or ((Get-Item $error_log).Length -eq 0))
$total_attempts = $GlobalRetryAttempts - $attempt_count + 1
if($maintenance_success -eq $true) {
"[[Maintenance]] Succeeded after $total_attempts attempt(s)" | Tee-Object -Append $success_log | Write-Host
}
else {
"[[Maintenance]] Ran with errors on attempt $total_attempts" | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
$error_count++
}
}
else {
"[[Maintenance]] Failed - cannot access repository." | Tee-Object -Append $success_log | Tee-Object -Append $error_log | Write-Host
$error_count++
}
$attempt_count--
# update logs prior to sending email
if($maintenance_success -eq $false) {
if($attempt_count -gt 0) {
"[[Maintenance]] Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log | Write-Host
}
else {
"[[Maintenance]] Retry limit has been reached. No more attempts to run maintenance will be made." | Tee-Object -Append $success_log | Write-Host
}
}
Invoke-HistoryCheck $success_log $error_log "Maintenance"
Send-Email $success_log $error_log "Maintenance"
# update the state of the last maintenance success or failure
$Script:ResticStateLastMaintenanceSuccessful = $maintenance_success
# Save state to file
Set-BackupState
# loop exit/wait condition
if(($maintenance_success -eq $false) -and ($attempt_count -gt 0)) {
Start-Sleep (15*60)
}
else {
break
}
}
# custom end actions
if((-not $backup_success) -or ($maintenance_needed -and -not $maintenance_success)) {
# call the custom error action if backup failed and/or maintenance was needed and failed
if($null -ne $CustomActionEndError) {
Invoke-Expression $CustomActionEndError
}
}
else {
# call custom success action if backup & maintenance were successful
if($null -ne $CustomActionEndSuccess) {
Invoke-Expression $CustomActionEndSuccess
}
}
# Save state to file
Set-BackupState
# cleanup older log files
Get-ChildItem $Script:LogPath | Where-Object {$_.CreationTime -lt $(Get-Date).AddDays(-$LogRetentionDays)} | Remove-Item
exit $error_count
}
Invoke-Main

68
devices/merganser/config/C/restic/config.ps1 Normal file
View File

@@ -0,0 +1,68 @@
# Sample configuration file
# Update this file to control how the restic backup, forget, and purge operations are run
# Rename to `config.ps1`
# general configuration
$InstallPath = "C:\restic"
$ExeName = "restic.exe"
$GlobalParameters = @()
$LogRetentionDays = 30
$BackupOnMeteredNetwork = $true
$InternetTestAttempts = 10
$GlobalRetryAttempts = 4
# email configuration
$SendEmailOnSuccess = $false
$SendEmailOnError = $true
# backup configuration
$WindowsExcludeFile = Join-Path $InstallPath "windows.exclude"
$LocalExcludeFile = Join-Path $InstallPath "local.exclude"
$IgnoreMissingBackupSources = $false
$AdditionalBackupParameters = @("--exclude-if-present", ".nobackup", "--no-scan")
# Paths to backup
$BackupSources = @{}
# Paths to backup
$BackupSources = @{}
$BackupSources["C:\"] = @(
'Users\chris'
)
$BackupSources["D:\"] = @(
'workspace',
'Mikrotik'
)
#$BackupSources["DRIVE_LABEL_NAME_OR_SERIAL_NUMBER"] = @(
# "Example\FolderName"
#)
# maintenance configuration
$SnapshotMaintenanceEnabled = $true
$SnapshotRetentionPolicy = @("--host", $env:COMPUTERNAME, "--group-by", "host,tags", "--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
# $SnapshotRetentionPolicy = @("--group-by", "host,tags", "--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
$SnapshotPrunePolicy = @("--max-unused", "1%")
$SnapshotMaintenanceInterval = 7
$SnapshotMaintenanceDays = 30
$SnapshotDeepMaintenanceDays = 90
# restic.exe self update configuration
$SelfUpdateEnabled = $true
# (optional) custom actions
# Define commands to pass to Invoke-Expression at script start and script end
# note: Errors will only be reported if the script does not eventually succeed. Errors
# from unsuccessful attempts to backup or maintain the repository will not result
# in the custom error action being called unless all attempts to backup or maintain failed.
$CustomActionStart = $null
$CustomActionEndError = $null
$CustomActionEndSuccess = $null
# Example: Calling a healthcheck remote service
# $healthCheckURL = "https://healthcheckservice.com/etc/etc"
# $CustomActionStart = "Invoke-RestMethod $healthCheckURL/start"
# $CustomActionEndError = "Invoke-RestMethod $healthCheckURL/fail"
# $CustomActionEndSuccess = "Invoke-RestMethod $healthCheckURL"
# Example: Invoking a script
# $successScript = Join-Path $InstallPath "mySuccessScript.ps1"
# $CustomActionEndSuccess = "& $successScript"

108
devices/merganser/config/C/restic/install.ps1 Normal file
View File

@@ -0,0 +1,108 @@
#
# Restic Windows Backup - Installation Script
#
# =========== start configuration =========== #
# load restic configuration parmeters (destination, passwords, etc.)
$SecretsScript = Join-Path $PSScriptRoot "secrets.ps1"
# load backup configuration variables
$ConfigScript = Join-Path $PSScriptRoot "config.ps1"
# initialize secrets
. $SecretsScript
# initialize config
. $ConfigScript
# apply global configuration
$ResticExe = Join-Path $InstallPath $ExeName
$LogPath = Join-Path $InstallPath "logs"
# make LASTEXITCODE global to enable error checking for Invoke-Expression commands
$global:LASTEXITCODE=0
# =========== end configuration =========== #
# download restic
if(-not (Test-Path $ResticExe)) {
$url = $null
if([Environment]::Is64BitOperatingSystem){
$url = "https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_windows_amd64.zip"
}
else {
$url = "https://github.com/restic/restic/releases/download/v0.17.3/restic_0.17.3_windows_386.zip"
}
try {
$output = Join-Path $InstallPath "restic.zip"
Invoke-WebRequest -Uri $url -OutFile $output
Expand-Archive -LiteralPath $output $InstallPath
Remove-Item $output
Get-ChildItem *.exe | Rename-Item -NewName $ExeName
}
catch {
Write-Error "[[Install]] restic.exe download failed. Check errors and resolve: $_"
exit 1
}
}
# Apply global paramters to $ResticExe, after the $ResticExe has been downloaded/confirmed to exist
if(-not [String]::IsNullOrEmpty($GlobalParameters)) {
$ResticExe = "$ResticExe $GlobalParameters"
}
# Invoke restic self-update to check for a newer version
# This is enabled by default unless configuration disables self-update
if ([String]::IsNullOrEmpty($SelfUpdateEnabled) -or ($SelfUpdateEnabled -eq $true)) {
Invoke-Expression "$ResticExe self-update"
if($LASTEXITCODE) {
Write-Warning "[[Update]] Restic self-update failed. Check errors and resolve."
}
}
# Create log directory if it doesn't exit
if(-not (Test-Path $LogPath)) {
New-Item -ItemType Directory -Force -Path $LogPath | Out-Null
Write-Output "[[Init]] Created log directory: $LogPath"
}
# Create the local exclude file
if(-not (Test-Path $LocalExcludeFile)) {
New-Item -Type File -Path $LocalExcludeFile | Out-Null
}
# Initialize the restic repository
Invoke-Expression "$ResticExe --verbose init"
if($LASTEXITCODE) {
Write-Warning "[[Init]] Repository initialization failed. Check errors and resolve."
}
else {
Write-Output "[[Init]] Repository successfully initialized."
}
# Scheduled Windows Task Scheduler to run the backup
$backup_task_name = "Restic Backup"
$backup_task = Get-ScheduledTask $backup_task_name -ErrorAction SilentlyContinue
if($null -eq $backup_task) {
try {
$task_action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-ExecutionPolicy Bypass -NonInteractive -NoLogo -NoProfile -Command ".\backup.ps1; exit $LASTEXITCODE"' -WorkingDirectory $InstallPath
$task_user = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount -RunLevel Highest
$task_settings = New-ScheduledTaskSettingsSet -RestartCount 4 -RestartInterval (New-TimeSpan -Minutes 15) -ExecutionTimeLimit (New-TimeSpan -Days 3) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -DontStopOnIdleEnd -MultipleInstances IgnoreNew -IdleDuration 0 -IdleWaitTimeout 0 -StartWhenAvailable -RestartOnIdle
$task_trigger = New-ScheduledTaskTrigger -Daily -At 4:00am
Register-ScheduledTask $backup_task_name -Action $task_action -Principal $task_user -Settings $task_settings -Trigger $task_trigger | Out-Null
Write-Output "[[Scheduler]] Backup task scheduled."
}
catch {
Write-Error "[[Scheduler]] Setting up backup task schedule failed: $_"
}
}
else {
Write-Warning "[[Scheduler]] Backup task not scheduled: there is already a task with the name '$backup_task_name'."
}
# Install NuGet and Send-MailKitMessage module (by force)
if ($PSVersionTable.PSVersion.Major -eq 5) {
Install-PackageProvider -Name NuGet -Force
}
Install-Module Send-MailKitMessage -Repository PSGallery -Scope AllUsers -Force

0
devices/merganser/config/C/restic/local.exclude Normal file
View File

BIN
devices/merganser/config/C/restic/restic.exe Normal file
View File

Binary file not shown.

BIN
devices/merganser/config/C/restic/restic.exe.bak Normal file
View File

Binary file not shown.

17
devices/merganser/config/C/restic/secrets.ps1 Normal file
View File

@@ -0,0 +1,17 @@
# Template file for backup destination configuration and email passwords.
# Update this file to point to your restic repository and email service.
# Rename to `secrets.ps1`
# restic backup repository configuration
$Env:AWS_ACCESS_KEY_ID='<KEY>'
$Env:AWS_SECRET_ACCESS_KEY='<KEY>'
$Env:RESTIC_REPOSITORY='sftp:sftpuser@teal:/srv/restic/merganser-backup'
$Env:RESTIC_PASSWORD='<LambDuck1977>'
# email configuration
$ResticEmailServer='smtp.gmail.com'
$ResticEmailPort='465'
$ResticEmailTo='jones.chrisk@gmail.com'
$ResticEmailFrom='jones.chrisk@gmail.com'
$ResticEmailUsername='jones.chrisk@gmail.com'
$ResticEmailPassword='nvooxavrzeskrzlg'

BIN
devices/merganser/config/C/restic/state.xml Normal file
View File

Binary file not shown.

337
devices/merganser/config/C/restic/update.ps1 Normal file
View File

@@ -0,0 +1,337 @@
<#
.SYNOPSIS
Updates the local installed restic backup scripts from GitHub,
either using the latest tagged release or by targeting a specific branch.
.DESCRIPTION
This script supports two modes:
1. **Release mode (default):**
- Fetches the latest release info via GitHub’s API.
- Compares the release tag (after normalization) against a locally stored version (in state.xml).
- If the GitHub release is newer, downloads the release zip, extracts it, copies the files
over the local installation.
2. **Branch mode:**
- Targets a specific branch (default "main") by retrieving branch information from GitHub.
- Compares the latest commit SHA on that branch against a locally stored SHA (in state.xml).
- If the remote commit SHA differs, downloads the branch zip archive, extracts it,
copies the files over the local installation.
.NOTES
Example 1 - update scripts to the latest tagged release
.\update.ps1
Example 2 - update scripts from a branch
.\update.ps1 -Mode branch -BranchName 'release_1.8'
Example 3 - download a new copy of the update scripts and run it
1. Change your directory to your installation directory (e.g. `cd c:\restic`)
2. Invoke-WebRequest "https://raw.githubusercontent.com/kmwoley/restic-windows-backup/main/update.ps1" -OutFile update.ps1
3. .\update.ps1
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $false)]
[ValidateSet("release", "branch")]
[string]$Mode = "release",
[Parameter(Mandatory = $false)]
[string]$BranchName = "main",
[Parameter(Mandatory = $false)]
[string]$InstallPath = $null
)
# ====================================
# Configuration and Setup
# ====================================
# GitHub repository details
$repoOwner = "kmwoley"
$repoName = "restic-windows-backup"
# User-Agent header (GitHub requires this)
$headers = @{ "User-Agent" = "PowerShell" }
# default the installation directory to the location of the running script
if([string]::IsNullOrEmpty($InstallPath)) {
# default to the script's location, if running as a script
$InstallPath = $PSScriptRoot
if([string]::IsNullOrEmpty($InstallPath)) {
# default to the current working directory, if not running as a script
$InstallPath = Get-Location
}
}
# ====================================
# Functions for state management
# ====================================
function Get-State {
if(Test-Path $Script:StateFile) {
Import-Clixml $Script:StateFile | ForEach-Object{ Set-Variable -Scope Script $_.Name $_.Value }
}
}
function Set-State {
Get-Variable ResticState* | Export-Clixml $Script:StateFile
}
# ===========================================
# Functions for file management and download
# ===========================================
function Get-ModifiedFiles {
param(
[Parameter(Mandatory = $true)]
[string]$Source,
[Parameter(Mandatory = $true)]
[string]$Destination,
[Parameter(Mandatory = $true)]
[string]$DateTime
)
$modifiedFiles = New-Object System.Collections.Generic.List[System.Object]
if(-not (Test-Path $Source)) {
Write-Error "Source does not exist ($Source)"
exit 1
}
if(-not (Test-Path $Destination)) {
Write-Error "Destination does not exist ($Destination)"
exit 1
}
$sourceFiles = Get-ChildItem $Source
ForEach ($sourceFile in $sourceFiles) {
# find if there's a corrosponding file in the destination
$destFileName = Join-Path $Destination $sourceFile.Name
if(Test-Path $destFileName) {
$destFile = Get-ChildItem $destFileName
if($destFile.LastWriteTime -gt $DateTime) {
# destination file has been modified after $DateTime
$modifiedFiles.Add($destFile.FullName)
}
}
}
return $modifiedFiles
}
function Update-InstalledScripts {
param(
[Parameter(Mandatory=$true)][string]$ZipUrl,
[Parameter(Mandatory=$true)][string]$DestinationFolder
)
$timestamp = Get-Date -Format FileDateTime
$tempExtractDir = Join-Path $env:TEMP ("restic-windows-backup." + $timestamp)
$tempZipPath = Join-Path $env:TEMP ("restic-windows-backup." + $timestamp + ".zip")
# test temp location, fail if in use
if (Test-Path $tempExtractDir) {
Write-Error "Temporary directory already exists: $tempExtractDir"
exit 1
}
if (Test-Path $tempZipPath) {
Write-Error "Temporary directory already exists: $tempZipPath"
exit 1
}
# Create a temporary folder for extraction
New-Item -ItemType Directory -Path $tempExtractDir | Out-Null
Write-Host "Downloading from: $ZipUrl"
try {
Invoke-WebRequest -Uri $ZipUrl -OutFile $tempZipPath -Headers $headers
} catch {
Write-Error "Failed to download the file: $_"
exit 1
}
try {
Expand-Archive -LiteralPath $tempZipPath $tempExtractDir
} catch {
Write-Error "Error extracting zip file: $_"
exit 1
}
# Determine the actual folder containing the repository files.
$extractedContent = Get-ChildItem -Path $tempExtractDir | Where-Object { $_.PSIsContainer }
if ($extractedContent.Count -eq 1) {
$extractedFolder = $extractedContent[0].FullName
} else {
$extractedFolder = $tempExtractDir
}
# Check to make sure not to overwrite modified files
$installedDate = $Script:ResticStateInstalledDate
if([string]::IsNullOrEmpty($installedDate)) {
# unkown install date; setting the date
$installedDate = [datetime]::MinValue
}
$modifiedFiles = Get-ModifiedFiles -Source $extractedFolder -Destination $DestinationFolder -DateTime $installedDate
if($modifiedFiles) {
if([string]::IsNullOrEmpty($Script:ResticStateInstalledDate)) {
Write-Host "WARNING: The following files already exist in the target directory"
}
else {
Write-Host "WARNING: The following files have been modified since they were installed on $installedDate"
}
ForEach ($fileName in $modifiedFiles) {
Write-Host " - " $fileName
}
# TODO: add a "-Force" parameter to skip this check/question
Write-Host "Continuing will overwrite these files."
Write-host "Do you want to continue?"
$userInput = Read-Host "[Y] Yes [N] No (default is ""Y"")"
if ($userInput -ieq 'n') {
Write-Host "Operation cancelled."
exit 0
}
}
Write-Host "Updating files in installation directory ($DestinationFolder)..."
try {
# Recursively copy all content from the extracted folder to the local directory.
Copy-Item -Path (Join-Path $extractedFolder "*") -Destination $DestinationFolder -Recurse -Force
} catch {
Write-Error "Error copying files: $_"
exit 1
}
# Clean up temporary files
Remove-Item $tempZipPath -Force
Remove-Item $tempExtractDir -Recurse -Force
}
# ====================================
# Main
# ====================================
# load restic state
$Script:ResticStateInstalledVersion = $null
$Script:ResticStateInstalledBranchSHA = $null
$Script:ResticStateInstalledDate = $null
$Script:StateFile = Join-Path $InstallPath "state.xml"
Get-State
# ====================================
# Release mode
# ====================================
if ($Mode -eq "release") {
# Read the version of the scripts installed
$localVersion = $Script:ResticStateInstalledVersion
if ([string]::IsNullOrEmpty($localVersion)) {
# No version information stored locally
$localVersion = "0.0.0"
}
# Get the Latest Release Info from GitHub
$releaseApiUrl = "https://api.github.com/repos/$repoOwner/$repoName/releases/latest"
try {
Write-Host "Checking GitHub for latest release of '$repoOwner/$repoName'..."
$release = Invoke-RestMethod -Uri $releaseApiUrl -Headers $headers
} catch {
Write-Error "Error fetching release information from GitHub: $_"
exit 1
}
$latestTagRaw = $release.tag_name
$latestTag = $latestTagRaw.Trim()
# Normalize versions (remove leading "v" if present)
function Get-NormalizedVersion($versionString) {
if ($versionString.StartsWith("v", [System.StringComparison]::InvariantCultureIgnoreCase)) {
return $versionString.Substring(1)
}
return $versionString
}
$normalizedLocalVersion = Get-NormalizedVersion $localVersion
$normalizedLatestVersion = Get-NormalizedVersion $latestTag
try {
$localVersionObj = [Version]$normalizedLocalVersion
$latestVersionObj = [Version]$normalizedLatestVersion
} catch {
Write-Error "Error parsing version strings. Local: $normalizedLocalVersion, Latest: $normalizedLatestVersion. $_"
exit 1
}
if ($latestVersionObj -le $localVersionObj) {
Write-Host "Installed version ($localVersionObj) is up-to-date. No update needed."
exit 0
} else {
Write-Host "Newer release available: $latestVersionObj (installed: $localVersionObj). Proceeding with update..."
}
# get the zip URL from the release info
$zipUrl = $release.zipball_url
# Download and update the installed scripts
Update-InstalledScripts -ZipUrl $zipUrl -DestinationFolder $InstallPath
# Store the installed version number and time installed
$Script:ResticStateInstalledVersion = $normalizedLatestVersion
$Script:ResticStateInstalledDate = Get-Date
$Script:ResticStateInstalledBranchSHA = $null
Set-State
Write-Host "Update successful. Installed version is now $normalizedLatestVersion."
}
# ====================================
# Branch mode
# ====================================
elseif ($Mode -eq "branch") {
# Read the SHA of the branch source installed
$localCommitSHA = $Script:ResticStateInstalledBranchSHA
if ([string]::IsNullOrEmpty($localCommitSHA)) {
# Write-Host "No branch information stored locally."
$localCommitSHA = "unknown"
}
# Retrieve branch information from GitHub
$branchApiUrl = "https://api.github.com/repos/$repoOwner/$repoName/branches/$BranchName"
try {
Write-Host "Checking GitHub for latest commit of '$repoOwner/$repoName' on branch '$BranchName'..."
$branchInfo = Invoke-RestMethod -Uri $branchApiUrl -Headers $headers
} catch {
Write-Error "Error fetching branch information from GitHub: $_"
exit 1
}
$latestCommitSHA = $branchInfo.commit.sha
if ($localCommitSHA -eq $latestCommitSHA) {
Write-Host "Installed commit ($latestCommitSHA) is up-to-date. No update needed."
exit 0
} else {
Write-Host "Latest commit: $latestCommitSHA (installed: $localCommitSHA). Proceeding with update..."
}
# Construct the zip URL for the branch.
# GitHub provides branch archives at:
# https://github.com/{owner}/{repo}/archive/refs/heads/{branch}.zip
$zipUrl = "https://github.com/$repoOwner/$repoName/archive/refs/heads/$BranchName.zip"
# Download and update the installed scripts
Update-InstalledScripts -ZipUrl $zipUrl -DestinationFolder $InstallPath
# Store the installed branch commit SHA and time installed
$Script:ResticStateInstalledVersion = $null
$Script:ResticStateInstalledDate = Get-Date
$Script:ResticStateInstalledBranchSHA = $latestCommitSHA
Set-State
Write-Host "Update successful. Local branch is now at commit $latestCommitSHA."
}
else {
Write-Error "Unsupported mode."
exit 1
}

65
devices/merganser/config/C/restic/windows.exclude Normal file
View File

@@ -0,0 +1,65 @@
# default excludes
# examples https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Utility/FilterGroups.cs
C:\hiberfil.sys
C:\pagefile.sys
C:\swapfile.sys
C:\$Recycle.Bin
C:\autoexec.bat
C:\Config.Msi
C:\Documents and Settings
C:\Recycled
C:\Recycler
C:\$$Recycle.Bin
C:\System Volume Information
C:\Recovery
C:\Program Files
C:\Program Files (x86)
C:\ProgramData
C:\PerfLogs
C:\Windows
C:\Windows.old
C:\$$WINDOWS.~BT
C:\$$WinREAgent
Microsoft\Windows\Recent
Microsoft\**\RecoveryStore*
Microsoft\**\Windows\*.edb
Microsoft\**\Windows\*.log
Microsoft\**\Windows\Cookies*
MSOCache
NTUSER*
ntuser*
UsrClass.dat
# cloud services
Dropbox
AppData\Local\Google\Drive
Google Drive\.tmp.drivedownload
C:\OneDriveTemp
Users\**\Nextcloud
# browsers
Google\Chrome
# AppData
AppData\Local\Microsoft
AppData\Local\Duplicati
AppData\Local\D3DSCache
AppData\Local\ConnectedDevicesPlatform
AppData\Local\Packages
AppData\Roaming\Signal
AppData\Local\ElevatedDiagnostics
AppData\Local\restic
AppData\LocalLow\Microsoft\CryptnetUrlCache
AppData\Local\IsolatedStorage
AppData\Local\Spotify
AppData\Local\Programs\signal-desktop
AppData\Roaming\Code
AppData\Roaming\Slack
AppData\Roaming\Spotify
AppData\Roaming\Zoom
# misc. temporary files
Temporary Internet Files
Thumbs.db
AppData\Local\Temp
Users\Public\AccountPictures