Merge pull request #42 from kmwoley/release_1_4_1

Release 1.4.1

CHANGELOG.md

@@ -1,16 +1,76 @@
 # Changelog
 
-## [1.2.1](https://github.com/kmwoley/restic-windows-backup/tree/HEAD) (2020-06-08)
+## [1.4.1](https://github.com/kmwoley/restic-windows-backup/tree/1.4.1) (2021-05-29)
+[Full Changelog](https://github.com/kmwoley/restic-windows-backup/compare/1.4...1.4.1)
 
-[Full Changelog](https://github.com/kmwoley/restic-windows-backup/compare/1.1...HEAD)
+Bugfix release.
 
-* Fix/improve internet connectivity checks for azure: gs: b2:
-* Internet connectivity test now supports more repository types (s3:, sftp:, rest:, azure:, gs:), and ignores unsupported (swift:, rclone: and local) 
-* Add 32-bit support in the `install.ps1`
+## Fixes
+- Improved URL parsing so that the internet connectivity check works if the URL doesn't provide a protocol
+- Add PowerShell 7.1 support to internet connectivity check
 
+## Enhancements
+- Setting $InternetTestAttempts to 0 will now bypass the internet connectivity checks entirely
+
+## [1.4](https://github.com/kmwoley/restic-windows-backup/tree/1.4) (2021-02-24)
+[Full Changelog](https://github.com/kmwoley/restic-windows-backup/compare/1.3...1.4)
+
+Moved to using Restic's inbuilt filesystem shadow copy creation (VSS).
+
+## Breaking Change
+`local.exclude` file changes that previously referenced the `resticVSS` directory will need to be changed to `C:\` or the relevant root drive letter.
+
+## Other enhancements
+- Future snapshot grouping (and cleanup) will be better since the root-level folders included in the backup won't change (instead, the script targets the root drive letter instead of a list of folders under the drive letter).
+- Added the ability to set prune parameters via `.\config.ps1`, and defaulted the settings to `--group-by host` to clean up the aforementioned snapshot grouping & pruning.
+- Updated the `windows.exclude` to include additional directories (most notably, the Recycle Bin is no longer backed up) 
 
 **Closed issues:**
 
+- Remove VSS Operations, Switch to `--use-fs-snapshot` [\#32](https://github.com/kmwoley/restic-windows-backup/issues/32)
+- powershell execution policy is blocking the scheduled task [\#27](https://github.com/kmwoley/restic-windows-backup/issues/27)
+- VSS Cleanup Upon Errors [\#8](https://github.com/kmwoley/restic-windows-backup/issues/8)
+
+**Merged pull requests:**
+
+- Release 1.4 [\#33](https://github.com/kmwoley/restic-windows-backup/pull/33) ([kmwoley](https://github.com/kmwoley))
+
+## [1.3](https://github.com/kmwoley/restic-windows-backup/tree/1.3) (2021-02-23)
+
+[Full Changelog](https://github.com/kmwoley/restic-windows-backup/compare/1.2.1...1.3)
+
+Improvements for Restic 0.12 and additional error logging.
+
+**Closed issues:**
+
+- backup errors after update to restic 0.12.0 due to --quiet and --verbose being used simultaneously [\#29](https://github.com/kmwoley/restic-windows-backup/issues/29)
+- Restic + rclone errors [\#26](https://github.com/kmwoley/restic-windows-backup/issues/26)
+- E-Mail sending errors are not logged [\#25](https://github.com/kmwoley/restic-windows-backup/issues/25)
+- FYI: Restic now has built-in VSS support [\#23](https://github.com/kmwoley/restic-windows-backup/issues/23)
+- SFTP backup [\#22](https://github.com/kmwoley/restic-windows-backup/issues/22)
+- Dirrectory/Folder Backup [\#21](https://github.com/kmwoley/restic-windows-backup/issues/21)
+- Docker format [\#20](https://github.com/kmwoley/restic-windows-backup/issues/20)
+- Filtering out errors before deciding to retry ? [\#19](https://github.com/kmwoley/restic-windows-backup/issues/19)
+- Backup task stucked [\#18](https://github.com/kmwoley/restic-windows-backup/issues/18)
+
+**Merged pull requests:**
+
+- Release 1.4 [\#31](https://github.com/kmwoley/restic-windows-backup/pull/31) ([kmwoley](https://github.com/kmwoley))
+- Add '-ExecutionPolicy Bypass' to the task scheduler arguments to avoi… [\#28](https://github.com/kmwoley/restic-windows-backup/pull/28) ([scelfo](https://github.com/scelfo))
+- Fix URI parsing [\#24](https://github.com/kmwoley/restic-windows-backup/pull/24) ([Phlogi](https://github.com/Phlogi))
+
+## [1.2.1](https://github.com/kmwoley/restic-windows-backup/tree/1.2.1) (2020-06-08)
+
+[Full Changelog](https://github.com/kmwoley/restic-windows-backup/compare/1.1...1.2.1)
+
+* Internet connectivity test now supports more repository types (s3:, sftp:, rest:, azure:, gs:), and ignores unsupported (swift:, rclone: and local) 
+* Add 32-bit support in the `install.ps1`
+
+* Fix/improve internet connectivity checks for azure: gs: b2:
+
+**Closed issues:**
+
+- azure repo could not be parsed [\#15](https://github.com/kmwoley/restic-windows-backup/issues/15)
 - Need to strip rest: in addition to s3: from RESTIC\_REPOSITORY [\#14](https://github.com/kmwoley/restic-windows-backup/issues/14)
 - Use non-s3 repos [\#10](https://github.com/kmwoley/restic-windows-backup/issues/10)
 - Test-Connection fails [\#9](https://github.com/kmwoley/restic-windows-backup/issues/9)
@@ -19,6 +79,7 @@
 
 **Merged pull requests:**
 
+- Release 1 3 [\#17](https://github.com/kmwoley/restic-windows-backup/pull/17) ([kmwoley](https://github.com/kmwoley))
 - 1.2 Release [\#13](https://github.com/kmwoley/restic-windows-backup/pull/13) ([kmwoley](https://github.com/kmwoley))
 
 ## [1.1](https://github.com/kmwoley/restic-windows-backup/tree/1.1) (2020-02-15)

backup.ps1

@@ -67,7 +67,7 @@ function Invoke-Maintenance {
 
     # forget snapshots based upon the retention policy
     Write-Output "[[Maintenance]] Start forgetting..." | Tee-Object -Append $SuccessLog
-    & $ResticExe --verbose -q forget $SnapshotRetentionPolicy 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
+    & $ResticExe forget $SnapshotRetentionPolicy 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
     if(-not $?) {
         Write-Output "[[Maintenance]] Forget operation completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog
         $maintenance_success = $false
@@ -76,7 +76,7 @@ function Invoke-Maintenance {
     # prune (remove) data from the backup step. Running this separate from `forget` because
     #   `forget` only prunes when it detects removed snapshots upon invocation, not previously removed
     Write-Output "[[Maintenance]] Start pruning..." | Tee-Object -Append $SuccessLog
-    & $ResticExe --verbose -q prune 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
+    & $ResticExe prune $SnapshotPrunePolicy 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
     if(-not $?) {
         Write-Output "[[Maintenance]] Prune operation completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog
         $maintenance_success = $false
@@ -103,7 +103,7 @@ function Invoke-Maintenance {
         $Script:ResticStateLastDeepMaintenance = Get-Date
     }
 
-    & $ResticExe --verbose -q check @data_check 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
+    & $ResticExe check @data_check 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
     if(-not $?) {
         Write-Output "[[Maintenance]] Check completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog
         $maintenance_success = $false
@@ -113,7 +113,7 @@ function Invoke-Maintenance {
     
     if($maintenance_success -eq $true) {
         $Script:ResticStateLastMaintenance = Get-Date
-        $Script:ResticStateMaintenanceCounter = 0;
+        $Script:ResticStateMaintenanceCounter = 0
     }
 }
 
@@ -127,60 +127,38 @@ function Invoke-Backup {
     $starting_location = Get-Location
     ForEach ($item in $BackupSources.GetEnumerator()) {
 
-        $ShadowPath = Join-Path $item.Key 'resticVSS'
+        # Get the source drive letter and set as the root path
+        $root_path = $item.Key
 
-        # check for existance of previous, orphaned VSS directory (and remove it) before creating the shadow copy
-        if(Test-Path $ShadowPath) {
-            Write-Output "[[Backup]] VSS directory exists: '$ShadowPath' - removing. Past script failure?" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog
-            cmd /c rmdir $ShadowPath
-        }
-        
-        # Create the shadow copy
-        $s1 = (Get-WmiObject -List Win32_ShadowCopy).Create($item.Key, "ClientAccessible")
-        $s2 = Get-WmiObject -Class Win32_ShadowCopy | Where-Object { $_.ID -eq $s1.ShadowID }
-        
-        # Create a symbolic link to the shadow copy
-        $device  = $s2.DeviceObject + "\"
-        cmd /c mklink /d $ShadowPath "$device" 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
-
-        # Build the new list of folders
-        $root_path = $ShadowPath
+        # Avoid storing the drive letter in the backup path if only backing up a single drive
+        # FIXME: this doesn't really work. "C:\" still gets stored
         if($drive_count -eq 1) {
+            Set-Location $root_path
             $root_path = "."
-            Set-Location $ShadowPath
         }
-
+        
+        # Build the new list of folders from settings (if there are any)
         $folder_list = New-Object System.Collections.Generic.List[System.Object]
         ForEach ($path in $item.Value) {
-            $p = Join-Path $root_path $path
+            $p = '"{0}"' -f ((Join-Path $root_path $path) -replace "\\$")
             $folder_list.Add($p)
         }
 
         # backup everything in the root if no folders are provided
-        # note this won't select items with hidden attributes (a good thing to avoid)
         if (-not $folder_list) {
-            ForEach ($path in Get-ChildItem $ShadowPath) {
-                $p = Join-Path $root_path $path
-                $folder_list.Add($p)
-            }
+            $folder_list.Add($root_path)
         }
 
         # Launch Restic
-        & $ResticExe --verbose -q backup $folder_list --exclude-file=$WindowsExcludeFile --exclude-file=$LocalExcludeFile 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
+        & $ResticExe backup $folder_list --use-fs-snapshot --exclude-file=$WindowsExcludeFile --exclude-file=$LocalExcludeFile 3>&1 2>> $ErrorLog | Tee-Object -Append $SuccessLog
         if(-not $?) {
             Write-Output "[[Backup]] Completed with errors" | Tee-Object -Append $ErrorLog | Tee-Object -Append $SuccessLog
             $return_value = $false
         }
-
-        # Delete the shadow copy and remove the symbolic link
-        if($drive_count -eq 1) {
-            Set-Location $starting_location
-        }
-        $s2.Delete()
-        cmd /c rmdir $ShadowPath
-
-        Write-Output "[[Backup]] End $(Get-Date)" | Tee-Object -Append $SuccessLog
     }
+    
+    Set-Location $starting_location
+    Write-Output "[[Backup]] End $(Get-Date)" | Tee-Object -Append $SuccessLog
 
     return $return_value
 }
@@ -213,15 +191,33 @@ function Send-Email {
     }
     if((($status -eq "SUCCESS") -and ($SendEmailOnSuccess -ne $false)) -or ((($status -eq "ERROR") -or $success_after_failure) -and ($SendEmailOnError -ne $false))) {
         $subject = "$env:COMPUTERNAME Restic Backup Report [$status]"
-        Send-MailMessage @ResticEmailConfig -From $ResticEmailFrom -To $ResticEmailTo -Credential $credentials -Subject $subject -Body $body @attachments
+
+        # create a temporary error log to log errors; can't write to the same file that Send-MailMessage is reading
+        $temp_error_log = $ErrorLog + "_temp"
+
+        Send-MailMessage @ResticEmailConfig -From $ResticEmailFrom -To $ResticEmailTo -Credential $credentials -Subject $subject -Body $body @attachments 3>&1 2>> $temp_error_log
+
+        if(-not $?) {
+            Write-Output "[[Email]] Sending email completed with errors" | Tee-Object -Append $temp_error_log | Tee-Object -Append $SuccessLog
+        }
+
+        # join error logs and remove the temporary
+        Get-Content $temp_error_log | Add-Content $ErrorLog
+        Remove-Item $temp_error_log
     }
 }
 
 function Invoke-ConnectivityCheck {
     Param($SuccessLog, $ErrorLog)
+    
+    if($InternetTestAttempts -le 0) {
+        Write-Output "[[Internet]] Internet connectivity check disabled. Skipping." | Tee-Object -Append $SuccessLog    
+        return $true
+    }
+
     # skip the internet connectivity check for local repos
     if(Test-Path $env:RESTIC_REPOSITORY) {
-        Write-Output "[[Internet]] Skipping internet connectivity check." | Tee-Object -Append $SuccessLog    
+        Write-Output "[[Internet]] Local repository. Skipping internet connectivity check." | Tee-Object -Append $SuccessLog    
         return $true
     }
 
@@ -243,9 +239,13 @@ function Invoke-ConnectivityCheck {
     }
     else {
         # parse connection string for hostname
-        #   Uri parser doesn't handle leading connection type info (s3:, sftp:, rest:)
+        # Uri parser doesn't handle leading connection type info (s3:, sftp:, rest:)
         $connection_string = $env:RESTIC_REPOSITORY -replace "^s3:" -replace "^sftp:" -replace "^rest:"
-        $repository_host = ([System.Uri]$connection_string).host
+        if(-not ($connection_string -match "://")) {
+            # Uri parser expects to have a protocol. Add 'https://' to make it parse correctly.
+            $connection_string = "https://" + $connection_string
+        }
+        $repository_host = ([System.Uri]$connection_string).DnsSafeHost
     }
 
     if([string]::IsNullOrEmpty($repository_host)) {
@@ -266,7 +266,7 @@ function Invoke-ConnectivityCheck {
             Write-Output "[[Internet]] Waiting for internet connectivity... $sleep_count" | Tee-Object -Append $SuccessLog
             Start-Sleep 30
         }
-        elseif(!(Test-Connection -Server $repository_host -Quiet)) {
+        elseif(!(Test-Connection -ComputerName $repository_host -Quiet)) {
             Write-Output "[[Internet]] Waiting for connection to repository ($repository_host)... $sleep_count" | Tee-Object -Append $SuccessLog
             Start-Sleep 30
         }
@@ -337,15 +337,15 @@ function Invoke-Main {
             }
         }
 
-        Write-Warning "Errors found! Error Log: $error_log"
+        Write-Output "[[General]] Errors found. Log: $error_log" | Tee-Object -Append $success_log | Tee-Object -Append $error_log
         $error_count++
         
         $attempt_count--
         if($attempt_count -gt 0) {
-            Write-Output "Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log
+            Write-Output "[[Retry]] Sleeping for 15 min and then retrying..." | Tee-Object -Append $success_log
         }
         else {
-            Write-Output "Retry limit has been reached. No more attempts to backup will be made." | Tee-Object -Append $success_log
+            Write-Output "[[Retry]] Retry limit has been reached. No more attempts to backup will be made." | Tee-Object -Append $success_log
         }
         if($internet_available -eq $true) {
             Invoke-HistoryCheck $success_log $error_log
@@ -364,4 +364,4 @@ function Invoke-Main {
     exit $error_count
 }
 
-Invoke-Main
+Invoke-Main

config.ps1

@@ -12,7 +12,8 @@ $GlobalRetryAttempts = 4
 
 # maintenance configuration
 $SnapshotMaintenanceEnabled = $true
-$SnapshotRetentionPolicy = @("--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
+$SnapshotRetentionPolicy = @("--group-by", "host", "--keep-daily", "30", "--keep-weekly", "52", "--keep-monthly", "24", "--keep-yearly", "10")
+$SnapshotPrunePolicy = @("--max-unused", "1%")
 $SnapshotMaintenanceInterval = 7
 $SnapshotMaintenanceDays = 30
 $SnapshotDeepMaintenanceDays = 90;

install.ps1

@@ -5,10 +5,10 @@
 if(-not (Test-Path $ResticExe)) {
     $url = $null
     if([Environment]::Is64BitOperatingSystem){
-        $url = "https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_windows_amd64.zip"
+        $url = "https://github.com/restic/restic/releases/download/v0.12.0/restic_0.12.0_windows_amd64.zip"
     }
     else {
-        $url = "https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_windows_386.zip"
+        $url = "https://github.com/restic/restic/releases/download/v0.12.0/restic_0.12.0_windows_386.zip"
     }
     $output = Join-Path $InstallPath "restic.zip"
     Invoke-WebRequest -Uri $url -OutFile $output
@@ -43,7 +43,7 @@ $backup_task_name = "Restic Backup"
 $backup_task = Get-ScheduledTask $backup_task_name -ErrorAction SilentlyContinue
 if($null -eq $backup_task) {
     try {
-        $task_action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-NonInteractive -NoLogo -NoProfile -Command ".\backup.ps1; exit $LASTEXITCODE"' -WorkingDirectory $InstallPath
+        $task_action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-ExecutionPolicy Bypass -NonInteractive -NoLogo -NoProfile -Command ".\backup.ps1; exit $LASTEXITCODE"' -WorkingDirectory $InstallPath
         $task_user = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -RunLevel Highest
         $task_settings = New-ScheduledTaskSettingsSet -RestartCount 4 -RestartInterval (New-TimeSpan -Minutes 15) -ExecutionTimeLimit (New-TimeSpan -Days 3) -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -DontStopOnIdleEnd -MultipleInstances IgnoreNew -IdleDuration 0 -IdleWaitTimeout 0 -StartWhenAvailable -RestartOnIdle
         $task_trigger = New-ScheduledTaskTrigger -Daily -At 4:00am

windows.exclude

@@ -1,23 +1,25 @@
 # default excludes
 # examples https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Utility/FilterGroups.cs
-# note, because we're using a VSS directory, we can use that as the root dir for exclude rules (i.e. resticVSS)
-resticVSS\hiberfil.sys
-resticVSS\pagefile.sys
-resticVSS\swapfile.sys
-resticVSS\$Recycle.Bin
-resticVSS\autoexec.bat
-resticVSS\Config.Msi
-resticVSS\Documents and Settings
-resticVSS\Recycled
-resticVSS\Recycler
-resticVSS\System Volume Information
-resticVSS\Recovery
-resticVSS\Program Files
-resticVSS\Program Files (x86)
-resticVSS\ProgramData
-resticVSS\PerfLogs
-resticVSS\Windows
-resticVSS\Windows.old
+C:\hiberfil.sys
+C:\pagefile.sys
+C:\swapfile.sys
+C:\$Recycle.Bin
+C:\autoexec.bat
+C:\Config.Msi
+C:\Documents and Settings
+C:\Recycled
+C:\Recycler
+C:\$$Recycle.Bin
+C:\System Volume Information
+C:\Recovery
+C:\Program Files
+C:\Program Files (x86)
+C:\ProgramData
+C:\PerfLogs
+C:\Windows
+C:\Windows.old
+C:\$$WINDOWS.~BT
+C:\$$WinREAgent
 Microsoft\Windows\Recent
 Microsoft\**\RecoveryStore*
 Microsoft\**\Windows\*.edb
@@ -32,7 +34,7 @@ UsrClass.dat
 Dropbox
 AppData\Local\Google\Drive
 Google Drive\.tmp.drivedownload
-resticVSS\OneDriveTemp
+C:\OneDriveTemp
 
 # browsers
 Google\Chrome