feat(server): near-duplicate detection (#8228)

* duplicate detection job, entity, config

* queueing

* job panel, update api

* use embedding in db instead of fetching

* disable concurrency

* only queue visible assets

* handle multiple duplicateIds

* update concurrent queue check

* add provider

* add web placeholder, server endpoint, migration, various fixes

* update sql

* select embedding by default

* rename variable

* simplify

* remove separate entity, handle re-running with different threshold, set default back to 0.02

* fix tests

* add tests

* add index to entity

* formatting

* update asset mock

* fix `upsertJobStatus` signature

* update sql

* formatting

* default to 0.03

* optimize clustering

* use asset's `duplicateId` if present

* update sql

* update tests

* expose admin setting

* refactor

* formatting

* skip if ml is disabled

* debug trash e2e

* remove from web

* remove from sidebar

* test if ml is disabled

* update sql

* separate duplicate detection from clip in config, disable by default for now

* fix doc

* lower minimum `maxDistance`

* update api

* Add and Use Duplicate Detection Feature Flag (#9364)

* Add Duplicate Detection Flag

* Use Duplicate Detection Flag

* Attempt Fixes for Failing Checks

* lower minimum `maxDistance`

* fix tests

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>

* chore: fixes and additions after rebase

* chore: update api (remove new Role enum)

* fix: left join smart search so getAll works without machine learning

* test: trash e2e go back to checking length of assets is zero

* chore: regen api after rebase

* test: fix tests after rebase

* redundant join

---------

Co-authored-by: Nicholas Flamy <30300649+NicholasFlamy@users.noreply.github.com>
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack@futo.org>

mobile/openapi/test/all_job_status_response_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // JobStatusDto duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // JobStatusDto faceDetection
     test('to test the property `faceDetection`', () async {
       // TODO

mobile/openapi/test/asset_api_test.dart

@@ -50,6 +50,11 @@ void main() {
       // TODO
     });
 
+    //Future<List<AssetResponseDto>> getAssetDuplicates() async
+    test('test getAssetDuplicates', () async {
+      // TODO
+    });
+
     //Future<AssetResponseDto> getAssetInfo(String id, { String key }) async
     test('test getAssetInfo', () async {
       // TODO

mobile/openapi/test/duplicate_detection_config_test.dart

@@ -0,0 +1,32 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+import 'package:openapi/api.dart';
+import 'package:test/test.dart';
+
+// tests for DuplicateDetectionConfig
+void main() {
+  // final instance = DuplicateDetectionConfig();
+
+  group('test DuplicateDetectionConfig', () {
+    // bool enabled
+    test('to test the property `enabled`', () async {
+      // TODO
+    });
+
+    // double maxDistance
+    test('to test the property `maxDistance`', () async {
+      // TODO
+    });
+
+
+  });
+
+}

mobile/openapi/test/server_features_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // bool duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // bool email
     test('to test the property `email`', () async {
       // TODO

mobile/openapi/test/system_config_machine_learning_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // DuplicateDetectionConfig duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // bool enabled
     test('to test the property `enabled`', () async {
       // TODO