feat(server): near-duplicate detection (#8228)

* duplicate detection job, entity, config * queueing * job panel, update api * use embedding in db instead of fetching * disable concurrency * only queue visible assets * handle multiple duplicateIds * update concurrent queue check * add provider * add web placeholder, server endpoint, migration, various fixes * update sql * select embedding by default * rename variable * simplify * remove separate entity, handle re-running with different threshold, set default back to 0.02 * fix tests * add tests * add index to entity * formatting * update asset mock * fix `upsertJobStatus` signature * update sql * formatting * default to 0.03 * optimize clustering * use asset's `duplicateId` if present * update sql * update tests * expose admin setting * refactor * formatting * skip if ml is disabled * debug trash e2e * remove from web * remove from sidebar * test if ml is disabled * update sql * separate duplicate detection from clip in config, disable by default for now * fix doc * lower minimum `maxDistance` * update api * Add and Use Duplicate Detection Feature Flag (#9364) * Add Duplicate Detection Flag * Use Duplicate Detection Flag * Attempt Fixes for Failing Checks * lower minimum `maxDistance` * fix tests --------- Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com> * chore: fixes and additions after rebase * chore: update api (remove new Role enum) * fix: left join smart search so getAll works without machine learning * test: trash e2e go back to checking length of assets is zero * chore: regen api after rebase * test: fix tests after rebase * redundant join --------- Co-authored-by: Nicholas Flamy <30300649+NicholasFlamy@users.noreply.github.com> Co-authored-by: Zack Pollard <zackpollard@ymail.com> Co-authored-by: Zack Pollard <zack@futo.org>
2024-05-16 13:08:37 -04:00
parent 673e97e71d
commit 64636c0618
61 changed files with 1254 additions and 61 deletions
@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **backgroundTask** | [**JobStatusDto**](JobStatusDto.md) |  | 
+**duplicateDetection** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **faceDetection** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **facialRecognition** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **library_** | [**JobStatusDto**](JobStatusDto.md) |  | 
@@ -14,6 +14,7 @@ Method | HTTP request | Description
 [**deleteAssets**](AssetApi.md#deleteassets) | **DELETE** /asset | 
 [**getAllAssets**](AssetApi.md#getallassets) | **GET** /asset | 
 [**getAllUserAssetsByDeviceId**](AssetApi.md#getalluserassetsbydeviceid) | **GET** /asset/device/{deviceId} | 
+[**getAssetDuplicates**](AssetApi.md#getassetduplicates) | **GET** /asset/duplicates | 
 [**getAssetInfo**](AssetApi.md#getassetinfo) | **GET** /asset/{id} | 
 [**getAssetStatistics**](AssetApi.md#getassetstatistics) | **GET** /asset/statistics | 
 [**getAssetThumbnail**](AssetApi.md#getassetthumbnail) | **GET** /asset/thumbnail/{id} | 
@@ -324,6 +325,43 @@ Name | Type | Description  | Notes

 [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)

+# **getAssetDuplicates**
+> List<AssetResponseDto> getAssetDuplicates()
+
+
+
+### Example
+```dart
+import 'package:openapi/api.dart';
+
+final api_instance = AssetApi();
+
+try {
+    final result = api_instance.getAssetDuplicates();
+    print(result);
+} catch (e) {
+    print('Exception when calling AssetApi->getAssetDuplicates: $e\n');
+}
+```
+
+### Parameters
+This endpoint does not need any parameter.
+
+### Return type
+
+[**List<AssetResponseDto>**](AssetResponseDto.md)
+
+### Authorization
+
+No authorization required
+
+### HTTP request headers
+
+ - **Content-Type**: Not defined
+ - **Accept**: application/json
+
+[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
+
 # **getAssetInfo**
 > AssetResponseDto getAssetInfo(id, key)

@@ -0,0 +1,16 @@
+# openapi.model.DuplicateDetectionConfig
+
+## Load the model package
+```dart
+import 'package:openapi/api.dart';
+```
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**enabled** | **bool** |  | 
+**maxDistance** | **double** |  | 
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **configFile** | **bool** |  | 
+**duplicateDetection** | **bool** |  | 
 **email** | **bool** |  | 
 **facialRecognition** | **bool** |  | 
 **map** | **bool** |  | 
@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **clip** | [**CLIPConfig**](CLIPConfig.md) |  | 
+**duplicateDetection** | [**DuplicateDetectionConfig**](DuplicateDetectionConfig.md) |  | 
 **enabled** | **bool** |  | 
 **facialRecognition** | [**RecognitionConfig**](RecognitionConfig.md) |  | 
 **url** | **String** |  |