Commit Graph

19 Commits

Author SHA1 Message Date
Daniel Dietzler 84f7ca855a chore(server): move domain interfaces (#8124)
move domain interfaces
2024-03-20 16:42:58 -04:00
Daniel Dietzler 30f499cf2e chore(server): use absolute import paths (#8080)
update server to use absolute import paths
2024-03-20 14:32:04 -04:00
Jonathan Jogenfors 5d377e5b0f chore(server): eslint await-thenable (#7545)
* await-thenable

* fix library watchers

* moar eslint

* fix test

* fix typo

* try to remove check void return

* fix checksVoidReturn

* move to domain utils

* remove eslint ignores

* chore: cleanup types

* chore: use logger

* fix: e2e

---------

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-03-05 17:23:06 -05:00
Mert fb18129843 fix(server): truncate embedding tables (#7449)
truncate
2024-02-27 09:24:23 -06:00
Mert e334443919 feat(server, web): smart search filtering and pagination (#6525)
* initial pagination impl

* use limit + offset instead of take + skip

* wip web pagination

* working infinite scroll

* update api

* formatting

* fix rebase

* search refactor

* re-add runtime config for vector search

* fix rebase

* fixes

* useless omitBy

* unnecessary handling

* add sql decorator for `searchAssets`

* fixed search builder

* fixed sql

* remove mock method

* linting

* fixed pagination

* fixed unit tests

* formatting

* fix e2e tests

* re-flatten search builder

* refactor endpoints

* clean up dto

* refinements

* don't break everything just yet

* update openapi spec & sql

* update api

* linting

* update sql

* fixes

* optimize web code

* fix typing

* add page limit

* make limit based on asset count

* increase limit

* simpler import
2024-02-12 20:50:47 -05:00
Mert ae7f174948 refactor: rename clip -> smart search (#6713) 2024-01-29 09:51:22 -05:00
Michael Manganiello 4a5b8c3770 feat(server): Enqueue jobs in bulk (#5974)
* feat(server): Enqueue jobs in bulk

The Job Repository now has a `queueAll` method, that enqueues messages
in bulk (using BullMQ's
[`addBulk`](https://docs.bullmq.io/guide/queues/adding-bulks)),
improving performance when many jobs must be enqueued within the same
operation.

Primary change is in `src/domain/job/job.service.ts`, and other services
have been refactored to use `queueAll` when useful.

As a simple local benchmark, triggering a full thumbnail generation
process over a library of ~1,200 assets and ~350 faces went from
**~600ms** to **~250ms**.

* fix: Review feedback
2024-01-01 15:45:42 -05:00
Mert 8119d4bb26 chore(server): refactor locks (#5953)
* lock refactor

* add mocks

* add await

* move database repo injection to service

* update tests

* add mock implementation

* remove unused imports

* this
2023-12-27 18:36:51 -05:00
Mert 092a23fd7f feat(server,ml): remove image tagging (#5903)
* remove image tagging

* updated lock

* fixed tests, improved logging

* be nice

* fixed tests
2023-12-20 20:47:56 -05:00
Mert 458257847e fix(server): disable classification by default (#5708)
* disable classification by default

* fixed tests
2023-12-14 23:34:18 -06:00
Jason Rasmussen 1e99ba8167 feat: use pgvecto.rs (#3605) 2023-12-08 11:15:46 -05:00
Mert 87a0ba3db3 feat(ml): export clip models to ONNX and host models on Hugging Face (#4700)
* export clip models

* export to hf

refactored export code

* export mclip, general refactoring

cleanup

* updated conda deps

* do transforms with pillow and numpy, add tokenization config to export, general refactoring

* moved conda dockerfile, re-added poetry

* minor fixes

* updated link

* updated tests

* removed `requirements.txt` from workflow

* fixed mimalloc path

* removed torchvision

* cleaner np typing

* review suggestions

* update default model name

* update test
2023-10-31 05:02:04 -05:00
Jason Rasmussen 2ea080cacd refactor: domain repositories (#4403) 2023-10-09 14:25:03 +00:00
Daniel Dietzler 7173af60e4 chore(server): Improve test coverage! (#3889)
* tests for person service

* tests for auth service

* tests for access core

* improve tests for album service

* fix missing brackets and remove comments

* tests for asset service

* tests for face recognition

* tests for job service

* feedback

* tests for search service (broken)

* fix: disabled search test

* tests for smart-info service

* tests for storage template service

* tests for user service

* fix formatting of untouched files LOL

* attempt to fix formatting

* streamline api utils, add asset api for uploading files

* test upload of assets

* fix formatting

* move test-utils to correct folder

* test add assets to album

* use random bytes instead of test image

* (e2e) test albums with assets

* (e2e) complete tests for album endpoints

* (e2e) tests for asset endpoint

* fix: asset upload/import dto validation

* (e2e) tests for statistics asset endpoint

* fix wrong describe text

* (e2e) tests for people with faces

* (e2e) clean up person tests

* (e2e) tests for partner sharing endpoints

* (e2e) tests for link sharing

* (e2e) tests for the asset time bucket endpoint

* fix minor issues

* remove access.core.spec.ts

* chore: wording

* chore: organize test api files

* chore: fix test describe

* implement feedback

* fix race condition in album tests

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-11 11:56:38 -04:00
Mert bcc36d14a1 feat(ml)!: customizable ML settings (#3891)
* consolidated endpoints, added live configuration

* added ml settings to server

* added settings dashboard

* updated deps, fixed typos

* simplified modelconfig

updated tests

* Added ml setting accordion for admin page

updated tests

* merge `clipText` and `clipVision`

* added face distance setting

clarified setting

* add clip mode in request, dropdown for face models

* polished ml settings

updated descriptions

* update clip field on error

* removed unused import

* add description for image classification threshold

* pin safetensors for arm wheel

updated poetry lock

* moved dto

* set model type only in ml repository

* revert form-data package install

use fetch instead of axios

* added slotted description with link

updated facial recognition description

clarified effect of disabling tasks

* validation before model load

* removed unnecessary getconfig call

* added migration

* updated api

updated api

updated api

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-08-29 08:58:00 -05:00
Jason Rasmussen 8211afb726 feat(web,server)!: configure machine learning via the UI (#3768) 2023-08-25 06:15:03 +02:00
Jason Rasmussen 9e085c1071 refactor(server): test fixtures (#3491) 2023-07-31 20:28:07 -05:00
Mert 017214fd56 fix(server): empty tag responses should be considered valid (#2993)
* accept empty tag array

* renamed test
2023-06-28 09:40:30 -05:00
Jason Rasmussen 8ebac41318 refactor(server)*: tsconfigs (#2689)
* refactor(server): tsconfigs

* chore: dummy commit

* fix: start.sh

* chore: restore original entry scripts
2023-06-08 10:01:07 -05:00