Commit Graph

41 Commits

Author SHA1 Message Date
Mert 2daed747cd chore(server): change save -> update in asset repository (#8055)
* `save` -> `update`

* change return type

* include relations

* fix tests

* remove when mocks

* fix

* stricter typing

* simpler type
2024-03-19 22:42:10 -04:00
Daniel Dietzler a46366d336 chore(server): introduce proper job status (#7932)
* introduce proper job status

* fix condition for onDone jobs

* fix tests
2024-03-15 09:16:54 -04:00
Mert ee8e8a0c0f perf(server): optimize getByIds query (#7918)
* clean up usage

* i'm not updating all these tests

* update tests

* add indices

* add indices to entities

remove index from person entity

add to face entity

fix

* simplify query

* update sql

* missing await

* remove synchronize false
2024-03-14 01:58:09 -04:00
Fynn Petersen-Frey 3f1d37e556 feat(server): hardware HDR tonemapping for RKMPP (#7655)
* feat(server): hardware HDR tonemapping for RKMPP

* review feedback
2024-03-08 21:17:26 -05:00
Mert 5ff68d4cdb feat(server): only transcode streams that require it (#7106) 2024-02-14 11:24:39 -05:00
Jonathan Jogenfors f44fa45aa0 chore(server,cli,web): housekeeping and stricter code style (#6751)
* add unicorn to eslint

* fix lint errors for cli

* fix merge

* fix album name extraction

* Update cli/src/commands/upload.command.ts

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>

* es2k23

* use lowercase os

* return undefined album name

* fix bug in asset response dto

* auto fix issues

* fix server code style

* es2022 and formatting

* fix compilation error

* fix test

* fix config load

* fix last lint errors

* set string type

* bump ts

* start work on web

* web formatting

* Fix UUIDParamDto as UUIDParamDto

* fix library service lint

* fix web errors

* fix errors

* formatting

* wip

* lints fixed

* web can now start

* alphabetical package json

* rename error

* chore: clean up

---------

Co-authored-by: Ben McCann <322311+benmccann@users.noreply.github.com>
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-02-01 22:18:00 -05:00
Markus 87c38d1832 feat(server, web): Added TranscodePolicy "Bitrate higher than max bitrate or not in accepted format" (#6479)
* chore: rebase

* chore: open api

* Add Database-Migration for setting targetCodec as acceptedCodec if it was set by admin

* Add TranscodePolicy setting, to only transcode files with a bitrate higher than set max bitrate

* Rename enum value of TranscodePolicy

* calculate max_bitrate according to "k" and "m" suffix for comparison

* remove migration

* minor changes

* UnitTest for Bitrate Policy

* Fix UnitTest

* Add missing output options

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2024-01-30 20:25:07 -05:00
Markus 8aef92affc feat(server, web): accepted codecs (#6460)
* chore: rebase

* chore: open api

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2024-01-26 12:02:56 -05:00
Mert 68f52818ae feat(server): separate face clustering job (#5598)
* separate facial clustering job

* update api

* fixed some tests

* invert clustering

* hdbscan

* update api

* remove commented code

* wip dbscan

* cleanup

removed cluster endpoint

remove commented code

* fixes

updated tests

minor fixes and formatting

fixed queuing

refinements

* scale search range based on library size

* defer non-core faces

* optimizations

removed unused query option

* assign faces individually for correctness

fixed unit tests

remove unused method

* don't select face embedding

update sql

linting

fixed ml typing

* updated job mock

* paginate people query

* select face embeddings because typeorm

* fix setting face detection concurrency

* update sql

formatting

linting

* simplify logic

remove unused imports

* more specific delete signature

* more accurate typing for face stubs

* add migration

formatting

* chore: better typing

* don't select embedding by default

remove unused import

* updated sql

* use normal try/catch

* stricter concurrency typing and enforcement

* update api

* update job concurrency panel to show disabled queues

formatting

* check jobId in queueAll

fix tests

* remove outdated comment

* better facial recognition icon

* wording

wording

formatting

* fixed tests

* fix

* formatting & sql

* try to fix sql check

* more detailed description

* update sql

* formatting

* wording

* update `minFaces` description

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2024-01-18 00:08:48 -05:00
Michael Manganiello 4a5b8c3770 feat(server): Enqueue jobs in bulk (#5974)
* feat(server): Enqueue jobs in bulk

The Job Repository now has a `queueAll` method, that enqueues messages
in bulk (using BullMQ's
[`addBulk`](https://docs.bullmq.io/guide/queues/adding-bulks)),
improving performance when many jobs must be enqueued within the same
operation.

Primary change is in `src/domain/job/job.service.ts`, and other services
have been refactored to use `queueAll` when useful.

As a simple local benchmark, triggering a full thumbnail generation
process over a library of ~1,200 assets and ~350 faces went from
**~600ms** to **~250ms**.

* fix: Review feedback
2024-01-01 15:45:42 -05:00
Zack Pollard 2e38fa73bf feat: storage template file move hardening (#5917)
* fix: pgvecto.rs extension breaks typeorm schema:drop command

* fix: parse postgres bigints to javascript number types when selecting data

* feat: verify file size is the same as original asset after copying file for storage template job

* feat: allow disabling of storage template job, defaults to disabled for new instances

* fix: don't allow setting concurrency for storage template migration, can cause race conditions above 1

* feat: add checksum verification when file is copied for storage template job

* fix: extract metadata for assets that aren't visible on timeline
2023-12-29 18:41:33 +00:00
Mert a1e1f11399 feat(server): delete unnecessary encoded videos (#6027)
* delete unnecessary transcodes

* added test
2023-12-27 23:34:00 -06:00
Jason Rasmussen 9768931275 feat(web,server)!: runtime log level (#5672)
* feat: change log level at runtime

* chore: open api

* chore: prefer env over runtime

* chore: remove default env value
2023-12-14 16:55:40 +00:00
Fynn Petersen-Frey ce04e9e07a feat(server): hardware video acceleration for Rockchip SOCs via RKMPP (#4645)
* feat(server): hardware video acceleration for Rockchip SOCs via RKMPP

* add tests

* use LD_LIBRARY_PATH for custom ffmpeg

* incorporate review feedback

* code re-use for ffmpeg call

* review feedback
2023-10-30 09:39:37 -05:00
Daniel Dietzler 6b25435b4f refactor(server): make storage core singleton (#4608) 2023-10-23 11:52:21 -04:00
Daniel Dietzler 09bf1c9175 feat(server): harden move file (#4361)
Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-10-10 21:14:44 -05:00
Jason Rasmussen 2ea080cacd refactor: domain repositories (#4403) 2023-10-09 14:25:03 +00:00
Daniel Dietzler 0243570c0b fix(server): make system config core singleton (#4392)
* make system config core singleton

* refactor

* fix tests

* chore: fix tests

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-10-09 00:51:03 +00:00
Daniel Dietzler cc70f5f6a0 fix(server): Thumbnail migration creating unnecessary directories (#4251)
* fix: during migration folders will be created before checking if needed

* refactor
2023-09-27 15:29:07 -04:00
Jason Rasmussen 7bc6e9ef64 refactor(server): person thumbnail job (#4233)
* refactor(server): person thumbnail job

* fix(server): set feature photo
2023-09-26 14:03:22 +07:00
Mert 56cf9464af fix(server): use srgb pipeline for srgb images (#4101)
* added color-related exif fields

* remove metadata check, conditional pipe colorspace

* check exif metadata for srgb

* added migration

* updated e2e fixture

* uncased srgb check, search substrings

* extracted exif logic into separate function

* handle images with no bit depth or color metadata

* added unit tests
2023-09-25 19:18:47 -04:00
Daniel Dietzler 3053cbd4c8 chore(server): Store generated files (thumbnails, encoded video) in subdirectories (#4112)
* save thumbnails in subdirectories

* migration job, migrate assets and face thumbnails

* fix tests

* directory depth of two instead of three

* cleanup empty dirs after migration

* clean up empty dirs after migration, migrate people without assetId

* add job card for new migration job

* fix removeEmptyDirs race condition because of missing await

* cleanup empty directories after asset deletion

* move ensurePath to storage core

* rename jobs

* remove unnecessary property of IEntityJob

* use updated person getById, minor refactoring

* ensure that directory cleanup doesn't interfere with migration

* better description for job in ui

* fix remove directories when migration is done

* cleanup empty folders at start of migration

* fix: actually persist concurrency setting

* add comment explaining regex

* chore: cleanup

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-09-25 11:07:21 -04:00
Daniele Ricci 3432b4625f fix(server): regenerate missing person thumbnails (#3970)
* Regenerate missing person thumbnails

* Check for empty string instead of zero length

* Remember asset used as person face

* Define entity relation between person and asset via faceAssetId

* Typo

* Fix entity relation

* Tests

* Tests

* Fix code formatting

* Fix import path

* Fix migration

* format

* Fix entity and migration

* Linting

* Remove unneeded cast

* Conventions

* Simplify queries

* Simplify queries

* Remove unneeded typings from entity

* Remove unneeded cast

---------

Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-09-08 13:49:43 +07:00
Mert 04d4a30471 fix(server): await thumbnail generation before returning (#3975)
* await sharp command, minor fixes

* removed outdated test
2023-09-04 19:24:55 -04:00
Jason Rasmussen 816d040d81 fix(server): lint import order (#3974)
* fix: use prettier extension

* chore: format fix
2023-09-04 21:45:59 +02:00
Mert 2069293cc1 feat(server): wide gamut thumbnails (#3658) 2023-09-03 02:21:51 -04:00
Mert e510e733cd fix(server): extract motion photo android single frame (#3903) 2023-08-29 04:01:42 -05:00
Alex ddd4ec2d9e feat(web/server): webp thumbnail size configurable (#3598)
* feat(server/web): webp thumbnail size configurable

* update api

* add ui and fix test

* lint

* setting for jpeg size

* feat: coerce to number

* api

* jpeg resolution

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-08-08 09:39:51 -05:00
Mert 1d37d8cac0 feat(server): tone-mapping (#3512)
* added tonemapping

* check for hdr in transcode policy

* merged video thumbnail and transcoding logic

* updated tests

* removed log

* added dashboard option, updated api

* `out_color_matrix` for sdr video thumbs, cleanup

* updated tests & styling

* refactored tonemapping setting

* fixed tests

* formatting

* added tests

* updated api

* set target npl higher for mobius and reinhard

* convert to nv12 before nvenc

* fix test

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-08-07 15:35:25 -05:00
Mert ee49f470b7 feat(server): transcoding hardware acceleration (#3171)
* added transcode configs for nvenc,qsv and vaapi

* updated dev docker compose

* added software fallback

* working vaapi

* minor fixes and added tests

* updated api

* compile libvips

* move hwaccel settings to `hwaccel.yml`

* changed default dockerfile, moved `readdir` call

* removed unused import

* minor cleanup

* fix for arm build

* added documentation, minor fixes

* added intel driver

* updated docs

styling

* uppercase codec and api names

* formatting

* added tests

* updated docs

* removed semicolons

* added link to `hwaccel.yml`

* added newlines

* added `hwaccel` section to docker-compose.prod.yml

* ensure mesa drivers are installed

* switch to mimalloc for sharp

* moved build version and sha256 to json

* let libmfx set the render device

* possible fix for vp9 on qsv

* updated tests

* formatting

* review suggestions

* semicolon

* moved `LD_PRELOAD` to start script

* switched to jellyfin's ffmpeg package

* fixed dockerfile

* use cqp instead of icq for qsv vp9

* updated dockerfile

* added sha256sum for other platforms

* fixtures
2023-08-01 20:56:10 -05:00
Jason Rasmussen 6180828ed2 refactor(server): mime types (#3197)
* refactor(server): mime type check

* chore: open api

* chore: remove duplicate test
2023-07-10 12:56:45 -05:00
kasgel 50f26374e3 fix(server): enable transcoding of audioless videos (#3147)
* Fix: enable transcoding of audioless videos

* Fix: enable transcoding of audioless videos & typing

* Fix: enable transcoding of audioless videos & formatting

* fix: do not always transcode if there is no audio stream
2023-07-09 15:15:34 -05:00
Mert 8349a28ed8 refactor(server): modularize getFfmpegOptions (#3138)
* refactored `getFfmpegOptions`

refactor transcoding, make separate service

* fixed enum casing

* use `Logger` instead of `console.log`

* review suggestions

* use enum for `getHandler`

* fixed formatting

* Update server/src/domain/media/media.util.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* Update server/src/domain/media/media.util.ts

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

* More specific imports, renamed codec classes

* simplified code

* removed unused import

* added tests

* added base implementation for bitrate and threads

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>
2023-07-08 21:43:11 -05:00
Mert 71a2914f3e fix(server): transcodes failing due to storage migration happening simultaneously (#3071) 2023-07-05 00:36:16 -05:00
Mert f9671dfbf7 fix(server): h264 videos failing to transcode in two-pass mode (#3053)
* set `-fps_mode` to passthrough

* updated tests
2023-06-30 20:48:40 -05:00
Mert b1fcf02d13 fix(server): h264 and hevc not respecting max bitrate (#3052)
* added `-bufsize` flag

* updated test
2023-06-30 20:48:05 -05:00
Covalent 3e804f16df feat(web,server): add thumbhash support (#2649)
* add thumbhash: server generation and web impl

* move logic to infra & use byta in db

* remove unnecesary logs

* update generated API and simplify thumbhash gen

* fix check errors

* removed unnecessary library and css tag

* style edits

* syntax mistake

* update server test, change thumbhash job name

* fix tests

* Update server/src/domain/asset/response-dto/asset-response.dto.ts

Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>

* add unit test, change migration date

* change to official thumbhash impl

* update call method to not use eval

* "generate missing" looks for thumbhash

* improve queue & improve syntax

* update syntax again

* update tests

* fix thumbhash generation

* consolidate queueing to avoid duplication

* cover all types of incorrect thumbnail cases

* split out jest tasks

* put back thumbnail duration loading for images without thumbhash

* Remove stray package.json

---------

Co-authored-by: Luke McCarthy <mail@lukehmcc.com>
Co-authored-by: Thomas <9749173+uhthomas@users.noreply.github.com>
Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-06-17 22:22:31 -05:00
Thomas 3b4f6edbdb don't fallback to exiftool for embedded image previews (#2747)
Given #2668 introduced support for imagemagick and libraw, this should no
longer be necessary which allow for reduced code footprint and complexity.

Fixes: #2744
2023-06-14 22:42:35 -05:00
Thomas 41c2c8b82d use imagemagick and libraw for raw image support (#2668)
* use imagemagick and libraw for raw image support

imagemagick and libraw have generally good support for raw images, including
Sony's ARW format. These tools should also allow Immich to support many more
image formats in future without any major code changes.

https://www.libraw.org/supported-cameras

I've tested and verified this change with .ARW files and other standard formats.

Fixes: #2156

* Add additional type for awr

* pr feedback

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-06-14 21:34:03 -05:00
Mert 9cdec62918 feat(server): option to transcode to original resolution (#2709)
* option to transcode to original resolution

* changed value for target res setting

* updated test, clarified scaling condition
2023-06-09 23:15:12 -05:00
Jason Rasmussen 8ebac41318 refactor(server)*: tsconfigs (#2689)
* refactor(server): tsconfigs

* chore: dummy commit

* fix: start.sh

* chore: restore original entry scripts
2023-06-08 10:01:07 -05:00