Commit graph

21 commits

Author SHA1 Message Date
Andrey Antukh
0e92bcc0de 🎉 Add file-data offload mechanism 2024-08-09 14:28:18 +02:00
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
c64e14859c Simplify internal executor module 2023-11-27 14:25:12 +01:00
Andrey Antukh
dfdc9c9fa5 ♻️ Refactor storage internal concurrency model 2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
e657c1bbfa Start use datoteka.io ns 2022-09-09 12:19:49 +02:00
Andrey Antukh
ebcb385593 ♻️ Minor refactor on storages
Fix many issues on FS & S3 backend; removes the unused and broken
DB backend. Normalize operations on bytes and byte streams on a
separated namespace: app.util.bytes
2022-06-22 11:37:45 +02:00
Andrey Antukh
7f3c67724e 🐛 Fix svg media asset upload internal server error 2022-03-20 13:04:12 +01:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
219f9c478d 🎉 Add version 4 of blob encoding.
The version 4 starts using the hight performance fressian
binary encoding with very lightweight compression layer.
2021-12-21 16:19:25 +01:00
Andrey Antukh
2728fa2b8d Add proper fdata objects deletion. 2021-06-15 09:25:37 +02:00
Andrey Antukh
0c97a44a2a 🎉 Add file offloading to external storage mechanism. 2021-06-14 15:41:27 +02:00
Andrey Antukh
e15a212b14 🎉 Add dashboard custom fonts management. 2021-05-04 14:21:31 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
784a4f8ecd Add some type hints and remove legacy code. 2021-02-18 09:35:37 +01:00
Andrey Antukh
b80295a21c Fix all linter issues on backend code. 2021-02-01 09:48:28 +01:00
Andrey Antukh
a69a35a0b6 Improve storage recheck task and add more specs. 2021-01-26 11:13:21 +01:00
Andrey Antukh
d494e44df3 🎉 Add builtin copy fast path operation for storage. 2021-01-19 15:01:33 +01:00
Andrey Antukh
ab944fb9ae ♻️ Integrate new storage subsystem. 2021-01-07 11:29:57 +01:00
Andrey Antukh
760eb926bf 🎉 Add plugable storages abstraction layer (with support for fs, s3 and db). 2021-01-07 11:29:57 +01:00