Commit graph

102 commits

Author SHA1 Message Date
Andrey Antukh
893f19fa5e Remove automatic cascade on file_change table fk constraint 2025-02-21 14:24:07 +01:00
Andrey Antukh
f871f88f30
♻️ Refactor file data migrations subsystem (#5692)
* ♻️ Refactor file data migrations subsystem

* 📎 Add backend scripts/run helper script
2025-01-31 13:37:41 +01:00
Andrey Antukh
349552879e 🔥 Remove already broken v2 migration script 2025-01-27 12:53:11 +01:00
Andrey Antukh
bfe9a2b545 🎉 Add migration-like script for execute media-refs fix 2025-01-27 12:53:11 +01:00
alonso.torres
b1dda02b47 Add mentions to notifications 2025-01-09 11:55:53 +01:00
Pablo Alba
cbc92e9f1e Add created-by to invitations, and an event related 2024-11-11 17:00:54 +01:00
Andrey Antukh
32126d1874 ♻️ Refactor file changes gc tasks
Make it more friendly with the current snapshoting mechanism
2024-10-30 13:39:38 +01:00
alonso.torres
ecb7f0a2f6 File history versions management 2024-10-29 14:23:35 +01:00
Eva Marco
043c4105db Add viewer only mode on webhook 2024-10-15 13:38:46 +02:00
Andrey Antukh
50df2279a7 🐛 Make the media cleaning on file-gc task aware of snapshots
It now takes in account the snapshots, and prevents
deletion of media files used in snapshots.
2024-09-03 14:50:17 +02:00
Andrey Antukh
ceaafdbb1c Add offload mechanism for file snapshots 2024-08-26 13:52:42 +02:00
Andrey Antukh
215148ca81 Add better fillfactor setting for storage_object and task tables 2024-08-23 15:09:58 +02:00
Pablo Alba
6169f5c2e8 🎉 New oops page with login and request access 2024-08-14 15:32:04 +02:00
Andrey Antukh
3219c150d4 Add better internal fillfactor setting for file table
Increasing the change for HOT updates on db for this heavy-update
table
2024-08-09 14:28:18 +02:00
Andrey Antukh
ba167f256b Add performance enhancements on telemetry related queries 2024-08-09 14:28:18 +02:00
Andrey Antukh
0e92bcc0de 🎉 Add file-data offload mechanism 2024-08-09 14:28:18 +02:00
Andrey Antukh
86a732600b Add naming consistency changes for file_data_fragment table 2024-08-07 16:34:39 +02:00
Alejandro Alonso
f0eff95e18 🐛 Fix v2 components migration script 2024-04-15 20:26:51 +02:00
Andrey Antukh
fa00fed694 🐛 Fix issue with v2 manual migration script 2024-04-11 13:29:33 +02:00
Andrey Antukh
c55ceb4bca Add automatic v2 migration process on startup 2024-04-10 15:31:49 +02:00
Andrey Antukh
763fc3532e Simplify local audit table
Remove unnecessary partitioning
2024-03-25 17:58:39 +01:00
Andrey Antukh
b718a282e0 ♻️ Add minor refactor to file migrations
Relevant changes:

- Add the ability to create migration in both directions, defaulting
  to identity if not provided
- Move the version attribute to file table column for to make it more
  accessible (previously it was on data blob)
- Reduce db update operations on file-update rpc method
2024-02-19 09:20:47 +01:00
Andrey Antukh
a71e7f7906 Remove partitioning from task table
Which causes strange random delays when some row is moved from one
partition to other. Also, there are evidences that partitioning is
not aporting real value here.
2024-02-06 17:23:18 +01:00
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
6d49e1cac5 🐛 Add missing index on file_tagged_object_thumbnail media_id field 2023-11-24 10:41:27 +01:00
Aitor
7951350762 🐛 Fix db table tagged thumbnails 2023-11-07 16:01:00 +01:00
Andrey Antukh
6f93b41920 🎉 Add features assignation for teams 2023-11-07 12:48:31 +01:00
Aitor
c28c55bf0b 🎉 Add tag property to thumbnails 2023-11-02 11:06:30 +01:00
Andrey Antukh
80bf7cc1e5 Merge remote-tracking branch 'origin/staging' into develop 2023-08-07 12:59:17 +02:00
Andrey Antukh
1190cf837b Add an internal approach to prevent xlog gc to remove file changes 2023-08-03 16:40:42 +02:00
Andrey Antukh
1bfc28f63d Add missing index on server_error_report table 2023-08-02 13:43:53 +02:00
Andrey Antukh
e8ffcbae69 🎉 Add support for multipart upload of thumbnails
and improve the thumbnails storage to offloading it
to the storage subsystem
2023-05-05 17:00:35 +02:00
Alejandro Alonso
890583a13a Add mvp access-token support 2023-05-04 22:14:55 +02:00
Andrey Antukh
bb055a3c84 ♻️ Refactor logging subsystem and error reporting 2023-02-02 13:38:04 +01:00
Andrey Antukh
fa17ce5d40 📎 Avoid email index change on profile indexes migration 2023-01-23 09:56:21 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
bafe3ec087 Revert some changes related to admin that are no longer necessary 2023-01-13 10:19:39 +01:00
Andrey Antukh
73a3e0c0ae 🎉 Add usage quotes 2022-12-31 11:22:36 +01:00
Andrey Antukh
b929564fa7 ♻️ Add admin facilities on the code base
- Fix bugs related to orphan teams on profile deletion
- Separate session based profile-id param from api user provided
2022-12-22 16:42:45 +01:00
Andrey Antukh
d7459db292 🎉 Add task deduplication by label 2022-12-13 23:13:11 +01:00
Andrey Antukh
5b9f0ed0b1 🎉 Add webhook processing worker 2022-12-13 16:17:31 +01:00
Andrey Antukh
39b9daa3a7 🎉 Add webhooks rpc API 2022-12-05 15:20:29 +01:00
Andrey Antukh
76333cec26 🎉 Integrate storage/pointer-map file feature 2022-11-08 13:02:14 +01:00
Andrey Antukh
951b3eb4fe Integrate objects-map and introduce file feature flags 2022-10-18 15:49:18 +02:00
Andrey Antukh
69f084e1df Add deleted at index to file table 2022-10-18 15:49:18 +02:00
Andrey Antukh
757cee67fb 🎉 Add the ability to completly block access to a profile 2022-09-28 09:45:43 +02:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
b3623ed14c 🎉 Add migration for remove on cascade action on file-media-object table 2022-08-11 17:31:32 +02:00
Eva
1c446a011e Move comments 2022-08-01 09:53:55 +02:00
Andrey Antukh
dece149c9e 🎉 Add migration for fix legacy storage object backend names 2022-07-27 12:55:43 +02:00