Commit graph

199 commits

Author SHA1 Message Date
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrey Antukh
c64e14859c Simplify internal executor module 2023-11-27 14:25:12 +01:00
Andrey Antukh
1bd32327e5 🔥 Remove executor internal dependency on rpc routes module 2023-11-27 14:25:12 +01:00
Andrey Antukh
da7f88c7ca 🔥 Remove executor internal dependency on storage module 2023-11-27 14:25:12 +01:00
Andrey Antukh
97f8315cd0 🔥 Remove executor internal dependency from http assets module 2023-11-27 14:25:12 +01:00
Andrey Antukh
bc01afe158 🔥 Remove executor internal dependency from debug module 2023-11-27 14:25:12 +01:00
Andrey Antukh
d241b73940 🔥 Remove executor internal dependency on http client module 2023-11-27 14:25:12 +01:00
Andrey Antukh
ca6738d20c Remove executor dependency from awsns handlers 2023-11-27 14:25:12 +01:00
Andrey Antukh
54341d5b22 Make the RPC climit subsystem more robust 2023-11-27 14:25:12 +01:00
Andrey Antukh
bb5a4c0fa5 Update yetti and adapt for ring-2.0 2023-11-27 14:25:12 +01:00
Andrey Antukh
d10d8eed2b 🐛 Pass missing optimizer instance to rpc methods 2023-11-20 11:21:13 +01:00
Andrey Antukh
4c190e385e Make the s3 client io-threads configurable and adaptable 2023-11-15 13:49:26 +01:00
Andrey Antukh
99f39c9777 Add improved REPL support 2023-11-13 18:33:28 +01:00
Andrey Antukh
00afb841ac Add svg optimization on components migration process 2023-11-07 12:48:31 +01:00
Andrey Antukh
b70880420a Add improvements to dbg interface 2023-10-23 10:00:07 +02:00
Andrey Antukh
d0a422e8bd 💄 Add cosmetic improvement to backend main ns 2023-07-10 15:07:17 +02:00
Andrey Antukh
494c585e2f Make builtin templates download ondemand if cache is not present 2023-07-10 15:07:17 +02:00
Andrey Antukh
e8ffcbae69 🎉 Add support for multipart upload of thumbnails
and improve the thumbnails storage to offloading it
to the storage subsystem
2023-05-05 17:00:35 +02:00
Alejandro Alonso
408de63ea3 Merge remote-tracking branch 'origin/staging' into develop 2023-04-05 07:35:36 +02:00
Andrey Antukh
30e1c7d54e 🐛 Add missing storage dependency to dbg routes module 2023-04-04 19:20:10 +02:00
Andrey Antukh
aafbf6bc15 ♻️ Refactor cocurrency model on backend
Mainly the followin changes:

- Pass majority of code to the old and plain synchronous style
  and start using virtual threads for the RPC (and partially some
  HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
  unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
2023-03-14 12:30:27 +01:00
Andrey Antukh
2e717882f1 ♻️ Refactor websockets impl to use virtual threads
Removing the use of core.async code and implement code using
plain old and familiar synchronous code
2023-03-14 12:30:27 +01:00
Andrey Antukh
04b321caae Add several improvements to internal worker impl
Mainly for make the cron jobs do not block the scheduled executor
and offload all work to a separate threads
2023-03-14 12:30:27 +01:00
Andrey Antukh
cad1851e95 🔥 Replace own scheduled executor with the one defined in promesa lib 2023-03-14 12:30:27 +01:00
Andrey Antukh
d549fcb2ae 🐛 Pass a valid executor instance to yetti http server 2023-03-14 12:30:27 +01:00
Andrey Antukh
738cf6407c 📎 Fix liner issue 2023-02-25 10:24:22 +01:00
Andrey Antukh
1d21ee7089 Merge remote-tracking branch 'origin/staging' into develop 2023-02-24 18:30:05 +01:00
Andrey Antukh
f0c0e5e43a Add proper audit log for invitations 2023-02-24 10:28:07 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
4b4f78b4cc Add minor change to srepl module options
Replace unqualified attrs with fully qualified
2023-02-07 18:16:55 +01:00
Andrey Antukh
0c48f76911 Add better spec validation on http module 2023-02-07 18:16:55 +01:00
Andrey Antukh
bd2745d1fe ♻️ Clean email related namespaces
- Remove legacy and outdated mjml directory
- Rename namespace to a proper name
- Add more specs
2023-02-07 18:16:55 +01:00
Andrey Antukh
1325e46192 Improve internal state validation on db module 2023-02-02 14:20:13 +01:00
Andrey Antukh
bb055a3c84 ♻️ Refactor logging subsystem and error reporting 2023-02-02 13:38:04 +01:00
Andrey Antukh
2fef3dc881 🎉 Add prepl support
And rename the current repl to urepl (user-repl).
2023-01-19 15:41:12 +01:00
Andrey Antukh
8a5afefc1c 🎉 Add prepl support
And rename the current repl to urepl (user-repl).
2023-01-18 17:14:22 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
bafe3ec087 Revert some changes related to admin that are no longer necessary 2023-01-13 10:19:39 +01:00
Andrey Antukh
6cdf696fc4 🐛 Fix issues on ldap provider and rpc method 2023-01-09 11:57:13 +01:00
Andrey Antukh
853be27780 🐛 Fix issues on database logger 2023-01-09 11:43:26 +01:00
Andrey Antukh
b929564fa7 ♻️ Add admin facilities on the code base
- Fix bugs related to orphan teams on profile deletion
- Separate session based profile-id param from api user provided
2022-12-22 16:42:45 +01:00
Andrey Antukh
408d33bdec 🐛 Fix incorrect auth cookie decoding on first stage session middleware 2022-12-16 09:33:24 +01:00
Andrey Antukh
842463ed1b 🎉 Add the ability to copy team invitation link 2022-12-14 16:22:13 +01:00
Andrey Antukh
5b9f0ed0b1 🎉 Add webhook processing worker 2022-12-13 16:17:31 +01:00
Andrey Antukh
c0a4b7dc76 Improve worker queue management
and add specific worker instance for webhooks
2022-12-13 16:17:31 +01:00
Andrey Antukh
7f589b09ca ♻️ Move audit http handler to RPC 2022-12-13 16:17:31 +01:00
Andrey Antukh
39b9daa3a7 🎉 Add webhooks rpc API 2022-12-05 15:20:29 +01:00
Andrey Antukh
8bad9d8340 ♻️ Refactor loggers/audit, auth/oidc, and http/clent modules 2022-12-05 08:53:00 +01:00
Andrey Antukh
7f7efc5760 Improve exception formating on backend 2022-12-05 08:51:54 +01:00