Commit graph

67 commits

Author SHA1 Message Date
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrey Antukh
c64e14859c Simplify internal executor module 2023-11-27 14:25:12 +01:00
Andrey Antukh
a23d1908e9 Improve worker logging 2023-09-04 11:35:31 +02:00
Andrey Antukh
a3495800b5 Add minor logging improvements to worker namespace 2023-08-31 21:09:18 +02:00
Andrey Antukh
d2d9aeff25 📎 Reduce log level of worker submit operation
Start logging to as TRACE instead of DEBUG
2023-08-31 20:59:58 +02:00
Andrey Antukh
5275c35002 🐛 Prevent rollback for idle-in-transaction errors on cron tasks 2023-08-24 11:18:56 +02:00
Andrey Antukh
259b05db51 Add more improvements to error reporting 2023-08-04 13:10:36 +02:00
Andrey Antukh
aafbf6bc15 ♻️ Refactor cocurrency model on backend
Mainly the followin changes:

- Pass majority of code to the old and plain synchronous style
  and start using virtual threads for the RPC (and partially some
  HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
  unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
2023-03-14 12:30:27 +01:00
Andrey Antukh
04b321caae Add several improvements to internal worker impl
Mainly for make the cron jobs do not block the scheduled executor
and offload all work to a separate threads
2023-03-14 12:30:27 +01:00
Andrey Antukh
cad1851e95 🔥 Replace own scheduled executor with the one defined in promesa lib 2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
071ecca875 🐛 Fix internal executor naming issue 2023-02-02 13:38:04 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
226afe98e0 💄 Change worker logging 2022-12-15 11:33:10 +01:00
Andrey Antukh
d7459db292 🎉 Add task deduplication by label 2022-12-13 23:13:11 +01:00
Andrey Antukh
c0a4b7dc76 Improve worker queue management
and add specific worker instance for webhooks
2022-12-13 16:17:31 +01:00
Andrey Antukh
8bad9d8340 ♻️ Refactor loggers/audit, auth/oidc, and http/clent modules 2022-12-05 08:53:00 +01:00
Andrey Antukh
7f7efc5760 Improve exception formating on backend 2022-12-05 08:51:54 +01:00
Andrey Antukh
c451c7bb9d 🐛 Fix regression on worker task handling 2022-11-30 13:13:11 +01:00
Andrey Antukh
0600b2abe4 ♻️ Make the worker abstraction more scalable
Start using redis for dispatcher to worker communication
and add the ability to start multiple threads to worker
for increase the concurrency.
2022-11-28 12:21:44 +01:00
Andrey Antukh
13a092b192 ♻️ Normalize internal naming on the worker module 2022-11-28 12:20:58 +01:00
luz paz
e30bea0b6f 🔧 Fix typos in source code
Found via `codespell -q 3 -S *.po,./frontend/yarn.lock -L childs,clen,fpr,inflight,ody,ot,ro,te,trys,ue`
2022-10-04 10:40:34 +02:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
9bfdcc6277 Make the task retry algorithm use better backoff values 2022-09-20 23:04:37 +02:00
Andrey Antukh
6f42f4ec45 ♻️ Refactor semaphore and executors 2022-09-20 11:32:45 +02:00
Andrey Antukh
7fa609d5f4 Allow disable worker 2022-08-12 08:52:36 +02:00
Andrey Antukh
f2140a1421 🐛 Fix cron scheduler locking mechanism
And add improved logging to the worker/cron code
2022-07-04 11:32:36 +02:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
d1c834e647 🐛 Fix minor issue on executors monitor 2022-03-01 14:34:13 +01:00
Andrey Antukh
e60b8a7aef 🐛 Minor fix on worker executors monitor 2022-02-28 17:21:36 +01:00
Andrey Antukh
5d2715dd32 Improve monitors monitor 2022-02-28 15:29:30 +01:00
Andrey Antukh
7cf27ac86d ♻️ Refactor general resource and concurrency model on backend 2022-02-22 13:05:41 +01:00
Andrey Antukh
7eed8c5ee5 Merge remote-tracking branch 'origin/main' into develop 2022-02-16 11:23:26 +01:00
Andrey Antukh
9b78b2a432 Improve error reporting on background tasks 2022-02-15 12:26:36 +01:00
Andrey Antukh
321b2c7c23 🐛 Fix error handling on s3 delete-in-bulk operation 2022-02-15 12:26:36 +01:00
Andrey Antukh
741d2b3f3c Merge remote-tracking branch 'origin/main' into develop 2022-02-12 17:33:28 +01:00
Andrey Antukh
827c2140b7 ♻️ Refactor error reporting and logging context formatting
The prev approach uses clojure.pprint without any limit extensivelly
for format error context data and all this is done on the calling
thread. The clojure.pprint seems very inneficient in cpu and memory
usage on pprinting large data structures.

This is improved in the following way:

- All formatting and pretty printing is moved to logging thread,
  reducing unnecesary blocking and load on jetty http threads.
- Replace the clojure.pprint with fipp.edn that looks considerably
  faster than the first one.
- Add some safe limits on pretty printer for strip printing some
  data when the data structure is very large, very deep or both.
2022-02-10 15:12:35 +01:00
Andrey Antukh
bc2a0432b9 Allow connect to read-only databases. 2022-01-27 16:11:32 +01:00
Andrey Antukh
2d118ecc65 Merge remote-tracking branch 'origin/staging' into develop 2021-12-27 09:50:15 +01:00
Andrey Antukh
1839397ebc Minor enhacements on log processing. 2021-12-23 18:36:58 +01:00
Josh Soref
cd2d3d5fa3 🐛 Fix typos in backend 2021-11-15 09:53:10 -05:00
Andrey Antukh
b64d5ef357 🎉 Add unified logging api. 2021-09-30 11:48:14 +02:00
Andrey Antukh
d7eec3b92b Merge remote-tracking branch 'origin/main' into develop 2021-06-03 12:56:37 +02:00
Andrey Antukh
bae709df5b 🐛 Fix custom font deletion task. 2021-06-03 12:55:31 +02:00
Andrey Antukh
548664f6ce ♻️ Internal directory refactor.
Make common as first-class module.
2021-05-31 11:04:32 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
8daf6e822e 🎉 Add profile activity registry logger. 2021-04-09 15:28:18 +02:00
Andrey Antukh
e12a6e65a6 ♻️ Refactor logging. 2021-04-09 15:28:18 +02:00
Andrey Antukh
0926fbcbc6 ♻️ Minor code reorganization.
Improves modularity and reusability and allows usage
of backend code as a library.
2021-03-31 09:28:15 +02:00
Andrey Antukh
fbe2e2a285 Improve tasks metrics. 2021-02-22 14:50:01 +01:00