3pp-mirror/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2025-05-24 03:16:11 +02:00
Code Issues Projects Releases Packages Wiki Activity

♻️ Fix some fundamental bugs on climit module

Browse source 
The climit previously of this commit is heavily used inside a
transactions, so in heavy contention operation such that file thumbnail
creation can cause a db pool exhaust.

This commit fixes this issue setting up a better resource limiting
mechanism that works outside the transactions so, contention will
no longer hold an open connection/transaction.

It also adds general improvement to the traceability to the climit
mechanism: it now properly logs the profile-id that is currently
cause some contention on specific resources.

It also add a general/root climit that is applied to all requests
so if someone start making abussive requests, we can clearly detect
it.
This commit is contained in:
Andrey Antukh 2024-01-27 22:33:52 +01:00
parent 658c26014b
commit a5c6d78ee5
12 changed files with 291 additions and 212 deletions
Download patch file Download diff file Expand all files Collapse all files

18
backend/src/app/rpc/commands/files_update.clj
View file

@ -35,7 +35,8 @@
[app.util.services :as sv]
[app.util.time :as dt]
[app.worker :as-alias wrk]
[clojure.set :as set]))
[clojure.set :as set]
[promesa.exec :as px]))
;; --- SCHEMA
@ -132,8 +133,8 @@
;; database.
(sv/defmethod ::update-file
{::climit/id :update-file/by-profile
::climit/key-fn ::rpc/profile-id
{::climit/id [[:update-file/by-profile ::rpc/profile-id]
[:update-file/global]]
::webhooks/event? true
::webhooks/batch-timeout (dt/duration "2m")
::webhooks/batch-key (webhooks/key-fn ::rpc/profile-id :id)
@ -232,13 +233,9 @@
(defn- update-file*
[{:keys [::db/conn ::wrk/executor] :as cfg}
{:keys [profile-id file changes session-id ::created-at skip-validate] :as params}]
(let [;; Process the file data in the CLIMIT context; scheduling it
;; to be executed on a separated executor for avoid to do the
;; CPU intensive operation on vthread.
update-fdata-fn (partial update-file-data cfg file changes skip-validate)
file (-> (climit/configure cfg :update-file/global)
(climit/run! update-fdata-fn executor))]
(let [;; Process the file data on separated thread for avoid to do
;; the CPU intensive operation on vthread.
file (px/invoke! executor (partial update-file-data cfg file changes skip-validate))]
(db/insert! conn :file-change
{:id (uuid/next)
@ -306,7 +303,6 @@
(fmg/migrate-file))
file)
;; WARNING: this ruins performance; maybe we need to find
;; some other way to do general validation
libs (when (and (or (contains? cf/flags :file-validation)