diff --git a/docs/img/abstraction-levels/abstraction-levels.png b/docs/img/abstraction-levels/abstraction-levels.png new file mode 100644 index 000000000..fbb2c31ae Binary files /dev/null and b/docs/img/abstraction-levels/abstraction-levels.png differ diff --git a/docs/technical-guide/developer/abstraction-levels.md b/docs/technical-guide/developer/abstraction-levels.md new file mode 100644 index 000000000..40ba1c499 --- /dev/null +++ b/docs/technical-guide/developer/abstraction-levels.md @@ -0,0 +1,371 @@ +--- +title: 3.07. Abstraction levels +--- + +# Code organization in abstraction levels + +Initially, Penpot data model implementation was organized in a different way. +We are currently in a process of reorganization. The objective is to have data +manipulation code structured in abstraction layers, with well-defined +boundaries, and a hierarchical structure (each level may only use same or +lower levels, but not higher). + +![Abstraction levels](/img/abstraction-levels/abstraction-levels.png) + +At this moment the namespace structure is already organized as described here, +but there is much code that does not comply with these rules, and needs to be +moved or refactored. We expect to be refactoring existing modules incrementally, +each time we do an important functionality change. + +## Basic data + +```text +▾ common/ + ▾ src/app/common/ + data.cljc + ▾ src/app/data/ + macros.cljc +``` + +A level for generic data structures or operations, that are not specifically part +of the domain model (e.g. trees, strings, maps, iterators, etc.). Also may belong +here some functions in app.common.geom/ and app.common.files.helpers.cljc. + +We need to create a new directory for this and move there all functions in this +leve. + + +## Abstract data types + +```text +▾ common/ + ▾ src/app/common/ + ▾ types/ + file.cljc + page.cljc + shape.cljc + color.cljc + component.cljc + tokens_lib.cljc + ... +``` + +Namespaces here represent a single data entity of the domain model, or a +fragment of one, as an [Abstract Data Type](https://www.geeksforgeeks.org/abstract-data-types/). +An ADT is a data component that is defined through a series of properties and +operations, and that abstracts out the details of how it's implemented and what +is the internal structure. This allows to simplify the logic of the client +code, and also to make future changes in the ADT without affecting callers (if +the abstract interface does not change). + +Each structure in this module has: + +* A **schema spec** that defines the structure of the type and its values: + + ```clojure + (def ::fill + [:map {:title "Fill"} + [:fill-color {:optional true} ::ctc/rgb-color] + [:fill-opacity {:optional true} ::sm/safe-number] + ...) + + (def ::shape-attrs + [:map {:title "ShapeAttrs"} + [:name {:optional true} :string] + [:selrect {:optional true} ::grc/rect] + [:points {:optional true} ::points] + [:blocked {:optional true} :boolean] + [:fills {:optional true} + [:vector {:gen/max 2} ::fill]] + ...) + + (def schema:token-set-attrs + [:map {:title "TokenSet"} + [:name :string] + [:description {:optional true} :string] + [:modified-at {:optional true} ::sm/inst] + [:tokens {:optional true} [:and + [:map-of {:gen/max 5} + :string + schema:token] + [:fn d/ordered-map?]]]]) + ``` + +* A **protocol** that define the external interface to be used for this entity. + + (NOTE: this is currently only implemented in Design Tokens subsystem). + + ```clojure + (defprotocol ITokenSet + (update-name [_ set-name] "change a token set name while keeping the path") + (add-token [_ token] "add a token at the end of the list") + (update-token [_ token-name f] "update a token in the list") + (delete-token [_ token-name] "delete a token from the list") + (get-token [_ token-name] "return token by token-name") + (get-tokens [_] "return an ordered sequence of all tokens in the set")) + ``` + +* A **custom data type** that implements this protocol. __Functions here are the only + ones allowed to modify the internal structure of the type__. + + Clojure allows us two kinds of custom data types: + * [**`deftype`**](https://funcool.github.io/clojurescript-unraveled/#deftype). We'll + use it when we want the internal structure to be completely opaque and + data accessed through protocol functions. Clojure allows access to the + attributes with the (.-attr) + syntax, but we prefer not to use it. + * [**`defrecord`**](https://funcool.github.io/clojurescript-unraveled/#defrecord). + We'll use it when we want the structure to be exposed as a plain clojure + map, and thus allowing to read attributes with (:attr t), to use get, keys, vals, etc. Note that this also allows + modifying the object with assoc, + update, etc. But in general we + prefer to do all modification via protocol methods, because this way + it's easier to track down where the failure is if an invalid structure + is detected in a validation check, and add business logic like "update + modified-at whenever any other + attribute is changed". + + ```clojure + (defrecord TokenSet [name description modified-at tokens] + ITokenSet + (add-token [_ token] + (let [token (check-token token)] + (TokenSet. name + description + (dt/now) + (assoc tokens (:name token) token)))) + ``` + + +* **Additional helper functions** the protocol should be made as small and compact + as possible. If we need functions for business logic or complex queries that + do not need to directly access the internal structure, but can be implemented by + only calling the abstract procotol, they should be created as standalone functions. + At this level, they must be functions that operate only on instances of the given + domain model entity. They must always ensure the internal integrity of the data. + + ```clojure + (defn sets-tree-seq + "Get tokens sets tree as a flat list" + [token-sets] + ...) + +> IMPORTANT SUMMARY +> * Code in this level only knows about one domain model entity. +> * All functions ensure the internal integrity of the data. +> * For this, the schema is used, and the functions must check parameters +> and output values as needed. +> * No outside code should get any knowledge of the internal structure, so it +> can be changed in the future without breaking cliente code. +> * All modifications of the data should be done via protocol methods (even in +> defrecords). This allows a) more +> control of the internal data dependencies, b) easier bug tracking of +> corrupted data, and c) easier refactoring when the structure is modified. + +Currently most of Penpot code does not follow those requirements, but it +should do in new code or in any refactor. + +## File operations + +```text +▾ common/ + ▾ src/app/common/ + ▾ files/ + helpers.cljc + shapes_helpers.cljc + ... +``` + +Functions that modify a file object (or a part of it) in place, returning the +file object changed. They ensure the referential integrity within the file, or +between a file and its libraries. + +**These functions are used when we need to manipulate objects of different +domain entities inside a file.** + +```clojure +(defn resolve-component + "Retrieve the referenced component, from the local file or from a library" + [shape file libraries & {:keys [include-deleted?] :or {include-deleted? False}}] + (if (= (:component-file shape) (:id file)) + (ctkl/get-component (:data file) (:component-id shape) include-deleted?) + (get-component libraries + (:component-file shape) + (:component-id shape) + :include-deleted? include-deleted?))) + +(defn delete-component +"Mark a component as deleted and store the main instance shapes inside it, to +be able to be recovered later." +[file-data component-id skip-undelete? Main-instance] +(let [components-v2 (dm/get-in file-data [:options :components-v2])] + (if (or (not components-v2) skip-undelete?) + (ctkl/delete-component file-data component-id) + (let [set-main-instance ;; If there is a saved main-instance, restore it. + #(if main-instance + (assoc-in % [:objects (:main-instance-id %)] main-instance) + %)] + (-> file-data + (ctkl/update-component component-id load-component-objects) + (ctkl/update-component component-id set-main-instance) + (ctkl/mark-component-deleted component-id)))))) +``` + +> This module is still needing an important refactor. Mainly to take functions +> from common.types and move them here. + +### File validation and repair + +There is a function in app.common.files.validate that checks a file for +referential and semantic integrity. It's called automatically when file changes +are sent to backend, but may be invoked manually whenever it's needed. + +## File changes objects + +```text +▾ common/ + ▾ src/app/common/ + ▾ files/ + changes_builder.cljc + changes.cljc + ... +``` + +This layer is how we adopt the [Event sourcing pattern](https://www.geeksforgeeks.org/event-sourcing-pattern/). +Instead of directly modifying files, we create changes +objects, that represent one modification, and that can be serialized, stored, +send to backend, logged, etc. Then it can be *materialized* by a **processing +function**, that takes a file and a change, and returns the updated file. + +This also allows an important feature: undoing changes. + +Processing functions should not contain business logic or algorithms. Just +adapt the change interface to the operations in **File** or **Abstract Data +Types** levels. + +There exists a changes-builder module +with helper functions to conveniently build changes objects, and to +automatically calculate the reverse undo change. + +```clojure +(sm/define! ::changes + [:map {:title "changes"} + [:redo-changes vector?] + [:undo-changes seq?] + [:origin {:optional true} any?] + [:save-undo? {:optional true} boolean?] + [:stack-undo? {:optional true} boolean?] + [:undo-group {:optional true} any?]]) + +(defmethod process-change :add-component + [file-data params] + (ctkl/add-component file-data params)) +``` + +> IMPORTANT RULES +> +> All changes must satisfy two properties: +> * **[Idempotence](https://en.wikipedia.org/wiki/Idempotence)**. The event +> sourcing architecture and multiuser capability may cause that the same +> change may be applied more than once to a file. So changes must not, for +> example, be like *increment counter* but rather *set counter to value x*. +> * **Minimal scope**. To reduce conflicts, changes should only modify the +> relevant part of the domain entity. This way, if a concurrent change on +> the same entity arrives, from another user, and it modifies a different +> part of the data, they may ve processed without overriding. + +## Business logic + +```text +▾ common/ + ▾ src/app/common/ + ▾ logic/ + shapes.cljc + libraries.cljc +``` + +At this level there are functions that implement high level user actions, in an +abstract way (independent of UI). Here may be complex business logic (eg. to +create a component copy we must clone all shapes, assign new ids, relink +parents, change the head structure to be a copy and link each shape in the copy +with the corresponding one in the main). + +They don't directly modify files, but generate changes objects, that may be +executed in frontend or sent to backend. + +Those functions may also be composed in even higher level actions. For example +a "update shape attr" action may use "unapply token" actions when the attribute +has an applied token. + +```clojure +(defn generate-instantiate-component +"Generate changes to create a new instance from a component." +[changes objects file-id component-id position page libraries old-id parent-id + frame-id {:keys [force-frame?] :or {force-frame? False}}] + (let [component (ctf/get-component libraries file-id component-id) + parent (when parent-id (get objects parent-id)) + library (get libraries file-id) + components-v2 (dm/get-in library [:data :options :components-v2]) + [new-shape new-shapes]º + (ctn/make-component-instance page + Component + (:data library) + Position + Components-v2 + (cond-> {} + force-frame? (assoc :force-frame-id frame-id))) + changes (cond-> (pcb/add-object changes first-shape {:ignore-touched true}) + (some? old-id) (pcb/amend-last-change #(assoc % :old-id old-id))) + changes (reduce #(pcb/add-object %1 %2 {:ignore-touched true}) + changes + (rest new-shapes))] +[new-shape changes])) +``` + +## Data events + +```text +▾ frontend/ + ▾ src/app/main/data/ + ▾ dashboard/ + ▾ viewer/ + ▾ workspace/ +``` + +This is the intersection of the logic and the presentation in Penpot. Data +events belong to the presentation interface and they manage the global state of +the application. But they may also work on loaded files by using **File** or +**Abstract Data Types** operations to query the data, and by creating and +commiting **changes** via the **Business logic** generate functions. + +**IMPORTANT: data events must not contain business logic theirselves**, or +directly manipulate data structures. They only may modify or query the global +state, and delegate all logic to lower level functions. + +In current Penpot code, there is some quantity of business logic in data events, +that should be progressively moved elsewhere as we keep refactoring. + +```clojure +(defn detach-component + "Remove all references to components in the shape with the given id, + and all its children, at the current page." + [id] + (dm/assert! (uuid? id)) + (ptk/reify ::detach-component + ptk/WatchEvent + (watch [it state _] + (let [page-id (:current-page-id state) + file-id (:current-file-id state) + + fdata (dsh/lookup-file-data state file-id) + libraries (dsh/lookup-libraries state) + + changes (-> (pcb/empty-changes it) + (cll/generate-detach-component id fdata page-id libraries))] + + (rx/of (dch/commit-changes changes)))))) +``` diff --git a/docs/technical-guide/developer/architecture/index.md b/docs/technical-guide/developer/architecture/index.md index 19ea7e4c2..1160393d9 100644 --- a/docs/technical-guide/developer/architecture/index.md +++ b/docs/technical-guide/developer/architecture/index.md @@ -1,5 +1,5 @@ --- -title: 3.1. Architecture +title: 3.01. Architecture desc: Dive into architecture, backend, frontend, data models, and development environments. Contribute and self-host for free! See Penpot's technical guide. --- diff --git a/docs/technical-guide/developer/backend.md b/docs/technical-guide/developer/backend.md index 672b71879..1a0c69d01 100644 --- a/docs/technical-guide/developer/backend.md +++ b/docs/technical-guide/developer/backend.md @@ -1,5 +1,5 @@ --- -title: 3.6. Backend Guide +title: 3.06. Backend Guide --- # Backend guide # diff --git a/docs/technical-guide/developer/common.md b/docs/technical-guide/developer/common.md index 5b921dc74..ab2f51292 100644 --- a/docs/technical-guide/developer/common.md +++ b/docs/technical-guide/developer/common.md @@ -1,5 +1,5 @@ --- -title: 3.4. Common Guide +title: 3.04. Common Guide desc: "View Penpot's technical guide: self-hosting, configuration, developer insights, architecture, data model, integration, and troubleshooting." --- diff --git a/docs/technical-guide/developer/data-guide.md b/docs/technical-guide/developer/data-guide.md index 24157f012..793ff365c 100644 --- a/docs/technical-guide/developer/data-guide.md +++ b/docs/technical-guide/developer/data-guide.md @@ -1,5 +1,5 @@ --- -title: 3.7. Data Guide +title: 3.08. Data Guide desc: Learn about data structures, code organization, file operations, migrations, shape editing, and component syncing. See Penpot's technical guide. Try it free! --- @@ -30,217 +30,6 @@ all of this is important in general. Clojure (for example ending it with ? for boolean values), because this may cause problems when exporting. -## Code organization in abstraction levels - -Initially, Penpot data model implementation was organized in a different way. -We are currently in a process of reorganization. The objective is to have data -manipulation code structured in abstraction layers, with well-defined -boundaries. - -At this moment the namespace structure is already organized as described here, -but there is much code that does not comply with these rules, and needs to be -moved or refactored. We expect to be refactoring existing modules incrementally, -each time we do an important functionality change. - -### Abstract data types - -```text -▾ common/ - ▾ src/app/common/ - ▾ types/ - file.cljc - page.cljc - shape.cljc - color.cljc - component.cljc - ... -``` - -Namespaces here represent a single data structure, or a fragment of one, as an -abstract data type. Each structure has: - -* A **schema spec** that defines the structure of the type and its values: - - ```clojure - (sm/define! ::fill - [:map {:title "Fill"} - [:fill-color {:optional true} ::ctc/rgb-color] - [:fill-opacity {:optional true} ::sm/safe-number] - ...) - - (sm/define! ::shape-attrs - [:map {:title "ShapeAttrs"} - [:name {:optional true} :string] - [:selrect {:optional true} ::grc/rect] - [:points {:optional true} ::points] - [:blocked {:optional true} :boolean] - [:fills {:optional true} - [:vector {:gen/max 2} ::fill]] - ...) - ``` - -* **Helper functions** to create, query and manipulate the structure. Helpers - at this level only are allowed to see the internal attributes of a type. - Updaters receive an object of the type, and return a new object modified, - also ensuring the internal integrity of the data after the change. - - ```clojure - (defn setup-shape - "A function that initializes the geometric data of the shape. The props must - contain at least :x :y :width :height." - [{:keys [type] :as props}] - ...) - - (defn has-direction? - [interaction] - (#{:slide :push} (-> interaction :animation :animation-type))) - - (defn set-direction - [interaction direction] - (dm/assert! - "expected valid interaction map" - (check-interaction! interaction)) - (dm/assert! - "expected valid direction" - (contains? direction-types direction)) - (dm/assert! - "expected compatible interaction map" - (has-direction? interaction)) - (update interaction :animation assoc :direction direction)) - ``` - -> IMPORTANT: we should always use helper functions to access and modify these data -> structures. Avoid direct attribute read or using functions like assoc or -> update, even if the information is contained in a single attribute. This way -> it will be much simpler to add validation checks or to modify the internal -> representation of a type, and will be easier to search for places in the code -> where this data item is used. -> -> Currently much of Penpot code does not follow this requirement, but it -should do in new code or in any refactor. - -### File operations - -```text -▾ common/ - ▾ src/app/common/ - ▾ files/ - helpers.cljc - shapes_helpers.cljc - ... -``` - -Functions that modify a file object (or a part of it) in place, returning the -file object changed. They ensure the referential integrity within the file, or -between a file and its libraries. - -```clojure -(defn resolve-component - "Retrieve the referenced component, from the local file or from a library" - [shape file libraries & {:keys [include-deleted?] :or {include-deleted? False}}] - (if (= (:component-file shape) (:id file)) - (ctkl/get-component (:data file) (:component-id shape) include-deleted?) - (get-component libraries - (:component-file shape) - (:component-id shape) - :include-deleted? include-deleted?))) - -(defn delete-component -"Mark a component as deleted and store the main instance shapes inside it, to -be able to be recovered later." -[file-data component-id skip-undelete? Main-instance] -(let [components-v2 (dm/get-in file-data [:options :components-v2])] - (if (or (not components-v2) skip-undelete?) - (ctkl/delete-component file-data component-id) - (let [set-main-instance ;; If there is a saved main-instance, restore it. - #(if main-instance - (assoc-in % [:objects (:main-instance-id %)] main-instance) - %)] - (-> file-data - (ctkl/update-component component-id load-component-objects) - (ctkl/update-component component-id set-main-instance) - (ctkl/mark-component-deleted component-id)))))) -``` - -> This module is still needing an important refactor. Mainly to take functions -> from common.types and move them here. - -#### File validation and repair - -There is a function in app.common.files.validate that checks a file for -referential and semantic integrity. It's called automatically when file changes -are sent to backend, but may be invoked manually whenever it's needed. - -### File changes objects - -```text -▾ common/ - ▾ src/app/common/ - ▾ files/ - changes_builder.cljc - changes.cljc - ... -``` - -Wrap the update functions in file operations module into changes objects, that -may be serialized, stored, sent to backend and executed to actually modify a file -object. They should not contain business logic or algorithms. Only adapt the -interface to the file operations or types. - -```clojure -(sm/define! ::changes - [:map {:title "changes"} - [:redo-changes vector?] - [:undo-changes seq?] - [:origin {:optional true} any?] - [:save-undo? {:optional true} boolean?] - [:stack-undo? {:optional true} boolean?] - [:undo-group {:optional true} any?]]) - -(defmethod process-change :add-component - [file-data params] - (ctkl/add-component file-data params)) -``` - -### Business logic - -```text -▾ common/ - ▾ src/app/common/ - ▾ logic/ - shapes.cljc - libraries.cljc -``` - -Functions that implement semantic user actions, in an abstract way (independent -of UI). They don't directly modify files, but generate changes objects, that -may be executed in frontend or sent to backend. - -```clojure -(defn generate-instantiate-component -"Generate changes to create a new instance from a component." -[changes objects file-id component-id position page libraries old-id parent-id - frame-id {:keys [force-frame?] :or {force-frame? False}}] - (let [component (ctf/get-component libraries file-id component-id) - parent (when parent-id (get objects parent-id)) - library (get libraries file-id) - components-v2 (dm/get-in library [:data :options :components-v2]) - [new-shape new-shapes]º - (ctn/make-component-instance page - Component - (:data library) - Position - Components-v2 - (cond-> {} - force-frame? (assoc :force-frame-id frame-id))) - changes (cond-> (pcb/add-object changes first-shape {:ignore-touched true}) - (some? old-id) (pcb/amend-last-change #(assoc % :old-id old-id))) - changes (reduce #(pcb/add-object %1 %2 {:ignore-touched true}) - changes - (rest new-shapes))] -[new-shape changes])) -``` - ## Data migrations ```text diff --git a/docs/technical-guide/developer/data-model/index.md b/docs/technical-guide/developer/data-model/index.md index 21999f93c..6d384825f 100644 --- a/docs/technical-guide/developer/data-model/index.md +++ b/docs/technical-guide/developer/data-model/index.md @@ -1,5 +1,5 @@ --- -title: 3.2. Data model +title: 3.02. Data model desc: Learn about self-hosting, configuration, developer tools, data models, architecture, and integrations. View Penpot's technical guide. Free to use! --- diff --git a/docs/technical-guide/developer/devenv.md b/docs/technical-guide/developer/devenv.md index 27c42c201..61eb4f92b 100644 --- a/docs/technical-guide/developer/devenv.md +++ b/docs/technical-guide/developer/devenv.md @@ -1,5 +1,5 @@ --- -title: 3.3. Dev environment +title: 3.03. Dev environment desc: Dive into Penpot's development environment. Learn about self-hosting, configuration, developer tools, architecture, and more. See the Penpot Technical Guide! --- diff --git a/docs/technical-guide/developer/frontend.md b/docs/technical-guide/developer/frontend.md index 9b5feb6ba..bbab7a6eb 100644 --- a/docs/technical-guide/developer/frontend.md +++ b/docs/technical-guide/developer/frontend.md @@ -1,5 +1,5 @@ --- -title: 3.5. Frontend Guide +title: 3.05. Frontend Guide desc: "See Penpot's technical guide: self-hosting, configuration, developer insights (architecture, data model), frontend, backend, and integrations & more!" --- diff --git a/docs/technical-guide/developer/subsystems/index.md b/docs/technical-guide/developer/subsystems/index.md index aea446a12..f53b59adb 100644 --- a/docs/technical-guide/developer/subsystems/index.md +++ b/docs/technical-guide/developer/subsystems/index.md @@ -1,5 +1,5 @@ --- -title: 3.8. Penpot subsystems +title: 3.09. Penpot subsystems desc: Learn about architecture, data models, and subsystems. View Penpot's technical guide for self-hosting, configuration, and development insights. Free! --- diff --git a/docs/technical-guide/developer/ui.md b/docs/technical-guide/developer/ui.md index 103bc4025..5105e2bc5 100644 --- a/docs/technical-guide/developer/ui.md +++ b/docs/technical-guide/developer/ui.md @@ -1,5 +1,5 @@ --- -title: 3.9. UI Guide +title: 3.10. UI Guide desc: Learn UI development with React & Rumext, design system implementation, and performance considerations. See Penpot's technical guide. Free to use! ---