3pp-mirror/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2025-05-10 11:46:37 +02:00
Code Issues Projects Releases Packages Wiki Activity

📚 Merge penpot/penpot-docs repository

Browse source
This commit is contained in:
David Barragán Merino 2024-10-30 12:49:46 +01:00 • committed by Andrey Antukh
parent 3932054ea6
commit 88296480ec
665 changed files with 17621 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

136
docs/technical-guide/developer/architecture/backend.md Normal file
View file

@ -0,0 +1,136 @@
---
title: Backend app
---
# Backend app
This app is in charge of CRUD of data, integrity validation and persistence
into a database and also into a file system for media attachments.
To handle deletions it uses a garbage collector mechanism: no object in the
database is deleted instantly. Instead, a field <code class="language-bash">deleted_at</code> is set with the
date and time of the deletion, and every query ignores db rows that have this
field set. Then, an async task that runs periodically, locates rows whose
deletion date is older than a given threshold and permanently deletes them.
For this, and other possibly slow tasks, there is an internal async tasks
worker, that may be used to queue tasks to be scheduled and executed when the
backend is idle. Other tasks are email sending, collecting data for telemetry
and detecting unused media attachment, for removing them from the file storage.
## Backend structure
Penpot backend app code resides under <code class="language-text">backend/src/app</code> path in the main repository.
@startuml BackendGeneral
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
!include DEVICONS/react.puml
!include DEVICONS/java.puml
!include DEVICONS/clojure.puml
!include DEVICONS/postgresql.puml
!include DEVICONS/redis.puml
!include DEVICONS/chrome.puml
HIDE_STEREOTYPE()
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
System_Boundary(backend, "Backend") {
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
ContainerDb(redis, "Broker", "Redis", "", "redis")
}
BiRel(frontend_app, backend_app, "Open", "websocket")
Rel(frontend_app, backend_app, "Uses", "RPC API")
Rel(backend_app, db, "Uses", "SQL")
Rel(redis, backend_app, "Subscribes", "pub/sub")
Rel(backend_app, redis, "Notifies", "pub/sub")
@enduml
```
▾ backend/src/app/
▸ cli/
▸ http/
▸ migrations/
▸ rpc/
▸ setup/
▸ srepl/
▸ util/
▸ tasks/
main.clj
config.clj
http.clj
metrics.clj
migrations.clj
notifications.clj
rpc.clj
setup.clj
srepl.clj
worker.clj
...
```
* <code class="language-text">main.clj</code> defines the app global settings and the main entry point of the
application, served by a JVM.
* <code class="language-text">config.clj</code> defines of the configuration options read from linux
environment.
* <code class="language-text">http</code> contains the HTTP server and the backend routes list.
* <code class="language-text">migrations</code> contains the SQL scripts that define the database schema, in
the form of a sequence of migrations.
* <code class="language-text">rpc</code> is the main module to handle the RPC API calls.
* <code class="language-text">notifications.clj</code> is the main module that manages the websocket. It allows
clients to subscribe to open files, intercepts update RPC calls and notify
them to all subscribers of the file.
* <code class="language-text">setup</code> initializes the environment (loads config variables, sets up the
database, executes migrations, loads initial data, etc).
* <code class="language-text">srepl</code> sets up an interactive REPL shell, with some useful commands to be
used to debug a running instance.
* <code class="language-text">cli</code> sets a command-line interface, with some more maintenance commands.
* <code class="language-text">metrics.clj</code> has some interceptors that watches RPC calls, calculate
statistics and other metrics, and send them to external systems to store and
analyze.
* <code class="language-text">worker.clj</code> and <code class="language-text">tasks</code> define some async tasks that are executed in
parallel to the main http server (using java threads), and scheduled in a
cron-like table. They are useful to do some garbage collection, data packing
and similar periodic maintenance tasks.
* <code class="language-text">db.clj</code>, <code class="language-text">emails.clj</code>, <code class="language-text">media.clj</code>, <code class="language-text">msgbus.clj</code>, <code class="language-text">storage.clj</code>,
<code class="language-text">rlimits.clj</code> are general libraries to use I/O resources (SQL database,
send emails, handle multimedia objects, use REDIS messages, external file
storage and semaphores).
* <code class="language-text">util/</code> has a collection of generic utility functions.
### RPC calls
The RPC (Remote Procedure Call) subsystem consists of a mechanism that allows
to expose clojure functions as an HTTP endpoint. We take advantage of being
using Clojure at both front and back ends, to avoid needing complex data
conversions.
1. Frontend initiates a "query" or "mutation" call to <code class="language-text">:xxx</code> method, and
passes a Clojure object as params.
2. Params are string-encoded using
[transit](https://github.com/cognitect/transit-clj), a format similar to
JSON but more powerful.
3. The call is mapped to <code class="language-text"><backend-host>/api/rpc/query/xxx</code> or
<code class="language-text"><backend-host>/api/rpc/mutation/xxx</code>.
4. The <code class="language-text">rpc</code> module receives the call, decode the parameters and executes the
corresponding method inside <code class="language-text">src/app/rpc/queries/</code> or <code class="language-text">src/app/rpc/mutations/</code>.
We have created a <code class="language-text">defmethod</code> macro to declare an RPC method and its
parameter specs.
5. The result value is also transit-encoded and returned to the frontend.
This way, frontend can execute backend calls like it was calling an async function,
with all the power of Clojure data structures.
### PubSub
To manage subscriptions to a file, to be notified of changes, we use a redis
server as a pub/sub broker. Whenever a user visits a file and opens a
websocket, the backend creates a subscription in redis, with a topic that has
the id of the file. If the user sends any change to the file, backend sends a
notification to this topic, that is received by all subscribers. Then the
notification is retrieved and sent to the user via the websocket.

84
docs/technical-guide/developer/architecture/common.md Normal file
View file

@ -0,0 +1,84 @@
---
title: Common code
---
# Common code
In penpot, we take advantage of using the same language in frontend and
backend, to have a bunch of shared code.
Sometimes, we use conditional compilation, for small chunks of code that
are different in a Clojure+Java or ClojureScript+JS environments. We use
the <code class="language-clojure">#?</code> construct, like this, for example:
```clojure
(defn ordered-set?
[o]
#?(:cljs (instance? lks/LinkedSet o)
:clj (instance? LinkedSet o)))
```
```text
▾ common/src/app/common/
▸ geom/
▸ pages/
▸ path/
▸ types/
...
```
Some of the modules need some refactoring, to organize them more cleanly.
## Data model and business logic
* **geom** contains functions to manage 2D geometric entities.
- **point** defines the 2D Point type and many geometric transformations.
- **matrix** defines the [2D transformation
matrix](https://www.alanzucconi.com/2016/02/10/tranfsormation-matrix/)
type and its operations.
- **shapes** manages shapes as a collection of points with a bounding
rectangle.
* **path** contains functions to manage SVG paths, transform them and also
convert other types of shapes into paths.
* **pages** contains the definition of the [Penpot data model](/technical-guide/developer/data-model/) and
the conceptual business logic (transformations of the model entities,
independent of the user interface or data storage).
- **spec** has the definitions of data structures of files and shapes, and
also of the transformation operations in **changes** module. Uses [Clojure
spec](https://github.com/clojure/spec.alpha) to define the structure and
validators.
- **init** defines the default content of files, pages and shapes.
- **helpers** are some functions to help manipulating the data structures.
- **migrations** is in charge to manage the evolution of the data model
structure over time. It contains a function that gets a file data
content, identifies its version, and applies the needed migrations. Much
like the SQL database migrations scripts.
- **changes** and **changes_builder** define a set of transactional
operations, that receive a file data content, and perform a semantic
operation following the business logic (add a page or a shape, change a
shape attribute, modify some file asset, etc.).
* **types** we are currently in process of refactoring **pages** module, to
organize it in a way more compliant of [Abstract Data
Types](https://en.wikipedia.org/wiki/Abstract_data_type) paradigm. We are
approaching the process incrementally, rewriting one module each time, as
needed.
## Utilities
The main ones are:
* **data** basic data structures and utility functions that could be added to
Clojure standard library.
* **math** some mathematic functions that could also be standard.
* **file_builder** functions to parse the content of a <code class="language-text">.penpot</code> exported file
and build a File data structure from it.
* **logging** functions to generate traces for debugging and usage analysis.
* **text** an adapter layer over the [DraftJS editor](https://draftjs.org) that
we use to edit text shapes in workspace.
* **transit** functions to encode/decode Clojure objects into
[transit](https://github.com/cognitect/transit-clj), a format similar to JSON
but more powerful.
* **uuid** functions to generate [Universally Unique Identifiers
(UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier), used
over all Penpot models to have identifiers for objects that are practically
ensured to be unique, without having a central control.

68
docs/technical-guide/developer/architecture/exporter.md Normal file
View file

@ -0,0 +1,68 @@
---
title: Exporter app
---
# Exporter app
When exporting file contents to a file, we want the result to be exactly the
same as the user sees in screen. To achieve this, we use a headless browser
installed in the backend host, and controled via puppeteer automation. The
browser loads the frontend app from the static webserver, and executes it like
a normal user browser. It visits a special endpoint that renders one shape
inside a file. Then, if takes a screenshot if we are exporting to a bitmap
image, or extract the svg from the DOM if we want a vectorial export, and write
it to a file that the user can download.
## Exporter structure
Penpot exporter app code resides under <code class="language-text">exporter/src/app</code> path in the main repository.
@startuml Exporter
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
!include DEVICONS/react.puml
!include DEVICONS/clojure.puml
!include DEVICONS/chrome.puml
HIDE_STEREOTYPE()
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
System_Boundary(backend, "Backend") {
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
Container(browser, "Headless browser", "Chrome", "", "chrome")
}
Rel_D(frontend_app, exporter, "Uses", "HTTPS")
Rel_R(exporter, browser, "Uses", "puppeteer")
Rel_U(browser, frontend_app, "Uses", "HTTPS")
@enduml
```text
▾ exporter/src/app/
▸ http/
▸ renderer/
▸ util/
core.cljs
http.cljs
browser.cljs
config.cljs
```
## Exporter namespaces
* **core** has the setup and run functions of the nodejs app.
* **http** exposes a basic http server, with endpoints to export a shape or a
file.
* **browser** has functions to control a local Chromium browser via
[puppeteer](https://puppeteer.github.io/puppeteer).
* **renderer** has functions to tell the browser to render an object and make a
screenshot, and then convert it to bitmap, pdf or svg as needed.
* **config** gets configuration settings from the linux environment.
* **util** has some generic utility functions.

258
docs/technical-guide/developer/architecture/frontend.md Normal file
View file

@ -0,0 +1,258 @@
---
title: Frontend app
---
### Frontend app
The main application, with the user interface and the presentation logic.
To talk with backend, it uses a custom RPC-style API: some functions in the
backend are exposed through an HTTP server. When the front wants to execute a
query or data mutation, it sends a HTTP request, containing the name of the
function to execute, and the ascii-encoded arguments. The resulting data is
also encoded and returned. This way we don't need any data type conversion,
besides the transport encoding, as there is Clojure at both ends.
When the user opens any file, a persistent websocket is opened with the backend
and associated to the file id. It is used to send presence events, such as
connection, disconnection and mouse movements. And also to receive changes made
by other users that are editing the same file, so it may be updated in real
time.
## Frontend structure
Penpot frontend app code resides under <code class="language-text">frontend/src/app</code> path in the main repository.
@startuml FrontendGeneral
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
!include DEVICONS/react.puml
HIDE_STEREOTYPE()
Person(user, "User")
System_Boundary(frontend, "Frontend") {
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
Container(worker, "Worker", "Web worker")
}
Rel(user, frontend_app, "Uses", "HTTPS")
BiRel_L(frontend_app, worker, "Works with")
@enduml
```text
▾ frontend/src/app/
▸ main/
▸ util/
▸ worker/
main.cljs
worker.cljs
```
* <code class="language-text">main.cljs</code> and <code class="language-text">main/</code> contain the main frontend app, written in
ClojureScript language and using React framework, wrapped in [rumext
library](https://github.com/funcool/rumext).
* <code class="language-text">worker.cljs</code> and <code class="language-text">worker/</code> contain the web worker, to make expensive
calculations in background.
* <code class="language-text">util/</code> contains many generic utilities, non dependant on the user
interface.
@startuml FrontendMain
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
HIDE_STEREOTYPE()
Component(ui, "ui", "main web component")
Component(store, "store", "module")
Component(refs, "refs", "module")
Component(repo, "repo", "module")
Component(streams, "streams", "module")
Component(errors, "errors", "module")
Boundary(ui_namespaces, "ui namespaces") {
Component(ui_auth, "auth", "web component")
Component(ui_settings, "settings", "web component")
Component(ui_dashboard, "dashboard", "web component")
Component(ui_workspace, "workspace", "web component")
Component(ui_viewer, "viewer", "web component")
Component(ui_render, "render", "web component")
Component(ui_exports, "exports", "web component")
Component(ui_shapes, "shapes", "component library")
Component(ui_components, "components", "component library")
}
Boundary(data_namespaces, "data namespaces") {
Component(data_common, "common", "events")
Component(data_users, "users", "events")
Component(data_dashboard, "dashboard", "events")
Component(data_workspace, "workspace", "events")
Component(data_viewer, "viewer", "events")
Component(data_comments, "comments", "events")
Component(data_fonts, "fonts", "events")
Component(data_messages, "messages", "events")
Component(data_modal, "modal", "events")
Component(data_shortcuts, "shortcuts", "utilities")
}
Lay_D(ui_exports, data_viewer)
Lay_D(ui_settings, ui_components)
Lay_D(data_viewer, data_common)
Lay_D(data_fonts, data_messages)
Lay_D(data_dashboard, data_modal)
Lay_D(data_workspace, data_shortcuts)
Lay_L(data_dashboard, data_fonts)
Lay_L(data_workspace, data_comments)
Rel_Up(refs, store, "Watches")
Rel_Up(streams, store, "Watches")
Rel(ui, ui_auth, "Routes")
Rel(ui, ui_settings, "Routes")
Rel(ui, ui_dashboard, "Routes")
Rel(ui, ui_workspace, "Routes")
Rel(ui, ui_viewer, "Routes")
Rel(ui, ui_render, "Routes")
Rel(ui_render, ui_exports, "Uses")
Rel(ui_workspace, ui_shapes, "Uses")
Rel(ui_viewer, ui_shapes, "Uses")
Rel_Right(ui_exports, ui_shapes, "Uses")
Rel(ui_auth, data_users, "Uses")
Rel(ui_settings, data_users, "Uses")
Rel(ui_dashboard, data_dashboard, "Uses")
Rel(ui_dashboard, data_fonts, "Uses")
Rel(ui_workspace, data_workspace, "Uses")
Rel(ui_workspace, data_comments, "Uses")
Rel(ui_viewer, data_viewer, "Uses")
@enduml
### General namespaces
* **store** contains the global state of the application. Uses an event loop
paradigm, similar to Redux, with a global state object and a stream of events
that modify it. Made with [potok library](https://funcool.github.io/potok/latest/).
* **refs** has the collection of references or lenses: RX streams that you can
use to subscribe to parts of the global state, and be notified when they
change.
* **streams** has some streams, derived from the main event stream, for keyboard
and mouse events. Used mainly from the workspace viewport.
* **repo** contains the functions to make calls to backend.
* **errors** has functions with global error handlers, to manage exceptions or other
kinds of errors in the ui or the data events, notify the user in a useful way,
and allow to recover and continue working.
### UI namespaces
* **ui** is the root web component. It reads the current url and mounts the needed
subcomponent depending on the route.
* **auth** has the web components for the login, register, password recover,
etc. screens.
* **settings** has the web comonents for the user profile and settings screens.
* **dashboard** has the web components for the dashboard and its subsections.
* **workspace** has the web components for the file workspace and its subsections.
* **viewer** has the web components for the viewer and its subsections.
* **render** contain special web components to render one page or one specific
shape, to be used in exports.
* **export** contain basic web components that display one shape or frame, to
be used from exports render or else from dashboard and viewer thumbnails and
other places.
* **shapes** is the basic collection of web components that convert all types of
shapes in the corresponding svg elements, without adding any extra function.
* **components** a library of generic UI widgets, to be used as building blocks
of penpot screens (text or numeric inputs, selects, forms, buttons...).
### Data namespaces
* **users** has events to login and register, fetch the user profile and update it.
* **dashboard** has events to fetch and modify teams, projects and files.
* **fonts** has some extra events to manage uploaded fonts from dashboard.
* **workspace** has a lot of events to manage the current file and do all kinds of
edits and updates.
* **comments** has some extra events to manage design comments.
* **viewer** has events to fetch a file contents to display, and manage the
interactive behavior and hand-off.
* **common** has some events used from several places.
* **modal** has some events to show modal popup windows.
* **messages** has some events to show non-modal informative messages.
* **shortcuts** has some utility functions, used in other modules to setup the
keyboard shortcuts.
## Worker app
Some operations are costly to make in real time, so we leave them to be
executed asynchronously in a web worker. This way they don't impact the user
experience. Some of these operations are generating file thumbnails for the
dashboard and maintaining some geometric indexes to speed up snap points while
drawing.
@startuml FrontendWorker
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
HIDE_STEREOTYPE()
Component(worker, "worker", "worker entry point")
Boundary(worker_namespaces, "worker namespaces") {
Component(thumbnails, "thumbnails", "worker methods")
Component(snaps, "snaps", "worker methods")
Component(selection, "selection", "worker methods")
Component(impl, "impl", "worker methods")
Component(import, "import", "worker methods")
Component(export, "export", "worker methods")
}
Rel(worker, thumbnails, "Uses")
Rel(worker, impl, "Uses")
Rel(worker, import, "Uses")
Rel(worker, export, "Uses")
Rel(impl, snaps, "Uses")
Rel(impl, selection, "Uses")
@enduml
* **worker** contains the worker setup code and the global handler that receives
requests from the main app, and process them.
* **thumbnails** has a method to generate the file thumbnails used in dashboard.
* **snaps** manages a distance index of shapes, and has a method to get
other shapes near a given one, to be used in snaps while drawing.
* **selection** manages a geometric index of shapes, with methods to get what
shapes are under the cursor at a given moment, for select.
* **impl** has a simple method to update all indexes in a page at once.
* **import** has a method to import a whole file from an external <code class="language-text">.penpot</code> archive.
* **export** has a method to export a whole file to an external <code class="language-text">.penpot</code> archive.

65
docs/technical-guide/developer/architecture/index.md Normal file
View file

@ -0,0 +1,65 @@
---
title: 3.1. Architecture
---
# Architecture
This section gives an overall structure of the system.
Penpot has the architecture of a typical SPA. There is a frontend application,
written in ClojureScript and using React framework, and served from a static
web server. It talks to a backend application, that persists data on a
PostgreSQL database.
The backend is written in Clojure, so front and back can share code and data
structures without problem. Then, the code is compiled into JVM bytecode and
run in a JVM environment.
There are some additional components, explained in subsections.
@startuml C4_Elements
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
!include DEVICONS/react.puml
!include DEVICONS/java.puml
!include DEVICONS/clojure.puml
!include DEVICONS/postgresql.puml
!include DEVICONS/redis.puml
!include DEVICONS/chrome.puml
HIDE_STEREOTYPE()
Person(user, "User")
System_Boundary(frontend, "Frontend") {
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
Container(worker, "Worker", "Web worker")
}
System_Boundary(backend, "Backend") {
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
ContainerDb(redis, "Broker", "Redis", "", "redis")
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
Container(browser, "Headless browser", "Chrome", "", "chrome")
}
Rel(user, frontend_app, "Uses", "HTTPS")
BiRel_L(frontend_app, worker, "Works with")
BiRel(frontend_app, backend_app, "Open", "websocket")
Rel(frontend_app, backend_app, "Uses", "RPC API")
Rel(backend_app, db, "Uses", "SQL")
Rel(redis, backend_app, "Subscribes", "pub/sub")
Rel(backend_app, redis, "Notifies", "pub/sub")
Rel(frontend_app, exporter, "Uses", "HTTPS")
Rel(exporter, browser, "Uses", "puppeteer")
Rel(browser, frontend_app, "Uses", "HTTPS")
@enduml
See more at
* [Frontend app](/technical-guide/developer/architecture/frontend/)
* [Backend app](/technical-guide/developer/architecture/backend/)
* [Exporter app](/technical-guide/developer/architecture/exporter/)
* [Common code](/technical-guide/developer/architecture/common/)