mirror of
https://github.com/penpot/penpot.git
synced 2025-06-24 15:57:01 +02:00
📚 Merge penpot/penpot-docs repository
This commit is contained in:
David Barragán Merino
Andrey Antukh
parent
3932054ea6
commit
88296480ec
665 changed files with 17621 additions and 0 deletions
136
docs/technical-guide/developer/architecture/backend.md
Normal file
136
docs/technical-guide/developer/architecture/backend.md
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
---
|
||||
title: Backend app
|
||||
---
|
||||
|
||||
# Backend app
|
||||
|
||||
This app is in charge of CRUD of data, integrity validation and persistence
|
||||
into a database and also into a file system for media attachments.
|
||||
|
||||
To handle deletions it uses a garbage collector mechanism: no object in the
|
||||
database is deleted instantly. Instead, a field <code class="language-bash">deleted_at</code> is set with the
|
||||
date and time of the deletion, and every query ignores db rows that have this
|
||||
field set. Then, an async task that runs periodically, locates rows whose
|
||||
deletion date is older than a given threshold and permanently deletes them.
|
||||
|
||||
For this, and other possibly slow tasks, there is an internal async tasks
|
||||
worker, that may be used to queue tasks to be scheduled and executed when the
|
||||
backend is idle. Other tasks are email sending, collecting data for telemetry
|
||||
and detecting unused media attachment, for removing them from the file storage.
|
||||
|
||||
## Backend structure
|
||||
|
||||
Penpot backend app code resides under <code class="language-text">backend/src/app</code> path in the main repository.
|
||||
|
||||
@startuml BackendGeneral
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/java.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/postgresql.puml
|
||||
!include DEVICONS/redis.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
||||
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
||||
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
||||
}
|
||||
|
||||
BiRel(frontend_app, backend_app, "Open", "websocket")
|
||||
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
||||
Rel(backend_app, db, "Uses", "SQL")
|
||||
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
||||
Rel(backend_app, redis, "Notifies", "pub/sub")
|
||||
|
||||
@enduml
|
||||
|
||||
```
|
||||
▾ backend/src/app/
|
||||
▸ cli/
|
||||
▸ http/
|
||||
▸ migrations/
|
||||
▸ rpc/
|
||||
▸ setup/
|
||||
▸ srepl/
|
||||
▸ util/
|
||||
▸ tasks/
|
||||
main.clj
|
||||
config.clj
|
||||
http.clj
|
||||
metrics.clj
|
||||
migrations.clj
|
||||
notifications.clj
|
||||
rpc.clj
|
||||
setup.clj
|
||||
srepl.clj
|
||||
worker.clj
|
||||
...
|
||||
```
|
||||
|
||||
* <code class="language-text">main.clj</code> defines the app global settings and the main entry point of the
|
||||
application, served by a JVM.
|
||||
* <code class="language-text">config.clj</code> defines of the configuration options read from linux
|
||||
environment.
|
||||
* <code class="language-text">http</code> contains the HTTP server and the backend routes list.
|
||||
* <code class="language-text">migrations</code> contains the SQL scripts that define the database schema, in
|
||||
the form of a sequence of migrations.
|
||||
* <code class="language-text">rpc</code> is the main module to handle the RPC API calls.
|
||||
* <code class="language-text">notifications.clj</code> is the main module that manages the websocket. It allows
|
||||
clients to subscribe to open files, intercepts update RPC calls and notify
|
||||
them to all subscribers of the file.
|
||||
* <code class="language-text">setup</code> initializes the environment (loads config variables, sets up the
|
||||
database, executes migrations, loads initial data, etc).
|
||||
* <code class="language-text">srepl</code> sets up an interactive REPL shell, with some useful commands to be
|
||||
used to debug a running instance.
|
||||
* <code class="language-text">cli</code> sets a command-line interface, with some more maintenance commands.
|
||||
* <code class="language-text">metrics.clj</code> has some interceptors that watches RPC calls, calculate
|
||||
statistics and other metrics, and send them to external systems to store and
|
||||
analyze.
|
||||
* <code class="language-text">worker.clj</code> and <code class="language-text">tasks</code> define some async tasks that are executed in
|
||||
parallel to the main http server (using java threads), and scheduled in a
|
||||
cron-like table. They are useful to do some garbage collection, data packing
|
||||
and similar periodic maintenance tasks.
|
||||
* <code class="language-text">db.clj</code>, <code class="language-text">emails.clj</code>, <code class="language-text">media.clj</code>, <code class="language-text">msgbus.clj</code>, <code class="language-text">storage.clj</code>,
|
||||
<code class="language-text">rlimits.clj</code> are general libraries to use I/O resources (SQL database,
|
||||
send emails, handle multimedia objects, use REDIS messages, external file
|
||||
storage and semaphores).
|
||||
* <code class="language-text">util/</code> has a collection of generic utility functions.
|
||||
|
||||
### RPC calls
|
||||
|
||||
The RPC (Remote Procedure Call) subsystem consists of a mechanism that allows
|
||||
to expose clojure functions as an HTTP endpoint. We take advantage of being
|
||||
using Clojure at both front and back ends, to avoid needing complex data
|
||||
conversions.
|
||||
|
||||
1. Frontend initiates a "query" or "mutation" call to <code class="language-text">:xxx</code> method, and
|
||||
passes a Clojure object as params.
|
||||
2. Params are string-encoded using
|
||||
[transit](https://github.com/cognitect/transit-clj), a format similar to
|
||||
JSON but more powerful.
|
||||
3. The call is mapped to <code class="language-text"><backend-host>/api/rpc/query/xxx</code> or
|
||||
<code class="language-text"><backend-host>/api/rpc/mutation/xxx</code>.
|
||||
4. The <code class="language-text">rpc</code> module receives the call, decode the parameters and executes the
|
||||
corresponding method inside <code class="language-text">src/app/rpc/queries/</code> or <code class="language-text">src/app/rpc/mutations/</code>.
|
||||
We have created a <code class="language-text">defmethod</code> macro to declare an RPC method and its
|
||||
parameter specs.
|
||||
5. The result value is also transit-encoded and returned to the frontend.
|
||||
|
||||
This way, frontend can execute backend calls like it was calling an async function,
|
||||
with all the power of Clojure data structures.
|
||||
|
||||
### PubSub
|
||||
|
||||
To manage subscriptions to a file, to be notified of changes, we use a redis
|
||||
server as a pub/sub broker. Whenever a user visits a file and opens a
|
||||
websocket, the backend creates a subscription in redis, with a topic that has
|
||||
the id of the file. If the user sends any change to the file, backend sends a
|
||||
notification to this topic, that is received by all subscribers. Then the
|
||||
notification is retrieved and sent to the user via the websocket.
|
||||
|
||||
84
docs/technical-guide/developer/architecture/common.md
Normal file
84
docs/technical-guide/developer/architecture/common.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
---
|
||||
title: Common code
|
||||
---
|
||||
|
||||
# Common code
|
||||
|
||||
In penpot, we take advantage of using the same language in frontend and
|
||||
backend, to have a bunch of shared code.
|
||||
|
||||
Sometimes, we use conditional compilation, for small chunks of code that
|
||||
are different in a Clojure+Java or ClojureScript+JS environments. We use
|
||||
the <code class="language-clojure">#?</code> construct, like this, for example:
|
||||
|
||||
```clojure
|
||||
(defn ordered-set?
|
||||
[o]
|
||||
#?(:cljs (instance? lks/LinkedSet o)
|
||||
:clj (instance? LinkedSet o)))
|
||||
```
|
||||
|
||||
```text
|
||||
▾ common/src/app/common/
|
||||
▸ geom/
|
||||
▸ pages/
|
||||
▸ path/
|
||||
▸ types/
|
||||
...
|
||||
```
|
||||
|
||||
Some of the modules need some refactoring, to organize them more cleanly.
|
||||
|
||||
## Data model and business logic
|
||||
|
||||
* **geom** contains functions to manage 2D geometric entities.
|
||||
- **point** defines the 2D Point type and many geometric transformations.
|
||||
- **matrix** defines the [2D transformation
|
||||
matrix](https://www.alanzucconi.com/2016/02/10/tranfsormation-matrix/)
|
||||
type and its operations.
|
||||
- **shapes** manages shapes as a collection of points with a bounding
|
||||
rectangle.
|
||||
* **path** contains functions to manage SVG paths, transform them and also
|
||||
convert other types of shapes into paths.
|
||||
* **pages** contains the definition of the [Penpot data model](/technical-guide/developer/data-model/) and
|
||||
the conceptual business logic (transformations of the model entities,
|
||||
independent of the user interface or data storage).
|
||||
- **spec** has the definitions of data structures of files and shapes, and
|
||||
also of the transformation operations in **changes** module. Uses [Clojure
|
||||
spec](https://github.com/clojure/spec.alpha) to define the structure and
|
||||
validators.
|
||||
- **init** defines the default content of files, pages and shapes.
|
||||
- **helpers** are some functions to help manipulating the data structures.
|
||||
- **migrations** is in charge to manage the evolution of the data model
|
||||
structure over time. It contains a function that gets a file data
|
||||
content, identifies its version, and applies the needed migrations. Much
|
||||
like the SQL database migrations scripts.
|
||||
- **changes** and **changes_builder** define a set of transactional
|
||||
operations, that receive a file data content, and perform a semantic
|
||||
operation following the business logic (add a page or a shape, change a
|
||||
shape attribute, modify some file asset, etc.).
|
||||
* **types** we are currently in process of refactoring **pages** module, to
|
||||
organize it in a way more compliant of [Abstract Data
|
||||
Types](https://en.wikipedia.org/wiki/Abstract_data_type) paradigm. We are
|
||||
approaching the process incrementally, rewriting one module each time, as
|
||||
needed.
|
||||
|
||||
## Utilities
|
||||
|
||||
The main ones are:
|
||||
|
||||
* **data** basic data structures and utility functions that could be added to
|
||||
Clojure standard library.
|
||||
* **math** some mathematic functions that could also be standard.
|
||||
* **file_builder** functions to parse the content of a <code class="language-text">.penpot</code> exported file
|
||||
and build a File data structure from it.
|
||||
* **logging** functions to generate traces for debugging and usage analysis.
|
||||
* **text** an adapter layer over the [DraftJS editor](https://draftjs.org) that
|
||||
we use to edit text shapes in workspace.
|
||||
* **transit** functions to encode/decode Clojure objects into
|
||||
[transit](https://github.com/cognitect/transit-clj), a format similar to JSON
|
||||
but more powerful.
|
||||
* **uuid** functions to generate [Universally Unique Identifiers
|
||||
(UUID)](https://en.wikipedia.org/wiki/Universally_unique_identifier), used
|
||||
over all Penpot models to have identifiers for objects that are practically
|
||||
ensured to be unique, without having a central control.
|
||||
68
docs/technical-guide/developer/architecture/exporter.md
Normal file
68
docs/technical-guide/developer/architecture/exporter.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
---
|
||||
title: Exporter app
|
||||
---
|
||||
|
||||
# Exporter app
|
||||
|
||||
When exporting file contents to a file, we want the result to be exactly the
|
||||
same as the user sees in screen. To achieve this, we use a headless browser
|
||||
installed in the backend host, and controled via puppeteer automation. The
|
||||
browser loads the frontend app from the static webserver, and executes it like
|
||||
a normal user browser. It visits a special endpoint that renders one shape
|
||||
inside a file. Then, if takes a screenshot if we are exporting to a bitmap
|
||||
image, or extract the svg from the DOM if we want a vectorial export, and write
|
||||
it to a file that the user can download.
|
||||
|
||||
## Exporter structure
|
||||
|
||||
Penpot exporter app code resides under <code class="language-text">exporter/src/app</code> path in the main repository.
|
||||
|
||||
@startuml Exporter
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
|
||||
Container(browser, "Headless browser", "Chrome", "", "chrome")
|
||||
}
|
||||
|
||||
Rel_D(frontend_app, exporter, "Uses", "HTTPS")
|
||||
Rel_R(exporter, browser, "Uses", "puppeteer")
|
||||
Rel_U(browser, frontend_app, "Uses", "HTTPS")
|
||||
|
||||
@enduml
|
||||
|
||||
```text
|
||||
▾ exporter/src/app/
|
||||
▸ http/
|
||||
▸ renderer/
|
||||
▸ util/
|
||||
core.cljs
|
||||
http.cljs
|
||||
browser.cljs
|
||||
config.cljs
|
||||
```
|
||||
|
||||
## Exporter namespaces
|
||||
|
||||
* **core** has the setup and run functions of the nodejs app.
|
||||
|
||||
* **http** exposes a basic http server, with endpoints to export a shape or a
|
||||
file.
|
||||
|
||||
* **browser** has functions to control a local Chromium browser via
|
||||
[puppeteer](https://puppeteer.github.io/puppeteer).
|
||||
|
||||
* **renderer** has functions to tell the browser to render an object and make a
|
||||
screenshot, and then convert it to bitmap, pdf or svg as needed.
|
||||
|
||||
* **config** gets configuration settings from the linux environment.
|
||||
|
||||
* **util** has some generic utility functions.
|
||||
258
docs/technical-guide/developer/architecture/frontend.md
Normal file
258
docs/technical-guide/developer/architecture/frontend.md
Normal file
|
|
@ -0,0 +1,258 @@
|
|||
---
|
||||
title: Frontend app
|
||||
---
|
||||
|
||||
### Frontend app
|
||||
|
||||
The main application, with the user interface and the presentation logic.
|
||||
|
||||
To talk with backend, it uses a custom RPC-style API: some functions in the
|
||||
backend are exposed through an HTTP server. When the front wants to execute a
|
||||
query or data mutation, it sends a HTTP request, containing the name of the
|
||||
function to execute, and the ascii-encoded arguments. The resulting data is
|
||||
also encoded and returned. This way we don't need any data type conversion,
|
||||
besides the transport encoding, as there is Clojure at both ends.
|
||||
|
||||
When the user opens any file, a persistent websocket is opened with the backend
|
||||
and associated to the file id. It is used to send presence events, such as
|
||||
connection, disconnection and mouse movements. And also to receive changes made
|
||||
by other users that are editing the same file, so it may be updated in real
|
||||
time.
|
||||
|
||||
## Frontend structure
|
||||
|
||||
Penpot frontend app code resides under <code class="language-text">frontend/src/app</code> path in the main repository.
|
||||
|
||||
@startuml FrontendGeneral
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Person(user, "User")
|
||||
System_Boundary(frontend, "Frontend") {
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
Container(worker, "Worker", "Web worker")
|
||||
}
|
||||
|
||||
Rel(user, frontend_app, "Uses", "HTTPS")
|
||||
BiRel_L(frontend_app, worker, "Works with")
|
||||
|
||||
@enduml
|
||||
|
||||
```text
|
||||
▾ frontend/src/app/
|
||||
▸ main/
|
||||
▸ util/
|
||||
▸ worker/
|
||||
main.cljs
|
||||
worker.cljs
|
||||
```
|
||||
|
||||
* <code class="language-text">main.cljs</code> and <code class="language-text">main/</code> contain the main frontend app, written in
|
||||
ClojureScript language and using React framework, wrapped in [rumext
|
||||
library](https://github.com/funcool/rumext).
|
||||
* <code class="language-text">worker.cljs</code> and <code class="language-text">worker/</code> contain the web worker, to make expensive
|
||||
calculations in background.
|
||||
* <code class="language-text">util/</code> contains many generic utilities, non dependant on the user
|
||||
interface.
|
||||
|
||||
@startuml FrontendMain
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Component(ui, "ui", "main web component")
|
||||
Component(store, "store", "module")
|
||||
Component(refs, "refs", "module")
|
||||
Component(repo, "repo", "module")
|
||||
Component(streams, "streams", "module")
|
||||
Component(errors, "errors", "module")
|
||||
|
||||
Boundary(ui_namespaces, "ui namespaces") {
|
||||
Component(ui_auth, "auth", "web component")
|
||||
Component(ui_settings, "settings", "web component")
|
||||
Component(ui_dashboard, "dashboard", "web component")
|
||||
Component(ui_workspace, "workspace", "web component")
|
||||
Component(ui_viewer, "viewer", "web component")
|
||||
Component(ui_render, "render", "web component")
|
||||
Component(ui_exports, "exports", "web component")
|
||||
Component(ui_shapes, "shapes", "component library")
|
||||
Component(ui_components, "components", "component library")
|
||||
}
|
||||
|
||||
Boundary(data_namespaces, "data namespaces") {
|
||||
Component(data_common, "common", "events")
|
||||
Component(data_users, "users", "events")
|
||||
Component(data_dashboard, "dashboard", "events")
|
||||
Component(data_workspace, "workspace", "events")
|
||||
Component(data_viewer, "viewer", "events")
|
||||
Component(data_comments, "comments", "events")
|
||||
Component(data_fonts, "fonts", "events")
|
||||
Component(data_messages, "messages", "events")
|
||||
Component(data_modal, "modal", "events")
|
||||
Component(data_shortcuts, "shortcuts", "utilities")
|
||||
}
|
||||
|
||||
Lay_D(ui_exports, data_viewer)
|
||||
Lay_D(ui_settings, ui_components)
|
||||
Lay_D(data_viewer, data_common)
|
||||
Lay_D(data_fonts, data_messages)
|
||||
Lay_D(data_dashboard, data_modal)
|
||||
Lay_D(data_workspace, data_shortcuts)
|
||||
Lay_L(data_dashboard, data_fonts)
|
||||
Lay_L(data_workspace, data_comments)
|
||||
|
||||
Rel_Up(refs, store, "Watches")
|
||||
Rel_Up(streams, store, "Watches")
|
||||
|
||||
Rel(ui, ui_auth, "Routes")
|
||||
Rel(ui, ui_settings, "Routes")
|
||||
Rel(ui, ui_dashboard, "Routes")
|
||||
Rel(ui, ui_workspace, "Routes")
|
||||
Rel(ui, ui_viewer, "Routes")
|
||||
Rel(ui, ui_render, "Routes")
|
||||
|
||||
Rel(ui_render, ui_exports, "Uses")
|
||||
Rel(ui_workspace, ui_shapes, "Uses")
|
||||
Rel(ui_viewer, ui_shapes, "Uses")
|
||||
Rel_Right(ui_exports, ui_shapes, "Uses")
|
||||
|
||||
Rel(ui_auth, data_users, "Uses")
|
||||
Rel(ui_settings, data_users, "Uses")
|
||||
Rel(ui_dashboard, data_dashboard, "Uses")
|
||||
Rel(ui_dashboard, data_fonts, "Uses")
|
||||
Rel(ui_workspace, data_workspace, "Uses")
|
||||
Rel(ui_workspace, data_comments, "Uses")
|
||||
Rel(ui_viewer, data_viewer, "Uses")
|
||||
|
||||
@enduml
|
||||
|
||||
### General namespaces
|
||||
|
||||
* **store** contains the global state of the application. Uses an event loop
|
||||
paradigm, similar to Redux, with a global state object and a stream of events
|
||||
that modify it. Made with [potok library](https://funcool.github.io/potok/latest/).
|
||||
|
||||
* **refs** has the collection of references or lenses: RX streams that you can
|
||||
use to subscribe to parts of the global state, and be notified when they
|
||||
change.
|
||||
|
||||
* **streams** has some streams, derived from the main event stream, for keyboard
|
||||
and mouse events. Used mainly from the workspace viewport.
|
||||
|
||||
* **repo** contains the functions to make calls to backend.
|
||||
|
||||
* **errors** has functions with global error handlers, to manage exceptions or other
|
||||
kinds of errors in the ui or the data events, notify the user in a useful way,
|
||||
and allow to recover and continue working.
|
||||
|
||||
### UI namespaces
|
||||
|
||||
* **ui** is the root web component. It reads the current url and mounts the needed
|
||||
subcomponent depending on the route.
|
||||
|
||||
* **auth** has the web components for the login, register, password recover,
|
||||
etc. screens.
|
||||
|
||||
* **settings** has the web comonents for the user profile and settings screens.
|
||||
|
||||
* **dashboard** has the web components for the dashboard and its subsections.
|
||||
|
||||
* **workspace** has the web components for the file workspace and its subsections.
|
||||
|
||||
* **viewer** has the web components for the viewer and its subsections.
|
||||
|
||||
* **render** contain special web components to render one page or one specific
|
||||
shape, to be used in exports.
|
||||
|
||||
* **export** contain basic web components that display one shape or frame, to
|
||||
be used from exports render or else from dashboard and viewer thumbnails and
|
||||
other places.
|
||||
|
||||
* **shapes** is the basic collection of web components that convert all types of
|
||||
shapes in the corresponding svg elements, without adding any extra function.
|
||||
|
||||
* **components** a library of generic UI widgets, to be used as building blocks
|
||||
of penpot screens (text or numeric inputs, selects, forms, buttons...).
|
||||
|
||||
|
||||
### Data namespaces
|
||||
|
||||
* **users** has events to login and register, fetch the user profile and update it.
|
||||
|
||||
* **dashboard** has events to fetch and modify teams, projects and files.
|
||||
|
||||
* **fonts** has some extra events to manage uploaded fonts from dashboard.
|
||||
|
||||
* **workspace** has a lot of events to manage the current file and do all kinds of
|
||||
edits and updates.
|
||||
|
||||
* **comments** has some extra events to manage design comments.
|
||||
|
||||
* **viewer** has events to fetch a file contents to display, and manage the
|
||||
interactive behavior and hand-off.
|
||||
|
||||
* **common** has some events used from several places.
|
||||
|
||||
* **modal** has some events to show modal popup windows.
|
||||
|
||||
* **messages** has some events to show non-modal informative messages.
|
||||
|
||||
* **shortcuts** has some utility functions, used in other modules to setup the
|
||||
keyboard shortcuts.
|
||||
|
||||
|
||||
## Worker app
|
||||
|
||||
Some operations are costly to make in real time, so we leave them to be
|
||||
executed asynchronously in a web worker. This way they don't impact the user
|
||||
experience. Some of these operations are generating file thumbnails for the
|
||||
dashboard and maintaining some geometric indexes to speed up snap points while
|
||||
drawing.
|
||||
|
||||
@startuml FrontendWorker
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Component(worker, "worker", "worker entry point")
|
||||
|
||||
Boundary(worker_namespaces, "worker namespaces") {
|
||||
Component(thumbnails, "thumbnails", "worker methods")
|
||||
Component(snaps, "snaps", "worker methods")
|
||||
Component(selection, "selection", "worker methods")
|
||||
Component(impl, "impl", "worker methods")
|
||||
Component(import, "import", "worker methods")
|
||||
Component(export, "export", "worker methods")
|
||||
}
|
||||
|
||||
Rel(worker, thumbnails, "Uses")
|
||||
Rel(worker, impl, "Uses")
|
||||
Rel(worker, import, "Uses")
|
||||
Rel(worker, export, "Uses")
|
||||
Rel(impl, snaps, "Uses")
|
||||
Rel(impl, selection, "Uses")
|
||||
|
||||
@enduml
|
||||
|
||||
* **worker** contains the worker setup code and the global handler that receives
|
||||
requests from the main app, and process them.
|
||||
|
||||
* **thumbnails** has a method to generate the file thumbnails used in dashboard.
|
||||
|
||||
* **snaps** manages a distance index of shapes, and has a method to get
|
||||
other shapes near a given one, to be used in snaps while drawing.
|
||||
|
||||
* **selection** manages a geometric index of shapes, with methods to get what
|
||||
shapes are under the cursor at a given moment, for select.
|
||||
|
||||
* **impl** has a simple method to update all indexes in a page at once.
|
||||
|
||||
* **import** has a method to import a whole file from an external <code class="language-text">.penpot</code> archive.
|
||||
|
||||
* **export** has a method to export a whole file to an external <code class="language-text">.penpot</code> archive.
|
||||
|
||||
65
docs/technical-guide/developer/architecture/index.md
Normal file
65
docs/technical-guide/developer/architecture/index.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
---
|
||||
title: 3.1. Architecture
|
||||
---
|
||||
|
||||
# Architecture
|
||||
|
||||
This section gives an overall structure of the system.
|
||||
|
||||
Penpot has the architecture of a typical SPA. There is a frontend application,
|
||||
written in ClojureScript and using React framework, and served from a static
|
||||
web server. It talks to a backend application, that persists data on a
|
||||
PostgreSQL database.
|
||||
|
||||
The backend is written in Clojure, so front and back can share code and data
|
||||
structures without problem. Then, the code is compiled into JVM bytecode and
|
||||
run in a JVM environment.
|
||||
|
||||
There are some additional components, explained in subsections.
|
||||
|
||||
@startuml C4_Elements
|
||||
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
||||
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
||||
!include DEVICONS/react.puml
|
||||
!include DEVICONS/java.puml
|
||||
!include DEVICONS/clojure.puml
|
||||
!include DEVICONS/postgresql.puml
|
||||
!include DEVICONS/redis.puml
|
||||
!include DEVICONS/chrome.puml
|
||||
|
||||
HIDE_STEREOTYPE()
|
||||
|
||||
Person(user, "User")
|
||||
System_Boundary(frontend, "Frontend") {
|
||||
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
||||
Container(worker, "Worker", "Web worker")
|
||||
}
|
||||
|
||||
System_Boundary(backend, "Backend") {
|
||||
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
||||
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
||||
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
||||
Container(exporter, "Exporter", "ClojureScript / nodejs", "", "clojure")
|
||||
Container(browser, "Headless browser", "Chrome", "", "chrome")
|
||||
}
|
||||
|
||||
Rel(user, frontend_app, "Uses", "HTTPS")
|
||||
BiRel_L(frontend_app, worker, "Works with")
|
||||
BiRel(frontend_app, backend_app, "Open", "websocket")
|
||||
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
||||
Rel(backend_app, db, "Uses", "SQL")
|
||||
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
||||
Rel(backend_app, redis, "Notifies", "pub/sub")
|
||||
Rel(frontend_app, exporter, "Uses", "HTTPS")
|
||||
Rel(exporter, browser, "Uses", "puppeteer")
|
||||
Rel(browser, frontend_app, "Uses", "HTTPS")
|
||||
|
||||
@enduml
|
||||
|
||||
See more at
|
||||
|
||||
* [Frontend app](/technical-guide/developer/architecture/frontend/)
|
||||
* [Backend app](/technical-guide/developer/architecture/backend/)
|
||||
* [Exporter app](/technical-guide/developer/architecture/exporter/)
|
||||
* [Common code](/technical-guide/developer/architecture/common/)
|
||||
|
||||
111
docs/technical-guide/developer/backend.md
Normal file
111
docs/technical-guide/developer/backend.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
---
|
||||
title: 3.6. Backend Guide
|
||||
---
|
||||
|
||||
# Backend guide #
|
||||
|
||||
This guide intends to explain the essential details of the backend
|
||||
application.
|
||||
|
||||
|
||||
## REPL ##
|
||||
|
||||
In the devenv environment you can execute <code class="language-clojure">scripts/repl</code> to open a
|
||||
Clojure interactive shell ([REPL](https://codewith.mu/en/tutorials/1.0/repl)).
|
||||
|
||||
Once there, you can execute <code class="language-clojure">(restart)</code> to load and execute the backend
|
||||
process, or to reload it after making changes to the source code.
|
||||
|
||||
Then you have access to all backend code. You can import and use any function
|
||||
o read any global variable:
|
||||
|
||||
```clojure
|
||||
(require '[app.some.namespace :as some])
|
||||
(some/your-function arg1 arg2)
|
||||
```
|
||||
|
||||
There is a specific namespace <code class="language-clojure">app.srepl</code> with some functions useful to be
|
||||
executed from the repl and perform some tasks manually. Most of them accept
|
||||
a <code class="language-clojure">system</code> parameter. There is a global variable with this name, that contains
|
||||
the runtime information and configuration needed for the functions to run.
|
||||
|
||||
For example:
|
||||
|
||||
```clojure
|
||||
(require '[app.srepl.main :as srepl])
|
||||
(srepl/send-test-email! system "test@example.com")
|
||||
```
|
||||
|
||||
|
||||
## Fixtures ##
|
||||
|
||||
This is a development feature that allows populate the database with a
|
||||
good amount of content (usually used for just test the application or
|
||||
perform performance tweaks on queries).
|
||||
|
||||
In order to load fixtures, enter to the REPL environment with the <code class="language-clojure">scripts/repl</code>
|
||||
script, and then execute <code class="language-clojure">(app.cli.fixtures/run {:preset :small})</code>.
|
||||
|
||||
You also can execute this as a standalone script with:
|
||||
|
||||
```bash
|
||||
clojure -Adev -X:fn-fixtures
|
||||
```
|
||||
|
||||
NOTE: It is an optional step because the application can start with an
|
||||
empty database.
|
||||
|
||||
This by default will create a bunch of users that can be used to login
|
||||
in the application. All users uses the following pattern:
|
||||
|
||||
- Username: <code class="language-text">profileN@example.com</code>
|
||||
- Password: <code class="language-text">123123</code>
|
||||
|
||||
Where <code class="language-text">N</code> is a number from 0 to 5 on the default fixture parameters.
|
||||
|
||||
|
||||
## Migrations ##
|
||||
|
||||
The database migrations are located in two directories:
|
||||
|
||||
- <code class="language-text">src/app/migrations</code> (contains migration scripts in clojure)
|
||||
- <code class="language-text">src/app/migrations/sql</code> (contains the pure SQL migrations)
|
||||
|
||||
The SQL migration naming consists in the following:
|
||||
|
||||
```bash
|
||||
XXXX-<add|mod|del|drop|[...verb...]>-<table-name>-<any-additional-text>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
0025-del-generic-tokens-table
|
||||
0026-mod-profile-table-add-is-active-field
|
||||
```
|
||||
|
||||
**NOTE**: if table name has more than one word, we still use <code class="language-text">-</code> as a separator.
|
||||
|
||||
If you need to have a global overview of the all schema of the database you can extract it
|
||||
using postgresql:
|
||||
|
||||
```bash
|
||||
# (in the devenv environment)
|
||||
pg_dump -h postgres -s > schema.sql
|
||||
```
|
||||
|
||||
## Linter ##
|
||||
|
||||
There are no watch process for the linter; you will need to execute it
|
||||
manually. We use [clj-kondo][kondo] for linting purposes and the
|
||||
repository already comes with base configuration.
|
||||
|
||||
[kondo]: https://github.com/clj-kondo/clj-kondo
|
||||
|
||||
You can run **clj-kondo** as-is (is included in the devenv image):
|
||||
|
||||
```bash
|
||||
cd penpot/backend;
|
||||
clj-kondo --lint src
|
||||
```
|
||||
|
||||
494
docs/technical-guide/developer/common.md
Normal file
494
docs/technical-guide/developer/common.md
Normal file
|
|
@ -0,0 +1,494 @@
|
|||
---
|
||||
title: 3.4. Common Guide
|
||||
---
|
||||
|
||||
# Common guide
|
||||
|
||||
This section has articles related to all submodules (frontend, backend and
|
||||
exporter) such as: code style hints, architecture decisions, etc...
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
Both in the backend, the frontend and the exporter subsystems, there are an
|
||||
<code class="language-text">app.config</code> namespace that defines the global configuration variables,
|
||||
their specs and the default values.
|
||||
|
||||
All variables have a conservative default, meaning that you can set up a Penpot
|
||||
instance without changing any configuration, and it will be reasonably safe
|
||||
and useful.
|
||||
|
||||
In backend and exporter, to change the runtime values you need to set them in
|
||||
the process environment, following the rule that an environment variable in the
|
||||
form <code class="language-bash">PENPOT_<VARIABLE_NAME_IN_UPPERCASE></code> correspond to a configuration
|
||||
variable named <code class="language-bash">variable-name-in-lowercase</code>. Example:
|
||||
|
||||
```bash
|
||||
(env)
|
||||
PENPOT_ASSETS_STORAGE_BACKEND=assets-s3
|
||||
|
||||
(config)
|
||||
assets-storage-backend :assets-s3
|
||||
```
|
||||
|
||||
In frontend, the main <code class="language-text">resources/public/index.html</code> file includes (if it
|
||||
exists) a file named <code class="language-text">js/config.js</code>, where you can set configuration values
|
||||
as javascript global variables. The file is not created by default, so if
|
||||
you need it you must create it blank, and set the variables you want, in
|
||||
the form <code class="language-bash">penpot\<VariableNameInCamelCase></code>:
|
||||
|
||||
```js
|
||||
(js/config.js)
|
||||
var penpotPublicURI = "https://penpot.example.com";
|
||||
|
||||
(config)
|
||||
public-uri "https://penpot.example.com"
|
||||
```
|
||||
|
||||
### On premise instances
|
||||
|
||||
If you use the official Penpot docker images, as explained in the [Getting
|
||||
Started](/technical-guide/getting-started/#start-penpot) section, there is a
|
||||
[config.env](https://github.com/penpot/penpot/blob/develop/docker/images/config.env)
|
||||
file that sets the configuration environment variables. It's the same file for
|
||||
backend, exporter and frontend.
|
||||
|
||||
For this last one, there is a script
|
||||
[nginx-entrypoint.sh](https://github.com/penpot/penpot/blob/develop/docker/images/files/nginx-entrypoint.sh)
|
||||
that reads the environment and generates the <code class="language-text">js/config.js</code> when the container
|
||||
is started. This way all configuration is made in the single <code class="language-text">config.env</code> file.
|
||||
|
||||
|
||||
### Dev environment
|
||||
|
||||
If you use the [developer docker images](/technical-guide/developer/devenv/),
|
||||
the [docker-compose.yaml](https://github.com/penpot/penpot/blob/develop/docker/devenv/docker-compose.yaml)
|
||||
directly sets the environment variables more appropriate for backend and
|
||||
exporter development.
|
||||
|
||||
Additionally, the backend [start script](https://github.com/penpot/penpot/blob/develop/backend/scripts/start-dev)
|
||||
and [repl script](https://github.com/penpot/penpot/blob/develop/backend/scripts/repl) set
|
||||
some more variables.
|
||||
|
||||
The frontend uses only the defaults.
|
||||
|
||||
If you want to change any variable for your local environment, you can change
|
||||
<code class="language-text">docker-compose.yaml</code> and shut down and start again the container. Or you can
|
||||
modify the start script or directly set the environment variable in your
|
||||
session, and restart backend or exporter processes.
|
||||
|
||||
For frontend, you can manually create <code class="language-text">resources/public/js/config.js</code> (it's
|
||||
ignored in git) and define your settings there. Then, just reload the page.
|
||||
|
||||
## System logging
|
||||
|
||||
In [app.common.logging](https://github.com/penpot/penpot/blob/develop/common/src/app/common/logging.cljc)
|
||||
we have a general system logging utility, that may be used throughout all our
|
||||
code to generate execution traces, mainly for debugging.
|
||||
|
||||
You can add a trace anywhere, specifying the log level (<code class="language-text">trace</code>, <code class="language-text">debug</code>,
|
||||
<code class="language-text">info</code>, <code class="language-text">warn</code>, <code class="language-text">error</code>) and any number of key-values:
|
||||
|
||||
```clojure
|
||||
(ns app.main.data.workspace.libraries-helpers
|
||||
(:require [app.common.logging :as log]))
|
||||
|
||||
(log/set-level! :warn)
|
||||
|
||||
...
|
||||
|
||||
(defn generate-detach-instance
|
||||
[changes container shape-id]
|
||||
(log/debug :msg "Detach instance"
|
||||
:shape-id shape-id
|
||||
:container (:id container))
|
||||
...)
|
||||
```
|
||||
|
||||
The current namespace is tracked within the log message, and you can configure
|
||||
at runtime, by namespace, the log level (by default <code class="language-clojure">:warn</code>). Any trace below
|
||||
this level will be ignored.
|
||||
|
||||
Some keys have a special meaning:
|
||||
* <code class="language-clojure">:msg</code> is the main trace message.
|
||||
* <code class="language-clojure">::log/raw</code> outputs the value without any processing or prettifying.
|
||||
* <code class="language-clojure">::log/context</code> append metadata to the trace (not printed, it's to be
|
||||
processed by other tools).
|
||||
* <code class="language-clojure">::log/cause</code> (only in backend) attach a java exception object that will
|
||||
be printed in a readable way with the stack trace.
|
||||
* <code class="language-clojure">::log/async</code> (only in backend) if set to false, makes the log processing
|
||||
synchronous. If true (the default), it's executed in a separate thread.
|
||||
* <code class="language-clojure">:js/\<key></code> (only in frontend) if you prefix the key with the <code class="language-text">js/</code>
|
||||
namespace, the value will be printed as a javascript interactively
|
||||
inspectionable object.
|
||||
* <code class="language-clojure">:err</code> (only in frontend) attach a javascript exception object, and it
|
||||
will be printed in a readable way with the stack trace.
|
||||
|
||||
### backend
|
||||
|
||||
The logging utility uses a different library for Clojure and Clojurescript. In
|
||||
the first case we use [log4j2](https://logging.apache.org/log4j/2.x) to have
|
||||
much flexibility.
|
||||
|
||||
The configuration is made in [log4j2.xml](https://github.com/penpot/penpot/blob/develop/backend/resources/log4j2.xml)
|
||||
file. The Logger used for this is named "app" (there are other loggers for
|
||||
other subsystems). The default configuration just outputs all traces of level
|
||||
<code class="language-clojure">debug</code> or higher to the console standard output.
|
||||
|
||||
There is a different [log4j2-devenv](https://github.com/penpot/penpot/blob/develop/backend/resources/log4j2-devenv.xml)
|
||||
for the development environment. This one outputs traces of level <code class="language-text">trace</code> or
|
||||
higher to a file, and <code class="language-text">debug</code> or higher to a <code class="language-text">zmq</code> queue, that may be
|
||||
subscribed for other parts of the application for further processing.
|
||||
|
||||
The ouput for a trace in <code class="language-text">logs/main.log</code> uses the format
|
||||
|
||||
```bash
|
||||
[<date time>] : <level> <namespace> - <key1=val1> <key2=val2> ...
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
[2022-04-27 06:59:08.820] T app.rpc - action="register", name="update-file"
|
||||
```
|
||||
|
||||
The <code class="language-text">zmq</code> queue is not used in the default on premise or devenv setups, but there
|
||||
are a number of handlers you can use in custom instances to save errors in the
|
||||
database, or send them to a [Sentry](https://sentry.io/welcome/) or similar
|
||||
service, for example.
|
||||
|
||||
### frontend and exporter
|
||||
|
||||
In the Clojurescript subservices, we use [goog.log](https://google.github.io/closure-library/api/goog.log.html)
|
||||
library. This is much simpler, and basically outputs the traces to the console
|
||||
standard output (the devtools in the browser or the console in the nodejs
|
||||
exporter).
|
||||
|
||||
In the browser, we have an utility [debug function](/technical-guide/developer/frontend/#console-debug-utility)
|
||||
that enables you to change the logging level of any namespace (or of the whole
|
||||
app) in a live environment:
|
||||
|
||||
```javascript
|
||||
debug.set_logging("namespace", "level")
|
||||
```
|
||||
|
||||
## Assertions
|
||||
|
||||
Penpot source code has this types of assertions:
|
||||
|
||||
### **assert**
|
||||
|
||||
Just using the clojure builtin `assert` macro.
|
||||
|
||||
Example:
|
||||
|
||||
```clojure
|
||||
(assert (number? 3) "optional message")
|
||||
```
|
||||
|
||||
This asserts are only executed in development mode. In production
|
||||
environment all asserts like this will be ignored by runtime.
|
||||
|
||||
### **spec/assert**
|
||||
|
||||
Using the <code class="language-text">app.common.spec/assert</code> macro.
|
||||
|
||||
This macro is based in <code class="language-text">cojure.spec.alpha/assert</code> macro, and it's
|
||||
also ignored in a production environment.
|
||||
|
||||
The Penpot variant doesn't have any runtime checks to know if asserts
|
||||
are disabled. Instead, the assert calls are completely removed by the
|
||||
compiler/runtime, thus generating simpler and faster code in production
|
||||
builds.
|
||||
|
||||
Example:
|
||||
|
||||
```clojure
|
||||
(require '[clojure.spec.alpha :as s]
|
||||
'[app.common.spec :as us])
|
||||
|
||||
(s/def ::number number?)
|
||||
|
||||
(us/assert ::number 3)
|
||||
```
|
||||
|
||||
### **spec/verify**
|
||||
|
||||
An assertion type that is always executed.
|
||||
|
||||
Example:
|
||||
|
||||
```clojure
|
||||
(require '[app.common.spec :as us])
|
||||
|
||||
(us/verify ::number 3)
|
||||
```
|
||||
|
||||
This macro enables you to have assertions on production code, that
|
||||
generate runtime exceptions when failed (make sure you handle them
|
||||
appropriately).
|
||||
|
||||
## Unit tests
|
||||
|
||||
We expect all Penpot code (either in frontend, backend or common subsystems) to
|
||||
have unit tests, i.e. the ones that test a single unit of code, in isolation
|
||||
from other blocks. Currently we are quite far from that objective, but we are
|
||||
working to improve this.
|
||||
|
||||
### Running tests with kaocha
|
||||
|
||||
Unit tests are executed inside the [development environment](/technical-guide/developer/devenv).
|
||||
|
||||
We can use [kaocha test runner](https://cljdoc.org/d/lambdaisland/kaocha/), and
|
||||
we have prepared, for convenience, some aliases in <code class="language-text">deps.edn</code> files. To run
|
||||
them, just go to <code class="language-text">backend</code>, <code class="language-text">frontend</code> or <code class="language-text">common</code> and execute:
|
||||
|
||||
```bash
|
||||
# To run all tests once
|
||||
clojure -M:dev:test
|
||||
|
||||
# To run all tests and keep watching for changes
|
||||
clojure -M:dev:test --watch
|
||||
|
||||
# To run a single tests module
|
||||
clojure -M:dev:test --focus common-tests.logic.comp-sync-test
|
||||
|
||||
# To run a single test
|
||||
clojure -M:dev:test --focus common-tests.logic.comp-sync-test/test-sync-when-changing-attribute
|
||||
```
|
||||
|
||||
Watch mode runs all tests when some file changes, except if some tests failed
|
||||
previously. In this case it only runs the failed tests. When they pass, then
|
||||
runs all of them again.
|
||||
|
||||
You can also mark tests in the code by adding metadata:
|
||||
|
||||
```clojure
|
||||
;; To skip a test, for example when is not working or too slow
|
||||
(deftest ^:kaocha/skip bad-test
|
||||
(is (= 2 1)))
|
||||
|
||||
;; To skip it but warn you during test run, so you don't forget it
|
||||
(deftest ^:kaocha/pending bad-test
|
||||
(is (= 2 1)))
|
||||
```
|
||||
|
||||
Please refer to the [kaocha manual](https://cljdoc.org/d/lambdaisland/kaocha/1.91.1392/doc/6-focusing-and-skipping)
|
||||
for how to define custom metadata and other ways of selecting tests.
|
||||
|
||||
**NOTE**: in <code class="language-text">frontend</code> we still can't use kaocha to run the tests. We are on
|
||||
it, but for now we use shadow-cljs with <code class="language-text">package.json</code> scripts:
|
||||
|
||||
```bash
|
||||
yarn run test
|
||||
yarn run test:watch
|
||||
```
|
||||
|
||||
#### Test output
|
||||
|
||||
The default kaocha reporter outputs a summary for the test run. There is a pair
|
||||
of brackets <code class="language-bash">[ ]</code> for each suite, a pair of parentheses <code class="language-bash">( )</code> for each test,
|
||||
and a dot <code class="language-bash">.</code> for each assertion <code class="language-bash">t/is</code> inside tests.
|
||||
|
||||
```bash
|
||||
penpot@c261c95d4623:~/penpot/common$ clojure -M:dev:test
|
||||
[(...)(............................................................
|
||||
.............................)(....................................
|
||||
..)(..........)(.................................)(.)(.............
|
||||
.......................................................)(..........
|
||||
.....)(......)(.)(......)(.........................................
|
||||
..............................................)(............)]
|
||||
190 tests, 3434 assertions, 0 failures.
|
||||
```
|
||||
|
||||
All standard output from the tests is captured and hidden, except if some test
|
||||
fails. In this case, the output for the failing test is shown in a box:
|
||||
|
||||
```bash
|
||||
FAIL in sample-test/stdout-fail-test (sample_test.clj:10)
|
||||
Expected:
|
||||
:same
|
||||
Actual:
|
||||
-:same +:not-same
|
||||
╭───── Test output ───────────────────────────────────────────────────────
|
||||
│ Can you see this?
|
||||
╰─────────────────────────────────────────────────────────────────────────
|
||||
2 tests, 2 assertions, 1 failures.
|
||||
```
|
||||
|
||||
You can bypass the capture with the command line:
|
||||
|
||||
```bash
|
||||
clojure -M:dev:test --no-capture-output
|
||||
```
|
||||
|
||||
Or for some specific output:
|
||||
|
||||
```clojure
|
||||
(ns sample-test
|
||||
(:require [clojure.test :refer :all]
|
||||
[kaocha.plugin.capture-output :as capture]))
|
||||
|
||||
(deftest stdout-pass-test
|
||||
(capture/bypass
|
||||
(println "This message should be displayed"))
|
||||
(is (= :same :same)))
|
||||
```
|
||||
|
||||
### Running tests in the REPL
|
||||
|
||||
An alternative way of running tests is to do it from inside the
|
||||
[REPL](/technical-guide/developer/backend/#repl) you can use in the backend and
|
||||
common apps in the development environment.
|
||||
|
||||
We have a helper function <code class="language-bash">(run-tests)</code> that refreshes the environment (to avoid
|
||||
having [stale tests](https://practical.li/clojure/testing/unit-testing/#command-line-test-runners))
|
||||
and runs all tests or a selection. It is defined in <code class="language-bash">backend/dev/user.clj</code> and
|
||||
<code class="language-text">common/dev/user.clj</code>, so it's available without importing anything.
|
||||
|
||||
First start a REPL:
|
||||
|
||||
```bash
|
||||
~/penpot/backend$ scripts/repl
|
||||
```
|
||||
|
||||
And then:
|
||||
|
||||
```clojure
|
||||
;; To run all tests
|
||||
(run-tests)
|
||||
|
||||
;; To run all tests in one namespace
|
||||
(run-tests 'some.namespace)
|
||||
|
||||
;; To run a single test
|
||||
(run-tests 'some.namespace/some-test)
|
||||
|
||||
;; To run all tests in one or several namespaces,
|
||||
;; selected by a regular expression
|
||||
(run-tests #"^backend-tests.rpc.*")
|
||||
```
|
||||
|
||||
### Writing unit tests
|
||||
|
||||
We write tests using the standard [Clojure test
|
||||
API](https://clojure.github.io/clojure/clojure.test-api.html). You can find a
|
||||
[guide to writing unit tests](https://practical.li/clojure/testing/unit-testing) at Practicalli
|
||||
Clojure, that we follow as much as possible.
|
||||
|
||||
#### Sample files helpers
|
||||
|
||||
An important issue when writing tests in Penpot is to have files with the
|
||||
specific configurations we need to test. For this, we have defined a namespace
|
||||
of helpers to easily create files and its elements with sample data.
|
||||
|
||||
To make handling of uuids more convenient, those functions have a uuid
|
||||
registry. Whenever you create an object, you may give a <code class="language-clojure">:label</code>, and the id of
|
||||
the object will be stored in the registry associated with this label, so you
|
||||
can easily recover it later.
|
||||
|
||||
You have functions to create files, pages and shapes, to connect them and
|
||||
specify their attributes, having all of them default values if not set.
|
||||
|
||||
Files also store in metadata the **current page**, so you can control in what
|
||||
page the <code class="language-clojure">add-</code> and <code class="language-clojure">get-</code> functions will operate.
|
||||
|
||||
```clojure
|
||||
(ns common-tests.sample-helpers-test
|
||||
(:require
|
||||
[app.common.test-helpers.files :as thf]
|
||||
[app.common.test-helpers.ids-map :as thi]
|
||||
[app.common.test-helpers.shapes :as ths]
|
||||
[clojure.test :as t]))
|
||||
|
||||
(t/deftest test-create-file
|
||||
(let [;; Create a file with one page
|
||||
f1 (thf/sample-file :file1)
|
||||
|
||||
;; Same but define the label of the page, to retrieve it later
|
||||
f2 (thf/sample-file :file2 :page-label :page1)
|
||||
|
||||
;; Set the :name attribute of the created file
|
||||
f3 (thf/sample-file :file3 :name "testing file")
|
||||
|
||||
;; Create an isolated page
|
||||
p2 (thf/sample-page :page2 :name "testing page")
|
||||
|
||||
;; Create a second page and add to the file
|
||||
f4 (-> (thf/sample-file :file4 :page-label :page3)
|
||||
(thf/add-sample-page :page4 :name "other testing page"))
|
||||
|
||||
;; Create an isolated shape
|
||||
p2 (thf/sample-shape :shape1 :type :rect :name "testing shape")
|
||||
|
||||
;; Add a couple of shapes to a previous file, in different pages
|
||||
f5 (-> f4
|
||||
(ths/add-sample-shape :shape2)
|
||||
(thf/switch-to-page :page4)
|
||||
(ths/add-sample-shape :shape3 :name "other testing shape"
|
||||
:width 100))
|
||||
|
||||
;; Retrieve created shapes
|
||||
s1 (ths/get-shape f4 :shape1)
|
||||
s2 (ths/get-shape f5 :shape2 :page-label :page3)
|
||||
s3 (ths/get-shape f5 :shape3)]
|
||||
|
||||
;; Check some values
|
||||
(t/is (= (:name f1) "Test file"))
|
||||
(t/is (= (:name f3) "testing file"))
|
||||
(t/is (= (:id f2) (thi/id :file2)))
|
||||
(t/is (= (:id (thf/current-page f2)) (thi/id :page1)))
|
||||
(t/is (= (:id s1) (thi/id :shape1)))
|
||||
(t/is (= (:name s1) "Rectangle"))
|
||||
(t/is (= (:name s3) "testing shape"))
|
||||
(t/is (= (:width s3) 100))
|
||||
(t/is (= (:width (:selrect s3)) 100))))
|
||||
```
|
||||
|
||||
Also there are functions to make some transformations, like creating a
|
||||
component, instantiating it or swapping a copy.
|
||||
|
||||
```clojure
|
||||
(ns app.common-tests.sample-components-test
|
||||
(:require
|
||||
[app.common.test-helpers.components :as thc]
|
||||
[app.common.test-helpers.files :as thf]
|
||||
[app.common.test-helpers.shapes :as ths]))
|
||||
|
||||
(t/deftest test-create-component
|
||||
(let [;; Create a file with one component
|
||||
f1 (-> (thf/sample-file :file1)
|
||||
(ths/add-sample-shape :frame1 :type :frame)
|
||||
(ths/add-sample-shape :rect1 :type :rect
|
||||
:parent-label :frame1)
|
||||
(thc/make-component :component1 :frame1))]))
|
||||
```
|
||||
|
||||
Finally, there are composition helpers, to build typical structures with a
|
||||
single line of code. And the files module has some functions to display the
|
||||
contents of a file, in a way similar to `debug/dump-tree` but showing labels
|
||||
instead of ids:
|
||||
|
||||
```clojure
|
||||
(ns app.common-tests.sample-compositions-test
|
||||
(:require
|
||||
[app.common.test-helpers.compositions :as tho]
|
||||
[app.common.test-helpers.files :as thf]))
|
||||
|
||||
(t/deftest test-create-composition
|
||||
(let [f1 (-> (thf/sample-file :file1)
|
||||
(tho/add-simple-component-with-copy :component1
|
||||
:main-root
|
||||
:main-child
|
||||
:copy-root))]
|
||||
(ctf/dump-file f1 :show-refs? true)))
|
||||
|
||||
;; {:main-root} [:name Frame1] # [Component :component1]
|
||||
;; :main-child [:name Rect1]
|
||||
;;
|
||||
;; :copy-root [:name Frame1] #--> [Component :component1] :main-root
|
||||
;; <no-label> [:name Rect1] ---> :main-child
|
||||
```
|
||||
|
||||
You can see more examples of usage by looking at the existing unit tests.
|
||||
|
||||
441
docs/technical-guide/developer/data-guide.md
Normal file
441
docs/technical-guide/developer/data-guide.md
Normal file
|
|
@ -0,0 +1,441 @@
|
|||
---
|
||||
title: 3.7. Data Guide
|
||||
---
|
||||
|
||||
# Data Guide
|
||||
|
||||
The data structures are one of the most complex and important parts of Penpot.
|
||||
It's critical that the data integrity is always maintained throughout the whole
|
||||
usage, and also file exports & imports and data model evolution.
|
||||
|
||||
To modify the data structure (the most typical case will be to add a new attribute
|
||||
to the shapes), this list must be checked. This is not an exhaustive list, but
|
||||
all of this is important in general.
|
||||
|
||||
## General considerations
|
||||
|
||||
* We prefer that the page and shape attributes are optional. I.E. there is a
|
||||
default object behavior, that occurs when the attribute is not present, and
|
||||
its presence activates some feature (example: if there is no <code class="language-clojure">fill-color</code>,
|
||||
the shape is not filled). When you revert to the default state, it's better
|
||||
to remove the attribute than leaving it with <code class="language-clojure">null</code> value. There are some
|
||||
process (for example import & export) that filter out and remove all
|
||||
attributes that are <code class="language-clojure">null</code>.
|
||||
|
||||
* So never expect that attribute with <code class="language-clojure">null</code> value is a different state that
|
||||
without the attribute.
|
||||
|
||||
* In objects attribute names we don't use special symbols that are allowed by
|
||||
Clojure (for example ending it with ? for boolean values), because this may
|
||||
cause problems when exporting.
|
||||
|
||||
## Code organization in abstraction levels
|
||||
|
||||
Initially, Penpot data model implementation was organized in a different way.
|
||||
We are currently in a process of reorganization. The objective is to have data
|
||||
manipulation code structured in abstraction layers, with well-defined
|
||||
boundaries.
|
||||
|
||||
At this moment the namespace structure is already organized as described here,
|
||||
but there is much code that does not comply with these rules, and needs to be
|
||||
moved or refactored. We expect to be refactoring existing modules incrementally,
|
||||
each time we do an important functionality change.
|
||||
|
||||
### Abstract data types
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ types/
|
||||
file.cljc
|
||||
page.cljc
|
||||
shape.cljc
|
||||
color.cljc
|
||||
component.cljc
|
||||
...
|
||||
```
|
||||
|
||||
Namespaces here represent a single data structure, or a fragment of one, as an
|
||||
abstract data type. Each structure has:
|
||||
|
||||
* A **schema spec** that defines the structure of the type and its values:
|
||||
|
||||
```clojure
|
||||
(sm/define! ::fill
|
||||
[:map {:title "Fill"}
|
||||
[:fill-color {:optional true} ::ctc/rgb-color]
|
||||
[:fill-opacity {:optional true} ::sm/safe-number]
|
||||
...)
|
||||
|
||||
(sm/define! ::shape-attrs
|
||||
[:map {:title "ShapeAttrs"}
|
||||
[:name {:optional true} :string]
|
||||
[:selrect {:optional true} ::grc/rect]
|
||||
[:points {:optional true} ::points]
|
||||
[:blocked {:optional true} :boolean]
|
||||
[:fills {:optional true}
|
||||
[:vector {:gen/max 2} ::fill]]
|
||||
...)
|
||||
```
|
||||
|
||||
* **Helper functions** to create, query and manipulate the structure. Helpers
|
||||
at this level only are allowed to see the internal attributes of a type.
|
||||
Updaters receive an object of the type, and return a new object modified,
|
||||
also ensuring the internal integrity of the data after the change.
|
||||
|
||||
```clojure
|
||||
(defn setup-shape
|
||||
"A function that initializes the geometric data of the shape. The props must
|
||||
contain at least :x :y :width :height."
|
||||
[{:keys [type] :as props}]
|
||||
...)
|
||||
|
||||
(defn has-direction?
|
||||
[interaction]
|
||||
(#{:slide :push} (-> interaction :animation :animation-type)))
|
||||
|
||||
(defn set-direction
|
||||
[interaction direction]
|
||||
(dm/assert!
|
||||
"expected valid interaction map"
|
||||
(check-interaction! interaction))
|
||||
(dm/assert!
|
||||
"expected valid direction"
|
||||
(contains? direction-types direction))
|
||||
(dm/assert!
|
||||
"expected compatible interaction map"
|
||||
(has-direction? interaction))
|
||||
(update interaction :animation assoc :direction direction))
|
||||
```
|
||||
|
||||
> IMPORTANT: we should always use helper functions to access and modify these data
|
||||
> structures. Avoid direct attribute read or using functions like <code class="language-clojure">assoc</code> or
|
||||
> <code class="language-clojure">update</code>, even if the information is contained in a single attribute. This way
|
||||
> it will be much simpler to add validation checks or to modify the internal
|
||||
> representation of a type, and will be easier to search for places in the code
|
||||
> where this data item is used.
|
||||
>
|
||||
> Currently much of Penpot code does not follow this requirement, but it
|
||||
should do in new code or in any refactor.
|
||||
|
||||
### File operations
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ files/
|
||||
helpers.cljc
|
||||
shapes_helpers.cljc
|
||||
...
|
||||
```
|
||||
|
||||
Functions that modify a file object (or a part of it) in place, returning the
|
||||
file object changed. They ensure the referential integrity within the file, or
|
||||
between a file and its libraries.
|
||||
|
||||
```clojure
|
||||
(defn resolve-component
|
||||
"Retrieve the referenced component, from the local file or from a library"
|
||||
[shape file libraries & {:keys [include-deleted?] :or {include-deleted? False}}]
|
||||
(if (= (:component-file shape) (:id file))
|
||||
(ctkl/get-component (:data file) (:component-id shape) include-deleted?)
|
||||
(get-component libraries
|
||||
(:component-file shape)
|
||||
(:component-id shape)
|
||||
:include-deleted? include-deleted?)))
|
||||
|
||||
(defn delete-component
|
||||
"Mark a component as deleted and store the main instance shapes inside it, to
|
||||
be able to be recovered later."
|
||||
[file-data component-id skip-undelete? Main-instance]
|
||||
(let [components-v2 (dm/get-in file-data [:options :components-v2])]
|
||||
(if (or (not components-v2) skip-undelete?)
|
||||
(ctkl/delete-component file-data component-id)
|
||||
(let [set-main-instance ;; If there is a saved main-instance, restore it.
|
||||
#(if main-instance
|
||||
(assoc-in % [:objects (:main-instance-id %)] main-instance)
|
||||
%)]
|
||||
(-> file-data
|
||||
(ctkl/update-component component-id load-component-objects)
|
||||
(ctkl/update-component component-id set-main-instance)
|
||||
(ctkl/mark-component-deleted component-id))))))
|
||||
```
|
||||
|
||||
> This module is still needing an important refactor. Mainly to take functions
|
||||
> from common.types and move them here.
|
||||
|
||||
#### File validation and repair
|
||||
|
||||
There is a function in <code class="language-clojure">app.common.files.validate</code> that checks a file for
|
||||
referential and semantic integrity. It's called automatically when file changes
|
||||
are sent to backend, but may be invoked manually whenever it's needed.
|
||||
|
||||
### File changes objects
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ files/
|
||||
changes_builder.cljc
|
||||
changes.cljc
|
||||
...
|
||||
```
|
||||
|
||||
Wrap the update functions in file operations module into <code class="language-clojure">changes</code> objects, that
|
||||
may be serialized, stored, sent to backend and executed to actually modify a file
|
||||
object. They should not contain business logic or algorithms. Only adapt the
|
||||
interface to the file operations or types.
|
||||
|
||||
```clojure
|
||||
(sm/define! ::changes
|
||||
[:map {:title "changes"}
|
||||
[:redo-changes vector?]
|
||||
[:undo-changes seq?]
|
||||
[:origin {:optional true} any?]
|
||||
[:save-undo? {:optional true} boolean?]
|
||||
[:stack-undo? {:optional true} boolean?]
|
||||
[:undo-group {:optional true} any?]])
|
||||
|
||||
(defmethod process-change :add-component
|
||||
[file-data params]
|
||||
(ctkl/add-component file-data params))
|
||||
```
|
||||
|
||||
### Business logic
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ logic/
|
||||
shapes.cljc
|
||||
libraries.cljc
|
||||
```
|
||||
|
||||
Functions that implement semantic user actions, in an abstract way (independent
|
||||
of UI). They don't directly modify files, but generate changes objects, that
|
||||
may be executed in frontend or sent to backend.
|
||||
|
||||
```clojure
|
||||
(defn generate-instantiate-component
|
||||
"Generate changes to create a new instance from a component."
|
||||
[changes objects file-id component-id position page libraries old-id parent-id
|
||||
frame-id {:keys [force-frame?] :or {force-frame? False}}]
|
||||
(let [component (ctf/get-component libraries file-id component-id)
|
||||
parent (when parent-id (get objects parent-id))
|
||||
library (get libraries file-id)
|
||||
components-v2 (dm/get-in library [:data :options :components-v2])
|
||||
[new-shape new-shapes]º
|
||||
(ctn/make-component-instance page
|
||||
Component
|
||||
(:data library)
|
||||
Position
|
||||
Components-v2
|
||||
(cond-> {}
|
||||
force-frame? (assoc :force-frame-id frame-id)))
|
||||
changes (cond-> (pcb/add-object changes first-shape {:ignore-touched true})
|
||||
(some? old-id) (pcb/amend-last-change #(assoc % :old-id old-id)))
|
||||
changes (reduce #(pcb/add-object %1 %2 {:ignore-touched true})
|
||||
changes
|
||||
(rest new-shapes))]
|
||||
[new-shape changes]))
|
||||
```
|
||||
|
||||
## Data migrations
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ files/
|
||||
migrations.cljc
|
||||
```
|
||||
|
||||
When changing the model it's essential to take into account that the existing
|
||||
Penpot files must keep working without changes. If you follow the general
|
||||
considerations stated above, usually this is automatic, since the objects
|
||||
already in the database just have the default behavior, that should be the same
|
||||
as before the change. And the new features apply to new or edited objects.
|
||||
|
||||
But if this is not possible, and we are talking of a breaking change, you can
|
||||
write a data migration. Just define a new data version and a migration script
|
||||
in <code class="language-text">migrations.cljc</code> and increment <code class="language-text">file-version</code> in <code class="language-text">common.cljc</code>.
|
||||
|
||||
From then on, every time a file is loaded from the database, if its version
|
||||
number is lower than the current version in the app, the file data will be
|
||||
handled to all the needed migration functions. If you later modify and save
|
||||
the file, it will be now updated in database.
|
||||
|
||||
## Shape edit forms
|
||||
|
||||
|
||||
|
||||
```text
|
||||
▾ frontend/
|
||||
▾ src/
|
||||
▾ app/
|
||||
▾ main/
|
||||
▾ ui/
|
||||
▾ workspace/
|
||||
▾ sidebar/
|
||||
▾ options/
|
||||
▸ menus/
|
||||
▸ rows/
|
||||
▾ shapes/
|
||||
bool.cljs
|
||||
circle.cljs
|
||||
frame.cljs
|
||||
group.cljs
|
||||
image.cljs
|
||||
multiple.cljs
|
||||
path.cljs
|
||||
rect.cljs
|
||||
svg_raw.cljs
|
||||
text.cljs
|
||||
```
|
||||
|
||||
* In <code class="language-text">shapes/*.cljs</code> there are the components that show the edit menu of each
|
||||
shape type.
|
||||
|
||||
* In <code class="language-text">menus/*.cljs</code> there are the building blocks of these menus.
|
||||
|
||||
* And in <code class="language-text">rows/*.cljs</code> there are some pieces, for example color input and
|
||||
picker.
|
||||
|
||||
## Multiple edit
|
||||
|
||||
When modifying the shape edit forms, you must take into account that these
|
||||
forms may edit several shapes at the same time, even of different types.
|
||||
|
||||
When more than one shape is selected, the form inside <code class="language-text">multiple.cljs</code> is used.
|
||||
At the top of this module, a couple of maps define what attributes may be edited
|
||||
and how, for each type of shape.
|
||||
|
||||
Then, the blocks in <code class="language-text">menus/*.cljs</code> are used, but they are not given a shape, but
|
||||
a values map. For each attribute, if all shapes have the same value, it is taken;
|
||||
if not, the attribute will have the value <code class="language-clojure">:multiple</code>.
|
||||
|
||||
The form blocks must be prepared for this value, display something useful to
|
||||
the user in this case, and do a meaningful action when changing the value.
|
||||
Usually this will be to set the attribute to a fixed value in all selected
|
||||
shapes, but **only** those that may have the attribute (for example, only text
|
||||
shapes have font attributes, or only rects has border radius).
|
||||
|
||||
## Component synchronization
|
||||
|
||||
```text
|
||||
▾ common/
|
||||
▾ src/app/common/
|
||||
▾ types/
|
||||
component.cljc
|
||||
```
|
||||
|
||||
For all shape attributes, you must take into account what happens when the
|
||||
attribute in a main component is changed and then the copies are synchronized.
|
||||
|
||||
In <code class="language-text">component.cljc</code> there is a structure <code class="language-clojure">sync-attrs</code> that maps shape
|
||||
attributes to sync groups. When an attribute is changed in a main component,
|
||||
the change will be propagated to its copies. If the change occurs in a copy,
|
||||
the group will be marked as *touched* in the copy shape, and from then on,
|
||||
further changes in the main to this attribute, or others in the same group,
|
||||
will not be propagated.
|
||||
|
||||
Any attribute that is not in this map will be ignored in synchronizations.
|
||||
|
||||
## Render shapes, export & import
|
||||
|
||||
```text
|
||||
▾ frontend/
|
||||
▾ src/
|
||||
▾ app/
|
||||
▾ main/
|
||||
▾ ui/
|
||||
▾ shapes/
|
||||
▸ text/
|
||||
attrs.cljs
|
||||
bool.cljs
|
||||
circle.cljs
|
||||
custom_stroke.cljs
|
||||
embed.cljs
|
||||
export.cljs
|
||||
fill_image.cljs
|
||||
filters.cljs
|
||||
frame.cljs
|
||||
gradients.cljs
|
||||
group.cljs
|
||||
image.cljs
|
||||
mask.cljs
|
||||
path.cljs
|
||||
rect.cljs
|
||||
shape.cljs
|
||||
svg_defs.cljs
|
||||
svg_raw.cljs
|
||||
text.cljs
|
||||
▾ worker/
|
||||
▾ import/
|
||||
parser.cljs
|
||||
```
|
||||
|
||||
To export a penpot file, basically we use the same system that is used to
|
||||
display shapes in the workspace or viewer. In <code class="language-text">shapes/*.cljs</code> there are
|
||||
components that render one shape of each type into a SVG node.
|
||||
|
||||
But to be able to import the file later, some attributes that not match
|
||||
directly to SVG properties need to be added as metadata (for example,
|
||||
proportion locks, constraints, stroke alignment...). This is done in the
|
||||
<code class="language-text">export.cljs</code> module.
|
||||
|
||||
Finally, to import a file, we make use of <code class="language-text">parser.cljs</code>, a module that
|
||||
contains the <code class="language-clojure">parse-data</code> function. It receives a SVG node (possibly with
|
||||
children) and converts it into a Penpot shape object. There are auxiliary
|
||||
functions to read and convert each group of attributes, from the node
|
||||
properties or the metadata (with the <code class="language-clojure">get-meta</code> function).
|
||||
|
||||
Any attribute that is not included in the export and import functions
|
||||
will not be exported and will be lost if reimporting the file again.
|
||||
|
||||
## Code generation
|
||||
|
||||
```text
|
||||
▾ frontend/
|
||||
▾ src/
|
||||
▾ app/
|
||||
▾ main/
|
||||
▾ ui/
|
||||
▾ viewer/
|
||||
▾ inspect/
|
||||
▾ attributes/
|
||||
blur.cljs
|
||||
common.cljs
|
||||
fill.cljs
|
||||
image.cljs
|
||||
layout.cljs
|
||||
shadow.cljs
|
||||
stroke.cljs
|
||||
svg.cljs
|
||||
text.cljs
|
||||
attributes.cljs
|
||||
code.cljs
|
||||
▾ util/
|
||||
code_gen.cljs
|
||||
markup_html.cljs
|
||||
markup_svg.cljs
|
||||
style_css.cljs
|
||||
style_css_formats.cljs
|
||||
style_css_values.cljs
|
||||
```
|
||||
|
||||
In the inspect panel we have two modes:
|
||||
|
||||
|
||||
|
||||
For the Info tab, the <code class="language-text">attributes.cljs</code> module and all modules under
|
||||
<code class="language-text">attributes/*.cljs</code> have the components that extract the attributes to inspect
|
||||
each type of shape.
|
||||
|
||||
|
||||
|
||||
For the Code tab, the <code class="language-text">util/code_gen.cljs</code> module is in charge. It calls the
|
||||
other modules in <code class="language-text">util/</code> depending on the format.
|
||||
|
||||
For HTML and CSS, there are functions that generate the code as needed from the
|
||||
shapes. For SVG, it simply takes the nodes from the viewer main viewport and
|
||||
prettily formats it.
|
||||
199
docs/technical-guide/developer/data-model/index.md
Normal file
199
docs/technical-guide/developer/data-model/index.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
---
|
||||
title: 3.2. Data model
|
||||
---
|
||||
|
||||
# Penpot Data Model
|
||||
|
||||
This is the conceptual data model. The actual representations of those entities
|
||||
slightly differ, depending on the environment (frontend app, backend RPC calls
|
||||
or the SQL database, for example). But the concepts are always the same.
|
||||
|
||||
The diagrams use [basic UML notation with PlantUML](https://plantuml.com/en/class-diagram).
|
||||
|
||||
## Users, teams and projects
|
||||
|
||||
@startuml TeamModel
|
||||
|
||||
hide members
|
||||
|
||||
class Profile
|
||||
class Team
|
||||
class Project
|
||||
class File
|
||||
class StorageObject
|
||||
class CommentThread
|
||||
class Comment
|
||||
class ShareLink
|
||||
|
||||
Profile "*" -right- "*" Team
|
||||
Team *--> "*" Project
|
||||
Profile "*" -- "*" Project
|
||||
Project *--> "*" File
|
||||
Profile "*" -- "*" File
|
||||
File "*" <-- "*" File : libraries
|
||||
File *--> "*" StorageObject : media_objects
|
||||
File *--> "*" CommentThread : comment_threads
|
||||
CommentThread *--> "*" Comment
|
||||
File *--> "*" ShareLink : share_links
|
||||
|
||||
@enduml
|
||||
|
||||
A <code class="language-text">Profile</code> holds the personal info of any user of the system. Users belongs to
|
||||
<code class="language-text">Teams</code> and may create <code class="language-text">Projects</code> inside them.
|
||||
|
||||
Inside the projects, there are <code class="language-text">Files</code>. All users of a team may see the projects
|
||||
and files inside the team. Also, any project and file has at least one user that
|
||||
is the owner, but may have more relationships with users with other roles.
|
||||
|
||||
Files may use other files as shared <code class="language-text">libraries</code>.
|
||||
|
||||
The main content of the file is in the "file data" attribute (see next section).
|
||||
But there are some objects that reside in separate entities:
|
||||
|
||||
* A <code class="language-text">StorageObject</code> represents a file in an external storage, that is embedded
|
||||
into a file (currently images and SVG icons, but we may add other media
|
||||
types in the future).
|
||||
|
||||
* <code class="language-text">CommentThreads</code>and <code class="language-text">Comments</code> are the comments that any user may add to a
|
||||
file.
|
||||
|
||||
* A <code class="language-text">ShareLink</code> contains a token, an URL and some permissions to share the file
|
||||
with external users.
|
||||
|
||||
## File data
|
||||
|
||||
@startuml FileModel
|
||||
|
||||
hide members
|
||||
|
||||
class File
|
||||
class Page
|
||||
class Component
|
||||
class Color
|
||||
class MediaItem
|
||||
class Typography
|
||||
|
||||
File *--> "*" Page : pages
|
||||
(File, Page) .. PagesList
|
||||
|
||||
File *--> "*" Component : components
|
||||
(File, Component) .. ComponentsList
|
||||
|
||||
File *--> "*" Color : colors
|
||||
(File, Color) .. ColorsList
|
||||
|
||||
File *--> "*" MediaItem : colors
|
||||
(File, MediaItem) .. MediaItemsList
|
||||
|
||||
File *--> "*" Typography : colors
|
||||
(File, Typography) .. TypographiesList
|
||||
|
||||
@enduml
|
||||
|
||||
The data attribute contains the <code class="language-text">Pages</code> and the library assets in the file
|
||||
(<code class="language-text">Components</code>, <code class="language-text">MediaItems</code>, <code class="language-text">Colors</code> and <code class="language-text">Typographies</code>).
|
||||
|
||||
The lists of pages and assets are modelled also as entities because they have a
|
||||
lot of functions and business logic.
|
||||
|
||||
## Pages and components
|
||||
|
||||
@startuml PageModel
|
||||
|
||||
hide members
|
||||
|
||||
class Container
|
||||
class Page
|
||||
class Component
|
||||
class Shape
|
||||
|
||||
Container <|-left- Page
|
||||
Container <|-right- Component
|
||||
|
||||
Container *--> "*" Shape : objects
|
||||
(Container, Shape) .. ShapeTree
|
||||
|
||||
Shape <-- Shape : parent
|
||||
|
||||
@enduml
|
||||
|
||||
Both <code class="language-text">Pages</code> and <code class="language-text">Components</code> contains a tree of shapes, and share many
|
||||
functions and logic. So, we have modelled a <code class="language-text">Container</code> entity, that is an
|
||||
abstraction that represents both a page or a component, to use it whenever we
|
||||
have code that fits the two.
|
||||
|
||||
A <code class="language-text">ShapeTree</code> represents a set of shapes that are hierarchically related: the top
|
||||
frame contains top-level shapes (frames and other shapes). Frames and groups may
|
||||
contain any non frame shape.
|
||||
|
||||
## Shapes
|
||||
|
||||
@startuml ShapeModel
|
||||
|
||||
hide members
|
||||
|
||||
class Shape
|
||||
class Selrect
|
||||
class Transform
|
||||
class Constraints
|
||||
class Interactions
|
||||
class Fill
|
||||
class Stroke
|
||||
class Shadow
|
||||
class Blur
|
||||
class Font
|
||||
class Content
|
||||
class Exports
|
||||
|
||||
Shape o--> Selrect
|
||||
Shape o--> Transform
|
||||
Shape o--> Constraints
|
||||
Shape o--> Interactions
|
||||
Shape o--> Fill
|
||||
Shape o--> Stroke
|
||||
Shape o--> Shadow
|
||||
Shape o--> Blur
|
||||
Shape o--> Font
|
||||
Shape o--> Content
|
||||
Shape o--> Exports
|
||||
|
||||
Shape <-- Shape : parent
|
||||
|
||||
@enduml
|
||||
|
||||
A <code class="language-text">Shape</code> is the most important entity of the model. Represents one of the
|
||||
[layers of our design](https://help.penpot.app/user-guide/layer-basics), and it
|
||||
corresponds with one SVG node, augmented with Penpot special features.
|
||||
|
||||
We have code to render a <code class="language-text">Shape</code> into a SVG tag, with more or less additions
|
||||
depending on the environment (editable in the workspace, interactive in the
|
||||
viewer, minimal in the shape exporter or the handoff, or with metadata in the
|
||||
file export).
|
||||
|
||||
Also have code that imports any SVG file and convert elements back into shapes.
|
||||
If it's a SVG exported by Penpot, it reads the metadata to reconstruct the
|
||||
shapes exactly as they were. If not, it infers the atributes with a best effort
|
||||
approach.
|
||||
|
||||
In addition to the identifier ones (the id, the name and the type of element),
|
||||
a shape has a lot of attributes. We tend to group them in related clusters.
|
||||
Those are the main ones:
|
||||
|
||||
* <code class="language-text">Selrect</code> and other geometric attributes (x, y, width, height...) define the
|
||||
position in the diagram and the bounding box.
|
||||
* <code class="language-text">Transform</code> is a [2D transformation matrix](https://www.alanzucconi.com/2016/02/10/tranfsormation-matrix/)
|
||||
to rotate or stretch the shape.
|
||||
* <code class="language-text">Constraints</code> explains how the shape changes when the container shape resizes
|
||||
(kind of "responsive" behavior).
|
||||
* <code class="language-text">Interactions</code> describe the interactive behavior when the shape is displayed
|
||||
in the viewer.
|
||||
* <code class="language-text">Fill</code> contains the shape fill color and options.
|
||||
* <code class="language-text">Stroke</code> contains the shape stroke color and options.
|
||||
* <code class="language-text">Shadow</code> contains the shape shadow options.
|
||||
* <code class="language-text">Blur</code> contains the shape blur options.
|
||||
* <code class="language-text">Font</code> contains the font options for a shape of type text.
|
||||
* <code class="language-text">Content</code> contains the text blocks for a shape of type text.
|
||||
* <code class="language-text">Exports</code> are the defined export settings for the shape.
|
||||
|
||||
Also a shape contains a reference to its containing shape (parent) and of all
|
||||
the children.
|
||||
146
docs/technical-guide/developer/devenv.md
Normal file
146
docs/technical-guide/developer/devenv.md
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
---
|
||||
title: 3.3. Dev environment
|
||||
---
|
||||
|
||||
# Development environment
|
||||
|
||||
## System requirements
|
||||
|
||||
You need to have <code class="language-bash">docker</code> and <code class="language-bash">docker-compose V2</code> installed on your system
|
||||
in order to correctly set up the development environment.
|
||||
|
||||
You can [look here][1] for complete instructions.
|
||||
|
||||
[1]: /technical-guide/getting-started/#install-with-docker
|
||||
|
||||
|
||||
Optionally, to improve performance, you can also increase the maximum number of
|
||||
user files able to be watched for changes with inotify:
|
||||
|
||||
```bash
|
||||
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
|
||||
```
|
||||
|
||||
|
||||
## Getting Started
|
||||
|
||||
**The interactive development environment requires some familiarity of [tmux](https://github.com/tmux/tmux/wiki).**
|
||||
|
||||
To start it, clone penpot repository, and execute:
|
||||
|
||||
```bash
|
||||
./manage.sh pull-devenv
|
||||
./manage.sh run-devenv
|
||||
```
|
||||
|
||||
This will do the following:
|
||||
|
||||
1. Pull the latest devenv image from dockerhub.
|
||||
2. Start all the containers in the background.
|
||||
3. Attach the terminal to the **devenv** container and execute the tmux session.
|
||||
4. The tmux session automatically starts all the necessary services.
|
||||
|
||||
This is an incomplete list of devenv related subcommands found on
|
||||
manage.sh script:
|
||||
|
||||
```bash
|
||||
./manage.sh build-devenv # builds the devenv docker image (called by run-devenv automatically when needed)
|
||||
./manage.sh start-devenv # starts background running containers
|
||||
./manage.sh run-devenv # enters to new tmux session inside of one of the running containers
|
||||
./manage.sh stop-devenv # stops background running containers
|
||||
./manage.sh drop-devenv # removes all the containers, volumes and networks used by the devenv
|
||||
```
|
||||
|
||||
Having the container running and tmux opened inside the container,
|
||||
you are free to execute commands and open as many shells as you want.
|
||||
|
||||
You can create a new shell just pressing the **Ctr+b c** shortcut. And
|
||||
**Ctrl+b w** for switch between windows, **Ctrl+b &** for kill the
|
||||
current window.
|
||||
|
||||
For more info: https://tmuxcheatsheet.com/
|
||||
|
||||
It may take a minute or so, but once all of the services have started, you can
|
||||
connect to penpot by browsing to http://localhost:3449 .
|
||||
|
||||
<!-- ## Inside the tmux session -->
|
||||
|
||||
<!-- By default, the tmux session opens 4 windows: -->
|
||||
|
||||
<!-- - **gulp** (0): responsible of build, watch (and other related) of -->
|
||||
<!-- styles, images, fonts and templates. -->
|
||||
<!-- - **frontend** (1): responsible of cljs compilation process of frontend. -->
|
||||
<!-- - **exporter** (2): responsible of cljs compilation process of exporter. -->
|
||||
<!-- - **backend** (3): responsible of starting the backend jvm process. -->
|
||||
|
||||
|
||||
### Frontend
|
||||
|
||||
The frontend build process is located on the tmux **window 0** and
|
||||
**window 1**. On the **window 0** we have the gulp process responsible
|
||||
of watching and building styles, fonts, icon-spreads and templates.
|
||||
|
||||
On the **window 1** we can found the **shadow-cljs** process that is
|
||||
responsible on watch and build frontend clojurescript code.
|
||||
|
||||
Additionally to the watch process you probably want to be able open a REPL
|
||||
process on the frontend application, for this case you can split the window
|
||||
and execute this:
|
||||
|
||||
```bash
|
||||
npx shadow-cljs cljs-repl main
|
||||
```
|
||||
|
||||
### Exporter
|
||||
|
||||
The exporter build process is located in the **window 2** and in the
|
||||
same way as frontend application, it is built and watched using
|
||||
**shadow-cljs**.
|
||||
|
||||
The main difference is that exporter will be executed in a nodejs, on
|
||||
the server side instead of browser.
|
||||
|
||||
The window is split into two slices. The top slice shows the build process and
|
||||
on the bottom slice has a shell ready to execute the generated bundle.
|
||||
|
||||
You can start the exporter process executing:
|
||||
|
||||
```bash
|
||||
node target/app.js
|
||||
```
|
||||
|
||||
This process does not start automatically.
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
The backend related process is located in the tmux **window 3**, and
|
||||
you can go directly to it using <code class="language-bash">ctrl+b 3</code> shortcut.
|
||||
|
||||
By default the backend will be started in a non-interactive mode for convenience
|
||||
but you can press <code class="language-bash">Ctrl+c</code> to exit and execute the following to start the repl:
|
||||
|
||||
```bash
|
||||
./scripts/repl
|
||||
```
|
||||
|
||||
On the REPL you have these helper functions:
|
||||
- <code class="language-bash">(start)</code>: start all the environment
|
||||
- <code class="language-bash">(stop)</code>: stops the environment
|
||||
- <code class="language-bash">(restart)</code>: stops, reload and start again.
|
||||
|
||||
And many other that are defined in the <code class="language-bash">dev/user.clj</code> file.
|
||||
|
||||
If an exception is raised or an error occurs when code is reloaded, just use
|
||||
<code class="language-bash">(repl/refresh-all)</code> to finish loading the code correctly and then use
|
||||
<code class="language-bash">(restart)</code> again.
|
||||
|
||||
## Email
|
||||
|
||||
To test email sending, the devenv includes [MailCatcher](https://mailcatcher.me/),
|
||||
a SMTP server that is used for develop. It does not send any mail outbounds.
|
||||
Instead, it stores them in memory and allows to browse them via a web interface
|
||||
similar to a webmail client. Simply navigate to:
|
||||
|
||||
[http://localhost:1080](http://localhost:1080)
|
||||
|
||||
647
docs/technical-guide/developer/frontend.md
Normal file
647
docs/technical-guide/developer/frontend.md
Normal file
|
|
@ -0,0 +1,647 @@
|
|||
---
|
||||
title: 3.5. Frontend Guide
|
||||
---
|
||||
|
||||
# Frontend Guide
|
||||
|
||||
This guide intends to explain the essential details of the frontend
|
||||
application.
|
||||
|
||||
## UI
|
||||
|
||||
Please refer to the [UI Guide](/technical-guide/developer/ui) to learn about implementing UI components and our design system.
|
||||
|
||||
## Logging, Tracing & Debugging
|
||||
|
||||
### Logging framework
|
||||
|
||||
To trace and debug the execution of the code, one method is to enable the log
|
||||
traces that currently are in the code using the [Logging
|
||||
framework](/technical-guide/developer/common/#system-logging). You can edit a
|
||||
module and set a lower log level, to see more traces in console. Search for
|
||||
this kind of line and change to <code class="language-clojure">:info</code> or <code class="language-clojure">:debug</code>:
|
||||
|
||||
```clojure
|
||||
(ns some.ns
|
||||
(:require [app.util.logging :as log]))
|
||||
|
||||
(log/set-level! :info)
|
||||
```
|
||||
|
||||
Or you can change it live with the debug utility (see below):
|
||||
|
||||
```javascript
|
||||
debug.set_logging("namespace", "level");
|
||||
```
|
||||
|
||||
### Temporary traces
|
||||
|
||||
Of course, you have the traditional way of inserting temporary traces inside
|
||||
the code to output data to the devtools console. There are several ways of
|
||||
doing this.
|
||||
|
||||
#### Use clojurescript helper <code class="language-clojure">prn</code>
|
||||
|
||||
This helper automatically formats the clojure and js data structures as plain
|
||||
[EDN](https://clojuredocs.org/clojure.edn) for visual inspection and to know
|
||||
the exact type of the data.
|
||||
|
||||
```clojure
|
||||
(prn "message" expression)
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Use <code class="language-clojure">pprint</code> function
|
||||
|
||||
We have set up a wrapper over [fipp](https://github.com/brandonbloom/fipp)
|
||||
<code class="language-clojure">pprint</code> function, that gives a human-readable formatting to the data, useful
|
||||
for easy understanding of larger data structures.
|
||||
|
||||
The wrapper allows to easily specify <code class="language-clojure">level</code>, <code class="language-clojure">length</code> and <code class="language-clojure">width</code> parameters,
|
||||
with reasonable defaults, to control the depth level of objects to print, the
|
||||
number of attributes to show and the display width.
|
||||
|
||||
```clojure
|
||||
(:require [app.common.pprint :refer [pprint]])
|
||||
|
||||
;; On the code
|
||||
(pprint shape {:level 2
|
||||
:length 21
|
||||
:width 30})
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### Use the js native functions
|
||||
|
||||
The <code class="language-clojure">clj->js</code> function converts the clojure data structure into a javacript
|
||||
object, interactively inspectable in the devtools.console.
|
||||
|
||||
```clojure
|
||||
(js/console.log "message" (clj->js expression))
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Breakpoints
|
||||
|
||||
You can insert standard javascript debugger breakpoints in the code, with this
|
||||
function:
|
||||
|
||||
```clojure
|
||||
(js-debugger)
|
||||
```
|
||||
|
||||
The Clojurescript environment generates source maps to trace your code step by
|
||||
step and inspect variable values. You may also insert breakpoints from the
|
||||
sources tab, like when you debug javascript code.
|
||||
|
||||
One way of locating a source file is to output a trace with <code class="language-clojure">(js/console.log)</code>
|
||||
and then clicking in the source link that shows in the console at the right
|
||||
of the trace.
|
||||
|
||||
### Access to clojure from js console
|
||||
|
||||
The penpot namespace of the main application is exported, so that is
|
||||
accessible from javascript console in Chrome developer tools. Object
|
||||
names and data types are converted to javascript style. For example
|
||||
you can emit the event to reset zoom level by typing this at the
|
||||
console (there is autocompletion for help):
|
||||
|
||||
```javascript
|
||||
app.main.store.emit_BANG_(app.main.data.workspace.reset_zoom);
|
||||
```
|
||||
|
||||
### Debug utility
|
||||
|
||||
We have defined, at <code class="language-clojure">src/debug.cljs</code>, a <code class="language-clojure">debug</code> namespace with many functions
|
||||
easily accesible from devtools console.
|
||||
|
||||
#### Change log level
|
||||
|
||||
You can change the [log level](/technical-guide/developer/common/#system-logging)
|
||||
of one namespace without reloading the page:
|
||||
|
||||
```javascript
|
||||
debug.set_logging("namespace", "level");
|
||||
```
|
||||
|
||||
#### Dump state and objects
|
||||
|
||||
There are some functions to inspect the global state or parts of it:
|
||||
|
||||
```javascript
|
||||
// print the whole global state
|
||||
debug.dump_state();
|
||||
|
||||
// print the latest events in the global stream
|
||||
debug.dump_buffer();
|
||||
|
||||
// print a key of the global state
|
||||
debug.get_state(":workspace-data :pages 0");
|
||||
|
||||
// print the objects list of the current page
|
||||
debug.dump_objects();
|
||||
|
||||
// print a single object by name
|
||||
debug.dump_object("Rect-1");
|
||||
|
||||
// print the currently selected objects
|
||||
debug.dump_selected();
|
||||
|
||||
// print all objects in the current page and local library components.
|
||||
// Objects are displayed as a tree in the same order of the
|
||||
// layers tree, and also links to components are shown.
|
||||
debug.dump_tree();
|
||||
|
||||
// This last one has two optional flags. The first one displays the
|
||||
// object ids, and the second one the {touched} state.
|
||||
debug.dump_tree(true, true);
|
||||
```
|
||||
|
||||
And a bunch of other utilities (see the file for more).
|
||||
|
||||
## Workspace visual debug
|
||||
|
||||
Debugging a problem in the viewport algorithms for grouping and
|
||||
rotating is difficult. We have set a visual debug mode that displays
|
||||
some annotations on screen, to help understanding what's happening.
|
||||
This is also in the <code class="language-clojure">debug</code> namespace.
|
||||
|
||||
To activate it, open the javascript console and type:
|
||||
|
||||
```js
|
||||
debug.toggle_debug("option");
|
||||
```
|
||||
|
||||
Current options are <code class="language-clojure">bounding-boxes</code>, <code class="language-clojure">group</code>, <code class="language-clojure">events</code> and
|
||||
<code class="language-clojure">rotation-handler</code>.
|
||||
|
||||
You can also activate or deactivate all visual aids with
|
||||
|
||||
```js
|
||||
debug.debug_all();
|
||||
debug.debug_none();
|
||||
```
|
||||
|
||||
## Translations (I18N)
|
||||
|
||||
### How it works
|
||||
|
||||
All the translation strings of this application are stored in
|
||||
standard _gettext_ files in <code class="language-bash">frontend/translations/*.po</code>.
|
||||
|
||||
They have a self explanatory format that looks like this:
|
||||
|
||||
```bash
|
||||
#: src/app/main/ui/auth/register.cljs, src/app/main/ui/auth/login.cljs
|
||||
msgid "auth.create-demo-account"
|
||||
msgstr "Create demo account"
|
||||
```
|
||||
|
||||
The files are automatically bundled into the <code class="language-bash">index.html</code> file on
|
||||
compile time (in development and production). The bundled content is a
|
||||
simplified version of this data structure to avoid loading unnecesary
|
||||
data. The development environment has a watch process that detect
|
||||
changes on that file and recompiles the <code class="language-bash">index.html</code>.
|
||||
|
||||
**There are no hot reload for translations strings**, you just need to
|
||||
refresh the browser tab to refresh the translations in the running the
|
||||
application.
|
||||
|
||||
Finally, when you have finished adding texts, execute the following command
|
||||
inside the devenv, to reformat the file before commiting the file into the
|
||||
repository:
|
||||
|
||||
```bash
|
||||
# cd <repo>/frontend
|
||||
yarn run validate-translations
|
||||
```
|
||||
|
||||
At Penpot core team we maintain manually the english and spanish .po files. All
|
||||
the others are managed in https://weblate.org.
|
||||
|
||||
**When a new language is available in weblate**, to enable it in the application
|
||||
you need to add it in two places:
|
||||
|
||||
```bash
|
||||
frontend/src/app/util/i18n.cljs (supported-locales)
|
||||
frontend/gulpfile.js (const langs)
|
||||
```
|
||||
|
||||
### How to use it
|
||||
|
||||
You need to use the <code class="language-bash">app.util.i18n/tr</code> function for lookup translation
|
||||
strings:
|
||||
|
||||
```clojure
|
||||
(require [app.util.i18n :as i18n :refer [tr]])
|
||||
|
||||
(tr "auth.create-demo-account")
|
||||
;; => "Create demo account"
|
||||
```
|
||||
|
||||
If you want to insert a variable into a translated text, use <code class="language-clojure">%s</code> as
|
||||
placeholder, and then pass the variable value to the <code class="language-clojure">(tr ...)</code> call.:
|
||||
|
||||
```bash
|
||||
#: src/app/main/ui/settings/change_email.cljs
|
||||
msgid "notifications.validation-email-sent"
|
||||
msgstr "Verification email sent to %s. Check your email!"
|
||||
```
|
||||
|
||||
```clojure
|
||||
(require [app.util.i18n :as i18n :refer [tr]])
|
||||
|
||||
(tr "notifications.validation-email-sent" email)
|
||||
;; => "Verification email sent to test@example.com. Check your email!"
|
||||
```
|
||||
|
||||
If you have defined plurals for some translation resource, then you
|
||||
need to pass an additional parameter marked as counter in order to
|
||||
allow the system know when to show the plural:
|
||||
|
||||
```bash
|
||||
#: src/app/main/ui/dashboard/team.cljs
|
||||
msgid "labels.num-of-projects"
|
||||
msgid_plural "labels.num-of-projects"
|
||||
msgstr[0] "1 project"
|
||||
msgstr[1] "%s projects"
|
||||
```
|
||||
|
||||
```clojure
|
||||
(require [app.util.i18n :as i18n :refer [tr]])
|
||||
|
||||
(tr "labels.num-of-projects" (i18n/c 10))
|
||||
;; => "10 projects"
|
||||
|
||||
(tr "labels.num-of-projects" (i18n/c 1))
|
||||
;; => "1 project"
|
||||
```
|
||||
|
||||
## Integration tests
|
||||
|
||||
### Setup
|
||||
|
||||
To run integration tests locally, follow these steps.
|
||||
|
||||
Ensure your development environment docker image is up to date.
|
||||
|
||||
1. If it is not up to date, run:
|
||||
|
||||
```bash
|
||||
./manage.sh pull-devenv
|
||||
```
|
||||
|
||||
2. Once the update is complete, start the environment:
|
||||
|
||||
```bash
|
||||
./manage.sh start-devenv
|
||||
```
|
||||
|
||||
**NOTE** You can learn more about how to set up, start and stop our development environment [here](/technical-guide/developer/devenv)
|
||||
|
||||
### Running the integration tests
|
||||
|
||||
#### Headless mode
|
||||
|
||||
Here's how to run the tests with a headless browser (i.e. within the terminal, no UI):
|
||||
|
||||
1. With the developer environment tmux session opened, create a new tab with <code class="language-bash">Ctrl + b c</code>.
|
||||
|
||||
2. Go to the frontend folder:
|
||||
|
||||
```bash
|
||||
cd penpot/frontend
|
||||
```
|
||||
|
||||
3. Run the tests with <code class="language-bash">yarn</code>:
|
||||
|
||||
```bash
|
||||
yarn e2e:test
|
||||
```
|
||||
|
||||
> 💡 **TIP:** By default, the tests will _not_ run in parallel. You can set the amount of workers to run the tests with <code class="language-bash">--workers</code>. Note that, depending on your machine, this might make some tests flaky.
|
||||
|
||||
```bash
|
||||
# run in parallel with 4 workers
|
||||
yarn e2e:test --workers 4
|
||||
```
|
||||
|
||||
#### Running the tests in Chromium
|
||||
|
||||
To access the testing UI and run the tests in a real browser, follow these steps:
|
||||
|
||||
1. In a terminal _in your host machine_, navigate to the <code class="language-bash">frontend</code> folder, then run:
|
||||
|
||||
```bash
|
||||
# cd <repo>/frontend
|
||||
npx playwright test --ui
|
||||
```
|
||||
|
||||
> ⚠️ **WARNING:** It is important to be in the right folder (<code class="language-bash">frontend</code>) to launch the command above, or you may have errors trying to run the tests.
|
||||
|
||||
> ❗️ **IMPORTANT**: You might need to [install Playwright's browsers and dependencies](https://playwright.dev/docs/intro) in your host machine with: <code class="language-bash">npx playwright install --with-deps</code>. In case you are using a Linux distribution other than Ubuntu, [you might need to install the dependencies manually](https://github.com/microsoft/playwright/issues/11122).
|
||||
|
||||
### How to write a test
|
||||
|
||||
When writing integration tests, we are simulating user actions and events triggered by them, in other to mirror real-world user interactions. The difference with fully end-to-end tests is that here we are faking the backend by intercepting the network requests, so tests can run faster and more tied to the front-end.
|
||||
|
||||
Keep in mind:
|
||||
|
||||
- **Use Realistic User Scenarios:** Design test cases that mimic real user scenarios and interactions with the application.
|
||||
|
||||
- **Simulate User Inputs**: Such as mouse clicks, keyboard inputs, form submissions, or touch gestures, using the testing framework's API. Mimic user interactions as closely as possible to accurately simulate user behavior.
|
||||
|
||||
- **Intercept the network**: Playwright offers ways to fake network responses to API calls, websocket messages, etc. Remember that there is no backend here, so you will need to intercept every request made by the front-end app.
|
||||
|
||||
#### Page Object Model
|
||||
|
||||
When writing a significant number of tests, encountering repetitive code and common actions is typical. To address this issue, we recommend leveraging **Page Object Models** (POM), which is a single class that encapsulates common locators, user interactions, etc.
|
||||
|
||||
POMs do not necessarily refer to entire pages but can also represent specific regions of a page that are the focus of our tests. For example, we may have a POM for the login form, or the projects section.
|
||||
|
||||
In a POM, we can define locators in the constructor itself — remember that locators will be accessed when interacted with (with a <code class="language-js">click()</code>, for instance) or when asserting expectations.
|
||||
|
||||
```js
|
||||
class LoginPage {
|
||||
constructor(page) {
|
||||
super(page);
|
||||
this.loginButton = page.getByRole("button", { name: "Login" });
|
||||
this.passwordInput = page.getByLabel("Password");
|
||||
this.emailInput = page.getByLabel("Email");
|
||||
}
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
We can later use this POM and its locators:
|
||||
|
||||
```js
|
||||
test("Sample test", async ({ page }) => {
|
||||
const loginPage = new loginPage(page);
|
||||
// ...
|
||||
await expect(loginPage.loginButton).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
> 💡 **TIP**: Locators that are generic and meant to be used in multiple tests should be part of the POM.
|
||||
>
|
||||
> If your locator is ad-hoc for a specific test, there's no need to add it to the POM.
|
||||
|
||||
In addition to locators, POMs also include methods that perform common actions on those elements, like filling out a group of related input fields.
|
||||
|
||||
```js
|
||||
class LoginPage {
|
||||
// ...
|
||||
async fillEmailAndPasswordInputs(email, password) {
|
||||
await this.emailInput.fill(email);
|
||||
await this.passwordInput.fill(password);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
POMs can also include the interception of network requests (but only include interceptions commont to multiple tests in the POM):
|
||||
|
||||
```js
|
||||
class LoginPage {
|
||||
// ...
|
||||
async setupLoginSuccess() {
|
||||
await this.mockRPC(
|
||||
"login-with-password",
|
||||
"logged-in-user/login-with-password-success.json"
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Here's an example of a test that uses a POM:
|
||||
|
||||
```js
|
||||
test("User submits a wrong formatted email", async ({ page }) => {
|
||||
const loginPage = new LoginPage(page);
|
||||
await loginPage.setupLoginSuccess();
|
||||
|
||||
await loginPage.fillEmailAndPasswordInputs("foo", "lorenIpsum");
|
||||
|
||||
await expect(loginPage.errorLoginMessage).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
#### Mocking the back-end
|
||||
|
||||
In the penpot repository there are some POMs that are meant to be extended by more specific pages. These include methods that should be useful when you write your own POMs.
|
||||
|
||||
- <code class="language-bash">BasePage</code> contains methods to intercept network requests and return JSON data fixtures.
|
||||
|
||||
- <code class="language-bash">BaseWebSocketPage</code> also can intercept websocket connections, which are a must for tests in the workspace, or any other Penpot page that uses a WebSocket.
|
||||
|
||||
##### API calls
|
||||
|
||||
In order to mock API calls we just need to extend from the <code class="language-bash">BasePage</code> POM and then call its method <code class="language-bash">mockRPC</code>:
|
||||
|
||||
```js
|
||||
export class FooPage extends BasePage {
|
||||
setupNetworkResponses() {
|
||||
this.mockRPC("lorem/ipsum", "json-file-with-fake-response.json");
|
||||
|
||||
// Regexes are supported too
|
||||
this.mockRPC(
|
||||
/a\-regex$/
|
||||
"json-file-with-fake-response.json"
|
||||
);
|
||||
|
||||
// ...You can also pass custom status code and override other options
|
||||
this.mockRPC("something/not/found", "json-file-with-fake-response.json", {
|
||||
status: 404,
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> ❗️ **IMPORTANT:** The <code class="language-bash">mockRPC</code> method is meant to intercept calls to Penpot's RPC API, and already prefixes the path you provide with <code class="language-bash">/api/rpc/command/</code>. So, if you need to intercept <code class="language-bash">/api/rpc/command/get-profile</code> you would just need to call <code class="language-bash">mockRPC("get-profile", "json-data.json")</code>.
|
||||
|
||||
##### WebSockets
|
||||
|
||||
Any Penpot page that uses a WebSocket requires it to be intercepted and mocked. To do that, you can extend from the POM <code class="language-bash">BaseWebSocketPage</code> _and_ call its <code class="language-bash">initWebSockets()</code> methods before each test.
|
||||
|
||||
Here's an an actual example from the Penpot repository:
|
||||
|
||||
```js
|
||||
// frontend/playwright/ui/pages/WorkspacePage.js
|
||||
export class WorkspacePage extends BaseWebSocketPage {
|
||||
static async init(page) {
|
||||
await BaseWebSocketPage.init(page);
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// frontend/playwright/ui/specs/workspace.spec.js
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await WorkspacePage.init(page);
|
||||
});
|
||||
```
|
||||
|
||||
<code class="language-bash">BaseWebSocketPage</code> also includes methods to wait for a specific WebSocket connection and to fake sending/receiving messages.
|
||||
|
||||
When testing the workspace, you will want to wait for the <code class="language-bash">/ws/notifications</code> WebSocket. There's a convenience method, <code class="language-bash">waitForNotificationsWebSocket</code> to do that:
|
||||
|
||||
```js
|
||||
// frontend/playwright/ui/pages/WorkspacePage.js
|
||||
export class WorkspacePage extends BaseWebSocketPage {
|
||||
// ...
|
||||
|
||||
// browses to the Workspace and waits for the /ws/notifications socket to be ready
|
||||
// to be listened to.
|
||||
async goToWorkspace() {
|
||||
// ...
|
||||
this.#ws = await this.waitForNotificationsWebSocket();
|
||||
await this.#ws.mockOpen();
|
||||
// ...
|
||||
}
|
||||
|
||||
// sends a message over the notifications websocket
|
||||
async sendPresenceMessage(fixture) {
|
||||
await this.#ws.mockMessage(JSON.stringify(fixture));
|
||||
}
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
```js
|
||||
// frontend/playwright/ui/specs/workspace.spec.js
|
||||
test("User receives presence notifications updates in the workspace", async ({
|
||||
page,
|
||||
}) => {
|
||||
const workspacePage = new WorkspacePage(page);
|
||||
// ...
|
||||
|
||||
await workspacePage.goToWorkspace();
|
||||
await workspacePage.sendPresenceMessage(presenceFixture);
|
||||
|
||||
await expect(
|
||||
page.getByTestId("active-users-list").getByAltText("Princesa Leia")
|
||||
).toHaveCount(2);
|
||||
});
|
||||
```
|
||||
|
||||
### Best practices for writing tests
|
||||
|
||||
Our best practices are based on [Testing library documentation](https://testing-library.com/docs/).
|
||||
|
||||
This is a summary of the most important points to take into account:
|
||||
|
||||
#### Query priority for locators
|
||||
|
||||
For our integration tests we use Playwright, you can find more info about this library and the different locators [here](https://playwright.dev/docs/intro).
|
||||
|
||||
Locator queries are the methods to find DOM elements in the page. Your test should simulate as closely as possible the way users interact with the application. Depending on the content of the page and the element to be selected, we will choose one method or the other following these priorities:
|
||||
|
||||
1. **Queries accessible to everyone**: Queries that simulate the experience of visual users or use assistive technologies.
|
||||
|
||||
- [<code class="language-js">page.getByRole</code>](https://playwright.dev/docs/locators#locate-by-role): To locate exposed elements in the [accessibility tree](https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree).
|
||||
|
||||
- [<code class="language-js">page.getByLabel</code>](https://playwright.dev/docs/locators#locate-by-label): For querying form fields.
|
||||
|
||||
- [<code class="language-js">page.getByPlaceholder</code>](https://playwright.dev/docs/locators#locate-by-placeholder): For when the placeholder text is more relevant than the label (or the label does not exist).
|
||||
|
||||
- [<code class="language-js">page.getByText</code>](https://playwright.dev/docs/locators#locate-by-text): For the non-form elements that also do not have a role in the accesibility tree, but have a distintive text.
|
||||
|
||||
2. **Semantic queries**: Less preferable than the above, since the user experience when interacting with these attributes may differ significantly depending on the browser and assistive technology being used.
|
||||
|
||||
- [<code class="language-js">page.byAltText</code>](https://playwright.dev/docs/locators#locate-by-alt-text): For elements that support <code class="language-js">alt</code> text (<code class="language-js">\<img></code>, <code class="language-js">\<area></code>, a custom element, etc.).
|
||||
|
||||
- [<code class="language-js">page.byTitle</code>](https://playwright.dev/docs/locators#locate-by-title): For elements with a <code class="language-html">title</code>.
|
||||
|
||||
3. **Test IDs**: If none of the queries above are feasible, we can locate by the <code class="language-html">data-testid</code> attribute. This locator is the least preffered since it's not user-interaction oriented.
|
||||
|
||||
- [<code class="language-js">page.getByTestId</code>](https://playwright.dev/docs/locators#locate-by-test-id): For elements with a <code class="language-html">data-testid</code> attribute.
|
||||
|
||||
#### A practical example for using locator queries.
|
||||
|
||||
Given this DOM structure:
|
||||
|
||||
```html
|
||||
<form>
|
||||
<p>Penpot is the free open-...</p>
|
||||
<label for="email">
|
||||
Email
|
||||
<input placeholder="Email" name="email" type="email" id="email" value="" />
|
||||
</label>
|
||||
|
||||
<label for="password">
|
||||
Password
|
||||
<input
|
||||
placeholder="Password"
|
||||
name="password"
|
||||
type="password"
|
||||
id="password"
|
||||
value=""
|
||||
/>
|
||||
</label>
|
||||
<button type="submit">Login</button>
|
||||
</form>
|
||||
```
|
||||
|
||||
The DOM above represents this part of the app:
|
||||
|
||||
|
||||
|
||||
Our first task will be to locate the **login button**:
|
||||
|
||||
|
||||
|
||||
Our initial approach involves following the instructions of the first group of locators, "Queries accessible to everyone". To achieve this, we inspect the accessibility tree to gather information:
|
||||
|
||||
|
||||
|
||||
Having examined the accessibility tree, we identify that the button can be located by its role and name, which is our primary option:
|
||||
|
||||
```js
|
||||
page.getByRole("button", { name: "Login" });
|
||||
```
|
||||
|
||||
For selecting the <code class="language-js">\<input></code> within the form, we opt for <code class="language-js">getByLabel</code>, as it is the recommended method for locating form inputs:
|
||||
|
||||
|
||||
|
||||
```js
|
||||
page.getByLabel("Password");
|
||||
```
|
||||
|
||||
If we need to locate a text with no specific role, we can use the <code class="language-js">getByText</code> method:
|
||||
|
||||
```js
|
||||
page.getByText("Penpot is the free open-");
|
||||
```
|
||||
|
||||
To locate the rest of the elements we continue exploring the list of queries according to the order of priority. If none of the above options match the item, we resort to <code class="language-js">getByTestId</code> as a last resort.
|
||||
|
||||
#### Assertions
|
||||
|
||||
Assertions use Playwright's <code class="language-js">expect</code> method. Here are some tips for writing your assertions:
|
||||
|
||||
- **Keep assertions clear and concise:** Each assertion should verify a single expected behavior or outcome. Avoid combining multiple assertions into a single line, to maintain clarity and readability.
|
||||
|
||||
- **Use descriptive assertions:** Use assertion messages that clearly communicate the purpose of the assertion.
|
||||
|
||||
- **Favor writing assertions from the user's point of view:** For instance, whenever possible, assert things about elements that the user can see or interact with.
|
||||
|
||||
- **Cover the error state of a page**: Verify that the application handles errors gracefully by asserting the presence of error messages. We do not have to cover all error cases, that will be taken care of by the unit tests.
|
||||
|
||||
- **Prefer positive assertions:** Avoid using <code class="language-js">.not</code> in your assertions (i.e. <code class="language-js">expect(false).not.toBeTruthy()</code>) —it helps with readability.
|
||||
|
||||
#### Naming tests
|
||||
|
||||
- **User-centric approach:** Tests should be named from the perspective of user actions. For instance, <code class="language-js">"User logs in successfully"</code> instead of <code class="language-js">"Test login"</code>.
|
||||
|
||||
- **Descriptive names:** Test names should be descriptive, clearly indicating the action being tested.
|
||||
|
||||
- **Clarity and conciseness:** Keep test names clear and concise.
|
||||
|
||||
- **Use action verbs:** Start test names with action verbs to denote the action being tested. Example: <code class="language-js">"Adds a new file to the project"</code>.
|
||||
21
docs/technical-guide/developer/index.md
Normal file
21
docs/technical-guide/developer/index.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
---
|
||||
title: 3. Developer Guide
|
||||
---
|
||||
|
||||
# Developer Guide
|
||||
|
||||
This section is intended for people wanting to mess with the code or the inners
|
||||
of Penpot application.
|
||||
|
||||
The [Architecture][1] and [Data model][2] sections provide a bird's eye view of
|
||||
the whole system, to better understand how is structured.
|
||||
|
||||
The [Dev Env][3] section explains how to setup the development enviroment that
|
||||
we (the core team) use internally.
|
||||
|
||||
And the rest of sections are a list categorized of probably not deeply
|
||||
related HOWTO articles about dev-centric subjects.
|
||||
|
||||
[1]: /technical-guide/developer/architecture/
|
||||
[2]: /technical-guide/developer/data-model/
|
||||
[3]: /technical-guide/developer/devenv/
|
||||
119
docs/technical-guide/developer/subsystems/assets-storage.md
Normal file
119
docs/technical-guide/developer/subsystems/assets-storage.md
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
---
|
||||
title: Assets storage
|
||||
---
|
||||
|
||||
# Assets storage
|
||||
|
||||
The [storage.clj](https://github.com/penpot/penpot/blob/develop/backend/src/app/storage.clj)
|
||||
is a module that manages storage of binary objects. It's a generic utility
|
||||
that may be used for any kind of user uploaded files. Currently:
|
||||
|
||||
* Image assets in Penpot files.
|
||||
* Uploaded fonts.
|
||||
* Profile photos of users and teams.
|
||||
|
||||
There is an abstract interface and several implementations (or **backends**),
|
||||
depending on where the objects are actually stored:
|
||||
|
||||
* <code class="language-clojure">:assets-fs</code> stores ojects in the file system, under a given base path.
|
||||
* <code class="language-clojure">:assets-s3</code> stores them in any cloud storage with an AWS-S3 compatible
|
||||
interface.
|
||||
* <code class="language-clojure">:assets-db</code> stores them inside the PostgreSQL database, in a special table
|
||||
with a binary column.
|
||||
|
||||
## Storage API
|
||||
|
||||
The **StorageObject** record represents one stored object. It contains the
|
||||
metadata, that is always stored in the database (table <code class="language-clojure">storage_object</code>),
|
||||
while the actual object data goes to the backend.
|
||||
|
||||
* <code class="language-clojure">:id</code> is the identifier you use to reference the object, may be stored
|
||||
in other places to represent the relationship with other element.
|
||||
* <code class="language-clojure">:backend</code> points to the backend where the object data resides.
|
||||
* <code class="language-clojure">:created-at</code> is the date/time of object creation.
|
||||
* <code class="language-clojure">:deleted-at</code> is the date/time of object marked for deletion (see below).
|
||||
* <code class="language-clojure">:expired-at</code> allows to create objects that are automatically deleted
|
||||
at some time (useful for temporary objects).
|
||||
* <code class="language-clojure">:touched-at</code> is used to check objects that may need to be deleted (see
|
||||
below).
|
||||
|
||||
Also more metadata may be attached to objects, such as the <code class="language-clojure">:content-type</code> or
|
||||
the <code class="language-clojure">:bucket</code> (see below).
|
||||
|
||||
You can use the API functions to manipulate objects. For example <code class="language-clojure">put-object!</code>
|
||||
to create a new one, <code class="language-clojure">get-object</code> to retrieve the StorageObject,
|
||||
<code class="language-clojure">get-object-data</code> or <code class="language-clojure">get-object-bytes</code> to read the binary contents, etc.
|
||||
|
||||
For profile photos or fonts, the object id is stored in the related table,
|
||||
without further ado. But for file images, one more indirection is used. The
|
||||
**file-media-object** is an abstraction that represents one image uploaded
|
||||
by the user (in the future we may support other multimedia types). It has its
|
||||
own database table, and references two <code class="language-clojure">StorageObjects</code>, one for the original
|
||||
file and another one for the thumbnail. Image shapes contains the id of the
|
||||
<code class="language-clojure">file-media-object</code> with the <code class="language-clojure">:is-local</code> property as true. Image assets in the
|
||||
file library also have a <code class="language-clojure">file-media-object</code> with <code class="language-clojure">:is-local</code> false,
|
||||
representing that the object may be being used in other files.
|
||||
|
||||
## Serving objects
|
||||
|
||||
Stored objects are always served by Penpot (even if they have a public URL,
|
||||
like when <code class="language-clojure">:s3</code> storage are used). We have an endpoint <code class="language-text">/assets</code> with three
|
||||
variants:
|
||||
|
||||
```bash
|
||||
/assets/by-id/<uuid>
|
||||
/assets/by-file-media-id/<uuid>
|
||||
/assets/by-file-media-id/<uuid>/thumbnail
|
||||
```
|
||||
|
||||
They take an object and retrieve its data to the user. For <code class="language-clojure">:db</code> backend, the
|
||||
data is extracted from the database and served by the app. For the other ones,
|
||||
we calculate the real url of the object, and pass it to our **nginx** server,
|
||||
via special HTTP headers, for it to retrieve the data and serve it to the user.
|
||||
|
||||
This is the same in all environments (devenv, production or on premise).
|
||||
|
||||
## Object buckets
|
||||
|
||||
Obects may be organized in **buckets**, that are a kind of "intelligent" folders
|
||||
(not related to AWS-S3 buckets, this is a Penpot internal concept).
|
||||
|
||||
The storage module may use the bucket (hardcoded) to make special treatment to
|
||||
object, such as storing in a different path, or guessing how to know if an object
|
||||
is referenced from other place.
|
||||
|
||||
## Sharing and deleting objects
|
||||
|
||||
To save storage space, duplicated objects wre shared. So, if for example
|
||||
several users upload the same image, or a library asset is instantiated many
|
||||
times, even by different users, the object data is actuall stored only once.
|
||||
|
||||
To achieve this, when an object is uploaded, its content is hashed, and the
|
||||
hash compared with other objects in the same bucket. If there is a match,
|
||||
the <code class="language-clojure">StorabeObject</code> is reused. Thus, there may be different, unrelated, shapes
|
||||
or library assets whose <code class="language-clojure">:object-id</code> is the same.
|
||||
|
||||
### Garbage collector and reference count
|
||||
|
||||
Of course, when objects are shared, we cannot delete it directly when the
|
||||
associated item is removed or unlinked. Instead, we need some mechanism to
|
||||
track the references, and a garbage collector that deletes any object that
|
||||
is no longer referenced.
|
||||
|
||||
We don't use explicit reference counts or indexes. Instead, the storage system
|
||||
is intelligent enough to search, depending on the bucket (one for profile
|
||||
photos, other for file media objects, etc.) if there is any element that is
|
||||
using the object. For example, in the first case we look for user or team
|
||||
profiles where the <code class="language-clojure">:photo-id</code> field matches the object id.
|
||||
|
||||
When one item stops using one storage object (e. g. an image shape is deleted),
|
||||
we mark the object as <code class="language-clojure">:touched</code>. A periodic task revises all touched objectsm
|
||||
checking if they are still referenced in other places. If not, they are marked
|
||||
as :deleted. They're preserved in this state for some time (to allow "undeletion"
|
||||
if the user undoes the change), and eventually, another garbage collection task
|
||||
definitively deletes it, both in the backend and in the database table.
|
||||
|
||||
For <code class="language-clojure">file-media-objects</code>, there is another collector, that periodically checks
|
||||
if a media object is referenced by any shape or asset in its file. If not, it
|
||||
marks the object as <code class="language-clojure">:touched</code> triggering the process described above.
|
||||
|
||||
149
docs/technical-guide/developer/subsystems/authentication.md
Normal file
149
docs/technical-guide/developer/subsystems/authentication.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
---
|
||||
title: Authentication
|
||||
---
|
||||
|
||||
# User authentication
|
||||
|
||||
Users in Penpot may register via several different methods (if enabled in the
|
||||
configuration of the Penpot instance). We have implemented this as a series
|
||||
of "authentication backends" in our code:
|
||||
|
||||
* **penpot**: internal registration with email and password.
|
||||
* **ldap**: authentication over an external LDAP directory.
|
||||
* **oidc**, **google**, **github**, **gitlab**: authentication over an external
|
||||
service using the [OpenID Connect](https://openid.net/connect) protocol. We
|
||||
have a generic handler, and other ones already preconfigured for popular
|
||||
services.
|
||||
|
||||
The main logic resides in the following files:
|
||||
|
||||
```text
|
||||
backend/src/app/rpc/mutations/profile.clj
|
||||
backend/src/app/rpc/mutations/ldap.clj
|
||||
backend/src/app/rpc/mutations/verify-token.clj
|
||||
backend/src/app/http/oauth.clj
|
||||
backend/src/app/http/session.clj
|
||||
frontend/src/app/main/ui/auth/verify-token.cljs
|
||||
```
|
||||
|
||||
We store in the user profiles in the database the auth backend used to register
|
||||
first time (mainly for audit). A user may login with other methods later, if the
|
||||
email is the same.
|
||||
|
||||
## Register and login
|
||||
|
||||
The code is organized to try to reuse functions and unify processes as much as
|
||||
possible for the different auth systems.
|
||||
|
||||
|
||||
### Penpot backend
|
||||
|
||||
When a user types an email and password in the basic Penpot registration page,
|
||||
frontend calls <code class="language-clojure">:prepare-register-profile</code> method. It generates a "register
|
||||
token", a temporary JWT token that includes the login data.
|
||||
|
||||
This is used in the second registration page, that finally calls
|
||||
<code class="language-clojure">:register-profile</code> with the token and the rest of profile data. This function
|
||||
is reused in all the registration methods, and it's responsible of creating the
|
||||
user profile in the database. Then, it sends the confirmation email if using
|
||||
penpot backend, or directly opens a session (see below) for othe methods or if
|
||||
the user has been invited from a team.
|
||||
|
||||
The confirmation email has a link to <code class="language-clojure">/auth/verify-token</code>, that has a handler
|
||||
in frontend, that is a hub for different kinds of tokens (registration email,
|
||||
email change and invitation link). This view uses <code class="language-clojure">:verify-token</code> RPC call and
|
||||
redirects to the corresponding page with the result.
|
||||
|
||||
To login with the penpot backend, the user simply types the email and password
|
||||
and they are sent to <code class="language-clojure">:login</code> method to check and open session.
|
||||
|
||||
### OIDC backend
|
||||
|
||||
When the user press one of the "Log in with XXX" button, frontend calls
|
||||
<code class="language-clojure">/auth/oauth/:provider</code> (provider is google, github or gitlab). The handler
|
||||
generates a request token and redirects the user to the service provider to
|
||||
authenticate in it.
|
||||
|
||||
If succesful, the provider redirects to the<code class="language-clojure">/auth/oauth/:provider/callback</code>.
|
||||
This verifies the call with the request token, extracts another access token
|
||||
from the auth response, and uses it to request the email and full name from the
|
||||
service provider.
|
||||
|
||||
Then checks if this is an already registered profile or not. In the first case
|
||||
it opens a session, and in the second one calls<code class="language-clojure">:register-profile</code> to create a
|
||||
new user in the sytem.
|
||||
|
||||
For the known service providers, the addresses of the protocol endpoints are
|
||||
hardcoded. But for a generic OIDC service, there is a discovery protocol to ask
|
||||
the provider for them, or the system administrator may set them via configuration
|
||||
variables.
|
||||
|
||||
### LDAP
|
||||
|
||||
Registration is not possible by LDAP (we use an external user directory managed
|
||||
outside of Penpot). Typically when LDAP registration is enabled, the plain user
|
||||
& password login is disabled.
|
||||
|
||||
When the user types their user & password and presses "Login with LDAP" button,
|
||||
the <code class="language-clojure">:login-with-ldap</code> method is called. It connects with the LDAP service to
|
||||
validate credentials and retrieve email and full name.
|
||||
|
||||
Similarly as the OIDC backend, it checks if the profile exists, and calls
|
||||
<code class="language-clojure">:login</code> or <code class="language-clojure">:register-profile</code> as needed.
|
||||
|
||||
## Sessions
|
||||
|
||||
User sessions are created when a user logs in via any one of the backends. A
|
||||
session token is generated (a JWT token that does not currently contain any data)
|
||||
and returned to frontend as a cookie.
|
||||
|
||||
Normally the session is stored in a DB table with the information of the user
|
||||
profile and the session expiration. But if a frontend connects to the backend in
|
||||
"read only" mode (for example, to debug something in production with the local
|
||||
devenv), sessions are stored in memory (may be lost if the backend restarts).
|
||||
|
||||
## Team invitations
|
||||
|
||||
The invitation link has a call to <code class="language-clojure">/auth/verify-token</code> frontend view (explained
|
||||
above) with a token that includes the invited email.
|
||||
|
||||
When a user follows it, the token is verified and then the corresponding process
|
||||
is routed, depending if the email corresponds to an existing account or not. The
|
||||
<code class="language-clojure">:register-profile</code> or <code class="language-clojure">:login</code> services are used, and the invitation token is
|
||||
attached so that the profile is linked to the team at the end.
|
||||
|
||||
## Handling unfinished registrations and bouncing users
|
||||
|
||||
All tokens have an expiration date, and when they are put in a permanent
|
||||
storage, a garbage colector task visits it periodically to cleand old items.
|
||||
|
||||
Also our email sever registers email bounces and spam complaint reportings
|
||||
(see <code class="language-text">backend/src/app/emails.clj</code>). When the email of one profile receives too
|
||||
many notifications, it becames blocked. From this on, the user cannot login or
|
||||
register with this email, and no message will be sent to it. If it recovers
|
||||
later, it needs to be unlocked manually in the database.
|
||||
|
||||
## How to test in devenv
|
||||
|
||||
To test all normal registration process you can use the devenv [Mail
|
||||
catcher](/technical-guide/developer/devenv/#email) utility.
|
||||
|
||||
To test OIDC, you need to register an application in one of the providers:
|
||||
|
||||
* [Github](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app)
|
||||
* [Gitlab](https://docs.gitlab.com/ee/integration/oauth_provider.html)
|
||||
* [Google](https://support.google.com/cloud/answer/6158849)
|
||||
|
||||
The URL of the app will be the devenv frontend: [http://localhost:3449]().
|
||||
|
||||
And then put the credentials in <code class="language-text">backend/scripts/repl</code> and
|
||||
<code class="language-text">frontend/resources/public/js/config.js</code>.
|
||||
|
||||
Finally, to test LDAP, in the devenv we include a [test LDAP](https://github.com/rroemhild/docker-test-openldap)
|
||||
server, that is already configured, and only needs to be enabled in frontend
|
||||
<code class="language-text">config.js</code>:
|
||||
|
||||
```js
|
||||
var penpotFlags = "enable-login-with-ldap";
|
||||
```
|
||||
|
||||
15
docs/technical-guide/developer/subsystems/index.md
Normal file
15
docs/technical-guide/developer/subsystems/index.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
title: 3.8. Penpot subsystems
|
||||
---
|
||||
|
||||
# Penpot subsystems
|
||||
|
||||
This section groups articles about several Penpot subsystems that have enough
|
||||
complexity not to be easy to understand by only looking at the source code.
|
||||
|
||||
Each article gives an overview of how a particular functionality has been
|
||||
implemented, over the whole app (backend, frontend or exporter), and points to
|
||||
the most relevant source files to look at to start exploring it. When some
|
||||
special considerations are needed (performance questions, limits, common
|
||||
"gotchas", historic reasons of some decisions, etc.) they are also noted.
|
||||
|
||||
756
docs/technical-guide/developer/ui.md
Normal file
756
docs/technical-guide/developer/ui.md
Normal file
|
|
@ -0,0 +1,756 @@
|
|||
---
|
||||
title: 3.9. UI Guide
|
||||
---
|
||||
|
||||
# UI Guide
|
||||
|
||||
These are the guidelines for developing UI in Penpot, including the design system.
|
||||
|
||||
## React & Rumext
|
||||
|
||||
The UI in Penpot uses React v18 , with the help of [rumext](https://github.com/funcool/rumext) for providing Clojure bindings. See [Rumext's User Guide](https://funcool.github.io/rumext/latest/user-guide.html) to learn how to create React components with Clojure.
|
||||
|
||||
## General guidelines
|
||||
|
||||
We want to hold our UI code to the same quality standards of the rest of the codebase. In practice, this means:
|
||||
|
||||
- UI components should be easy to maintain over time, especially because our design system is ever-changing.
|
||||
- UI components should be accessible, and use the relevant HTML elements and/or Aria roles when applicable.
|
||||
- We need to apply the rules for good software design:
|
||||
- The code should adhere to common patterns.
|
||||
- UI components should offer an ergonomic "API" (i.e. props).
|
||||
- UI components should favor composability.
|
||||
- Try to have loose coupling.
|
||||
|
||||
### Composability
|
||||
|
||||
**Composability is a common pattern** in the Web. We can see it in the standard HTML elements, which are made to be nested one inside another to craft more complex content. Standard Web components also offer slots to make composability more flexible.
|
||||
|
||||
<mark>Our UI components must be composable</mark>. In React, this is achieved via the <code class="language-clojure">children</code> prop, in addition to pass slotted components via regular props.
|
||||
|
||||
#### Use of <code class="language-clojure">children</code>
|
||||
|
||||
> **⚠️ NOTE**: Avoid manipulating <code class="language-clojure">children</code> in your component. See [React docs](https://react.dev/reference/react/Children#alternatives) about the topic.
|
||||
|
||||
✅ **DO: Use children when we need to enable composing**
|
||||
|
||||
```clojure
|
||||
(mf/defc primary-button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [children] :rest props}]
|
||||
[:> "button" props children])
|
||||
```
|
||||
|
||||
❓**Why?**
|
||||
|
||||
By using children, we are signaling the users of the component that they can put things _inside_, vs a regular prop that only works with text, etc. For example, it’s obvious that we can do things like this:
|
||||
|
||||
```clojure
|
||||
[:> button* {}
|
||||
[:*
|
||||
"Subscribe for "
|
||||
[:& money-amount {:currency "EUR" amount: 3000}]]]
|
||||
```
|
||||
|
||||
#### Use of slotted props
|
||||
|
||||
When we need to either:
|
||||
|
||||
- Inject multiple (and separate) groups of elements.
|
||||
- Manipulate the provided components to add, remove, filter them, etc.
|
||||
|
||||
Instead of <code class="language-clojure">children</code>, we can pass the component(s) via a regular a prop.
|
||||
|
||||
#### When _not_ to pass a component via a prop
|
||||
|
||||
It's about **ownership**. By allowing the passing of a full component, the responsibility of styling and handling the events of that component belong to whoever instantiated that component and passed it to another one.
|
||||
|
||||
For instance, here the user would be in total control of the <code class="language-clojure">icon</code> component for styling (and for choosing which component to use as an icon, be it another React component, or a plain SVG, etc.)
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [icon children] :rest props}]
|
||||
[:> "button" props
|
||||
icon
|
||||
children])
|
||||
```
|
||||
|
||||
However, we might want to control the aspect of the icons, or limit which icons are available for this component, or choose which specific React component should be used. In this case, instead of passing the component via a prop, we'd want to provide the data we need for the icon component to be instantiated:
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [icon children] :rest props}]
|
||||
(assert (or (nil? icon) (contains? valid-icon-list icon) "expected valid icon id"))
|
||||
[:> "button" props
|
||||
(when icon [:> icon* {:id icon :size "m"}])
|
||||
children])
|
||||
```
|
||||
|
||||
### Our components should have a clear responsibility
|
||||
|
||||
It's important we are aware of:
|
||||
|
||||
- What are the **boundaries** of our component (i.e. what it can and cannot do)
|
||||
- Like in regular programming, it's good to keep all the inner elements at the same level of abstraction.
|
||||
- If a component grows too big, we can split it in several ones. Note that we can mark components as private with the <code class="language-clojure">::mf/private true</code> meta tag.
|
||||
- Which component is **responsible for what**.
|
||||
|
||||
As a rule of thumb:
|
||||
|
||||
- Components own the stuff they instantiate themselves.
|
||||
- Slotted components or <code class="language-clojure">children</code> belong to the place they have been instantiated.
|
||||
|
||||
This ownership materializes in other areas, like **styles**. For instance, parent components are usually reponsible for placing their children into a layout. Or, as mentioned earlier, we should avoid manipulating the styles of a component we don't have ownership over.
|
||||
|
||||
## Styling components
|
||||
|
||||
We use **CSS modules** and **Sass** to style components. Use the <code class="language-clojure">(stl/css)</code> and <code class="language-clojure">(stl/css-case)</code> functions to generate the class names for the CSS modules.
|
||||
|
||||
### Allow passing a class name
|
||||
|
||||
Our components should allow some customization by whoever is instantiating them. This is useful for positioning elements in a layout, providing CSS properties, etc.
|
||||
|
||||
This is achieved by accepting a <code class="language-clojure">class</code> prop (equivalent to <code class="language-clojure">className</code> in JSX). Then, we need to join the class name we have received as a prop with our own class name for CSS modules.
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [children class] :rest props}]
|
||||
(let [class (dm/str class " " (stl/css :primary-button))
|
||||
props (mf/spread-props props {:class class})]
|
||||
[:> "button" props children]))
|
||||
```
|
||||
|
||||
### About nested selectors
|
||||
|
||||
Nested styles for DOM elements that are not instantiated by our component should be avoided. Otherwise, we would be leaking CSS out of the component scope, which can lead to hard-to-maintain code.
|
||||
|
||||
❌ **AVOID: Styling elements that don’t belong to the component**
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [children] :rest props}]
|
||||
(let [props (mf/spread-props props {:class (stl/css :primary-button)})]
|
||||
;; note that we are NOT instantiating a <svg> here.
|
||||
[:> "button" props children]))
|
||||
|
||||
;; later in code
|
||||
[:> button* {}
|
||||
[:> icon {:id "foo"}]
|
||||
"Lorem ipsum"]
|
||||
```
|
||||
|
||||
```scss
|
||||
.button {
|
||||
// ...
|
||||
svg {
|
||||
fill: var(--icon-color);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
✅ **DO: Take ownership of instantiating the component we need to style**
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [icon children class] :rest props}]
|
||||
(let [props (mf/spread-props props {:class (stl/css :button)})]
|
||||
[:> "button" props
|
||||
(when icon [:> icon* {:id icon :size "m"}])
|
||||
[:span {:class (stl/css :label-wrapper)} children]]))
|
||||
|
||||
;; later in code
|
||||
[:> button* {:icon "foo"} "Lorem ipsum"]
|
||||
```
|
||||
|
||||
```scss
|
||||
.button {
|
||||
// ...
|
||||
}
|
||||
|
||||
.icon {
|
||||
fill: var(--icon-color);
|
||||
}
|
||||
```
|
||||
|
||||
### Favor lower specificity
|
||||
|
||||
This helps with maintanibility, since lower [specificity](https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity) styles are easier to override.
|
||||
|
||||
Remember that nesting selector increases specificity, and it's usually not needed. However, pseudo-classes and pseudo-elements don't.
|
||||
|
||||
❌ **AVOID: Using a not-needed high specificity**
|
||||
|
||||
```scss
|
||||
.btn {
|
||||
// ...
|
||||
.icon {
|
||||
fill: var(--icon-color);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
✅ **DO: Choose selectors with low specificity**
|
||||
|
||||
```scss
|
||||
.btn {
|
||||
// ...
|
||||
}
|
||||
|
||||
.icon {
|
||||
fill: var(--icon-color);
|
||||
}
|
||||
```
|
||||
|
||||
## Accessibility
|
||||
|
||||
### Let the browser do the heavy lifting
|
||||
|
||||
Whenever possible, leverage HTML semantic elements, which have been implemented by browsers and are accessible out of the box.
|
||||
|
||||
This includes:
|
||||
|
||||
- Using <code class="language-html">\<a></code> for link (navigation, downloading files, sending e-mails via <code class="language-html">mailto:</code>, etc.)
|
||||
- Using <code class="language-html">\<button></code> for triggering actions (submitting a form, closing a modal, selecting a tool, etc.)
|
||||
- Using the proper heading level.
|
||||
- Etc.
|
||||
|
||||
Also, elements **should be focusable** with keyboard. Pay attention to <code class="language-html">tabindex</code> and the use of focus.
|
||||
|
||||
### Aria roles
|
||||
|
||||
If you cannot use a native element because of styling (like a <code class="language-html">\<select></code> for a dropdown menu), consider either adding one that is hidden (except for assistive software) or use relevant [aria roles](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles) in your custom markup.
|
||||
|
||||
When using images as icons, they should have an <code class="language-html">aria-label</code>, <code class="language-html">alt</code>, or similar if they are not decorative and there's no text around to tag the button. Think, for instance, of a generic toolbar without text labels, just icon buttons.
|
||||
|
||||
For decorative images, they don't need to be anounced to assistive devices and should have <code class="language-html">aria-hidden</code> set to <code class="language-html">true</code>.
|
||||
|
||||
## Clojure / Rumext implementation notes
|
||||
|
||||
Please refer to the [Rumext User Guide](https://funcool.github.io/rumext/latest/user-guide.html) for important information, like naming conventions, available functions and macros, transformations done to props, etc.
|
||||
|
||||
Some things to have in mind:
|
||||
|
||||
- When you want to use JavaScript props, use the meta <code class="language-clojure">{::mf/props :obj}</code>. In this case, avoid using <code class="language-clojure">?</code> for boolean props, since they don't get a clean translation to JavaScript.
|
||||
- You can use type hints such as <code class="language-clojure">^boolean</code> to get JS semantics.
|
||||
- Split big components into smaller ones. You can mark components as private with the <code class="language-clojure">::mf/private true</code> meta.
|
||||
|
||||
### Delegating props
|
||||
|
||||
There is a mechanism to [delegate props](https://react.dev/learn/passing-props-to-a-component#forwarding-props-with-the-jsx-spread-syntax) equivalent to this:
|
||||
|
||||
```jsx
|
||||
const Button => ({children, ...other}) {
|
||||
return <button {...other}>{children}</button>
|
||||
};
|
||||
```
|
||||
|
||||
We just need to use `:rest ` when declaring the component props.
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [children] :rest other}]
|
||||
[:> "button" other children])
|
||||
```
|
||||
|
||||
If we need to augment this props object, we can use <code class="language-clojure">spread-props</code> and the usual transformations that Rumext does (like <code class="language-clojure">class</code> -> <code class="language-clojure">className</code>, for instance) will be applied too.
|
||||
|
||||
```clojure
|
||||
(mf/defc button*
|
||||
{::mf/props :obj}
|
||||
[{:keys [children class] :rest props}]
|
||||
(let [class (dm/str class " " (stl/css :button))
|
||||
props (mf/spread-props props {:class class})]
|
||||
[:> "button" props children]))
|
||||
```
|
||||
|
||||
### Performance considerations
|
||||
|
||||
For components that belong to the “hot path” of rendering (like those in the sidebar, for instance), it’s worth avoiding some pitfalls that make rendering slower and/or will trigger a re-render.
|
||||
|
||||
Most of this techniques revolve around achieving one of these:
|
||||
|
||||
- Avoid creating brand new objects and functions in each render.
|
||||
- Avoid needlessly operations that can be costly.
|
||||
- Avoid a re-render.
|
||||
|
||||
#### Use of a JS object as props
|
||||
|
||||
It's faster to use a JS Object for props instead of a native Clojure map, because then that conversion will not happen in runtime in each re-render.
|
||||
|
||||
✅ **DO: Use the metadata <code class="language-clojure">::mf/props :obj</code> when creating a component**
|
||||
|
||||
```clojure
|
||||
(mf/defc icon*
|
||||
{::mf/props :obj}
|
||||
[props]
|
||||
;; ...
|
||||
)
|
||||
```
|
||||
|
||||
#### Split large and complex components into smaller parts
|
||||
|
||||
This can help to avoid full re-renders.
|
||||
|
||||
#### Avoid creating anonymous functions as callback handlers, etc.
|
||||
|
||||
This creates a brand new function every render. Instead, create the function on its own and memoize it when needed.
|
||||
|
||||
❌ **AVOID: Creating anonymous functions for handlers**
|
||||
|
||||
```clojure
|
||||
(mf/defc login-button {::mf/props obj} []
|
||||
[:button {:on-click (fn []
|
||||
;; emit event to login, etc.
|
||||
)}
|
||||
"Login"])
|
||||
```
|
||||
|
||||
✅ **DO: Use named functions as callback handlers**
|
||||
|
||||
```clojure
|
||||
(defn- login []
|
||||
;; ...
|
||||
)
|
||||
|
||||
(mf/defc login-button
|
||||
{::mf/props :obj}
|
||||
[]
|
||||
[:button {:on-click login} "Login"])
|
||||
|
||||
```
|
||||
|
||||
#### Avoid defining functions inside of a component (via <code class="language-clojure">let</code>)
|
||||
|
||||
When we do this inside of a component, a brand new function is created in every render.
|
||||
|
||||
❌ \*\*AVOID: Using <code class="language-clojure">let</code> to define functions
|
||||
|
||||
```clojure
|
||||
(mf/defc login-button
|
||||
{::mf/props :obj}
|
||||
[]
|
||||
(let [click-handler (fn []
|
||||
;; ...
|
||||
)]
|
||||
[:button {:on-click click-handler} "Login"]))
|
||||
```
|
||||
|
||||
✅ **DO: Define functions outside of the component**
|
||||
|
||||
```clojure
|
||||
(defn- login []
|
||||
;; ...
|
||||
)
|
||||
|
||||
(mf/defc login-button
|
||||
{::mf/props :obj}
|
||||
[]
|
||||
[:button {:on-click login} "Login"])
|
||||
```
|
||||
|
||||
#### Avoid defining functions with <code class="language-clojure">partial</code> inside of components
|
||||
|
||||
<code class="language-clojure">partial</code> returns a brand new anonymous function, so we should avoid using it in each render. For callback handlers that need parameters, a work around is to store these as <code class="language-clojure">data-*</code> attributes and retrieve them inside the function.
|
||||
|
||||
❌ **AVOID: Using `partial` inside of a component**
|
||||
|
||||
```clojure
|
||||
(defn- set-margin [side value]
|
||||
;; ...
|
||||
)
|
||||
|
||||
(mf/defc margins []
|
||||
[:*
|
||||
[:> numeric-input* {:on-change (partial set-margin :left)}]
|
||||
[:> numeric-input* {:on-change (partial set-margin :right)}] ])
|
||||
```
|
||||
|
||||
✅ **DO: Use <code class="language-clojure">data-*</code> attributes to modify a function (many uses)**
|
||||
|
||||
```clojure
|
||||
(defn- set-margin [value event]
|
||||
(let [side -> (dom/get-current-target event)
|
||||
(dom/get-data "side")
|
||||
(keyword)]
|
||||
;; ...
|
||||
)
|
||||
|
||||
(defc margins []
|
||||
[:*
|
||||
[:> numeric-input* {:data-side "left" :on-change set-margin}]
|
||||
[:> numeric-input* {:data-side "right" :on-change set-margin}]
|
||||
[:> numeric-input* {:data-side "top" :on-change set-margin}]
|
||||
[:> numeric-input* {:data-side "bottom" :on-change set-margin}]])
|
||||
|
||||
```
|
||||
|
||||
✅ **DO: Store the returned function from <code class="language-clojure">partial</code> (few uses)**
|
||||
|
||||
```clojure
|
||||
(defn- set-padding [sides value]
|
||||
;; ...
|
||||
)
|
||||
|
||||
(def set-block-padding (partial set-padding :block))
|
||||
(def set-inline-padding (partial set-padding :inline))
|
||||
|
||||
(defc paddings []
|
||||
[:*
|
||||
[:> numeric-input* {:on-change set-block-padding}]
|
||||
[:> numeric-input* {:on-change set-inline-padding}]])
|
||||
```
|
||||
|
||||
#### Store values you need to use multiple times
|
||||
|
||||
Often we need to access values from props. It's best if we destructure them (because it can be costly, especially if this adds up and we need to access them multiple times) and store them in variables.
|
||||
|
||||
##### Destructuring props
|
||||
|
||||
✅ **DO: Destructure props with <code class="language-clojure">:keys</code>**
|
||||
|
||||
```clojure
|
||||
(defc icon
|
||||
{::mf/props :obj}
|
||||
[{:keys [size img] :as props]
|
||||
[:svg {:width size
|
||||
:height size
|
||||
:class (stl/css-case icon true
|
||||
icon-large (> size 16))}
|
||||
[:use {:href img}]])
|
||||
```
|
||||
|
||||
❌ **AVOID: Accessing the object each time**
|
||||
|
||||
```clojure
|
||||
(defc icon
|
||||
{::mf/props :obj}
|
||||
[props]
|
||||
[:svg {:width (unchecked-get props "size")
|
||||
:height (unchecked-get props "size")
|
||||
:class (stl/css-case icon true
|
||||
icon-large (> (unchecked-get props "size") 16))}
|
||||
[:use {:href (unchecked-get props "img")}]])
|
||||
```
|
||||
|
||||
##### Storing state values
|
||||
|
||||
We can avoid multiple calls to <code class="language-clojure">(deref)</code> if we store the value in a variable.
|
||||
|
||||
✅ **DO: store state values**
|
||||
|
||||
```clojure
|
||||
(defc accordion
|
||||
{::mf/props :obj}
|
||||
[{:keys [^boolean default-open title children] :as props]
|
||||
|
||||
(let [
|
||||
open* (mf/use-state default-open)
|
||||
open? (deref open*)]
|
||||
[:details {:open open?}
|
||||
[:summary title]
|
||||
children]))
|
||||
```
|
||||
|
||||
##### Unroll loops
|
||||
|
||||
Creating an array of static elements and iterating over it to generate DOM may be more costly than manually unrolling the loop.
|
||||
|
||||
❌ **AVOID: iterating over a static array**
|
||||
|
||||
```clojure
|
||||
(defc shape-toolbar []
|
||||
(let tools ["rect" "circle" "text"]
|
||||
(for tool tools [:> tool-button {:tool tool}])))
|
||||
```
|
||||
|
||||
✅ **DO: unroll the loop**
|
||||
|
||||
```clojure
|
||||
(defc shape-toolbar []
|
||||
[:*
|
||||
[:> tool-button {:tool "rect"}]
|
||||
[:> tool-button {:tool "circle"}]
|
||||
[:> tool-button {:tool "text"}]])
|
||||
```
|
||||
|
||||
## Penpot Design System
|
||||
|
||||
Penpot has started to use a **design system**, which is located at <code class="language-bash">frontend/src/app/main/ui/ds</code>. The components of the design system is published in a Storybook at [hourly.penpot.dev/storybook/](https://hourly.penpot.dev/storybook/) with the contents of the <code class="language-bash">develop</code> branch of the repository.
|
||||
|
||||
<mark>When a UI component is **available in the design system**, use it!</mark>. If it's not available but it's part of the Design System (ask the design folks if you are unsure), then do add it to the design system and Storybook.
|
||||
|
||||
### Adding a new component
|
||||
|
||||
In order to implement a new component for the design system, you need to:
|
||||
|
||||
- Add a new <code class="language-bash">\<component>.cljs</code> file within the <code class="language-bash">ds/</code> folder tree. This contains the CLJS implementation of the component, and related code (props schemas, private components, etc.).
|
||||
- Add a <code class="language-bash">\<component>.css</code> file with the styles for the component. This is a CSS Module file, and the selectors are scoped to this component.
|
||||
- Add a <code class="language-bash">\<component>.stories.jsx</code> Storybook file (see the _Storybook_ section below).
|
||||
- (Optional) When available docs, add a <code class="language-bash">\<component>.mdx</code> doc file (see _Storybook_ section below).
|
||||
|
||||
In addition to the above, you also need to **specifically export the new component** with a JavaScript-friendly name in <code class="language-bash">frontend/src/app/main/ui/ds.cljs</code>.
|
||||
|
||||
### Tokens
|
||||
|
||||
We use three **levels of tokens**:
|
||||
|
||||
- **Primary** tokens, referring to raw values (i.e. pixels, hex colors, etc.) of color, sizes, borders, etc. These are implemented as Sass variables. Examples are: <code class="language-css">$mint-250</code>, <code class="language-css">$sz-16</code>, <code class="language-css">$br-circle</code>, etc.
|
||||
|
||||
- **Semantic** tokens, used mainly for theming. These are implemented with **CSS custom properties**. Depending on the theme, these semantic tokens would have different primary tokens as values. For instance, <code class="language-css">--color-accent-primary</code> is <code class="language-css">$purple-700</code> when the light theme is active, but <code class="language-css">$mint-150</code> in the default theme. These custom properties have **global scope**.
|
||||
|
||||
- **Component** tokens, defined at component level as **CSS custom properties**. These are very useful when implementing variants. Examples include <code class="language-css">--button-bg-color</code> or <code class="language-css">--toast-icon-color</code>. These custom properties are constrained to the **local scope** of its component.
|
||||
|
||||
### Implementing variants
|
||||
|
||||
We can leverage component tokens to easily implement variants, by overriding their values in each component variant.
|
||||
|
||||
For instance, this is how we handle the styles of <code class="language-clojure">\<Toast></code>, which have a different style depending on the level of the message (default, info, error, etc.)
|
||||
|
||||
```scss
|
||||
.toast {
|
||||
// common styles for all toasts
|
||||
// ...
|
||||
|
||||
--toast-bg-color: var(--color-background-primary);
|
||||
--toast-icon-color: var(--color-foreground-secondary);
|
||||
// ... more variables here
|
||||
|
||||
background-color: var(--toast-bg-color);
|
||||
}
|
||||
|
||||
.toast-icon {
|
||||
color: var(--toast-bg-color);
|
||||
}
|
||||
|
||||
.toast-info {
|
||||
--toast-bg-color: var(--color-background-info);
|
||||
--toast-icon-color: var(--color-accent-info);
|
||||
// ... override more variables here
|
||||
}
|
||||
|
||||
.toast-error {
|
||||
--toast-bg-color: var(--color-background-error);
|
||||
--toast-icon-color: var(--color-accent-error);
|
||||
// ... override more variables here
|
||||
}
|
||||
|
||||
// ... more variants here
|
||||
```
|
||||
|
||||
### Using icons and SVG assets
|
||||
|
||||
Please refer to the Storybook [documentation for icons](https://hourly.penpot.dev/storybook/?path=/docs/foundations-assets-icon--docs) and other [SVG assets](https://hourly.penpot.dev/storybook/?path=/docs/foundations-assets-rawsvg--docs) (logos, illustrations, etc.).
|
||||
|
||||
### Storybook
|
||||
|
||||
We use [Storybook](https://storybook.js.org/) to implement and showcase the components of the Design System.
|
||||
|
||||
The Storybook is available at the <code class="language-bash">/storybook</code> path in the URL for each environment. For instance, the one built out of our <code class="language-bash">develop</code> branch is available at [hourly.penpot.dev/storybook](https://hourly.penpot.dev/storybook).
|
||||
|
||||
#### Local development
|
||||
|
||||
Use <code class="language-bash">yarn watch:storybook</code> to develop the Design System components with the help of Storybook.
|
||||
|
||||
> **⚠️ WARNING**: Do stop any existing Shadow CLJS and asset compilation jobs (like the ones running at tabs <code class="language-bash">0</code> and <code class="language-bash">1</code> in the devenv tmux), because <code class="language-bash">watch:storybook</code> will spawn their own.
|
||||
|
||||
#### Writing stories
|
||||
|
||||
You should add a Storybook file for each design system component you implement. This is a <code class="language-bash">.jsx</code> file located at the same place as your component file, with the same name. For instance, a component defined in <code class="language-bash">loader.cljs</code> should have a <code class="language-bash">loader.stories.jsx</code> files alongside.
|
||||
|
||||
A **story showcases how to use** a component. For the most relevant props of your component, it's important to have at least one story to show how it's used and what effect it has.
|
||||
|
||||
Things to take into account when considering which stories to add and how to write them:
|
||||
|
||||
- Stories show have a <code class="language-bash">Default</code> story that showcases how the component looks like with default values for all the props.
|
||||
|
||||
- If a component has variants, we should show each one in its own story.
|
||||
|
||||
- Leverage setting base prop values in <code class="language-bash">args</code> and common rendering code in <code class="language-bash">render</code> to reuse those in the stories and avoid code duplication.
|
||||
|
||||
For instance, the stories file for the <code class="language-bash">button*</code> component looks like this:
|
||||
|
||||
```jsx
|
||||
// ...
|
||||
|
||||
export default {
|
||||
title: "Buttons/Button",
|
||||
component: Components.Button,
|
||||
// These are the props of the component, and we set here default values for
|
||||
// all stories.
|
||||
args: {
|
||||
children: "Lorem ipsum",
|
||||
disabled: false,
|
||||
variant: undefined,
|
||||
},
|
||||
// ...
|
||||
render: ({ ...args }) => <Button {...args} />,
|
||||
};
|
||||
|
||||
export const Default = {};
|
||||
|
||||
// An important prop: `icon`
|
||||
export const WithIcon = {
|
||||
args: {
|
||||
icon: "effects",
|
||||
},
|
||||
};
|
||||
|
||||
// A variant
|
||||
export const Primary = {
|
||||
args: {
|
||||
variant: "primary",
|
||||
},
|
||||
};
|
||||
|
||||
// Another variant
|
||||
export const Secondary = {
|
||||
args: {
|
||||
variant: "secondary",
|
||||
},
|
||||
};
|
||||
|
||||
// More variants here…
|
||||
```
|
||||
|
||||
In addition to the above, please **use the [Controls addon](https://storybook.js.org/docs/essentials/controls)** to let users change props and see their effect on the fly.
|
||||
|
||||
Controls are customized with <code class="language-bash">argTypes</code>, and you can control which ones to show / hide with <code class="language-bash">parameters.controls.exclude</code>. For instance, for the <code class="language-bash">button*</code> stories file, its relevant control-related code looks like this:
|
||||
|
||||
```jsx
|
||||
// ...
|
||||
const { icons } = Components.meta;
|
||||
|
||||
export default {
|
||||
// ...
|
||||
argTypes: {
|
||||
// Use the `icons` array for possible values for the `icon` prop, and
|
||||
// display them in a dropdown select
|
||||
icon: {
|
||||
options: icons,
|
||||
control: { type: "select" },
|
||||
},
|
||||
// Use a toggle for the `disabled` flag prop
|
||||
disabled: { control: "boolean" },
|
||||
// Show these values in a dropdown for the `variant` prop.
|
||||
variant: {
|
||||
options: ["primary", "secondary", "ghost", "destructive"],
|
||||
control: { type: "select" },
|
||||
},
|
||||
},
|
||||
parameters: {
|
||||
// Always hide the `children` controls.
|
||||
controls: { exclude: ["children"] },
|
||||
},
|
||||
// ...
|
||||
};
|
||||
```
|
||||
|
||||
#### Adding docs
|
||||
|
||||
Often, Design System components come along extra documentation provided by Design. Furthermore, they might be technical things to be aware of. For this, you can add documentation in [MDX format](https://storybook.js.org/docs/writing-docs/mdx).
|
||||
|
||||
You can use Storybook's <code class="language-bash">\<Canvas></code> element to showcase specific stories to enrich the documentation.
|
||||
|
||||
When including codeblocks, please add code in Clojure syntax (not JSX).
|
||||
|
||||
You can find an example MDX file in the [Buttons docs](https://hourly.penpot.dev/storybook/?path=/docs/buttons-docs--docs).
|
||||
|
||||
### Replacing a deprecated component
|
||||
|
||||
#### Run visual regression tests
|
||||
|
||||
We need to generate the screenshots for the visual regression tests _before_ making
|
||||
any changes, so we can compare the "before substitution" and "after substitution" states.
|
||||
|
||||
|
||||
Execute the tests in the playwright's <code class="language-bash">ds</code> project. In order to do so, stop the Shadow CLJS compiler in tmux tab <code class="language-bash">#1</code> and run;
|
||||
```bash
|
||||
clojure -M:dev:shadow-cljs release main
|
||||
```
|
||||
This will package the frontend in release mode so the tests run faster.
|
||||
|
||||
In your terminal, in the frontend folder, run:
|
||||
```bash
|
||||
npx playwright test --ui --project=ds
|
||||
```
|
||||
This will open the test runner UI in the selected project.
|
||||
|
||||
|
||||
|
||||
The first time you run these tests they'll fail because there are no screenshots yet, but the second time, they should pass.
|
||||
|
||||
#### Import the new component
|
||||
|
||||
In the selected file add the new namespace from the <code class="language-bash">ds</code> folder in alphabetical order:
|
||||
|
||||
```clojure
|
||||
[app.main.ui.ds.tab-switcher :refer [tab-switcher*]]
|
||||
...
|
||||
|
||||
[:> tab-switcher* {}]
|
||||
```
|
||||
|
||||
> **⚠️ NOTE**: Components with a <code class="language-bash">*</code> suffix are meant to be used with the <code class="language-clojure">[:></code> handler.
|
||||
|
||||
<small>Please refer to [Rumext User Guide](https://funcool.github.io/rumext/latest/user-guide.html) for more information.</small>
|
||||
|
||||
#### Pass props to the component
|
||||
|
||||
Check the props schema in the component’s source file
|
||||
|
||||
```clojure
|
||||
(def ^:private schema:tab-switcher
|
||||
[:map
|
||||
[:class {:optional true} :string]
|
||||
[:action-button-position {:optional true}
|
||||
[:enum "start" "end"]]
|
||||
[:default-selected {:optional true} :string]
|
||||
[:tabs [:vector {:min 1} schema:tab]]])
|
||||
|
||||
|
||||
(mf/defc tab-switcher*
|
||||
{::mf/props :obj
|
||||
::mf/schema schema:tab-switcher}...)
|
||||
```
|
||||
This schema shows which props are required and which are optional, so you can
|
||||
include the necessary values with the correct types.
|
||||
|
||||
Populate the component with the required props.
|
||||
|
||||
```clojure
|
||||
(let [tabs
|
||||
#js [#js {:label (tr "inspect.tabs.info")
|
||||
:id "info"
|
||||
:content info-content}
|
||||
|
||||
#js {:label (tr "inspect.tabs.code")
|
||||
:data-testid "code"
|
||||
:id "code"
|
||||
:content code-content}]]
|
||||
|
||||
[:> tab-switcher* {:tabs tabs
|
||||
:default-selected "info"
|
||||
:on-change-tab handle-change-tab
|
||||
:class (stl/css :viewer-tab-switcher)}])
|
||||
```
|
||||
|
||||
Once the component is rendering correctly, remove the old component and its imports.
|
||||
|
||||
#### Check tests after changes
|
||||
|
||||
Verify that everything looks the same after making the changes. To do this, run
|
||||
the visual tests again as previously described.
|
||||
|
||||
If the design hasn’t changed, the tests should pass without issues.
|
||||
|
||||
However, there are cases where the design might have changed from the original.
|
||||
In this case, first check the <code class="language-bash">diff</code> files provided by the test runner to ensure
|
||||
that the differences are expected (e.g., positioning, size, etc.).
|
||||
|
||||
Once confirmed, inform the QA team about these changes so they can review and take any necessary actions.
|
||||
Loading…
Add table
Add a link
Reference in a new issue