3pp-mirror/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2025-05-25 00:36:11 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #6486 from penpot/niwinz-library-export

Browse source 
✨ Add .penpot export support for penpot library
This commit is contained in:
Alejandro Alonso 2025-05-20 13:27:11 +02:00 • committed by GitHub
commit 0d60e3d997
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
55 changed files with 3199 additions and 3691 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored
View file

@ -68,6 +68,8 @@
/vendor/**/target
/vendor/svgclean/bundle*.js
/web
/library/target/
clj-profiler/
node_modules
/test-results/

80
CHANGES.md
View file

@ -8,33 +8,65 @@
**Breaking changes on penpot library:**
The library entrypoint API object has been changed. From now you start creating a new
build context, from where you can add multiple files and attach media. This change add the
ability to build more than one file at same time and export them in an unique .penpot
file.
```js
const context = penpot.createBuildContext()
context.addFile({name:"aa"})
context.addPage({name:"aa"})
context.closePage()
context.closeFile()
;; barray is instance of Uint8Array
const barray = penpot.exportAsBytes(context);
```
The previous `file.export()` method has been removed and several alternatives are
added as first level functions on penpot library API entrypoint:
- `exportAsBytes(BuildContext context) -> Promise<Uint8Array>`
- `exportAsBlob(BuildContext context) -> Promise<Blob>`
- `exportStream(BuildContext context, WritableStream stream) -> Promise<Void>`
The stream variant allows writting data as it is generated to the stream, without the need
to store the generated output entirelly in the memory.
There are also relevant semantic changes in how components should be created: this
refactor removes all notions of the old components (v1). Since v2, the shapes that are
part of a component live on a page. So, from now on, to create a component, you should
first create a frame, then add shapes and/or groups to that frame, and then create a
component by declaring that frame as the component root.
A non exhaustive list of changes:
- Change the signature of the `addPage` method: it now accepts an object (as a single argument) where you can pass `id`,
`name`, and `background` props (instead of the previous positional arguments)
- Rename the `file.createRect` method to `file.addRect`
- Rename the `file.createCircle` method to `file.addCircle`
- Rename the `file.createPath` method to `file.addPath`
- Rename the `file.createText` method to `file.addText`
- Rename `file.startComponent` to `file.addComponent` (to preserve the naming style)
- Rename `file.createComponentInstance` to `file.addComponentInstance` (to preserve the naming style)
- Rename `file.lookupShape` to `file.getShape`
- Rename `file.asMap` to `file.toMap`
- Remove `file.updateLibraryColor` (use `file.addLibraryColor` if you just need to replace a color)
- Remove `file.deleteLibraryColor` (this library is intended to build files)
- Remove `file.updateLibraryTypography` (use `file.addLibraryTypography` if you just need to replace a typography)
- Remove `file.deleteLibraryTypography` (this library is intended to build files)
- Remove `file.add/update/deleteLibraryMedia` (they are no longer supported by Penpot and have been replaced by components)
- Remove `file.deleteObject` (this library is intended to build files)
- Remove `file.updateObject` (this library is intended to build files)
- Remove `file.finishComponent` (it is no longer necessary; see below for more details on component creation changes)
- Change the `file.getCurrentPageId` function to a read-only `file.currentPageId` property
- Add `file.currentFrameId` read-only property
- Add `file.lastId` read-only property
- Rename the `createRect` method to `addRect`
- Rename the `createCircle` method to `addCircle`
- Rename the `createPath` method to `addPath`
- Rename the `createText` method to `addText`
- Rename the `addArtboard` method to `addBoard`
- Rename `startComponent` to `addComponent` (to preserve the naming style)
- Rename `createComponentInstance` to `addComponentInstance` (to preserve the naming style)
- Remove `lookupShape`
- Remove `asMap`
- Remove `updateLibraryColor` (use `addLibraryColor` if you just need to replace a color)
- Remove `deleteLibraryColor` (this library is intended to build files)
- Remove `updateLibraryTypography` (use `addLibraryTypography` if you just need to replace a typography)
- Remove `deleteLibraryTypography` (this library is intended to build files)
- Remove `add/update/deleteLibraryMedia` (they are no longer supported by Penpot and have been replaced by components)
- Remove `deleteObject` (this library is intended to build files)
- Remove `updateObject` (this library is intended to build files)
- Remove `finishComponent` (it is no longer necessary; see below for more details on component creation changes)
There are also relevant semantic changes in how components should be created: this refactor removes
all notions of the old components (v1). Since v2, the shapes that are part of a component live on a
page. So, from now on, to create a component, you should first create a frame, then add shapes
and/or groups to that frame, and then create a component by declaring that frame as the component
root.
- Change the `getCurrentPageId` function to a read-only `currentPageId` property
- Add `currentFileId` read-only property
- Add `currentFrameId` read-only property
- Add `lastId` read-only property
### :heart: Community contributions (Thank you!)

9
backend/src/app/binfile/migrations.clj
View file

@ -10,7 +10,6 @@
[app.binfile.common :as bfc]
[app.common.exceptions :as ex]
[app.common.features :as cfeat]
[app.features.components-v2 :as feat.compv2]
[clojure.set :as set]
[cuerdas.core :as str]))
@ -28,13 +27,11 @@
(defn apply-pending-migrations!
"Apply alredy registered pending migrations to files"
[cfg]
(doseq [[feature file-id] (-> bfc/*state* deref :pending-to-migrate)]
[_cfg]
(doseq [[feature _file-id] (-> bfc/*state* deref :pending-to-migrate)]
(case feature
"components/v2"
(feat.compv2/migrate-file! cfg file-id
:validate? (::validate cfg true)
:skip-on-graphic-error? true)
nil
"fdata/shape-data-type"
nil

45
backend/src/app/binfile/v3.clj
View file

@ -18,6 +18,7 @@
[app.common.files.migrations :as-alias fmg]
[app.common.json :as json]
[app.common.logging :as l]
[app.common.media :as cmedia]
[app.common.schema :as sm]
[app.common.thumbnails :as cth]
[app.common.types.color :as ctcl]
@ -73,7 +74,7 @@
[:size ::sm/int]
[:content-type :string]
[:bucket [::sm/one-of {:format :string} sto/valid-buckets]]
[:hash :string]])
[:hash {:optional true} :string]])
(def ^:private schema:file-thumbnail
[:map {:title "FileThumbnail"}
@ -88,13 +89,19 @@
ctf/schema:file
[:map [:options {:optional true} ctf/schema:options]]])
;; --- HELPERS
(defn- default-now
[o]
(or o (dt/now)))
;; --- ENCODERS
(def encode-file
(sm/encoder schema:file sm/json-transformer))
(def encode-page
(sm/encoder ::ctp/page sm/json-transformer))
(sm/encoder ctp/schema:page sm/json-transformer))
(def encode-shape
(sm/encoder ::cts/shape sm/json-transformer))
@ -129,7 +136,7 @@
(sm/decoder schema:manifest sm/json-transformer))
(def decode-media
(sm/decoder ::ctf/media sm/json-transformer))
(sm/decoder ctf/schema:media sm/json-transformer))
(def decode-component
(sm/decoder ::ctc/component sm/json-transformer))
@ -229,27 +236,13 @@
:always
(bfc/clean-file-features))))))
(defn- resolve-extension
[mtype]
(case mtype
"image/png" ".png"
"image/jpeg" ".jpg"
"image/gif" ".gif"
"image/svg+xml" ".svg"
"image/webp" ".webp"
"font/woff" ".woff"
"font/woff2" ".woff2"
"font/ttf" ".ttf"
"font/otf" ".otf"
"application/octet-stream" ".bin"))
(defn- export-storage-objects
[{:keys [::output] :as cfg}]
(let [storage (sto/resolve cfg)]
(doseq [id (-> bfc/*state* deref :storage-objects not-empty)]
(let [sobject (sto/get-object storage id)
smeta (meta sobject)
ext (resolve-extension (:content-type smeta))
ext (cmedia/mtype->extension (:content-type smeta))
path (str "objects/" id ".json")
params (-> (meta sobject)
(assoc :id (:id sobject))
@ -574,7 +567,14 @@
(let [object (->> (read-entry input entry)
(decode-media)
(validate-media))
object (assoc object :file-id file-id)]
object (-> object
(assoc :file-id file-id)
(update :created-at default-now)
(update :modified-at default-now)
;; FIXME: this is set default to true for
;; setting a value, this prop is no longer
;; relevant;
(assoc :is-local true))]
(if (= id (:id object))
(conj result object)
result)))
@ -800,7 +800,7 @@
:expected-id (str id)
:found-id (str (:id object))))
(let [ext (resolve-extension (:content-type object))
(let [ext (cmedia/mtype->extension (:content-type object))
path (str "objects/" id ext)
content (->> path
(get-zip-entry input)
@ -814,13 +814,14 @@
:expected-size (:size object)
:found-size (sto/get-size content)))
(when (not= (:hash object) (sto/get-hash content))
(when-let [hash (get object :hash)]
(when (not= hash (sto/get-hash content))
(ex/raise :type :validation
:code :inconsistent-penpot-file
:hint "found corrupted storage object: hash does not match"
:path path
:expected-hash (:hash object)
:found-hash (sto/get-hash content)))
:found-hash (sto/get-hash content))))
(let [params (-> object
(dissoc :id :size)

1845
backend/src/app/features/components_v2.clj
View file

File diff suppressed because it is too large Load diff

104
backend/src/app/media.clj
View file

@ -8,12 +8,11 @@
"Media & Font postprocessing."
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.media :as cm]
[app.common.schema :as sm]
[app.common.schema.openapi :as-alias oapi]
[app.common.spec :as us]
[app.common.svg :as csvg]
[app.config :as cf]
[app.db :as-alias db]
[app.storage :as-alias sto]
@ -22,30 +21,21 @@
[buddy.core.bytes :as bb]
[buddy.core.codecs :as bc]
[clojure.java.shell :as sh]
[clojure.spec.alpha :as s]
[clojure.xml :as xml]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
clojure.lang.XMLHandler
java.io.InputStream
javax.xml.XMLConstants
javax.xml.parsers.SAXParserFactory
org.apache.commons.io.IOUtils
org.im4java.core.ConvertCmd
org.im4java.core.IMOperation
org.im4java.core.Info))
(s/def ::path fs/path?)
(s/def ::filename string?)
(s/def ::size integer?)
(s/def ::headers (s/map-of string? string?))
(s/def ::mtype string?)
(s/def ::upload
(s/keys :req-un [::filename ::size ::path]
:opt-un [::mtype ::headers]))
;; A subset of fields from the ::upload spec
(s/def ::input
(s/keys :req-un [::path]
:opt-un [::mtype]))
(def schema:upload
(sm/register!
^{::sm/type ::upload}
[:map {:title "Upload"}
@ -54,7 +44,15 @@
[:path ::fs/path]
[:mtype {:optional true} :string]
[:headers {:optional true}
[:map-of :string :string]]])
[:map-of :string :string]]]))
(def ^:private schema:input
[:map {:title "Input"}
[:path ::fs/path]
[:mtype {:optional true} ::sm/text]])
(def ^:private check-input
(sm/check-fn schema:input))
(defn validate-media-type!
([upload] (validate-media-type! upload cm/valid-image-types))
@ -97,17 +95,44 @@
(catch Throwable e
(process-error e))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG PARSING
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- secure-parser-factory
[^InputStream input ^XMLHandler handler]
(.. (doto (SAXParserFactory/newInstance)
(.setFeature XMLConstants/FEATURE_SECURE_PROCESSING true)
(.setFeature "http://apache.org/xml/features/disallow-doctype-decl" true))
(newSAXParser)
(parse input handler)))
(defn- strip-doctype
[data]
(cond-> data
(str/includes? data "<!DOCTYPE")
(str/replace #"<\!DOCTYPE[^>]*>" "")))
(defn- parse-svg
[text]
(let [text (strip-doctype text)]
(dm/with-open [istream (IOUtils/toInputStream text "UTF-8")]
(xml/parse istream secure-parser-factory))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMAGE THUMBNAILS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(s/def ::width integer?)
(s/def ::height integer?)
(s/def ::format #{:jpeg :webp :png})
(s/def ::quality #(< 0 % 101))
(def ^:private schema:thumbnail-params
[:map {:title "ThumbnailParams"}
[:input schema:input]
[:format [:enum :jpeg :webp :png]]
[:quality [:int {:min 1 :max 100}]]
[:width :int]
[:height :int]])
(s/def ::thumbnail-params
(s/keys :req-un [::input ::format ::width ::height]))
(def ^:private check-thumbnail-params
(sm/check-fn schema:thumbnail-params))
;; Related info on how thumbnails generation
;; http://www.imagemagick.org/Usage/thumbnails/
@ -129,21 +154,28 @@
:data tmp)))
(defmethod process :generic-thumbnail
[{:keys [quality width height] :as params}]
(us/assert ::thumbnail-params params)
(let [op (doto (IMOperation.)
[params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)
operation
(doto (IMOperation.)
(.addImage)
(.autoOrient)
(.strip)
(.thumbnail ^Integer (int width) ^Integer (int height) ">")
(.quality (double quality))
(.addImage))]
(generic-process (assoc params :operation op))))
(generic-process (assoc params :operation operation))))
(defmethod process :profile-thumbnail
[{:keys [quality width height] :as params}]
(us/assert ::thumbnail-params params)
(let [op (doto (IMOperation.)
[params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)
operation
(doto (IMOperation.)
(.addImage)
(.autoOrient)
(.strip)
@ -152,7 +184,8 @@
(.extent (int width) (int height))
(.quality (double quality))
(.addImage))]
(generic-process (assoc params :operation op))))
(generic-process (assoc params :operation operation))))
(defn get-basic-info-from-svg
[{:keys [tag attrs] :as data}]
@ -184,10 +217,9 @@
(defmethod process :info
[{:keys [input] :as params}]
(us/assert ::input input)
(let [{:keys [path mtype]} input]
(let [{:keys [path mtype] :as input} (check-input input)]
(if (= mtype "image/svg+xml")
(let [info (some-> path slurp csvg/parse get-basic-info-from-svg)]
(let [info (some-> path slurp parse-svg get-basic-info-from-svg)]
(when-not info
(ex/raise :type :validation
:code :invalid-svg-file

6
backend/src/app/rpc/climit.clj
View file

@ -22,7 +22,8 @@
[datoteka.fs :as fs]
[integrant.core :as ig]
[promesa.exec :as px]
[promesa.exec.bulkhead :as pbh])
[promesa.exec.bulkhead :as pbh]
[promesa.protocols :as pt])
(:import
clojure.lang.ExceptionInfo
java.util.concurrent.atomic.AtomicLong))
@ -178,7 +179,8 @@
(measure metrics mlabels stats nil)
(log "enqueued" req-id stats limit-id limit-label limit-params nil))
(px/invoke! limiter (fn []
;; WORKAROUND: this is a temporal change until the bug is fixed in funcool/promesa
@(pt/-submit! limiter (fn []
(let [elapsed (tpoint)
stats (pbh/get-stats limiter)]
(measure metrics mlabels stats elapsed)

17
backend/src/app/rpc/commands/files_temp.clj
View file

@ -14,7 +14,6 @@
[app.config :as cf]
[app.db :as db]
[app.db.sql :as sql]
[app.features.components-v2 :as feat.compv2]
[app.features.fdata :as fdata]
[app.loggers.audit :as audit]
[app.rpc :as-alias rpc]
@ -110,7 +109,7 @@
;; --- MUTATION COMMAND: persist-temp-file
(defn persist-temp-file
[{:keys [::db/conn] :as cfg} {:keys [id ::rpc/profile-id] :as params}]
[{:keys [::db/conn] :as cfg} {:keys [id] :as params}]
(let [file (files/get-file cfg id
:migrate? false
:lock-for-update? true)]
@ -119,7 +118,6 @@
(ex/raise :type :validation
:code :cant-persist-already-persisted-file))
(let [changes (->> (db/cursor conn
(sql/select :file-change {:file-id id}
{:order-by [[:revn :asc]]})
@ -147,19 +145,6 @@
:revn 1
:data (blob/encode (:data file))}
{:id id})
(let [team (teams/get-team conn :profile-id profile-id :project-id (:project-id file))
file-features (:features file)
team-features (cfeat/get-team-enabled-features cf/flags team)]
(when (and (contains? team-features "components/v2")
(not (contains? file-features "components/v2")))
;; Migrate components v2
(feat.compv2/migrate-file! cfg
(:id file)
:max-procs 2
:validate? true
:throw-on-validate? true)))
nil)))
(def ^:private schema:persist-temp-file

306
backend/src/app/srepl/components_v2.clj
View file

@ -1,306 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns app.srepl.components-v2
(:require
[app.common.fressian :as fres]
[app.common.logging :as l]
[app.db :as db]
[app.features.components-v2 :as feat]
[app.main :as main]
[app.srepl.helpers :as h]
[app.util.events :as events]
[app.util.time :as dt]
[app.worker :as-alias wrk]
[datoteka.fs :as fs]
[datoteka.io :as io]
[promesa.exec :as px]
[promesa.exec.semaphore :as ps]
[promesa.util :as pu]))
(def ^:dynamic *scope* nil)
(def ^:dynamic *semaphore* nil)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PRIVATE HELPERS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private sql:get-files-by-created-at
"SELECT id, features,
row_number() OVER (ORDER BY created_at DESC) AS rown
FROM file
WHERE deleted_at IS NULL
ORDER BY created_at DESC")
(defn- get-files
[conn]
(->> (db/cursor conn [sql:get-files-by-created-at] {:chunk-size 500})
(map feat/decode-row)
(remove (fn [{:keys [features]}]
(contains? features "components/v2")))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PUBLIC API
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn migrate-file!
[file-id & {:keys [rollback? validate? label cache skip-on-graphic-error?]
:or {rollback? true
validate? false
skip-on-graphic-error? true}}]
(l/dbg :hint "migrate:start" :rollback rollback?)
(let [tpoint (dt/tpoint)
file-id (h/parse-uuid file-id)]
(binding [feat/*stats* (atom {})
feat/*cache* cache]
(try
(-> (assoc main/system ::db/rollback rollback?)
(feat/migrate-file! file-id
:validate? validate?
:skip-on-graphic-error? skip-on-graphic-error?
:label label))
(-> (deref feat/*stats*)
(assoc :elapsed (dt/format-duration (tpoint))))
(catch Throwable cause
(l/wrn :hint "migrate:error" :cause cause))
(finally
(let [elapsed (dt/format-duration (tpoint))]
(l/dbg :hint "migrate:end" :rollback rollback? :elapsed elapsed)))))))
(defn migrate-team!
[team-id & {:keys [rollback? skip-on-graphic-error? validate? label cache]
:or {rollback? true
validate? true
skip-on-graphic-error? true}}]
(l/dbg :hint "migrate:start" :rollback rollback?)
(let [team-id (h/parse-uuid team-id)
stats (atom {})
tpoint (dt/tpoint)]
(binding [feat/*stats* stats
feat/*cache* cache]
(try
(-> (assoc main/system ::db/rollback rollback?)
(feat/migrate-team! team-id
:label label
:validate? validate?
:skip-on-graphics-error? skip-on-graphic-error?))
(-> (deref feat/*stats*)
(assoc :elapsed (dt/format-duration (tpoint))))
(catch Throwable cause
(l/dbg :hint "migrate:error" :cause cause))
(finally
(let [elapsed (dt/format-duration (tpoint))]
(l/dbg :hint "migrate:end" :rollback rollback? :elapsed elapsed)))))))
(defn migrate-files!
"A REPL helper for migrate all files.
This function starts multiple concurrent file migration processes
until thw maximum number of jobs is reached which by default has the
value of `1`. This is controled with the `:max-jobs` option.
If you want to run this on multiple machines you will need to specify
the total number of partitions and the current partition.
In order to get the report table populated, you will need to provide
a correct `:label`. That label is also used for persist a file
snaphot before continue with the migration."
[& {:keys [max-jobs max-items rollback? validate?
cache skip-on-graphic-error?
label partitions current-partition]
:or {validate? false
rollback? true
max-jobs 1
current-partition 1
skip-on-graphic-error? true
max-items Long/MAX_VALUE}}]
(when (int? partitions)
(when-not (int? current-partition)
(throw (IllegalArgumentException. "missing `current-partition` parameter")))
(when-not (<= 0 current-partition partitions)
(throw (IllegalArgumentException. "invalid value on `current-partition` parameter"))))
(let [stats (atom {})
tpoint (dt/tpoint)
factory (px/thread-factory :virtual false :prefix "penpot/migration/")
executor (px/cached-executor :factory factory)
sjobs (ps/create :permits max-jobs)
migrate-file
(fn [file-id rown]
(try
(db/tx-run! (assoc main/system ::db/rollback rollback?)
(fn [system]
(db/exec-one! system ["SET LOCAL idle_in_transaction_session_timeout = 0"])
(feat/migrate-file! system file-id
:rown rown
:label label
:validate? validate?
:skip-on-graphic-error? skip-on-graphic-error?)))
(catch Throwable cause
(l/wrn :hint "unexpected error on processing file (skiping)"
:file-id (str file-id))
(events/tap :error
(ex-info "unexpected error on processing file (skiping)"
{:file-id file-id}
cause))
(swap! stats update :errors (fnil inc 0)))
(finally
(ps/release! sjobs))))
process-file
(fn [{:keys [id rown]}]
(ps/acquire! sjobs)
(px/run! executor (partial migrate-file id rown)))]
(l/dbg :hint "migrate:start"
:label label
:rollback rollback?
:max-jobs max-jobs
:max-items max-items)
(binding [feat/*stats* stats
feat/*cache* cache]
(try
(db/tx-run! main/system
(fn [{:keys [::db/conn] :as system}]
(db/exec! conn ["SET LOCAL statement_timeout = 0"])
(db/exec! conn ["SET LOCAL idle_in_transaction_session_timeout = 0"])
(run! process-file
(->> (get-files conn)
(filter (fn [{:keys [rown] :as row}]
(if (int? partitions)
(= current-partition (inc (mod rown partitions)))
true)))
(take max-items)))
;; Close and await tasks
(pu/close! executor)))
(-> (deref stats)
(assoc :elapsed (dt/format-duration (tpoint))))
(catch Throwable cause
(l/dbg :hint "migrate:error" :cause cause)
(events/tap :error cause))
(finally
(let [elapsed (dt/format-duration (tpoint))]
(l/dbg :hint "migrate:end"
:rollback rollback?
:elapsed elapsed)))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; CACHE POPULATE
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def sql:sobjects-for-cache
"SELECT id,
row_number() OVER (ORDER BY created_at) AS index
FROM storage_object
WHERE (metadata->>'~:bucket' = 'file-media-object' OR
metadata->>'~:bucket' IS NULL)
AND metadata->>'~:content-type' = 'image/svg+xml'
AND deleted_at IS NULL
AND size < 1135899
ORDER BY created_at ASC")
(defn populate-cache!
"A REPL helper for migrate all files.
This function starts multiple concurrent file migration processes
until thw maximum number of jobs is reached which by default has the
value of `1`. This is controled with the `:max-jobs` option.
If you want to run this on multiple machines you will need to specify
the total number of partitions and the current partition.
In order to get the report table populated, you will need to provide
a correct `:label`. That label is also used for persist a file
snaphot before continue with the migration."
[& {:keys [max-jobs] :or {max-jobs 1}}]
(let [tpoint (dt/tpoint)
factory (px/thread-factory :virtual false :prefix "penpot/cache/")
executor (px/cached-executor :factory factory)
sjobs (ps/create :permits max-jobs)
retrieve-sobject
(fn [id index]
(let [path (feat/get-sobject-cache-path id)
parent (fs/parent path)]
(try
(when-not (fs/exists? parent)
(fs/create-dir parent))
(if (fs/exists? path)
(l/inf :hint "create cache entry" :status "exists" :index index :id (str id) :path (str path))
(let [svg-data (feat/get-optimized-svg id)]
(with-open [^java.lang.AutoCloseable stream (io/output-stream path)]
(let [writer (fres/writer stream)]
(fres/write! writer svg-data)))
(l/inf :hint "create cache entry" :status "created"
:index index
:id (str id)
:path (str path))))
(catch Throwable cause
(l/wrn :hint "create cache entry"
:status "error"
:index index
:id (str id)
:path (str path)
:cause cause))
(finally
(ps/release! sjobs)))))
process-sobject
(fn [{:keys [id index]}]
(ps/acquire! sjobs)
(px/run! executor (partial retrieve-sobject id index)))]
(l/dbg :hint "migrate:start"
:max-jobs max-jobs)
(try
(binding [feat/*system* main/system]
(run! process-sobject
(db/exec! main/system [sql:sobjects-for-cache]))
;; Close and await tasks
(pu/close! executor))
{:elapsed (dt/format-duration (tpoint))}
(catch Throwable cause
(l/dbg :hint "populate:error" :cause cause))
(finally
(let [elapsed (dt/format-duration (tpoint))]
(l/dbg :hint "populate:end"
:elapsed elapsed))))))

31
backend/src/app/srepl/helpers.clj
View file

@ -13,7 +13,6 @@
[app.common.files.migrations :as fmg]
[app.common.files.validate :as cfv]
[app.db :as db]
[app.features.components-v2 :as feat.comp-v2]
[app.main :as main]
[app.rpc.commands.files :as files]
[app.rpc.commands.files-snapshot :as fsnap]
@ -62,6 +61,27 @@
{:id id})
team))
(def ^:private sql:get-and-lock-team-files
"SELECT f.id
FROM file AS f
JOIN project AS p ON (p.id = f.project_id)
WHERE p.team_id = ?
AND p.deleted_at IS NULL
AND f.deleted_at IS NULL
FOR UPDATE")
(defn get-team
[conn team-id]
(-> (db/get conn :team {:id team-id}
{::db/remove-deleted false
::db/check-deleted false})
(update :features db/decode-pgarray #{})))
(defn get-and-lock-team-files
[conn team-id]
(transduce (map :id) conj []
(db/plan conn [sql:get-and-lock-team-files team-id])))
(defn reset-file-data!
"Hardcode replace of the data of one file."
[system id data]
@ -96,7 +116,7 @@
(defn take-team-snapshot!
[system team-id label]
(let [conn (db/get-connection system)]
(->> (feat.comp-v2/get-and-lock-team-files conn team-id)
(->> (get-and-lock-team-files conn team-id)
(reduce (fn [result file-id]
(let [file (fsnap/get-file-snapshots system file-id)]
(fsnap/create-file-snapshot! system file
@ -108,19 +128,16 @@
(defn restore-team-snapshot!
[system team-id label]
(let [conn (db/get-connection system)
ids (->> (feat.comp-v2/get-and-lock-team-files conn team-id)
ids (->> (get-and-lock-team-files conn team-id)
(into #{}))
snap (search-file-snapshots conn ids label)
ids' (into #{} (map :file-id) snap)
team (-> (feat.comp-v2/get-team conn team-id)
(update :features disj "components/v2"))]
ids' (into #{} (map :file-id) snap)]
(when (not= ids ids')
(throw (RuntimeException. "no uniform snapshot available")))
(feat.comp-v2/update-team! conn team)
(reduce (fn [result {:keys [file-id id]}]
(fsnap/restore-file-snapshot! system file-id id)
(inc result))

3
backend/src/app/srepl/main.clj
View file

@ -22,7 +22,6 @@
[app.config :as cf]
[app.db :as db]
[app.db.sql :as-alias sql]
[app.features.components-v2 :as feat.comp-v2]
[app.features.fdata :as feat.fdata]
[app.loggers.audit :as audit]
[app.main :as main]
@ -439,7 +438,7 @@
(binding [h/*system* system
db/*conn* (db/get-connection system)]
(->> (feat.comp-v2/get-and-lock-team-files conn team-id)
(->> (h/get-and-lock-team-files conn team-id)
(reduce (fn [result file-id]
(if (h/process-file! system file-id update-fn opts)
(inc result)

1
backend/test/backend_tests/rpc_file_test.clj
View file

@ -1712,6 +1712,7 @@
[{:fill-image
{:id (:id fmedia)
:name "test"
:mtype "image/jpeg"
:width 200
:height 200}}]]

5
common/deps.edn
View file

@ -30,7 +30,7 @@
funcool/tubax {:mvn/version "2021.05.20-0"}
funcool/cuerdas {:mvn/version "2023.11.09-407"}
funcool/promesa
{:git/sha "0c5ed6ad033515a2df4b55addea044f60e9653d0"
{:git/sha "6c14b06d9d64fae6e43c1463ce313f2fdc0d989b"
:git/url "https://github.com/funcool/promesa"}
funcool/datoteka
@ -76,7 +76,8 @@
:extra-deps {lambdaisland/kaocha {:mvn/version "1.91.1392"}}}
:shadow-cljs
{:main-opts ["-m" "shadow.cljs.devtools.cli"]}
{:main-opts ["-m" "shadow.cljs.devtools.cli"]
:jvm-opts ["--sun-misc-unsafe-memory-access=allow"]}
:outdated
{:extra-deps {com.github.liquidz/antq {:mvn/version "RELEASE"}}

3
common/package.json
View file

@ -11,8 +11,7 @@
"url": "https://github.com/penpot/penpot"
},
"dependencies": {
"luxon": "^3.4.4",
"sax": "^1.4.1"
"luxon": "^3.4.4"
},
"devDependencies": {
"concurrently": "^9.0.1",

408
common/src/app/common/files/builder.cljc
View file

@ -13,17 +13,12 @@
[app.common.features :as cfeat]
[app.common.files.changes :as ch]
[app.common.files.migrations :as fmig]
[app.common.geom.point :as gpt]
[app.common.geom.shapes :as gsh]
[app.common.schema :as sm]
[app.common.svg :as csvg]
[app.common.types.color :as types.color]
[app.common.types.component :as types.component]
[app.common.types.components-list :as types.components-list]
[app.common.types.container :as types.container]
[app.common.types.file :as types.file]
[app.common.types.page :as types.page]
[app.common.types.pages-list :as types.pages-list]
[app.common.types.shape :as types.shape]
[app.common.types.typography :as types.typography]
[app.common.uuid :as uuid]
@ -37,41 +32,36 @@
(def ^:private conjv (fnil conj []))
(def ^:private conjs (fnil conj #{}))
(defn default-uuid
(defn- default-uuid
[v]
(or v (uuid/next)))
(defn- track-used-name
[file name]
(let [container-id (::current-page-id file)]
(update-in file [::unames container-id] conjs name)))
[state name]
(let [container-id (::current-page-id state)]
(update-in state [::unames container-id] conjs name)))
(defn- commit-change
[file change & {:keys [add-container]
:or {add-container false}}]
[state change & {:keys [add-container]}]
(let [file-id (get state ::current-file-id)]
(assert (uuid? file-id) "no current file id")
(let [change (cond-> change
add-container
(assoc :page-id (::current-page-id file)
:frame-id (::current-frame-id file)))]
(-> file
(update ::changes conjv change)
(update :data ch/process-changes [change] false))))
(defn- lookup-objects
[file]
(dm/get-in file [:data :pages-index (::current-page-id file) :objects]))
(assoc :page-id (::current-page-id state)
:frame-id (::current-frame-id state)))]
(update-in state [::files file-id :data] ch/process-changes [change] false))))
(defn- commit-shape
[file shape]
[state shape]
(let [parent-id
(-> file ::parent-stack peek)
(-> state ::parent-stack peek)
frame-id
(::current-frame-id file)
(get state ::current-frame-id)
page-id
(::current-page-id file)
(get state ::current-page-id)
change
{:type :add-obj
@ -82,39 +72,31 @@
:frame-id frame-id
:page-id page-id}]
(-> file
(-> state
(commit-change change)
(track-used-name (:name shape)))))
(defn- generate-name
[type data]
(if (= type :svg-raw)
(let [tag (dm/get-in data [:content :tag])]
(str "svg-" (cond (string? tag) tag
(keyword? tag) (d/name tag)
(nil? tag) "node"
:else (str tag))))
(str/capital (d/name type))))
(defn- unique-name
[name file]
(let [container-id (::current-page-id file)
unames (dm/get-in file [:unames container-id])]
[name state]
(let [container-id (::current-page-id state)
unames (dm/get-in state [:unames container-id])]
(d/unique-name name (or unames #{}))))
(defn- clear-names [file]
(dissoc file ::unames))
(defn- assign-name
(defn- assign-shape-name
"Given a tag returns its layer name"
[data file type]
(cond-> data
(nil? (:name data))
(assoc :name (generate-name type data))
[shape state]
(cond-> shape
(nil? (:name shape))
(assoc :name (let [type (get shape :type)]
(case type
:frame "Board"
(str/capital (d/name type)))))
:always
(update :name unique-name file)))
(update :name unique-name state)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SCHEMAS
@ -135,17 +117,32 @@
(def decode-library-typography
(sm/decode-fn types.typography/schema:typography sm/json-transformer))
(def decode-component
(sm/decode-fn types.component/schema:component sm/json-transformer))
(def schema:add-component
[:map
[:component-id ::sm/uuid]
[:file-id {:optional true} ::sm/uuid]
[:name {:optional true} ::sm/text]
[:path {:optional true} ::sm/text]
[:frame-id {:optional true} ::sm/uuid]
[:page-id {:optional true} ::sm/uuid]])
(def ^:private check-add-component
(sm/check-fn schema:add-component
:hint "invalid arguments passed for add-component"))
(def decode-add-component
(sm/decode-fn schema:add-component sm/json-transformer))
(def schema:add-component-instance
[:map
[:component-id ::sm/uuid]
[:x ::sm/safe-number]
[:y ::sm/safe-number]])
[:file-id {:optional true} ::sm/uuid]
[:frame-id {:optional true} ::sm/uuid]
[:page-id {:optional true} ::sm/uuid]])
(def check-add-component-instance
(sm/check-fn schema:add-component-instance))
(def ^:private check-add-component-instance
(sm/check-fn schema:add-component-instance
:hint "invalid arguments passed for add-component-instance"))
(def decode-add-component-instance
(sm/decode-fn schema:add-component-instance sm/json-transformer))
@ -158,37 +155,77 @@
(def decode-add-bool
(sm/decode-fn schema:add-bool sm/json-transformer))
(def check-add-bool
(def ^:private check-add-bool
(sm/check-fn schema:add-bool))
(def schema:add-file-media
[:map
[:id {:optional true} ::sm/uuid]
[:name ::sm/text]
[:width ::sm/int]
[:height ::sm/int]])
(def decode-add-file-media
(sm/decode-fn schema:add-file-media sm/json-transformer))
(def check-add-file-media
(sm/check-fn schema:add-file-media))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PUBLIC API
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn lookup-shape [file shape-id]
(-> (lookup-objects file)
(get shape-id)))
(defn create-state
[]
{})
(defn get-current-page
[file]
(let [page-id (::current-page-id file)]
(dm/get-in file [:data :pages-index page-id])))
[state]
(let [file-id (get state ::current-file-id)
page-id (get state ::current-page-id)]
(defn create-file
[params]
(assert (uuid? file-id) "expected current-file-id to be assigned")
(assert (uuid? page-id) "expected current-page-id to be assigned")
(dm/get-in state [::files file-id :data :pages-index page-id])))
(defn get-current-objects
[state]
(-> (get-current-page state)
(get :objects)))
(defn get-shape
[state shape-id]
(-> (get-current-objects state)
(get shape-id)))
(defn add-file
[state params]
(let [params (-> params
(assoc :features cfeat/default-features)
(assoc :migrations fmig/available-migrations))]
(types.file/make-file params :create-page false)))
(assoc :migrations fmig/available-migrations)
(update :id default-uuid))
file (types.file/make-file params :create-page false)]
(-> state
(update ::files assoc (:id file) file)
(assoc ::current-file-id (:id file)))))
(declare close-page)
(defn close-file
[state]
(let [state (-> state
(close-page)
(dissoc ::current-file-id))]
state))
(defn add-page
[file params]
[state params]
(let [page (-> (types.page/make-empty-page params)
(types.page/check-page))
change {:type :add-page
:page page}]
(-> file
(-> state
(commit-change change)
;; Current page being edited
@ -203,62 +240,62 @@
;; Last object id added
(assoc ::last-id nil))))
(defn close-page [file]
(-> file
(defn close-page [state]
(-> state
(dissoc ::current-page-id)
(dissoc ::parent-stack)
(dissoc ::last-id)
(clear-names)))
(defn add-artboard
[file data]
(defn add-board
[state params]
(let [{:keys [id] :as shape}
(-> data
(-> params
(update :id default-uuid)
(assoc :type :frame)
(assign-name file :frame)
(assign-shape-name state)
(types.shape/setup-shape)
(types.shape/check-shape))]
(-> file
(-> state
(commit-shape shape)
(update ::parent-stack conjv id)
(assoc ::current-frame-id id)
(assoc ::last-id id))))
(defn close-artboard
[file]
(let [parent-id (-> file ::parent-stack peek)
parent (lookup-shape file parent-id)]
(-> file
(defn close-board
[state]
(let [parent-id (-> state ::parent-stack peek)
parent (get-shape state parent-id)]
(-> state
(assoc ::current-frame-id (or (:frame-id parent) root-id))
(update ::parent-stack pop))))
(defn add-group
[file params]
[state params]
(let [{:keys [id] :as shape}
(-> params
(update :id default-uuid)
(assoc :type :group)
(assign-name file :group)
(assign-shape-name state)
(types.shape/setup-shape)
(types.shape/check-shape))]
(-> file
(-> state
(commit-shape shape)
(assoc ::last-id id)
(update ::parent-stack conjv id))))
(defn close-group
[file]
(let [group-id (-> file :parent-stack peek)
group (lookup-shape file group-id)
[state]
(let [group-id (-> state :parent-stack peek)
group (get-shape state group-id)
children (->> (get group :shapes)
(into [] (keep (partial lookup-shape file)))
(into [] (keep (partial get-shape state)))
(not-empty))]
(assert (some? children) "group expect to have at least 1 children")
(let [file (if (:masked-group group)
(let [state (if (:masked-group group)
(let [mask (first children)
change {:type :mod-obj
:id group-id
@ -271,7 +308,7 @@
{:type :set :attr :flip-y :val (-> mask :flip-y) :ignore-touched true}
{:type :set :attr :selrect :val (-> mask :selrect) :ignore-touched true}
{:type :set :attr :points :val (-> mask :points) :ignore-touched true}]}]
(commit-change file change :add-container true))
(commit-change state change :add-container true))
(let [group (gsh/update-group-selrect group children)
change {:type :mod-obj
:id group-id
@ -283,16 +320,16 @@
{:type :set :attr :width :val (-> group :selrect :width) :ignore-touched true}
{:type :set :attr :height :val (-> group :selrect :height) :ignore-touched true}]}]
(commit-change file change :add-container true)))]
(update file ::parent-stack pop))))
(commit-change state change :add-container true)))]
(update state ::parent-stack pop))))
(defn add-bool
[file params]
[state params]
(let [{:keys [group-id type]}
(check-add-bool params)
group
(lookup-shape file group-id)
(get-shape state group-id)
children
(->> (get group :shapes)
@ -300,7 +337,7 @@
(assert (some? children) "expect group to have at least 1 element")
(let [objects (lookup-objects file)
(let [objects (get-current-objects state)
bool (-> group
(assoc :type :bool)
(gsh/update-bool objects))
@ -317,101 +354,110 @@
{:type :set :attr :width :val (-> bool :selrect :width) :ignore-touched true}
{:type :set :attr :height :val (-> bool :selrect :height) :ignore-touched true}]}]
(-> file
(-> state
(commit-change change :add-container true)
(assoc ::last-id group-id)))))
(defn add-shape
[file params]
[state params]
(let [obj (-> params
(d/update-when :svg-attrs csvg/attrs->props)
(types.shape/setup-shape)
(assign-name file :type))]
(-> file
(assign-shape-name state))]
(-> state
(commit-shape obj)
(assoc ::last-id (:id obj)))))
(defn add-library-color
[file color]
[state color]
(let [color (-> color
(update :opacity d/nilv 1)
(update :id default-uuid)
(types.color/check-library-color color))
change {:type :add-color
:color color}]
(-> file
(-> state
(commit-change change)
(assoc ::last-id (:id color)))))
(defn add-library-typography
[file typography]
[state typography]
(let [typography (-> typography
(update :id default-uuid)
(d/without-nils))
change {:type :add-typography
:id (:id typography)
:typography typography}]
(-> file
(-> state
(commit-change change)
(assoc ::last-id (:id typography)))))
(defn add-component
[file params]
(let [change1 {:type :add-component
:id (or (:id params) (uuid/next))
:name (:name params)
:path (:path params)
:main-instance-id (:main-instance-id params)
:main-instance-page (:main-instance-page params)}
[state params]
(let [{:keys [component-id file-id page-id frame-id name path]}
(-> (check-add-component params)
(update :component-id default-uuid))
comp-id (get change1 :id)
change2 {:type :mod-obj
:id (:main-instance-id params)
:operations
[{:type :set :attr :component-root :val true}
{:type :set :attr :component-id :val comp-id}
{:type :set :attr :component-file :val (:id file)}]}]
(-> file
(commit-change change1)
(commit-change change2)
(assoc ::last-id comp-id)
(assoc ::current-frame-id comp-id))))
(defn add-component-instance
[{:keys [id data] :as file} params]
(let [{:keys [component-id x y]}
(check-add-component-instance params)
component
(types.components-list/get-component data component-id)
file-id
(or file-id (::current-file-id state))
page-id
(get file ::current-page-id)]
(or page-id (get state ::current-page-id))
(assert (uuid? page-id) "page-id is expected to be set")
(assert (uuid? component) "component is expected to exist")
frame-id
(or frame-id (get state ::current-frame-id))
;; FIXME: this should be on files and not in pages-list
(let [page (types.pages-list/get-page (:data file) page-id)
pos (gpt/point x y)
change1
(d/without-nils
{:type :add-component
:id component-id
:name (or name "anonmous")
:path path
:main-instance-id frame-id
:main-instance-page page-id})
[shape shapes]
(types.container/make-component-instance page component id pos)
change2
{:type :mod-obj
:id frame-id
:page-id page-id
:operations
[{:type :set :attr :component-root :val true}
{:type :set :attr :main-instance :val true}
{:type :set :attr :component-id :val component-id}
{:type :set :attr :component-file :val file-id}]}]
file
(reduce #(commit-change %1
{:type :add-obj
:id (:id %2)
:page-id (:id page)
:parent-id (:parent-id %2)
:frame-id (:frame-id %2)
:ignore-touched true
:obj %2})
file
shapes)]
(-> state
(commit-change change1)
(commit-change change2))))
(assoc file ::last-id (:id shape)))))
(defn add-component-instance
[state params]
(let [{:keys [component-id file-id frame-id page-id]}
(check-add-component-instance params)
file-id
(or file-id (get state ::current-file-id))
frame-id
(or frame-id (get state ::current-frame-id))
page-id
(or page-id (get state ::current-page-id))
change
{:type :mod-obj
:id frame-id
:page-id page-id
:operations
[{:type :set :attr :component-root :val true}
{:type :set :attr :component-id :val component-id}
{:type :set :attr :component-file :val file-id}]}]
(commit-change state change)))
(defn delete-shape
[file id]
@ -423,10 +469,12 @@
:id id}))
(defn update-shape
[file shape-id f]
(let [page-id (::current-page-id file)
objects (lookup-objects file)
[state shape-id f]
(let [page-id (get state ::current-page-id)
objects (get-current-objects state)
old-shape (get objects shape-id)
new-shape (f old-shape)
attrs (d/concat-set
(keys old-shape)
@ -440,7 +488,7 @@
changes
(conj changes {:type :set :attr attr :val new-val :ignore-touched true}))))]
(-> file
(-> state
(commit-change
{:type :mod-obj
:operations (reduce generate-operation [] attrs)
@ -449,12 +497,12 @@
(assoc ::last-id shape-id))))
(defn add-guide
[file guide]
[state guide]
(let [guide (cond-> guide
(nil? (:id guide))
(assoc :id (uuid/next)))
page-id (::current-page-id file)]
(-> file
(update :id default-uuid))
page-id (::current-page-id state)]
(-> state
(commit-change
{:type :set-guide
:page-id page-id
@ -463,24 +511,54 @@
(assoc ::last-id (:id guide)))))
(defn delete-guide
[file id]
(let [page-id (::current-page-id file)]
(commit-change file
[state id]
(let [page-id (::current-page-id state)]
(commit-change state
{:type :set-guide
:page-id page-id
:id id
:params nil})))
(defn update-guide
[file guide]
(let [page-id (::current-page-id file)]
(commit-change file
[state guide]
(let [page-id (::current-page-id state)]
(commit-change state
{:type :set-guide
:page-id page-id
:id (:id guide)
:params guide})))
(defn strip-image-extension [filename]
(let [image-extensions-re #"(\.png)|(\.jpg)|(\.jpeg)|(\.webp)|(\.gif)|(\.svg)$"]
(str/replace filename image-extensions-re "")))
(defrecord BlobWrapper [mtype size blob])
(defn add-file-media
[state params blob]
(assert (instance? BlobWrapper blob) "expect blob to be wrapped")
(let [media-id
(uuid/next)
file-id
(get state ::current-file-id)
{:keys [id width height name]}
(-> params
(update :id default-uuid)
(check-add-file-media params))]
(-> state
(update ::blobs assoc media-id blob)
(update ::media assoc media-id
{:id media-id
:bucket "file-media-object"
:content-type (get blob :mtype)
:size (get blob :size)})
(update ::file-media assoc id
{:id id
:name name
:width width
:height height
:file-id file-id
:media-id media-id
:mtype (get blob :mtype)})
(assoc ::last-id id))))

4
common/src/app/common/files/changes.cljc
View file

@ -310,12 +310,12 @@
[:add-media
[:map {:title "AddMediaChange"}
[:type [:= :add-media]]
[:object ::ctf/media-object]]]
[:object ctf/schema:media]]]
[:mod-media
[:map {:title "ModMediaChange"}
[:type [:= :mod-media]]
[:object ::ctf/media-object]]]
[:object ctf/schema:media]]]
[:del-media
[:map {:title "DelMediaChange"}

30
common/src/app/common/media.cljc
View file

@ -5,8 +5,8 @@
;; Copyright (c) KALEIDOS INC
(ns app.common.media
"Media assets helpers (images, fonts, etc)"
(:require
[clojure.spec.alpha :as s]
[cuerdas.core :as str]))
;; We have added ".ttf" as string to solve a problem with chrome input selector
@ -59,27 +59,17 @@
"application/penpot" ".penpot"
"application/pdf" ".pdf"
"text/plain" ".txt"
"font/woff" ".woff"
"font/woff2" ".woff2"
"font/ttf" ".ttf"
"font/otf" ".otf"
"application/octet-stream" ".bin"
nil))
(s/def ::id uuid?)
(s/def ::name string?)
(s/def ::width number?)
(s/def ::height number?)
(s/def ::created-at inst?)
(s/def ::modified-at inst?)
(s/def ::mtype string?)
(s/def ::uri string?)
(s/def ::media-object
(s/keys :req-un [::id
::name
::width
::height
::mtype
::created-at
::modified-at
::uri]))
(defn strip-image-extension
[filename]
(let [image-extensions-re #"(\.png)|(\.jpg)|(\.jpeg)|(\.webp)|(\.gif)|(\.svg)$"]
(str/replace filename image-extensions-re "")))
(defn parse-font-weight
[variant]

33
common/src/app/common/svg.cljc
View file

@ -6,8 +6,6 @@
(ns app.common.svg
(:require
#?(:clj [clojure.xml :as xml]
:cljs [tubax.core :as tubax])
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.geom.matrix :as gmt]
@ -15,15 +13,7 @@
[app.common.geom.shapes :as gsh]
[app.common.math :as mth]
[app.common.uuid :as uuid]
[cuerdas.core :as str])
#?(:clj
(:import
clojure.lang.XMLHandler
java.io.InputStream
javax.xml.XMLConstants
javax.xml.parsers.SAXParserFactory
org.apache.commons.io.IOUtils)))
[cuerdas.core :as str]))
;; Regex for XML ids per Spec
;; https://www.w3.org/TR/2008/REC-xml-20081126/#sec-common-syn
@ -1030,24 +1020,3 @@
:height (d/parse-integer (:height attrs) 0)})))]
(reduce-nodes redfn [] svg-data)))
#?(:clj
(defn- secure-parser-factory
[^InputStream input ^XMLHandler handler]
(.. (doto (SAXParserFactory/newInstance)
(.setFeature XMLConstants/FEATURE_SECURE_PROCESSING true)
(.setFeature "http://apache.org/xml/features/disallow-doctype-decl" true))
(newSAXParser)
(parse input handler))))
(defn strip-doctype
[data]
(cond-> data
(str/includes? data "<!DOCTYPE")
(str/replace #"<\!DOCTYPE[^>]*>" "")))
(defn parse
[text]
#?(:cljs (tubax/xml->clj text)
:clj (let [text (strip-doctype text)]
(dm/with-open [istream (IOUtils/toInputStream text "UTF-8")]
(xml/parse istream secure-parser-factory)))))

13
common/src/app/common/types/color.cljc
View file

@ -54,13 +54,13 @@
::oapi/type "integer"
::oapi/format "int64"}}))
(def schema:image-color
(def schema:image
[:map {:title "ImageColor"}
[:name {:optional true} :string]
[:width ::sm/int]
[:height ::sm/int]
[:mtype {:optional true} [:maybe :string]]
[:mtype ::sm/text]
[:id ::sm/uuid]
[:name {:optional true} ::sm/text]
[:keep-aspect-ratio {:optional true} :boolean]])
(def gradient-types
@ -93,7 +93,7 @@
[:ref-id {:optional true} ::sm/uuid]
[:ref-file {:optional true} ::sm/uuid]
[:gradient {:optional true} [:maybe schema:gradient]]
[:image {:optional true} [:maybe schema:image-color]]
[:image {:optional true} [:maybe schema:image]]
[:plugin-data {:optional true} ::ctpg/plugin-data]])
(def schema:color
@ -106,7 +106,7 @@
[:opacity {:optional true} [:maybe ::sm/safe-number]]
[:color {:optional true} [:maybe schema:rgb-color]]
[:gradient {:optional true} [:maybe schema:gradient]]
[:image {:optional true} [:maybe schema:image-color]]]
[:image {:optional true} [:maybe schema:image]]]
[::sm/contains-any {:strict true} [:color :gradient :image]]])
;; Same as color but with :id prop required
@ -115,9 +115,10 @@
(sm/required-keys schema:color-attrs [:id])
[::sm/contains-any {:strict true} [:color :gradient :image]]])
;; FIXME: revisit if we really need this all registers
(sm/register! ::color schema:color)
(sm/register! ::gradient schema:gradient)
(sm/register! ::image-color schema:image-color)
(sm/register! ::image-color schema:image)
(sm/register! ::recent-color schema:recent-color)
(sm/register! ::color-attrs schema:color-attrs)

11
common/src/app/common/types/file.cljc
View file

@ -38,18 +38,18 @@
(def schema:media
"A schema that represents the file media object"
[:map {:title "FileMediaObject"}
[:map {:title "FileMedia"}
[:id ::sm/uuid]
[:created-at ::sm/inst]
[:created-at {:optional true} ::sm/inst]
[:deleted-at {:optional true} ::sm/inst]
[:name :string]
[:width ::sm/safe-int]
[:height ::sm/safe-int]
[:mtype :string]
[:file-id {:optional true} ::sm/uuid]
[:media-id ::sm/uuid]
[:file-id {:optional true} ::sm/uuid]
[:thumbnail-id {:optional true} ::sm/uuid]
[:is-local :boolean]])
[:is-local {:optional true} :boolean]])
(def schema:colors
[:map-of {:gen/max 5} ::sm/uuid ::ctc/color])
@ -102,7 +102,6 @@
(sm/register! ::media schema:media)
(sm/register! ::colors schema:colors)
(sm/register! ::typographies schema:typographies)
(sm/register! ::media-object schema:media)
(def check-file
(sm/check-fn schema:file :hint "check error on validating file"))
@ -110,7 +109,7 @@
(def check-file-data
(sm/check-fn schema:data))
(def check-media-object
(def check-file-media
(sm/check-fn schema:media))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

7
common/src/app/common/types/tokens_lib.cljc
View file

@ -1506,12 +1506,13 @@ Will return a value that matches this schema:
(-> (make-tokens-lib)
(decode-dtcg-json encoded-json)))
(def type:tokens-lib
(def schema:tokens-lib
(sm/register!
{:type ::tokens-lib
:pred valid-tokens-lib?
:type-properties
{:encode/json encode-dtcg
:decode/json decode-dtcg}})
:decode/json decode-dtcg}}))
(defn duplicate-set [set-name lib & {:keys [suffix]}]
(let [sets (get-sets lib)
@ -1521,8 +1522,6 @@ Will return a value that matches this schema:
(assoc :name copy-name)
(assoc :modified-at (dt/now)))))
(sm/register! type:tokens-lib)
;; === Serialization handlers for RPC API (transit) and database (fressian)
(t/add-handlers!

26
common/test/common_tests/files_builder_test.cljc
View file

@ -1,26 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns common-tests.files-builder-test
(:require
[app.common.files.builder :as builder]
[clojure.test :as t]))
(t/deftest test-strip-image-extension
(t/testing "removes extension from supported image files"
(t/is (= (builder/strip-image-extension "foo.png") "foo"))
(t/is (= (builder/strip-image-extension "foo.webp") "foo"))
(t/is (= (builder/strip-image-extension "foo.jpg") "foo"))
(t/is (= (builder/strip-image-extension "foo.jpeg") "foo"))
(t/is (= (builder/strip-image-extension "foo.svg") "foo"))
(t/is (= (builder/strip-image-extension "foo.gif") "foo")))
(t/testing "does not remove extension for unsupported files"
(t/is (= (builder/strip-image-extension "foo.txt") "foo.txt"))
(t/is (= (builder/strip-image-extension "foo.bmp") "foo.bmp")))
(t/testing "leaves filename intact when it has no extension"
(t/is (= (builder/strip-image-extension "README") "README"))))

26
common/test/common_tests/media_test.cljc Normal file
View file

@ -0,0 +1,26 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns common-tests.media-test
(:require
[app.common.media :as media]
[clojure.test :as t]))
(t/deftest test-strip-image-extension
(t/testing "removes extension from supported image files"
(t/is (= (media/strip-image-extension "foo.png") "foo"))
(t/is (= (media/strip-image-extension "foo.webp") "foo"))
(t/is (= (media/strip-image-extension "foo.jpg") "foo"))
(t/is (= (media/strip-image-extension "foo.jpeg") "foo"))
(t/is (= (media/strip-image-extension "foo.svg") "foo"))
(t/is (= (media/strip-image-extension "foo.gif") "foo")))
(t/testing "does not remove extension for unsupported files"
(t/is (= (media/strip-image-extension "foo.txt") "foo.txt"))
(t/is (= (media/strip-image-extension "foo.bmp") "foo.bmp")))
(t/testing "leaves filename intact when it has no extension"
(t/is (= (media/strip-image-extension "README") "README"))))

14
common/test/common_tests/runner.cljc
View file

@ -9,7 +9,6 @@
[clojure.test :as t]
[common-tests.colors-test]
[common-tests.data-test]
[common-tests.files-builder-test]
[common-tests.files-changes-test]
[common-tests.files-migrations-test]
[common-tests.geom-point-test]
@ -29,6 +28,7 @@
[common-tests.logic.swap-and-reset-test]
[common-tests.logic.swap-as-override-test]
[common-tests.logic.token-test]
[common-tests.media-test]
[common-tests.pages-helpers-test]
[common-tests.record-test]
[common-tests.schema-test]
@ -58,7 +58,6 @@
(t/run-tests
'common-tests.colors-test
'common-tests.data-test
'common-tests.files-builder-test
'common-tests.files-changes-test
'common-tests.files-migrations-test
'common-tests.geom-point-test
@ -78,6 +77,7 @@
'common-tests.logic.swap-and-reset-test
'common-tests.logic.swap-as-override-test
'common-tests.logic.token-test
'common-tests.media-test
'common-tests.pages-helpers-test
'common-tests.record-test
'common-tests.schema-test
@ -85,11 +85,11 @@
'common-tests.svg-test
'common-tests.text-test
'common-tests.time-test
'common-tests.types.modifiers-test
'common-tests.types.shape-interactions-test
'common-tests.types.shape-decode-encode-test
'common-tests.types.tokens-lib-test
'common-tests.types.components-test
'common-tests.types.absorb-assets-test
'common-tests.types.components-test
'common-tests.types.modifiers-test
'common-tests.types.path-data-test
'common-tests.types.shape-decode-encode-test
'common-tests.types.shape-interactions-test
'common-tests.types.tokens-lib-test
'common-tests.uuid-test))

4
common/vendor/beicon/impl/rxjs.cljs vendored
View file

@ -1,4 +0,0 @@
(ns beicon.impl.rxjs
(:require ["rxjs" :as rx]))
(goog/exportSymbol "rxjsMain" rx)

4
common/vendor/beicon/impl/rxjs_operators.cljs vendored
View file

@ -1,4 +0,0 @@
(ns beicon.impl.rxjs-operators
(:require ["rxjs/operators" :as rxop]))
(goog/exportSymbol "rxjsOperators" rxop)

4
common/vendor/tubax/saxjs.cljs vendored
View file

@ -1,4 +0,0 @@
(ns tubax.saxjs
(:require ["sax" :as sax]))
(goog/exportSymbol "sax" sax)

8
common/yarn.lock
View file

@ -260,7 +260,6 @@ __metadata:
concurrently: "npm:^9.0.1"
luxon: "npm:^3.4.4"
nodemon: "npm:^3.1.7"
sax: "npm:^1.4.1"
shadow-cljs: "npm:3.0.5"
source-map-support: "npm:^0.5.21"
ws: "npm:^8.17.0"
@ -981,13 +980,6 @@ __metadata:
languageName: node
linkType: hard
"sax@npm:^1.4.1":
version: 1.4.1
resolution: "sax@npm:1.4.1"
checksum: 10c0/6bf86318a254c5d898ede6bd3ded15daf68ae08a5495a2739564eb265cd13bcc64a07ab466fb204f67ce472bb534eb8612dac587435515169593f4fffa11de7c
languageName: node
linkType: hard
"semver@npm:^7.3.5":
version: 7.6.2
resolution: "semver@npm:7.6.2"

18
frontend/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch Normal file
View file

@ -0,0 +1,18 @@
diff --git a/lib/zip-fs.js b/lib/zip-fs.js
index 1444c0f00e5f1ad6c13521f90a7f3c6659d81116..90e38baef5365c2abbcb9337f7ab37f800e883a4 100644
--- a/lib/zip-fs.js
+++ b/lib/zip-fs.js
@@ -33,12 +33,7 @@ import { initShimAsyncCodec } from "./core/util/stream-codec-shim.js";
import { terminateWorkers } from "./core/codec-pool.js";
let baseURL;
-try {
- baseURL = import.meta.url;
- // eslint-disable-next-line no-unused-vars
-} catch (_) {
- // ignored
-}
+
configure({ baseURL });
configureWebWorker(configure);

5
frontend/package.json
View file

@ -14,6 +14,7 @@
"url": "https://github.com/penpot/penpot"
},
"resolutions": {
"@zip.js/zip.js@npm:^2.7.44": "patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch",
"@vitejs/plugin-react": "^4.2.0"
},
"scripts": {
@ -25,7 +26,6 @@
"build:app:libs": "node ./scripts/build-libs.js",
"build:app:main": "clojure -M:dev:shadow-cljs release main worker",
"build:app": "yarn run clear:shadow-cache && yarn run build:app:main && yarn run build:app:libs",
"build:library": "yarn run clear:shadow-cache && clojure -M:dev:shadow-cljs release library",
"e2e:server": "node ./scripts/e2e-server.js",
"fmt:clj": "cljfmt fix --parallel=true src/ test/",
"fmt:clj:check": "cljfmt check --parallel=false src/ test/",
@ -45,7 +45,6 @@
"watch:app:main": "clojure -M:dev:shadow-cljs watch main worker storybook",
"clear:shadow-cache": "rm -rf .shadow-cljs",
"watch:app": "yarn run clear:shadow-cache && concurrently \"yarn run watch:app:main\" \"yarn run watch:app:libs\"",
"watch:library": "yarn run clear:shadow-cache && clojure -M:dev:shadow-cljs watch library",
"watch": "yarn run watch:app:assets",
"watch:storybook": "concurrently \"storybook dev -p 6006 --no-open\" \"yarn run watch:storybook:assets\"",
"watch:storybook:assets": "node ./scripts/watch-storybook.js"
@ -108,11 +107,11 @@
"@penpot/svgo": "penpot/svgo#v3.1",
"@penpot/text-editor": "portal:./text-editor",
"@tokens-studio/sd-transforms": "1.2.11",
"@zip.js/zip.js": "patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch",
"compression": "^1.7.5",
"date-fns": "^4.1.0",
"eventsource-parser": "^3.0.1",
"js-beautify": "^1.15.4",
"jszip": "^3.10.1",
"lodash": "^4.17.21",
"lodash.debounce": "^4.0.8",
"luxon": "^3.6.1",

26
frontend/shadow-cljs.edn
View file

@ -149,32 +149,6 @@
{:test {:init-fn frontend-tests.runner/init
:prepend-js ";if (typeof globalThis.navigator?.userAgent === 'undefined') { globalThis.navigator = {userAgent: ''}; };"}}}
:library
{:target :esm
:runtime :custom
:output-dir "target/library"
:devtools {:autoload false}
:modules
{:penpot
{:exports {BuilderError lib.file-builder/BuilderError
createFile lib.file-builder/create-file}}}
:compiler-options
{:output-feature-set :es2020
:output-wrapper false
:warnings {:fn-deprecated false}}
:release
{:compiler-options
{:fn-invoke-direct true
:optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :advanced]
:pretty-print false
:source-map true
:elide-asserts true
:anon-fn-naming-policy :off
:source-map-detail-level :all}}}
:bench
{:target :node-script
:output-to "target/bench.js"

17
frontend/src/app/main/data/exports/files.cljs
View file

@ -8,7 +8,6 @@
"The file exportation API and events"
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.schema :as sm]
[app.main.data.event :as ev]
[app.main.data.modal :as modal]
@ -35,23 +34,18 @@
(defn export-files
[files format]
(dm/assert!
"expected valid files param"
(check-export-files files))
(assert (contains? valid-formats format)
"expected valid export format")
(dm/assert!
"expected valid format"
(contains? valid-formats format))
(let [files (check-export-files files)]
(ptk/reify ::export-files
ptk/WatchEvent
(watch [_ state _]
(let [features (get state :features)
team-id (:current-team-id state)
(let [team-id (get state :current-team-id)
evname (if (= format :legacy-zip)
"export-standard-files"
"export-binary-files")]
(rx/merge
(rx/of (ptk/event ::ev/event {::ev/name evname
::ev/origin "dashboard"
@ -66,10 +60,9 @@
(rx/map (fn [files]
(modal/show
{:type ::export-files
:features features
:team-id team-id
:files files
:format format})))))))))
:format format}))))))))))
;;;;;;;;;;;;;;;;;;;;;;
;; Team Request

2
frontend/src/app/main/data/workspace/libraries.cljs
View file

@ -254,7 +254,7 @@
(defn add-media
[media]
(let [media (ctf/check-media-object media)]
(let [media (ctf/check-file-media media)]
(ptk/reify ::add-media
ev/Event
(-data [_] media)

4
frontend/src/app/main/data/workspace/media.cljs
View file

@ -10,11 +10,11 @@
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.files.builder :as fb]
[app.common.files.changes-builder :as pcb]
[app.common.files.shapes-builder :as sb]
[app.common.logging :as log]
[app.common.math :as mth]
[app.common.media :as media]
[app.common.schema :as sm]
[app.common.types.container :as ctn]
[app.common.types.shape :as cts]
@ -137,7 +137,7 @@
(= (.-type blob) "image/svg+xml")))
(prepare-blob [blob]
(let [name (or name (if (dmm/file? blob) (fb/strip-image-extension (.-name blob)) "blob"))]
(let [name (or name (if (dmm/file? blob) (media/strip-image-extension (.-name blob)) "blob"))]
{:file-id file-id
:name name
:is-local local?

7
frontend/src/app/main/ui/exports/files.cljs
View file

@ -67,11 +67,11 @@
[:div {:class (stl/css :file-name-label)}
(:name file)]]])
(mf/defc export-dialog*
(mf/defc export-dialog
{::mf/register modal/components
::mf/register-as ::fexp/export-files
::mf/props :obj}
[{:keys [team-id files features format]}]
[{:keys [team-id files format]}]
(let [state* (mf/use-state (partial initialize-state files))
has-libs? (some :has-libraries files)
@ -88,14 +88,13 @@
start-export
(mf/use-fn
(mf/deps team-id selected files features)
(mf/deps team-id selected files)
(fn []
(swap! state* assoc :status :exporting)
(->> (mw/ask-many!
{:cmd :export-files
:format format
:team-id team-id
:features features
:type selected
:files files})
(rx/mapcat #(->> (rx/of %)

124
frontend/src/app/util/zip.cljs
View file

@ -5,72 +5,82 @@
;; Copyright (c) KALEIDOS INC
(ns app.util.zip
"Helpers for make zip file (using jszip)."
"Helpers for make zip file."
(:require
["jszip" :as zip]
[app.util.http :as http]
[beicon.v2.core :as rx]
["@zip.js/zip.js" :as zip]
[app.util.array :as array]
[promesa.core :as p]))
(defn compress-files
[files]
(letfn [(attach-file [zobj [name content]]
(.file zobj name content))]
(let [zobj (zip.)]
(run! (partial attach-file zobj) files)
(->> (.generateAsync zobj #js {:type "blob"})
(rx/from)))))
(defn load-from-url
"Loads the data from a blob url"
[url]
(->> (http/send!
{:uri url
:response-type :blob
:method :get})
(rx/map :body)
(rx/merge-map zip/loadAsync)))
(defn- process-file
[entry path type]
;; (js/console.log "zip:process-file" entry path type)
(defn reader
[blob]
(cond
(nil? entry)
(p/rejected (str "File not found: " path))
(instance? js/Blob blob)
(let [breader (new zip/BlobReader blob)]
(new zip/ZipReader breader))
(.-dir ^js entry)
(p/resolved {:dir path})
(instance? js/Uint8Array blob)
(let [breader (new zip/Uint8ArrayReader blob)
zreader (new zip/ZipReader breader #js {:useWebWorkers false})]
zreader)
(instance? js/ArrayBuffer blob)
(reader (js/Uint8Array. blob))
:else
(->> (.async ^js entry type)
(p/fmap (fn [content]
;; (js/console.log "zip:process-file" 2 content)
{:path path
:content content})))))
(throw (ex-info "invalid arguments"
{:type :internal
:code :invalid-type}))))
(defn load
[data]
(rx/from (zip/loadAsync data)))
(defn blob-writer
[& {:keys [mtype]}]
(new zip/BlobWriter (or mtype "application/octet-stream")))
(defn get-file
"Gets a single file from the zip archive"
([zip path]
(get-file zip path "text"))
(defn bytes-writer
[]
(new zip/Uint8ArrayWriter))
([zip path type]
(-> (.file zip path)
(process-file path type)
(rx/from))))
(defn writer
[stream-writer]
(new zip/ZipWriter stream-writer))
(defn extract-files
"Creates a stream that will emit values for every file in the zip"
[zip]
(let [promises (atom [])
get-file
(fn [path entry]
(let [current (process-file entry path "text")]
(swap! promises conj current)))]
(.forEach zip get-file)
(defn add
[writer path content]
(assert (instance? zip/ZipWriter writer))
(cond
(instance? js/Uint8Array content)
(.add writer path (new zip/Uint8ArrayReader content))
(->> (rx/from (p/all @promises))
(rx/merge-map identity))))
(instance? js/ArrayBuffer content)
(.add writer path (new zip/Uint8ArrayReader
(new js/Uint8Array content)))
(instance? js/Blob content)
(.add writer path (new zip/BlobReader content))
(string? content)
(.add writer path (new zip/TextReader content))
:else
(throw (ex-info "invalid arguments"
{:type :internal
:code :invalid-type}))))
(defn get-entry
[reader path]
(assert (instance? zip/ZipReader reader))
(->> (.getEntries ^zip/ZipReader reader)
(p/fmap (fn [entries]
(array/find #(= (.-filename ^js %) path) entries)))))
(defn read-as-text
[entry]
(let [writer (new zip/TextWriter)]
(.getData entry writer)))
(defn close
[closeable]
(assert (or (instance? zip/ZipReader closeable)
(instance? zip/ZipWriter closeable)))
(.close ^js closeable))

417
frontend/src/app/worker/export.cljs
View file

@ -6,402 +6,18 @@
(ns app.worker.export
(:require
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.json :as json]
[app.common.media :as cm]
[app.common.text :as ct]
[app.common.types.components-list :as ctkl]
[app.common.types.file :as ctf]
[app.config :as cfg]
[app.main.features.pointer-map :as fpmap]
[app.main.render :as r]
[app.main.repo :as rp]
[app.util.http :as http]
[app.util.webapi :as wapi]
[app.util.zip :as uz]
[app.worker.impl :as impl]
[beicon.v2.core :as rx]
[cuerdas.core :as str]))
(def ^:const current-version 2)
(defn create-manifest
"Creates a manifest entry for the given files"
[team-id file-id export-type files features]
(letfn [(format-page [manifest page]
(-> manifest
(assoc (str (:id page))
{:name (:name page)})))
(format-file [manifest file]
(let [name (:name file)
is-shared (:is-shared file)
pages (->> (get-in file [:data :pages])
(mapv str))
index (->> (get-in file [:data :pages-index])
(vals)
(reduce format-page {}))]
(-> manifest
(assoc (str (:id file))
{:name name
:features features
:shared is-shared
:pages pages
:pagesIndex index
:version current-version
:libraries (->> (:libraries file) (into #{}) (mapv str))
:exportType (d/name export-type)
:hasComponents (d/not-empty? (ctkl/components-seq (:data file)))
:hasDeletedComponents (d/not-empty? (ctkl/deleted-components-seq (:data file)))
:hasMedia (d/not-empty? (get-in file [:data :media]))
:hasColors (d/not-empty? (get-in file [:data :colors]))
:hasTypographies (d/not-empty? (get-in file [:data :typographies]))}))))]
(let [manifest {:teamId (str team-id)
:fileId (str file-id)
:files (->> (vals files) (reduce format-file {}))}]
(json/encode manifest))))
(defn process-pages [file]
(let [pages (get-in file [:data :pages])
pages-index (get-in file [:data :pages-index])]
(->> pages
(map #(hash-map
:file-id (:id file)
:data (get pages-index %))))))
(defn get-page-data
[{file-id :file-id {:keys [id name] :as data} :data}]
(->> (r/render-page data)
(rx/map (fn [markup]
{:id id
:name name
:file-id file-id
:markup markup}))))
(defn collect-page
[{:keys [id file-id markup] :as page}]
[(str file-id "/" id ".svg") markup])
(defn collect-entries [result data keys]
(-> result
(assoc (str (:id data))
(->> (select-keys data keys)
(d/deep-mapm
(fn [[k v]]
[(-> k str/camel) v]))))))
(def ^:const color-keys
[:name :color :opacity :gradient :path])
(def ^:const image-color-keys
[:width :height :mtype :name :keep-aspect-ratio])
(def ^:const typography-keys
[:name :font-family :font-id :font-size :font-style :font-variant-id :font-weight
:letter-spacing :line-height :text-transform :path])
(def ^:const media-keys
[:name :mtype :width :height :path])
(defn collect-color
[result color]
(let [id (str (:id color))
basic-data (select-keys color color-keys)
image-color-data (when-let [image-color (:image color)]
(->> (select-keys image-color image-color-keys)))
color-data (cond-> basic-data
(some? image-color-data)
(->
(assoc :image image-color-data)
(assoc-in [:image :id] (str (get-in color [:image :id])))))]
(-> result
(assoc id
(->> color-data
(d/deep-mapm
(fn [[k v]]
[(-> k str/camel) v])))))))
(defn collect-typography
[result typography]
(collect-entries result typography typography-keys))
(defn collect-media
[result media]
(collect-entries result media media-keys))
(defn parse-library-color
[[file-id colors]]
(rx/merge
(let [markup
(->> (vals colors)
(reduce collect-color {})
(json/encode))]
(rx/of (vector (str file-id "/colors.json") markup)))
(->> (rx/from (vals colors))
(rx/map :image)
(rx/filter d/not-empty?)
(rx/merge-map
(fn [image-color]
(let [file-path (str/concat file-id "/colors/" (:id image-color) (cm/mtype->extension (:mtype image-color)))]
(->> (http/send!
{:uri (cfg/resolve-file-media image-color)
:response-type :blob
:method :get})
(rx/map :body)
(rx/map #(vector file-path %)))))))))
(defn parse-library-typographies
[[file-id typographies]]
(let [markup
(->> (vals typographies)
(reduce collect-typography {})
(json/encode))]
[(str file-id "/typographies.json") markup]))
(defn parse-library-media
[[file-id media]]
(rx/merge
(let [markup
(->> (vals media)
(reduce collect-media {})
(json/encode))]
(rx/of (vector (str file-id "/media.json") markup)))
(->> (rx/from (vals media))
(rx/map #(assoc % :file-id file-id))
(rx/merge-map
(fn [media]
(let [file-path (str/concat file-id "/media/" (:id media) (cm/mtype->extension (:mtype media)))]
(->> (http/send!
{:uri (cfg/resolve-file-media media)
:response-type :blob
:method :get})
(rx/map :body)
(rx/map #(vector file-path %)))))))))
(defn parse-library-components
[file]
(->> (r/render-components (:data file) false)
(rx/map #(vector (str (:id file) "/components.svg") %))))
(defn parse-deleted-components
[file]
(->> (r/render-components (:data file) true)
(rx/map #(vector (str (:id file) "/deleted-components.svg") %))))
(defn fetch-file-with-libraries
[file-id features]
(->> (rx/zip (->> (rp/cmd! :get-file {:id file-id :features features})
(rx/mapcat fpmap/resolve-file))
(rp/cmd! :get-file-libraries {:file-id file-id}))
(rx/map
(fn [[file file-libraries]]
(let [libraries-ids (->> file-libraries (map :id) (filterv #(not= (:id file) %)))]
(assoc file :libraries libraries-ids))))))
(defn make-local-external-references
[file file-id]
(let [change-fill
(fn [fill]
(cond-> fill
(not= file-id (:fill-color-ref-file fill))
(assoc :fill-color-ref-file file-id)))
change-stroke
(fn [stroke]
(cond-> stroke
(not= file-id (:stroke-color-ref-file stroke))
(assoc :stroke-color-ref-file file-id)))
change-text
(fn [content]
(->> content
(ct/transform-nodes
(fn [node]
(-> node
(d/update-when :fills #(mapv change-fill %))
(cond-> (not= file-id (:typography-ref-file node))
(assoc :typography-ref-file file-id)))))))
change-shape
(fn [shape]
(-> shape
(d/update-when :fills #(mapv change-fill %))
(d/update-when :strokes #(mapv change-stroke %))
(cond-> (not= file-id (:component-file shape))
(assoc :component-file file-id))
(cond-> (= :text (:type shape))
(update :content change-text))))
change-objects
(fn [objects]
(->> objects
(d/mapm #(change-shape %2))))
change-pages
(fn [pages-index]
(->> pages-index
(d/mapm
(fn [_ data]
(-> data
(update :objects change-objects))))))]
(-> file
(update-in [:data :pages-index] change-pages))))
(defn merge-assets [target-file assets-files]
(let [merge-file-assets
(fn [target file]
(let [colors (get-in file [:data :colors])
typographies (get-in file [:data :typographies])
media (get-in file [:data :media])
components (ctkl/components (:data file))]
(cond-> target
(d/not-empty? colors)
(update-in [:data :colors] merge colors)
(d/not-empty? typographies)
(update-in [:data :typographies] merge typographies)
(d/not-empty? media)
(update-in [:data :media] merge media)
(d/not-empty? components)
(update-in [:data :components] merge components))))]
(->> assets-files
(reduce merge-file-assets target-file))))
(defn process-export
[file-id export-type files]
(let [result
(case export-type
:all files
:merge (let [file-list (-> files (d/without-keys [file-id]) vals)]
(-> (select-keys files [file-id])
(update file-id merge-assets file-list)
(update file-id make-local-external-references file-id)
(update file-id dissoc :libraries)))
:detach (-> (select-keys files [file-id])
(update file-id ctf/detach-external-references file-id)
(update file-id dissoc :libraries)))]
;;(.log js/console (clj->js result))
result))
(defn collect-files
[file-id export-type features]
(letfn [(fetch-dependencies [[files pending]]
(if (empty? pending)
;; When not pending, we finish the generation
(rx/empty)
;; Still pending files, fetch the next one
(let [next (peek pending)
pending (pop pending)]
(if (contains? files next)
;; The file is already in the result
(rx/of [files pending])
(->> (fetch-file-with-libraries next features)
(rx/map
(fn [file]
[(-> files
(assoc (:id file) file))
(as-> pending $
(reduce conj $ (:libraries file)))])))))))]
(let [files {}
pending [file-id]]
(->> (rx/of [files pending])
(rx/expand fetch-dependencies)
(rx/last)
(rx/map first)
(rx/map #(process-export file-id export-type %))))))
(defn export-file
[team-id file-id export-type features]
(let [files-stream (->> (collect-files file-id export-type features)
(rx/share))
manifest-stream
(->> files-stream
(rx/map #(create-manifest team-id file-id export-type % features))
(rx/map #(vector "manifest.json" %)))
render-stream
(->> files-stream
(rx/merge-map vals)
(rx/merge-map process-pages)
(rx/observe-on :async)
(rx/merge-map get-page-data)
(rx/share))
colors-stream
(->> files-stream
(rx/merge-map vals)
(rx/map #(vector (:id %) (get-in % [:data :colors])))
(rx/filter #(d/not-empty? (second %)))
(rx/merge-map parse-library-color))
typographies-stream
(->> files-stream
(rx/merge-map vals)
(rx/map #(vector (:id %) (get-in % [:data :typographies])))
(rx/filter #(d/not-empty? (second %)))
(rx/map parse-library-typographies))
media-stream
(->> files-stream
(rx/merge-map vals)
(rx/map #(vector (:id %) (get-in % [:data :media])))
(rx/filter #(d/not-empty? (second %)))
(rx/merge-map parse-library-media))
components-stream
(->> files-stream
(rx/merge-map vals)
(rx/filter #(d/not-empty? (ctkl/components-seq (:data %))))
(rx/merge-map parse-library-components))
deleted-components-stream
(->> files-stream
(rx/merge-map vals)
(rx/filter #(d/not-empty? (ctkl/deleted-components-seq (:data %))))
(rx/merge-map parse-deleted-components))
pages-stream
(->> render-stream
(rx/map collect-page))]
(rx/merge
(->> render-stream
(rx/map #(hash-map
:type :progress
:file file-id
:data (str "Render " (:file-name %) " - " (:name %)))))
(->> (rx/merge
manifest-stream
pages-stream
components-stream
deleted-components-stream
media-stream
colors-stream
typographies-stream)
(rx/reduce conj [])
(rx/with-latest-from files-stream)
(rx/merge-map (fn [[data files]]
(->> (uz/compress-files data)
(rx/map #(vector (get files file-id) %)))))))))
[beicon.v2.core :as rx]))
(defmethod impl/handler :export-files
[{:keys [team-id files type format features] :as message}]
(cond
(or (= format :binfile-v1)
[{:keys [files type format] :as message}]
(assert (or (= format :binfile-v1)
(= format :binfile-v3))
"expected valid format")
(->> (rx/from files)
(rx/mapcat
(fn [file]
@ -419,31 +35,10 @@
"application/zip"
"application/penpot")
:uri uri}))
(rx/catch
(fn [cause]
(rx/of {:type :error
:file-id (:id file)
:hint (ex-message cause)})))))))
(= format :legacy-zip)
(->> (rx/from files)
(rx/mapcat
(fn [file]
(->> (export-file team-id (:id file) type features)
(rx/map
(fn [value]
(if (contains? value :type)
value
(let [[file export-blob] value]
{:type :finish
:file-id (:id file)
:filename (:name file)
:mtype "application/zip"
:uri (wapi/create-uri export-blob)}))))
(rx/catch
(fn [cause]
(rx/of (ex/raise :type :internal
:code :export-error
:hint "unexpected error on exporting file"
:file-id (:id file)
:cause cause))))))))))
:cause cause)))))))))

13
frontend/src/app/worker/import.cljs
View file

@ -30,9 +30,9 @@
(def conjv (fnil conj []))
(defn- read-zip-manifest
[zipfile]
(->> (uz/get-file zipfile "manifest.json")
(rx/map :content)
[zip-reader]
(->> (rx/from (uz/get-entry zip-reader "manifest.json"))
(rx/mapcat uz/read-as-text)
(rx/map json/decode)))
(defn slurp-uri
@ -121,14 +121,15 @@
(let [mtype (parse-mtype body)]
(cond
(= "application/zip" mtype)
(->> (uz/load body)
(rx/merge-map read-zip-manifest)
(let [zip-reader (uz/reader body)]
(->> (read-zip-manifest zip-reader)
(rx/map
(fn [manifest]
(if (= (:type manifest) "penpot/export-files")
(let [manifest (decode-manifest manifest)]
(assoc file :type :binfile-v3 :files (:files manifest)))
(assoc file :type :legacy-zip :body body)))))
(assoc file :type :legacy-zip :body body))))
(rx/finalize (partial uz/close zip-reader))))
(= "application/octet-stream" mtype)
(rx/of (assoc file :type :binfile-v1))

250
frontend/src/lib/file_builder.cljs
View file

@ -1,250 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns lib.file-builder
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.files.builder :as fb]
[app.common.json :as json]
[app.common.schema :as sm]
[app.common.uuid :as uuid]
[app.util.object :as obj]))
(def BuilderError
(obj/class
:name "BuilderError"
:extends js/Error
:constructor
(fn [this type code hint cause]
(.call js/Error this hint)
(set! (.-name this) (str "Exception: " hint))
(set! (.-type this) type)
(set! (.-code this) code)
(set! (.-hint this) hint)
(when (exists? js/Error.captureStackTrace)
(.captureStackTrace js/Error this))
(obj/add-properties!
this
{:name "cause"
:enumerable true
:this false
:get (fn [] cause)}
{:name "data"
:enumerable true
:this false
:get (fn []
(let [data (ex-data cause)]
(when-let [explain (::sm/explain data)]
(json/->js (sm/simplify explain)))))}))))
(defn- handle-exception
[cause]
(let [data (ex-data cause)]
(throw (new BuilderError
(d/name (get data :type :unknown))
(d/name (get data :code :unknown))
(or (get data :hint) (ex-message cause))
cause))))
(defn- decode-params
[params]
(if (obj/plain-object? params)
(json/->js params)
params))
(defn- create-file*
[file]
(let [state* (volatile! file)]
(obj/reify {:name "File"}
:id
{:get #(dm/str (:id @state*))}
:currentFrameId
{:get #(dm/str (::fb/current-frame-id @state*))}
:currentPageId
{:get #(dm/str (::fb/current-page-id @state*))}
:lastId
{:get #(dm/str (::fb/last-id @state*))}
:addPage
(fn [params]
(try
(let [params (-> params
(decode-params)
(fb/decode-page))]
(vswap! state* fb/add-page params)
(dm/str (::fb/current-page-id @state*)))
(catch :default cause
(handle-exception cause))))
:closePage
(fn []
(vswap! state* fb/close-page))
:addArtboard
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :frame)
(fb/decode-shape))]
(vswap! state* fb/add-artboard params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:closeArtboard
(fn []
(vswap! state* fb/close-artboard))
:addGroup
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :group)
(fb/decode-shape))]
(vswap! state* fb/add-group params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:closeGroup
(fn []
(vswap! state* fb/close-group))
:addBool
(fn [params]
(try
(let [params (-> params
(json/->clj)
(fb/decode-add-bool))]
(vswap! state* fb/add-bool params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addRect
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :rect)
(fb/decode-shape))]
(vswap! state* fb/add-shape params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addCircle
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :circle)
(fb/decode-shape))]
(vswap! state* fb/add-shape params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addPath
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :path)
(fb/decode-shape))]
(vswap! state* fb/add-shape params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addText
(fn [params]
(try
(let [params (-> params
(json/->clj)
(assoc :type :text)
(fb/decode-shape))]
(vswap! state* fb/add-shape params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addLibraryColor
(fn [params]
(try
(let [params (-> params
(json/->clj)
(fb/decode-library-color)
(d/without-nils))]
(vswap! state* fb/add-library-color params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addLibraryTypography
(fn [params]
(try
(let [params (-> params
(json/->clj)
(fb/decode-library-typography)
(d/without-nils))]
(vswap! state* fb/add-library-typography params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addComponent
(fn [params]
(try
(let [params (-> params
(json/->clj)
(fb/decode-component)
(d/without-nils))]
(vswap! state* fb/add-component params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:addComponentInstance
(fn [params]
(try
(let [params (-> params
(json/->clj)
(fb/decode-add-component-instance)
(d/without-nils))]
(vswap! state* fb/add-component-instance params)
(dm/str (::fb/last-id @state*)))
(catch :default cause
(handle-exception cause))))
:getShape
(fn [shape-id]
(let [shape-id (uuid/parse shape-id)]
(some-> (fb/lookup-shape @state* shape-id)
(json/->js))))
:toMap
(fn []
(-> @state*
(d/without-qualified)
(json/->js))))))
(defn create-file
[params]
(try
(let [params (-> params json/->clj fb/decode-file)
file (fb/create-file params)]
(create-file* file))
(catch :default cause
(handle-exception cause))))

30
frontend/src/lib/playground/sample1.js
View file

@ -1,30 +0,0 @@
import * as penpot from "../../../target/library/penpot.js";
console.log(penpot);
try {
const file = penpot.createFile({name: "Test"});
file.addPage({name: "Foo Page"})
const boardId = file.addArtboard({name: "Foo Board"})
const rectId = file.addRect({name: "Foo Rect", width:100, height: 200})
file.addLibraryColor({color: "#fabada", opacity: 0.5})
console.log("created board", boardId);
console.log("created rect", rectId);
const board = file.getShape(boardId);
console.log("=========== BOARD =============")
console.dir(board, {depth: 10});
const rect = file.getShape(rectId);
console.log("=========== RECT =============")
console.dir(rect, {depth: 10});
// console.dir(file.toMap(), {depth:10});
} catch (e) {
console.log(e);
// console.log(e.data);
}
process.exit(0);

175
frontend/test/frontend_tests/util_snap_data_test.cljs
View file

@ -27,7 +27,8 @@
(t/is (some? data))))
(t/testing "Add empty page (only root-frame)"
(let [page (-> (fb/create-file {:name "Test"})
(let [page (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/get-current-page))
@ -36,7 +37,8 @@
(t/is (some? data))))
(t/testing "Create simple shape on root"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-shape
{:type :rect
@ -44,7 +46,7 @@
:y 0
:width 100
:height 100}))
page (fb/get-current-page file)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
@ -66,17 +68,18 @@
(t/is (= (first (nth result-x 2)) 100))))
(t/testing "Add page with single empty frame"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard
(fb/add-board
{:x 0
:y 0
:width 100
:height 100})
(fb/close-artboard))
(fb/close-board))
frame-id (::fb/last-id file)
page (fb/get-current-page file)
frame-id (::fb/last-id state)
page (fb/get-current-page state)
;; frame-id (::fb/last-id file)
data (-> (sd/make-snap-data)
@ -91,26 +94,27 @@
(t/testing "Add page with some shapes inside frames"
(with-redefs [uuid/next (get-mocked-uuid)]
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard
(fb/add-board
{:x 0
:y 0
:width 100
:height 100}))
frame-id (::fb/last-id file)
frame-id (::fb/last-id state)
file (-> file
state (-> state
(fb/add-shape
{:type :rect
:x 25
:y 25
:width 50
:height 50})
(fb/close-artboard))
(fb/close-board))
page (fb/get-current-page file)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
@ -123,16 +127,16 @@
(t/is (= (count result-frame-x) 5)))))
(t/testing "Add a global guide"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-guide {:position 50 :axis :x})
(fb/add-artboard {:x 200 :y 200 :width 100 :height 100})
(fb/close-artboard))
(fb/add-board {:x 200 :y 200 :width 100 :height 100})
(fb/close-board))
frame-id (::fb/last-id file)
page (fb/get-current-page file)
frame-id (::fb/last-id state)
page (fb/get-current-page state)
;; frame-id (::fb/last-id file)
data (-> (sd/make-snap-data)
(sd/add-page page))
@ -151,17 +155,18 @@
(t/is (= (count result-frame-y) 0))))
(t/testing "Add a frame guide"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard {:x 200 :y 200 :width 100 :height 100})
(fb/close-artboard))
(fb/add-board {:x 200 :y 200 :width 100 :height 100})
(fb/close-board))
frame-id (::fb/last-id file)
frame-id (::fb/last-id state)
file (-> file
state (-> state
(fb/add-guide {:position 50 :axis :x :frame-id frame-id}))
page (fb/get-current-page file)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
@ -182,26 +187,26 @@
(t/deftest test-update-index
(t/testing "Create frame on root and then remove it."
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard
(fb/add-board
{:x 0
:y 0
:width 100
:height 100})
(fb/close-artboard))
(fb/close-board))
shape-id (::fb/last-id file)
page (fb/get-current-page file)
shape-id (::fb/last-id state)
page (fb/get-current-page state)
;; frame-id (::fb/last-id file)
data (-> (sd/make-snap-data)
(sd/add-page page))
file (-> file
state (-> state
(fb/delete-shape shape-id))
new-page (fb/get-current-page file)
new-page (fb/get-current-page state)
data (sd/update-page data page new-page)
result-x (sd/query data (:id page) uuid/zero :x [0 100])
@ -212,7 +217,8 @@
(t/is (= (count result-y) 0))))
(t/testing "Create simple shape on root. Then remove it"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-shape
{:type :rect
@ -221,16 +227,16 @@
:width 100
:height 100}))
shape-id (::fb/last-id file)
page (fb/get-current-page file)
shape-id (::fb/last-id state)
page (fb/get-current-page state)
;; frame-id (::fb/last-id file)
;; frame-id (::fb/last-id state)
data (-> (sd/make-snap-data)
(sd/add-page page))
file (fb/delete-shape file shape-id)
state (fb/delete-shape state shape-id)
new-page (fb/get-current-page file)
new-page (fb/get-current-page state)
data (sd/update-page data page new-page)
result-x (sd/query data (:id page) uuid/zero :x [0 100])
@ -241,26 +247,27 @@
(t/is (= (count result-y) 0))))
(t/testing "Create shape inside frame, then remove it"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard
(fb/add-board
{:x 0
:y 0
:width 100
:height 100}))
frame-id (::fb/last-id file)
frame-id (::fb/last-id state)
file (fb/add-shape file {:type :rect :x 25 :y 25 :width 50 :height 50})
shape-id (::fb/last-id file)
state (fb/add-shape state {:type :rect :x 25 :y 25 :width 50 :height 50})
shape-id (::fb/last-id state)
file (fb/close-artboard file)
state (fb/close-board state)
page (fb/get-current-page file)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
file (fb/delete-shape file shape-id)
new-page (fb/get-current-page file)
state (fb/delete-shape state shape-id)
new-page (fb/get-current-page state)
data (sd/update-page data page new-page)
@ -272,20 +279,22 @@
(t/is (= (count result-frame-x) 3))))
(t/testing "Create global guide then remove it"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-guide {:position 50 :axis :x}))
guide-id (::fb/last-id file)
guide-id (::fb/last-id state)
file (-> (fb/add-artboard file {:x 200 :y 200 :width 100 :height 100})
(fb/close-artboard))
state (-> (fb/add-board state {:x 200 :y 200 :width 100 :height 100})
(fb/close-board))
frame-id (::fb/last-id file)
page (fb/get-current-page file)
data (-> (sd/make-snap-data) (sd/add-page page))
frame-id (::fb/last-id state)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
new-page (-> (fb/delete-guide file guide-id)
new-page (-> (fb/delete-guide state guide-id)
(fb/get-current-page))
data (sd/update-page data page new-page)
@ -305,10 +314,11 @@
(t/is (= (count result-frame-y) 0))))
(t/testing "Create frame guide then remove it"
(let [file (-> (fb/create-file {:name "Test"})
(let [file (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard {:x 200 :y 200 :width 100 :height 100})
(fb/close-artboard))
(fb/add-board {:x 200 :y 200 :width 100 :height 100})
(fb/close-board))
frame-id (::fb/last-id file)
file (fb/add-guide file {:position 50 :axis :x :frame-id frame-id})
@ -336,20 +346,22 @@
(t/is (= (count result-frame-y) 0))))
(t/testing "Update frame coordinates"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-artboard
(fb/add-board
{:x 0
:y 0
:width 100
:height 100})
(fb/close-artboard))
(fb/close-board))
frame-id (::fb/last-id file)
page (fb/get-current-page file)
data (-> (sd/make-snap-data) (sd/add-page page))
frame-id (::fb/last-id state)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
file (fb/update-shape file frame-id
state (fb/update-shape state frame-id
(fn [shape]
(-> shape
(dissoc :selrect :points)
@ -357,8 +369,7 @@
(cts/setup-shape))))
new-page (fb/get-current-page file)
new-page (fb/get-current-page state)
data (sd/update-page data page new-page)
result-zero-x-1 (sd/query data (:id page) uuid/zero :x [0 100])
@ -373,7 +384,8 @@
(t/is (= (count result-frame-x-2) 3))))
(t/testing "Update shape coordinates"
(let [file (-> (fb/create-file {:name "Test"})
(let [state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-shape
{:type :rect
@ -382,19 +394,19 @@
:width 100
:height 100}))
shape-id (::fb/last-id file)
page (fb/get-current-page file)
shape-id (::fb/last-id state)
page (fb/get-current-page state)
data (-> (sd/make-snap-data)
(sd/add-page page))
file (fb/update-shape file shape-id
state (fb/update-shape state shape-id
(fn [shape]
(-> shape
(dissoc :selrect :points)
(assoc :x 200 :y 200)
(cts/setup-shape))))
new-page (fb/get-current-page file)
new-page (fb/get-current-page state)
;; FIXME: update
data (sd/update-page data page new-page)
@ -407,21 +419,22 @@
(t/testing "Update global guide"
(let [guide {:position 50 :axis :x}
file (-> (fb/create-file {:name "Test"})
state (-> (fb/create-state)
(fb/add-file {:name "Test"})
(fb/add-page {:name "Page 1"})
(fb/add-guide guide))
guide-id (::fb/last-id file)
guide-id (::fb/last-id state)
guide (assoc guide :id guide-id)
file (-> (fb/add-artboard file {:x 500 :y 500 :width 100 :height 100})
(fb/close-artboard))
state (-> (fb/add-board state {:x 500 :y 500 :width 100 :height 100})
(fb/close-board))
frame-id (::fb/last-id file)
page (fb/get-current-page file)
frame-id (::fb/last-id state)
page (fb/get-current-page state)
data (-> (sd/make-snap-data) (sd/add-page page))
new-page (-> (fb/update-guide file (assoc guide :position 150))
new-page (-> (fb/update-guide state (assoc guide :position 150))
(fb/get-current-page))
data (sd/update-page data page new-page)

52
frontend/yarn.lock
View file

@ -2881,10 +2881,17 @@ __metadata:
languageName: node
linkType: hard
"@zip.js/zip.js@npm:^2.7.44":
version: 2.7.53
resolution: "@zip.js/zip.js@npm:2.7.53"
checksum: 10c0/883527bf09ce7c312117536c79d5f07e736d87de802a6c19e39ba2e18027499dcb9359df94dfde13c9bcf6118a20b4f26a40f9892ee82d7cac3124d6986b15c8
"@zip.js/zip.js@npm:2.7.60":
version: 2.7.60
resolution: "@zip.js/zip.js@npm:2.7.60"
checksum: 10c0/466ff1729e36d9f500011475e230f2edb9c0e6e10f64d542e6ebc006dc70885bc909d69fd0c7b10126bdf722c761359ad1edfe295b6c7fed3169f0f63012a1cd
languageName: node
linkType: hard
"@zip.js/zip.js@patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch":
version: 2.7.60
resolution: "@zip.js/zip.js@patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch::version=2.7.60&hash=4a67b2"
checksum: 10c0/37e9a5dd708fd81b08d64b75ea44d70c903071165bbbc571fca7a1cb93f214fab6f63ed3c837a87a0205a6301bc78790c1505197570f112afa5311e3a16d2368
languageName: node
linkType: hard
@ -5705,6 +5712,7 @@ __metadata:
"@storybook/test-runner": "npm:^0.21.0"
"@tokens-studio/sd-transforms": "npm:1.2.11"
"@types/node": "npm:^22.12.0"
"@zip.js/zip.js": "patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch"
autoprefixer: "npm:^10.4.20"
compression: "npm:^1.7.5"
concurrently: "npm:^9.1.2"
@ -5724,7 +5732,6 @@ __metadata:
gulp-svg-sprite: "npm:^2.0.3"
js-beautify: "npm:^1.15.4"
jsdom: "npm:^26.1.0"
jszip: "npm:^3.10.1"
lodash: "npm:^4.17.21"
lodash.debounce: "npm:^4.0.8"
luxon: "npm:^3.6.1"
@ -6386,13 +6393,6 @@ __metadata:
languageName: node
linkType: hard
"immediate@npm:~3.0.5":
version: 3.0.6
resolution: "immediate@npm:3.0.6"
checksum: 10c0/f8ba7ede69bee9260241ad078d2d535848745ff5f6995c7c7cb41cfdc9ccc213f66e10fa5afb881f90298b24a3f7344b637b592beb4f54e582770cdce3f1f039
languageName: node
linkType: hard
"immutable@npm:^5.0.2":
version: 5.0.3
resolution: "immutable@npm:5.0.3"
@ -7686,18 +7686,6 @@ __metadata:
languageName: node
linkType: hard
"jszip@npm:^3.10.1":
version: 3.10.1
resolution: "jszip@npm:3.10.1"
dependencies:
lie: "npm:~3.3.0"
pako: "npm:~1.0.2"
readable-stream: "npm:~2.3.6"
setimmediate: "npm:^1.0.5"
checksum: 10c0/58e01ec9c4960383fb8b38dd5f67b83ccc1ec215bf74c8a5b32f42b6e5fb79fada5176842a11409c4051b5b94275044851814a31076bf49e1be218d3ef57c863
languageName: node
linkType: hard
"klaw-sync@npm:^6.0.0":
version: 6.0.0
resolution: "klaw-sync@npm:6.0.0"
@ -7728,15 +7716,6 @@ __metadata:
languageName: node
linkType: hard
"lie@npm:~3.3.0":
version: 3.3.0
resolution: "lie@npm:3.3.0"
dependencies:
immediate: "npm:~3.0.5"
checksum: 10c0/56dd113091978f82f9dc5081769c6f3b947852ecf9feccaf83e14a123bc630c2301439ce6182521e5fbafbde88e88ac38314327a4e0493a1bea7e0699a7af808
languageName: node
linkType: hard
"lilconfig@npm:^3.1.1":
version: 3.1.2
resolution: "lilconfig@npm:3.1.2"
@ -8812,13 +8791,6 @@ __metadata:
languageName: node
linkType: hard
"pako@npm:~1.0.2":
version: 1.0.11
resolution: "pako@npm:1.0.11"
checksum: 10c0/86dd99d8b34c3930345b8bbeb5e1cd8a05f608eeb40967b293f72fe469d0e9c88b783a8777e4cc7dc7c91ce54c5e93d88ff4b4f060e6ff18408fd21030d9ffbe
languageName: node
linkType: hard
"parse-json@npm:^4.0.0":
version: 4.0.0
resolution: "parse-json@npm:4.0.0"

18
library/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch Normal file
View file

@ -0,0 +1,18 @@
diff --git a/lib/zip-fs.js b/lib/zip-fs.js
index 1444c0f00e5f1ad6c13521f90a7f3c6659d81116..90e38baef5365c2abbcb9337f7ab37f800e883a4 100644
--- a/lib/zip-fs.js
+++ b/lib/zip-fs.js
@@ -33,12 +33,7 @@ import { initShimAsyncCodec } from "./core/util/stream-codec-shim.js";
import { terminateWorkers } from "./core/codec-pool.js";
let baseURL;
-try {
- baseURL = import.meta.url;
- // eslint-disable-next-line no-unused-vars
-} catch (_) {
- // ignored
-}
+
configure({ baseURL });
configureWebWorker(configure);

33
library/deps.edn Normal file
View file

@ -0,0 +1,33 @@
{:paths ["src" "vendor" "resources" "test"]
:deps
{penpot/common
{:local/root "../common"}
penpot/frontend
{:local/root "../frontend"}
}
:aliases
{:outdated
{:extra-deps {com.github.liquidz/antq {:mvn/version "RELEASE"}}
:main-opts ["-m" "antq.core"]}
:jvm-repl
{:extra-deps
{com.bhauman/rebel-readline {:mvn/version "RELEASE"}}
:main-opts ["-m" "rebel-readline.main"]
:jvm-opts ["--sun-misc-unsafe-memory-access=allow"]}
:dev
{:extra-paths ["dev"]
:extra-deps
{thheller/shadow-cljs {:mvn/version "3.0.5"}
com.bhauman/rebel-readline {:mvn/version "RELEASE"}
org.clojure/tools.namespace {:mvn/version "RELEASE"}
criterium/criterium {:mvn/version "RELEASE"}
cider/cider-nrepl {:mvn/version "0.48.0"}}}
:shadow-cljs
{:main-opts ["-m" "shadow.cljs.devtools.cli"]
:jvm-opts ["--sun-misc-unsafe-memory-access=allow"]}
}}

42
library/package.json Normal file
View file

@ -0,0 +1,42 @@
{
"name": "@penpotapp/library",
"version": "1.0.0",
"license": "MPL-2.0",
"author": "Kaleidos INC",
"private": true,
"packageManager": "yarn@4.9.1+sha512.f95ce356460e05be48d66401c1ae64ef84d163dd689964962c6888a9810865e39097a5e9de748876c2e0bf89b232d583c33982773e9903ae7a76257270986538",
"type": "module",
"repository": {
"type": "git",
"url": "https://github.com/penpot/penpot"
},
"resolutions": {
"@zip.js/zip.js@npm:^2.7.44": "patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch"
},
"imports": {
"#self": {
"default": "./target/library/penpot.js"
}
},
"scripts": {
"clear:shadow-cache": "rm -rf .shadow-cljs",
"build": "yarn run clear:shadow-cache && clojure -M:dev:shadow-cljs release library",
"fmt:clj": "cljfmt fix --parallel=true src/ test/",
"fmt:clj:check": "cljfmt check --parallel=false src/ test/",
"lint:clj": "clj-kondo --parallel --lint src/",
"test": "node --test",
"watch:test": "node --test --watch",
"watch": "yarn run clear:shadow-cache && clojure -M:dev:shadow-cljs watch library"
},
"devDependencies": {
"@types/node": "^22.12.0",
"concurrently": "^9.1.2",
"nodemon": "^3.1.9",
"shadow-cljs": "3.0.5"
},
"dependencies": {
"@zip.js/zip.js": "patch:@zip.js/zip.js@npm%3A2.7.60#~/.yarn/patches/@zip.js-zip.js-npm-2.7.60-b6b814410b.patch",
"luxon": "^3.6.1",
"source-map-support": "^0.5.21"
}
}

103
library/playground/components.js Normal file
View file

@ -0,0 +1,103 @@
import * as penpot from "#self";
import { createWriteStream } from 'fs';
import { Writable } from "stream";
// Example of creating component and instance out of order
(async function() {
const context = penpot.createBuildContext();
{
context.addFile({name: "Test File 1"});
context.addPage({name: "Foo Page"})
const mainBoardId = context.genId();
const mainRectId = context.genId();
// First create instance (just for with the purpose of teaching
// that it can be done, without putting that under obligation to
// do it in this order or the opposite)
context.addBoard({
name: "Board Instance 1",
x: 700,
y: 0,
width: 500,
height: 300,
shapeRef: mainBoardId,
touched: ["name-group"]
})
context.addRect({
name: "Rect Instance 1",
x: 800,
y: 20,
width:100,
height:200,
shapeRef: mainRectId,
touched: ["name-group"]
});
// this function call takes the current board from context, but it
// also can be passed as parameter on an explicit way if you
// prefer
context.addComponentInstance({
componentId: "00000000-0000-0000-0000-000000000001"
});
context.closeBoard();
// Then, create the main instance
context.addBoard({
id: mainBoardId,
name: "Board",
x: 0,
y: 0,
width: 500,
height: 300,
})
context.addRect({
id: mainRectId,
name: "Rect 1",
x: 20,
y: 20,
width:100,
height:200,
});
context.addComponent({
componentId: "00000000-0000-0000-0000-000000000001",
name: "Component 1",
});
context.closeBoard();
context.closeFile();
}
{
// Create a file stream to write the zip to
const output = createWriteStream('sample-with-components.zip');
// Wrap Node's stream in a WHATWG WritableStream
const writable = Writable.toWeb(output);
await penpot.exportStream(context, writable);
}
})().catch((cause) => {
console.error(cause);
const causeExplain = cause.explain;
if (causeExplain) {
console.log("EXPLAIN:")
console.error(cause.explain);
}
// const innerCause = cause.cause;
// if (innerCause) {
// console.log("INNER:");
// console.error(innerCause);
// }
process.exit(-1);
}).finally(() => {
process.exit(0);
})

BIN
library/playground/sample.jpg Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 21 KiB

88
library/playground/sample1.js Normal file
View file

@ -0,0 +1,88 @@
import * as penpot from "#self";
import { writeFile, readFile } from 'fs/promises';
import { createWriteStream } from 'fs';
import { Writable } from "stream";
// console.log(penpot);
(async function() {
const context = penpot.createBuildContext();
{
context.addFile({name: "Test File 1"});
context.addPage({name: "Foo Page"})
// Add image media
const buffer = await readFile("./playground/sample.jpg");
const blob = new Blob([buffer], { type: 'image/jpeg' });
const mediaId = context.addFileMedia({
name: "avatar.jpg",
width: 512,
height: 512
}, blob);
// Add image color asset
const assetColorId = context.addLibraryColor({
name: "Avatar",
opacity: 1,
image: {
...context.getMediaAsImage(mediaId),
keepAspectRatio: true
}
});
const boardId = context.addBoard({
name: "Foo Board",
x: 0,
y: 0,
width: 500,
height: 300,
})
const fill = {
fillColorRefId: assetColorId,
fillColorRefFile: context.currentFileId,
fillImage: {
...context.getMediaAsImage(mediaId),
keepAspectRatio: true
}
};
context.addRect({
name: "Rect 1",
x: 20,
y: 20,
width:100,
height:200,
fills: [fill]
});
context.closeBoard();
context.closeFile();
}
{
let result = await penpot.exportAsBytes(context)
await writeFile("sample-sync.zip", result);
}
// {
// // Create a file stream to write the zip to
// const output = createWriteStream('sample-stream.zip');
// // Wrap Node's stream in a WHATWG WritableStream
// const writable = Writable.toWeb(output);
// await penpot.exportStream(context, writable);
// }
})().catch((cause) => {
console.error(cause);
const innerCause = cause.cause;
if (innerCause) {
console.error("Inner cause:", innerCause);
}
process.exit(-1);
}).finally(() => {
process.exit(0);
})

6
library/scripts/repl Executable file
View file

@ -0,0 +1,6 @@
#!/usr/bin/env bash
export OPTIONS="-A:dev -J-XX:-OmitStackTraceInFastThrow";
set -ex
exec clojure $OPTIONS -M -m rebel-readline.main

55
library/shadow-cljs.edn Normal file
View file

@ -0,0 +1,55 @@
{:deps {:aliases [:dev]}
:http {:port #shadow/env ["HTTP_PORT" :as :int :default 4448]}
:dev-http {#shadow/env ["DEV_PORT" :as :int :default 8889] "classpath:public"}
:nrepl false
:socket-repl false
:cache-dir #shadow/env ["CACHE" :default ".shadow-cljs"]
:builds
{:test
{:target :esm
:output-dir "target/tests"
:runtime :custom
:js-options {:js-provider :import}
:modules
{:test {:init-fn lib.tests.runner/init
:prepend-js ";if (typeof globalThis.navigator?.userAgent === 'undefined') { globalThis.navigator = {userAgent: ''}; };"}}}
:library
{:target :esm
:runtime :custom
:output-dir "target/library"
:devtools {:autoload false}
:modules
{:penpot
{:exports {BuilderError lib.builder/BuilderError
createBuildContext lib.builder/create-build-context
exportAsBytes lib.export/export-bytes
exportAsBlob lib.export/export-blob
exportStream lib.export/export-stream
}}}
:js-options
{:entry-keys ["module" "browser" "main"]
:export-conditions ["module" "import", "browser" "require" "default"]
;; :js-provider :import
;; :external-index "target/library/dependencies.js"
;; :external-index-format :esm
}
:compiler-options
{:output-feature-set :es2020
:output-wrapper false
:warnings {:fn-deprecated false}}
:release
{:compiler-options
{:fn-invoke-direct true
:optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :advanced]
:pretty-print false
:source-map true
:elide-asserts true
:anon-fn-naming-policy :off
:source-map-detail-level :all}}}}}

292
library/src/lib/builder.cljs Normal file
View file

@ -0,0 +1,292 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns lib.builder
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.files.builder :as fb]
[app.common.json :as json]
[app.common.schema :as sm]
[app.common.uuid :as uuid]
[app.util.object :as obj]))
(def BuilderError
(obj/class
:name "BuilderError"
:extends js/Error
:constructor
(fn [this type code hint cause]
(.call js/Error this hint)
(set! (.-name this) (str "Exception: " hint))
(set! (.-type this) type)
(set! (.-code this) code)
(set! (.-hint this) hint)
(when (exists? js/Error.captureStackTrace)
(.captureStackTrace js/Error this))
(obj/add-properties!
this
{:name "cause"
:enumerable true
:this false
:get (fn [] cause)}
{:name "explain"
:enumerable true
:this false
:get (fn []
(let [data (ex-data cause)]
(when-let [explain (::sm/explain data)]
(json/->js (sm/simplify explain)))))}))))
(defn- handle-exception
[cause]
(let [data (ex-data cause)]
(throw (new BuilderError
(d/name (get data :type :unknown))
(d/name (get data :code :unknown))
(or (get data :hint) (ex-message cause))
cause))))
(defn- decode-params
[params]
(if (obj/plain-object? params)
(json/->clj params)
params))
(defn- get-current-page-id
[state]
(dm/str (get state ::fb/current-page-id)))
(defn- get-last-id
[state]
(dm/str (get state ::fb/last-id)))
(defn- create-builder-api
[state]
(obj/reify {:name "BuildContext"}
:currentFileId
{:get #(dm/str (get @state ::fb/current-file-id))}
:currentFrameId
{:get #(dm/str (get @state ::fb/current-frame-id))}
:currentPageId
{:get #(get-current-page-id @state)}
:lastId
{:get #(get-last-id @state)}
:addFile
(fn [params]
(try
(let [params (-> params decode-params fb/decode-file)]
(-> (swap! state fb/add-file params)
(get ::fb/current-file-id)))
(catch :default cause
(handle-exception cause))))
:closeFile
(fn []
(swap! state fb/close-file)
nil)
:addPage
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-page))]
(-> (swap! state fb/add-page params)
(get-current-page-id)))
(catch :default cause
(handle-exception cause))))
:closePage
(fn []
(swap! state fb/close-page)
nil)
:addBoard
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :frame)
(fb/decode-shape))]
(-> (swap! state fb/add-board params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:closeBoard
(fn []
(swap! state fb/close-board)
nil)
:addGroup
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :group)
(fb/decode-shape))]
(-> (swap! state fb/add-group params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:closeGroup
(fn []
(swap! state fb/close-group)
nil)
:addBool
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-add-bool))]
(-> (swap! state fb/add-bool params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addRect
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :rect)
(fb/decode-shape))]
(-> (swap! state fb/add-shape params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addCircle
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :circle)
(fb/decode-shape))]
(-> (swap! state fb/add-shape params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addPath
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :path)
(fb/decode-shape))]
(-> (swap! state fb/add-shape params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addText
(fn [params]
(try
(let [params (-> (decode-params params)
(assoc :type :text)
(fb/decode-shape))]
(-> (swap! state fb/add-shape params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addLibraryColor
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-library-color)
(d/without-nils))]
(-> (swap! state fb/add-library-color params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addLibraryTypography
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-library-typography)
(d/without-nils))]
(-> (swap! state fb/add-library-typography params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addComponent
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-add-component))]
(-> (swap! state fb/add-component params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addComponentInstance
(fn [params]
(try
(let [params (-> (decode-params params)
(fb/decode-add-component-instance))]
(-> (swap! state fb/add-component-instance params)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:addFileMedia
(fn [params blob]
(when-not (instance? js/Blob blob)
(throw (BuilderError. "validation"
"invalid-media"
"only Blob instance are soported")))
(try
(let [blob (fb/map->BlobWrapper
{:size (.-size ^js blob)
:mtype (.-type ^js blob)
:blob blob})
params
(-> (decode-params params)
(fb/decode-add-file-media))]
(-> (swap! state fb/add-file-media params blob)
(get-last-id)))
(catch :default cause
(handle-exception cause))))
:getMediaAsImage
(fn [id]
(let [id (uuid/parse id)]
(when-let [fmedia (get-in @state [::fb/file-media id])]
(let [image {:id (get fmedia :id)
:width (get fmedia :width)
:height (get fmedia :height)
:name (get fmedia :name)
:mtype (get fmedia :mtype)}]
(json/->js (d/without-nils image))))))
:genId
(fn []
(dm/str (uuid/next)))
:getInternalState
(fn []
(json/->js @state))))
(defn create-build-context
"Create an empty builder state context."
[]
(let [state (atom {})
api (create-builder-api state)]
(specify! api
cljs.core/IDeref
(-deref [_] @state))))

214
library/src/lib/export.cljs Normal file
View file

@ -0,0 +1,214 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns lib.export
"A .penpot export implementation"
(:require
[app.common.data :as d]
[app.common.files.builder :as fb]
[app.common.json :as json]
[app.common.media :as media]
[app.common.schema :as sm]
[app.common.types.color :as types.color]
[app.common.types.component :as types.component]
[app.common.types.file :as types.file]
[app.common.types.page :as types.page]
[app.common.types.plugins :as ctpg]
[app.common.types.shape :as types.shape]
[app.common.types.tokens-lib :as types.tokens-lib]
[app.common.types.typography :as types.typography]
[app.util.zip :as zip]
[cuerdas.core :as str]
[promesa.core :as p]))
(def ^:private schema:file
[:merge
types.file/schema:file
[:map [:options {:optional true} types.file/schema:options]]])
(def ^:private encode-file
(sm/encoder schema:file sm/json-transformer))
(def ^:private encode-page
(sm/encoder types.page/schema:page sm/json-transformer))
(def ^:private encode-shape
(sm/encoder types.shape/schema:shape sm/json-transformer))
(def ^:private encode-component
(sm/encoder types.component/schema:component sm/json-transformer))
(def encode-color
(sm/encoder types.color/schema:color sm/json-transformer))
(def encode-typography
(sm/encoder types.typography/schema:typography sm/json-transformer))
(def encode-tokens-lib
(sm/encoder types.tokens-lib/schema:tokens-lib sm/json-transformer))
(def encode-plugin-data
(sm/encoder ::ctpg/plugin-data sm/json-transformer))
(def ^:private valid-buckets
#{"file-media-object"
"team-font-variant"
"file-object-thumbnail"
"file-thumbnail"
"profile"
"file-data"
"file-data-fragment"
"file-change"})
(def ^:private schema:storage-object
[:map {:title "StorageObject"}
[:id ::sm/uuid]
[:size ::sm/int]
[:content-type :string]
[:bucket [::sm/one-of {:format :string} valid-buckets]]
[:hash :string]])
(def encode-storage-object
(sm/encoder schema:storage-object sm/json-transformer))
(def ^:private file-attrs
#{:id
:name
:migrations
:features
:is-shared
:version})
(defn- generate-file-export-procs
[{:keys [id data] :as file}]
(cons
(let [file (cond-> (select-keys file file-attrs)
(:options data)
(assoc :options (:options data)))]
[(str "files/" id ".json")
(delay (-> file encode-file json/encode))])
(concat
(let [pages (get data :pages)
pages-index (get data :pages-index)]
(->> (d/enumerate pages)
(mapcat
(fn [[index page-id]]
(let [page (get pages-index page-id)
objects (:objects page)
page (-> page
(dissoc :objects)
(assoc :index index))]
(cons
[(str "files/" id "/pages/" page-id ".json")
(delay (-> page encode-page json/encode))]
(map (fn [[shape-id shape]]
(let [shape (assoc shape :page-id page-id)]
[(str "files/" id "/pages/" page-id "/" shape-id ".json")
(delay (-> shape encode-shape json/encode))]))
objects)))))))
(->> (get data :components)
(map (fn [[component-id component]]
[(str "files/" id "/components/" component-id ".json")
(delay (-> component encode-component json/encode))])))
(->> (get data :colors)
(map (fn [[color-id color]]
[(str "files/" id "/colors/" color-id ".json")
(delay (let [color (-> color
encode-color
(dissoc :file-id))]
(cond-> color
(and (contains? color :path)
(str/empty? (:path color)))
(dissoc :path)
:always
(json/encode))))])))
(->> (get data :typographies)
(map (fn [[typography-id typography]]
[(str "files/" id "/typographies/" typography-id ".json")
(delay (-> typography
encode-typography
json/encode))])))
(when-let [tokens-lib (get data :tokens-lib)]
(list [(str "files/" id "/tokens.json")
(delay (-> tokens-lib
encode-tokens-lib
json/encode))])))))
(defn- generate-files-export-procs
[state]
(->> (vals (get state ::fb/files))
(mapcat generate-file-export-procs)))
(defn- generate-media-export-procs
[state]
(->> (get state ::fb/file-media)
(mapcat (fn [[file-media-id file-media]]
(let [media-id (get file-media :media-id)
media (get-in state [::fb/media media-id])
blob (get-in state [::fb/blobs media-id])]
(list
[(str "objects/" media-id (media/mtype->extension (:content-type media)))
(delay (get blob :blob))]
[(str "objects/" media-id ".json")
(delay (-> media
;; FIXME: proper encode?
(json/encode)))]
[(str "files/" (:file-id file-media) "/media/" file-media-id ".json")
(delay (-> file-media
(dissoc :file-id)
(json/encode)))]))))))
(defn- generate-manifest-procs
[state]
(let [files (->> (get state ::fb/files)
(mapv (fn [[file-id file]]
{:id file-id
:name (:name file)
:features (:features file)})))
params {:type "penpot/export-files"
:version 1
;; FIXME: set proper placeholder for replacement on build
:generated-by "penpot-lib/develop"
:files files
:relations []}]
["manifest.json" (delay (json/encode params))]))
(defn- export
[state writer]
(->> (p/reduce (fn [writer [path data]]
(let [data (if (delay? data) (deref data) data)]
(js/console.log "export" path)
(->> (zip/add writer path data)
(p/fmap (constantly writer)))))
writer
(cons (generate-manifest-procs @state)
(concat
(generate-files-export-procs @state)
(generate-media-export-procs @state))))
(p/mcat (fn [writer]
(zip/close writer)))))
(defn export-bytes
[state]
(export state (zip/writer (zip/bytes-writer))))
(defn export-blob
[state]
(export state (zip/writer (zip/blob-writer))))
(defn export-stream
[state stream]
(export state (zip/writer stream)))

77
library/test/builder.test.js Normal file
View file

@ -0,0 +1,77 @@
import assert from "node:assert/strict";
import test from "node:test";
import * as penpot from "#self";
test("create empty context", () => {
const context = penpot.createBuildContext();
assert.ok(context);
});
test("create context with single file", () => {
const context = penpot.createBuildContext();
context.addFile({name: "sample"});
const internalState = context.getInternalState();
// console.log(internalState);
assert.ok(internalState.files);
assert.equal(typeof internalState.files, "object");
assert.equal(typeof internalState.currentFileId, "string");
const file = internalState.files[internalState.currentFileId];
assert.ok(file);
});
test("create context with two file", () => {
const context = penpot.createBuildContext();
const fileId_1 = context.addFile({name: "sample 1"});
const fileId_2 = context.addFile({name: "sample 2"});
const internalState = context.getInternalState();
// console.log(internalState.files[fileId_1])
assert.ok(internalState.files[fileId_1]);
assert.ok(internalState.files[fileId_2]);
assert.equal(internalState.files[fileId_1].name, "sample 1");
assert.equal(internalState.files[fileId_2].name, "sample 2");
const file = internalState.files[fileId_2];
assert.ok(file.data);
assert.ok(file.data.pages);
assert.ok(file.data.pagesIndex);
assert.equal(file.data.pages.length, 0)
});
test("create context with file and page", () => {
const context = penpot.createBuildContext();
const fileId = context.addFile({name: "file 1"});
const pageId = context.addPage({name: "page 1"});
const internalState = context.getInternalState();
const file = internalState.files[fileId];
assert.ok(file, "file should exist");
assert.ok(file.data);
assert.ok(file.data.pages);
assert.equal(file.data.pages.length, 1);
const page = file.data.pagesIndex[pageId];
assert.ok(page, "page should exist");
assert.ok(page.objects, "page objects should exist");
assert.equal(page.id, pageId);
const rootShape = page.objects["00000000-0000-0000-0000-000000000000"];
assert.ok(rootShape, "root shape should exist");
assert.equal(rootShape.id, "00000000-0000-0000-0000-000000000000");
});

1352
library/yarn.lock Normal file
View file

File diff suppressed because it is too large Load diff

7
package.json
View file

@ -11,13 +11,14 @@
},
"type": "module",
"scripts": {
"fmt:clj:check": "cljfmt check --parallel=true common/src/ common/test/ frontend/src/ frontend/test/ backend/src/ backend/test/ exporter/src/",
"fmt:clj": "cljfmt fix --parallel=true common/src/ common/test/ frontend/src/ frontend/test/ backend/src/ backend/test/ exporter/src/",
"fmt:clj:check": "cljfmt check --parallel=true common/src/ common/test/ frontend/src/ frontend/test/ backend/src/ backend/test/ exporter/src/ library/src",
"fmt:clj": "cljfmt fix --parallel=true common/src/ common/test/ frontend/src/ frontend/test/ backend/src/ backend/test/ exporter/src/ library/src",
"lint:clj:common": "clj-kondo --parallel=true --lint common/src",
"lint:clj:frontend": "clj-kondo --parallel=true --lint frontend/src",
"lint:clj:backend": "clj-kondo --parallel=true --lint backend/src",
"lint:clj:exporter": "clj-kondo --parallel=true --lint exporter/src",
"lint:clj": "yarn run lint:clj:common && yarn run lint:clj:frontend && yarn run lint:clj:backend && yarn run lint:clj:exporter"
"lint:clj:library": "clj-kondo --parallel=true --lint library/src",
"lint:clj": "yarn run lint:clj:common && yarn run lint:clj:frontend && yarn run lint:clj:backend && yarn run lint:clj:exporter && yarn run lint:clj:library"
},
"devDependencies": {
"@playwright/test": "^1.43.1",