mirror of
https://github.com/pomerium/pomerium.git
synced 2025-07-13 23:08:17 +02:00
ff607fa018
## Summary
Add a new `SyncCache`:
```go
type SyncCache interface {
// Clear deletes all the data for the given record type in the sync cache.
Clear(recordType string) error
// Records yields the databroker records stored in the cache.
Records(recordType string) iter.Seq2[*Record, error]
// Sync syncs the cache with the databroker.
Sync(ctx context.Context, client DataBrokerServiceClient, recordType string) error
}
```
The cache maintains databroker records in a local pebble database (which
could be on-disk or in-memory). The way it's used is you first call
`.Sync(ctx, client, recordType)` and then `.Records(recordType)`, which
returns an iterator over all the records.
Internally we store the databroker records in a pebble key-value
database. Pebble was chosen because its fast and well-tested, but any
ordered key-value store would work. The first time we call `SyncLatest`
to retrieve all the records. Each subsequent time we call `Sync` with
the current server and record versions to retrieve only the changes.
This is significantly more efficient than calling `SyncLatest` every
time.
The primary use for this is in the enterprise-console as part of
directory sync to improve performance with large datasets.
## Related issues
-
[ENG-2401](https://linear.app/pomerium/issue/ENG-2401/enterprise-console-improve-performance-of-directory-sync-using-cached)
## Checklist
- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review
---------
Co-authored-by: Denis Mishin <dmishin@pomerium.com>
|
||
|---|---|---|
| .. | ||
| mock_databroker | ||
| changeset.go | ||
| changeset_test.go | ||
| databroker.go | ||
| databroker.pb.go | ||
| databroker.proto | ||
| databroker_grpc.pb.go | ||
| databroker_test.go | ||
| fast_forward.go | ||
| fast_forward_test.go | ||
| leaser.go | ||
| leaser_test.go | ||
| reconciler.go | ||
| recordset.go | ||
| recordset_test.go | ||
| sync.go | ||
| sync_cache.go | ||
| sync_cache_test.go | ||
| sync_test.go | ||
| syncer.go | ||
| syncer_test.go | ||