pomerium/pkg
Caleb Doxsey ff607fa018
databroker: add sync-cache (#5639)
## Summary
Add a new `SyncCache`:

```go
type SyncCache interface {
	// Clear deletes all the data for the given record type in the sync cache.
	Clear(recordType string) error
	// Records yields the databroker records stored in the cache.
	Records(recordType string) iter.Seq2[*Record, error]
	// Sync syncs the cache with the databroker.
	Sync(ctx context.Context, client DataBrokerServiceClient, recordType string) error
}
```

The cache maintains databroker records in a local pebble database (which
could be on-disk or in-memory). The way it's used is you first call
`.Sync(ctx, client, recordType)` and then `.Records(recordType)`, which
returns an iterator over all the records.

Internally we store the databroker records in a pebble key-value
database. Pebble was chosen because its fast and well-tested, but any
ordered key-value store would work. The first time we call `SyncLatest`
to retrieve all the records. Each subsequent time we call `Sync` with
the current server and record versions to retrieve only the changes.
This is significantly more efficient than calling `SyncLatest` every
time.

The primary use for this is in the enterprise-console as part of
directory sync to improve performance with large datasets.

## Related issues
-
[ENG-2401](https://linear.app/pomerium/issue/ENG-2401/enterprise-console-improve-performance-of-directory-sync-using-cached)

## Checklist

- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review

---------

Co-authored-by: Denis Mishin <dmishin@pomerium.com>
2025-06-02 13:42:30 -06:00
..
authenticateapi authorize: support authenticating with idp tokens (#5484) 2025-02-18 13:02:06 -07:00
cmd/pomerium logging: standardize on hyphens in attribute names (#5577) 2025-04-22 10:57:19 -07:00
contextutil core/lint: upgrade golangci-lint, replace interface{} with any (#5099) 2024-05-02 14:33:52 -06:00
counter upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
cryptutil chore(deps): bump the go group across 1 directory with 26 updates (#5385) 2024-12-02 15:24:28 -07:00
derivecert cryptutil: generate certificates from deriveca (#3992) 2023-02-23 08:38:56 -07:00
encoding/base58 core/ci: update linting (#4844) 2023-12-14 09:07:54 -08:00
envoy cleanup logs (#5571) 2025-04-14 08:20:10 -06:00
fanout Fix many instances of contexts and loggers not being propagated (#5340) 2024-10-25 14:50:56 -04:00
grpc databroker: add sync-cache (#5639) 2025-06-02 13:42:30 -06:00
grpcutil upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
health config: add support for http3 advertise port (#5466) 2025-02-03 13:58:57 -07:00
hpke Fix typo in Seal comment (#5620) 2025-05-18 18:56:59 -07:00
httputil core/lint: upgrade golangci-lint, replace interface{} with any (#5099) 2024-05-02 14:33:52 -06:00
identity identity: add access token support for github (#5615) 2025-05-13 10:59:47 -06:00
logutil protoutil: add NewAny method for deterministic serialization (#2462) 2021-08-09 17:51:57 -06:00
metrics Envoy resource monitoring & overload manager configuration (#5106) 2024-05-28 16:57:09 -04:00
netutil netutil: improve port allocation (#5485) 2025-02-19 09:45:21 -07:00
pebbleutil databroker: add sync-cache (#5639) 2025-06-02 13:42:30 -06:00
policy chore(deps): bump the go group with 39 updates (#5559) 2025-04-04 16:26:51 -07:00
protoutil upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
slices integration: check for profile cookies (#4847) 2023-12-12 10:07:13 -08:00
storage add metrics for cache (#5627) 2025-05-28 09:49:30 -06:00
telemetry storage: add sync querier (#5570) 2025-04-23 10:15:48 -06:00
webauthnutil upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
zero replace xxhash with xxh3 (#5457) 2025-01-31 08:44:08 -07:00