Pomerium is an identity and context-aware access proxy.
Find a file
Caleb Doxsey ff607fa018
databroker: add sync-cache (#5639)
## Summary
Add a new `SyncCache`:

```go
type SyncCache interface {
	// Clear deletes all the data for the given record type in the sync cache.
	Clear(recordType string) error
	// Records yields the databroker records stored in the cache.
	Records(recordType string) iter.Seq2[*Record, error]
	// Sync syncs the cache with the databroker.
	Sync(ctx context.Context, client DataBrokerServiceClient, recordType string) error
}
```

The cache maintains databroker records in a local pebble database (which
could be on-disk or in-memory). The way it's used is you first call
`.Sync(ctx, client, recordType)` and then `.Records(recordType)`, which
returns an iterator over all the records.

Internally we store the databroker records in a pebble key-value
database. Pebble was chosen because its fast and well-tested, but any
ordered key-value store would work. The first time we call `SyncLatest`
to retrieve all the records. Each subsequent time we call `Sync` with
the current server and record versions to retrieve only the changes.
This is significantly more efficient than calling `SyncLatest` every
time.

The primary use for this is in the enterprise-console as part of
directory sync to improve performance with large datasets.

## Related issues
-
[ENG-2401](https://linear.app/pomerium/issue/ENG-2401/enterprise-console-improve-performance-of-directory-sync-using-cached)

## Checklist

- [x] reference any related issues
- [x] updated unit tests
- [x] add appropriate label (`enhancement`, `bug`, `breaking`,
`dependencies`, `ci`)
- [x] ready for review

---------

Co-authored-by: Denis Mishin <dmishin@pomerium.com>
2025-06-02 13:42:30 -06:00
.github chore(deps): bump the docker group in /.github with 3 updates (#5603) 2025-05-01 11:54:12 -06:00
.vscode use tlsClientConfig instead of custom dialer (#3830) 2022-12-27 09:55:36 -07:00
authenticate core: more metrics (#5629) 2025-05-29 09:34:41 -06:00
authorize fix metric to use milliseconds (#5632) 2025-06-02 09:36:10 -06:00
changelogs add v0.29.0 release notes (#5515) 2025-04-08 11:34:20 -04:00
cmd/pomerium cleanup logs (#5571) 2025-04-14 08:20:10 -06:00
config core: more metrics (#5629) 2025-05-29 09:34:41 -06:00
databroker logging: standardize on hyphens in attribute names (#5577) 2025-04-22 10:57:19 -07:00
examples ci: address Dockerfile warnings (#5283) 2024-09-11 16:56:47 -07:00
integration upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
internal databroker: add a wait field to sync request (#5630) 2025-05-29 12:50:14 -06:00
k8s/zero zero: set k8s deployment to follow :latest tag (#5313) 2024-10-03 11:39:55 -07:00
ospkg move directory providers (#3633) 2022-11-03 11:33:56 -06:00
pkg databroker: add sync-cache (#5639) 2025-06-02 13:42:30 -06:00
proxy config: use stable route ids for authorize matching and order xds responses (#5618) 2025-05-19 10:52:15 -06:00
scripts proto: fix dependencies (#5450) 2025-01-27 14:41:08 -07:00
ui chore(deps): bump @babel/runtime from 7.24.4 to 7.26.10 in /ui (#5522) 2025-04-11 08:02:05 -06:00
.codecov.yml development: change codecov precision 2019-07-18 16:49:37 -07:00
.dockerignore frontend: react+mui (#3004) 2022-02-07 08:47:58 -07:00
.fossa.yml rm cli code (#2824) 2021-12-15 16:25:21 -05:00
.gitattributes assets: use embed instead of statik (#1960) 2021-03-03 18:56:55 -07:00
.gitignore core/ui: improve frontend build size (#5109) 2024-05-09 07:10:00 -06:00
.golangci.yml upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
.pre-commit-config.yaml integration: add single-cluster integration tests (#2516) 2021-08-24 15:35:05 -06:00
3RD-PARTY dependencies: vendor base58, remove shortuuid (#2739) 2021-11-02 09:23:15 -06:00
DEBUG.MD deplyoment: add debug build / container / docs (#1513) 2020-10-13 16:54:21 -04:00
Dockerfile chore(deps): bump the docker group with 2 updates (#5597) 2025-05-01 11:00:58 -06:00
Dockerfile.debug chore(deps): bump the docker group with 2 updates (#5597) 2025-05-01 11:00:58 -06:00
go.mod databroker: add sync-cache (#5639) 2025-06-02 13:42:30 -06:00
go.sum databroker: add sync-cache (#5639) 2025-06-02 13:42:30 -06:00
LICENSE initial release 2019-01-02 12:13:36 -08:00
Makefile upgrade to go v1.24 (#5562) 2025-04-02 15:53:09 -06:00
pomerium.go fix go get, improve redis test (#2450) 2021-08-06 12:07:20 -06:00
README.md core/ui: update logo (#5249) 2024-09-05 18:13:06 +02:00
RELEASING.md deployment: update RELEASING.md (#3503) 2022-08-16 10:40:03 -07:00
SECURITY.md Fix SECURITY.md treated as symlink (#5211) 2024-08-07 17:20:18 -04:00

pomerium logo

Go Report Card GoDoc LICENSE Docker Pulls

Pomerium builds secure, clientless connections to internal web apps and services without a corporate VPN.

Pomerium is:

Its not a VPN alternative its the trusted, foolproof way to protect your business. Want a hosted control plane and management GUI? Give Pomerium Zero a try today!

Docs

For comprehensive docs, and tutorials see our documentation.

Contributing

See Contributing for information on how you can contribute to Pomerium.