* authorize: add databroker server and record version to result, force sync via polling
* wrap inmem store to take read lock when grabbing databroker versions
* address code review comments
* reset max to 0
* refactor backend, implement encrypted store
* refactor in-memory store
* wip
* wip
* wip
* add syncer test
* fix redis expiry
* fix linting issues
* fix test by skipping non-config records
* fix backoff import
* fix init issues
* fix query
* wait for initial sync before starting directory sync
* add type to SyncLatest
* add more log messages, fix deadlock in in-memory store, always return server version from SyncLatest
* update sync types and tests
* add redis tests
* skip macos in github actions
* add comments to proto
* split getBackend into separate methods
* handle errors in initVersion
* return different error for not found vs other errors in get
* use exponential backoff for redis transaction retry
* rename raw to result
* use context instead of close channel
* store type urls as constants in databroker
* use timestampb instead of ptypes
* fix group merging not waiting
* change locked names
* update GetAll to return latest record version
* add method to grpcutil to get the type url for a protobuf type
- gofumpt everything
- fix TLS MinVersion to be at least 1.2
- add octal syntax
- remove newlines
- fix potential decompression bomb in ecjson
- remove implicit memory aliasing in for loops.
Signed-off-by: Bobby DeSimone <bobbydesimone@gmail.com>
* replace GetAllPages with InitialSync, improve merge performance
* fmt proto
* add test for base64 function
* add sync test
* go mod tidy
Co-authored-by: Bobby DeSimone <bobbydesimone@gmail.com>
* internal/databroker: make Sync send data in smaller batches
GRPC streaming is better at sending multiple smaller message instead of
a big one.
Benchmark result for sending 10k messages at once vs multiple batches,
each with 100 messages:
name old time/op new time/op delta
Sync-12 14.5ms ± 3% 12.4ms ± 2% -14.40% (p=0.000 n=10+9)
* cache: add test for databroker sync
Currently, we're doing "sync" in databroker server. If we're going to
support multiple databroker servers instance, this mechanism won't work.
This commit moves the "sync" to storage backend, by adding new Watch
method. The Watch method will return a channel for the caller. Everytime
something happens inside the storage, we notify the caller by sending a
message to this channel.
* config,docs: add databroker storage backend configuration
* cache: allow configuring which backend storage to use
Currently supported types are "memory", "redis".
Storing server version when creating new server. After then, we can
retrieve the version from backend when server restart.
With storage backend which supports persistent, the server version
won't change after restarting.
* pkg: add storage package
Which contains storage.Backend interface to initial support for multiple
backend storage.
* pkg/storage: add inmemory storage
* internal/databroker: use storage.Backend interface
Instead of implementing multiple databroker server implementation for
each kind of storage backend, we use only one databroker server
implementation, which is supported multiple storage backends, which
satisfy storage.Backend interface.
* databroker: add databroker, identity manager, update cache (#864)
* databroker: add databroker, identity manager, update cache
* fix cache tests
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* authorize: use databroker data for rego policy (#904)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix gitlab test
* use v4 backoff
* authenticate: databroker changes (#914)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix dashboard
* delete session on logout
* permanently delete sessions once they are marked as deleted
* remove permanent delete
* fix tests
* remove groups and refresh test
* databroker: remove dead code, rename cache url, move dashboard (#925)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix dashboard
* delete session on logout
* permanently delete sessions once they are marked as deleted
* remove permanent delete
* fix tests
* remove cache service
* remove kv
* remove refresh docs
* remove obsolete cache docs
* add databroker url option
* cache: use memberlist to detect multiple instances
* add databroker service url
* remove cache service
* remove kv
* remove refresh docs
* remove obsolete cache docs
* add databroker url option
* cache: use memberlist to detect multiple instances
* add databroker service url
* wip
* remove groups and refresh test
* fix redirect, signout
* remove databroker client from proxy
* remove unused method
* remove user dashboard test
* handle missing session ids
* session: reject sessions with no id
* sessions: invalidate old sessions via databroker server version (#930)
* session: add a version field tied to the databroker server version that can be used to invalidate sessions
* fix tests
* add log
* authenticate: create user record immediately, call "get" directly in authorize (#931)
