* internal/databroker: make Sync send data in smaller batches
GRPC streaming is better at sending multiple smaller message instead of
a big one.
Benchmark result for sending 10k messages at once vs multiple batches,
each with 100 messages:
name old time/op new time/op delta
Sync-12 14.5ms ± 3% 12.4ms ± 2% -14.40% (p=0.000 n=10+9)
* cache: add test for databroker sync
Currently, we're doing "sync" in databroker server. If we're going to
support multiple databroker servers instance, this mechanism won't work.
This commit moves the "sync" to storage backend, by adding new Watch
method. The Watch method will return a channel for the caller. Everytime
something happens inside the storage, we notify the caller by sending a
message to this channel.
* config,docs: add databroker storage backend configuration
* cache: allow configuring which backend storage to use
Currently supported types are "memory", "redis".
Storing server version when creating new server. After then, we can
retrieve the version from backend when server restart.
With storage backend which supports persistent, the server version
won't change after restarting.
* pkg: add storage package
Which contains storage.Backend interface to initial support for multiple
backend storage.
* pkg/storage: add inmemory storage
* internal/databroker: use storage.Backend interface
Instead of implementing multiple databroker server implementation for
each kind of storage backend, we use only one databroker server
implementation, which is supported multiple storage backends, which
satisfy storage.Backend interface.
* databroker: add databroker, identity manager, update cache (#864)
* databroker: add databroker, identity manager, update cache
* fix cache tests
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* authorize: use databroker data for rego policy (#904)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix gitlab test
* use v4 backoff
* authenticate: databroker changes (#914)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix dashboard
* delete session on logout
* permanently delete sessions once they are marked as deleted
* remove permanent delete
* fix tests
* remove groups and refresh test
* databroker: remove dead code, rename cache url, move dashboard (#925)
* wip
* add directory provider
* initialize before sync, upate google provider, remove dead code
* fix flaky test
* update authorize to use databroker data
* implement signed jwt
* wait for session and user to appear
* fix test
* directory service (#885)
* directory: add google and okta
* add onelogin
* add directory provider
* initialize before sync, upate google provider, remove dead code
* add azure provider
* fix azure provider
* fix gitlab
* add gitlab test, fix azure test
* hook up okta
* remove dead code
* fix tests
* fix flaky test
* remove log line
* only redirect when no session id exists
* prepare rego query as part of create
* return on ctx done
* retry on disconnect for sync
* move jwt signing
* use !=
* use parent ctx for wait
* remove session state, remove logs
* rename function
* add log message
* pre-allocate slice
* use errgroup
* return nil on eof for sync
* move check
* disable timeout on gRPC requests in envoy
* fix dashboard
* delete session on logout
* permanently delete sessions once they are marked as deleted
* remove permanent delete
* fix tests
* remove cache service
* remove kv
* remove refresh docs
* remove obsolete cache docs
* add databroker url option
* cache: use memberlist to detect multiple instances
* add databroker service url
* remove cache service
* remove kv
* remove refresh docs
* remove obsolete cache docs
* add databroker url option
* cache: use memberlist to detect multiple instances
* add databroker service url
* wip
* remove groups and refresh test
* fix redirect, signout
* remove databroker client from proxy
* remove unused method
* remove user dashboard test
* handle missing session ids
* session: reject sessions with no id
* sessions: invalidate old sessions via databroker server version (#930)
* session: add a version field tied to the databroker server version that can be used to invalidate sessions
* fix tests
* add log
* authenticate: create user record immediately, call "get" directly in authorize (#931)
