Commit graph

439 commits

Author SHA1 Message Date
Caleb Doxsey
8ab0dcb45b
logs: add new log scrubber (#1346) 2020-08-31 08:12:08 -06:00
bobby
fbd8c8f294
deployment: add goimports with path awareness (#1316)
Plus fix some spelling

Signed-off-by: Bobby DeSimone <bobbydesimone@gmail.com>
2020-08-24 13:04:55 -07:00
Travis Groth
d81cfb6e99
pkg/storage/redis: update tests to use local certs + upstream image (#1306) 2020-08-20 12:44:15 -04:00
Caleb Doxsey
c4c8ef8e53
azure: support deriving credentials from client id, client secret and provider url (#1300) 2020-08-18 10:17:28 -06:00
Caleb Doxsey
a1378c81f8
cache: support databroker option changes (#1294) 2020-08-18 07:27:20 -06:00
Caleb Doxsey
d9a224a5e8
proxy: move properties to atomically updated state (#1280)
* authenticate: remove cookie options

* authenticate: remove shared key field

* authenticate: remove shared cipher property

* authenticate: move properties to separate state struct

* proxy: allow local state to be updated on configuration changes

* fix test

* return new connection

* use warn, collapse to single line

* address concerns, fix tests
2020-08-14 11:44:58 -06:00
Caleb Doxsey
d608526998
authenticate: move properties to atomically updated state (#1277)
* authenticate: remove cookie options

* authenticate: remove shared key field

* authenticate: remove shared cipher property

* authenticate: move properties to separate state struct
2020-08-14 07:53:11 -06:00
Cuong Manh Le
9af2226b5b pkg/storage/redis: use SANs cert
Since go1.15, X.509 CommonName is deprecated, switch to a SANs
certificate for test redis TLS.

While at it, add instruction to genearte cert and build test image.

See: https://golang.org/doc/go1.15#commonname
2020-08-12 22:20:50 +07:00
Caleb Doxsey
1285a9d91d
databroker: add support for config settings (#1253) 2020-08-11 07:50:19 -06:00
Cuong Manh Le
ec52412d79
pkg/storage: make Watch returns receive only channel (#1211)
So the caller can not write to the channel, and insist that the channel
is for notifying only.
2020-08-05 23:49:28 +07:00
Cuong Manh Le
f4cb5ea6e9
pkg/storage/redis: fix multiple data race (#1210)
There are two data race in current code:

 - Call to close pub sub conn and renew pub sub conn
 - Call to close notify channel and send data to it

Fixing them by:

 - Moving pub sub conn creation/renew in the doNotifyLoop
 - Add a lock to guard before close/send data to channel, and also add
 another check for context was done at the beginning of notify loop.

Verifying by running:

	for _ in {1..100}; do
	  go test -race -count=1 ./pkg/storage/redis/...
	done

with no failure.
2020-08-05 22:35:14 +07:00
Travis Groth
3f9a5f8c32
pkg/storage/redis: metrics updates (#1195)
* pkg/storage/redis: add metric to subscribe

* pkg/storage/redis: fix pool stats nil pointer
2020-08-05 07:16:53 -04:00
Cuong Manh Le
fabf773e8b
pkg/storage/redis: handling connection to redis backend failure (#1174)
In case of connection to backend failure, we try reconnecting with
backoff, to re-subscribe to the notification channel.

Fixes #1167
2020-07-31 22:17:11 +07:00
Cuong Manh Le
bc61206b78
pkg/storage/redis: add redis TLS support (#1163)
Fixes #1156
2020-07-31 19:37:23 +07:00
Travis Groth
aab9ec413e
fix lint errors (#1171) 2020-07-31 00:00:06 -04:00
Travis Groth
3c4513a91e
telmetry: add databroker storage metrics and tracing (#1161)
* telmetry: add databroker storage metrics and tracing
2020-07-30 18:19:23 -04:00
Caleb Doxsey
29fb96a955
databroker: add encryption for records (#1168) 2020-07-30 14:04:31 -06:00
Caleb Doxsey
97f85481f8
fix redirect loop, remove user/session services, remove duplicate deleted_at fields (#1162)
* fix redirect loop, remove user/session services, remove duplicate deleted_at fields

* change loop

* reuse err variable

* wrap errors, use cookie timeout

* wrap error, duplicate if
2020-07-30 09:41:57 -06:00
Caleb Doxsey
714363fb07
grpcutil: add functions for JWTs in gRPC metadata (#1165) 2020-07-30 09:40:55 -06:00
Cuong Manh Le
3039407597
pkg/storage/redis: add authentication support (#1159)
Fixes #1157
2020-07-29 23:08:38 +07:00
Cuong Manh Le
05545b3e1d
pkg/storage/redis: do not use timeout to signal redis conn to stop (#1155)
Instead, we run the loop in goroutine, and when context was done,
closing the underlying connection of PubSubConn, so the Receive will
return.

Fixes #1154
2020-07-29 21:48:50 +07:00
Miguel
72b6347886
docs: Add required in cookie_secret (#1142) 2020-07-27 22:59:54 +07:00
Cuong Manh Le
a7bd2caae9
pkg/storage: introduce storage.Backend Watch method (#1135)
Currently, we're doing "sync" in databroker server. If we're going to
support multiple databroker servers instance, this mechanism won't work.

This commit moves the "sync" to storage backend, by adding new Watch
method. The Watch method will return a channel for the caller. Everytime
something happens inside the storage, we notify the caller by sending a
message to this channel.
2020-07-27 21:10:47 +07:00
Cuong Manh Le
a8fd23a2be
pkg/storage/redis: move last version to redis (#1134)
So we can support multiple databroker servers, we can't do it if we
store last version inside Server struct.
2020-07-24 11:27:11 +07:00
Cuong Manh Le
aedfbc4c71
pkg/storage: change backend interface to return error (#1131)
Since when storage backend like redis can be fault in many cases, the
interface should return error for the caller to handle.
2020-07-24 09:02:37 +07:00
Cuong Manh Le
1640151bc1
databroker server backend config (#1127)
* config,docs: add databroker storage backend configuration

* cache: allow configuring which backend storage to use

Currently supported types are "memory", "redis".
2020-07-23 10:42:43 +07:00
Caleb Doxsey
1ad243dfd1
directory.Group entry for groups (#1118)
* store directory groups separate from directory users

* fix group lookup, azure display name

* remove fields restriction

* fix test

* also support email

* use Email as name for google'

* remove changed file

* show groups on dashboard

* fix test

* re-add accidentally removed code
2020-07-22 11:28:53 -06:00
Caleb Doxsey
504197d83b
custom rego in databroker (#1124)
* add support for sub policies

* add support for sub policies

* update authz rego policy to support sub policies
2020-07-22 10:44:05 -06:00
Cuong Manh Le
99785cbb5b
internal/databroker: store server version (#1121)
Storing server version when creating new server. After then, we can
retrieve the version from backend when server restart.

With storage backend which supports persistent, the server version
won't change after restarting.
2020-07-22 03:50:22 +07:00
Cuong Manh Le
26f099b49d
redis storage backend (#1082)
* pkg/storage: add redis storage backend

* pkg/storage/redis: set record create time correctly

* pkg/storage/redis: add docs

* pkg/storage/redis: run test with redis tag only

* pkg/storage/redis: use localhost

* pkg/storage/redis: use 127.0.0.1

* pkg/storage/redis: honor REDIS_URL env

* .github/workflows: add missing config for redis service

* .github/workflows: map redis ports to host

* pkg/storage/redis: use proto marshaler instead of json one

* pkg/storage/redis: use better implementation

By using redis supported datastructure:

 - Hash for storing record
 - Sorted set for storing by version
 - Set for storing deleted ids

List operation will be now performed in O(log(N)+M) instead of O(N) like
previous implementation.

* pkg/storage/redis: add tx to wrap redis transaction

* pkg/storage/redis: set record type in New

* pkg/storage/redis: make sure tx commands appear in right order

* pkg/storage/redis: make deletePermanentAfter as argument

* pkg/storage/redis: make sure version is incremented when deleting

* pkg/storage/redis: fix linter

* pkg/storage/redis: fix cmd construction
2020-07-22 03:07:20 +07:00
bobby
e85226b609
grpc: use relative paths in codegen (#1106)
Signed-off-by: Bobby DeSimone <bobbydesimone@gmail.com>
2020-07-20 06:40:11 -07:00
Caleb Doxsey
b79e73b8b8
config: add support for policies stored in the databroker (#1099)
* wip

* always use databroker config source

* add test

* valid policy, remove debug lines
2020-07-17 10:35:29 -06:00
Cuong Manh Le
a5db94434d
pkg/storage: add package docs (#1078) 2020-07-15 10:30:37 +07:00
Cuong Manh Le
2f84dd2aff
Add storage backend interface (#1072)
* pkg: add storage package

Which contains storage.Backend interface to initial support for multiple
backend storage.

* pkg/storage: add inmemory storage

* internal/databroker: use storage.Backend interface

Instead of implementing multiple databroker server implementation for
each kind of storage backend, we use only one databroker server
implementation, which is supported multiple storage backends, which
satisfy storage.Backend interface.
2020-07-15 09:42:01 +07:00
Cuong Manh Le
6f3817aee5
pkg/grpc: fix wrong audit protoc gen file (#1048) 2020-07-08 22:57:12 +07:00
Caleb Doxsey
73105c0b08
audit: add protobuf definitions (#1047) 2020-07-08 07:23:03 -06:00
Caleb Doxsey
09621ee263
pkg: add grpcutil package (#1032) 2020-07-01 15:21:19 -06:00
Caleb Doxsey
fae02791f5
cryptutil: move to pkg dir, add token generator (#1029)
* cryptutil: move to pkg dir, add token generator

* add gitignored files

* add tests
2020-06-30 15:55:33 -06:00
Caleb Doxsey
091b71f12e
grpc: rename internal/grpc to pkg/grpc (#1010)
* grpc: rename internal/grpc to pkg/grpc

* don't ignore pkg dir

* remove debug line
2020-06-26 09:17:02 -06:00