diff --git a/docs/configuration/readme.md b/docs/configuration/readme.md index 54ff4081a..572cc31b7 100644 --- a/docs/configuration/readme.md +++ b/docs/configuration/readme.md @@ -1,7 +1,7 @@ --- title: Settings lang: en-US -sidebarDepth: 1 +sidebarDepth: 2 meta: - name: keywords content: configuration options settings pomerium @@ -24,16 +24,6 @@ Pomerium will automatically reload the configuration file if it is changed. At t These are configuration variables shared by all services, in all service modes. -### Service Mode - -- Environmental Variable: `SERVICES` -- Config File Key: `services` -- Type: `string` -- Default: `all` -- Options: `all` `authenticate` `authorize` `cache` or `proxy` - -Service mode sets the pomerium service(s) to run. If testing, you may want to set to `all` and run pomerium in "all-in-one mode." In production, you'll likely want to spin up several instances of each service mode for high availability. - ### Address - Environmental Variable: `ADDRESS` @@ -54,89 +44,6 @@ Address specifies the host and port to serve HTTP requests from. If empty, `:443 Administrative users are [super user](https://en.wikipedia.org/wiki/Superuser) that can sign in as another user or group. User impersonation allows administrators to temporarily impersonate a different user. -### Shared Secret - -- Environmental Variable: `SHARED_SECRET` -- Config File Key: `shared_secret` -- Type: [base64 encoded] `string` -- Required - -Shared Secret is the base64 encoded 256-bit key used to mutually authenticate requests between services. It's critical that secret keys are random, and stored safely. Use a key management system or `/dev/urandom/` to generate a key. For example: - -``` -head -c32 /dev/urandom | base64 -``` - -### Debug - -- Environmental Variable: `POMERIUM_DEBUG` -- Config File Key: `pomerium_debug` -- Type: `bool` -- Default: `false` - -::: danger - -Enabling the debug flag will result in sensitive information being logged!!! - -::: - -By default, JSON encoded logs are produced. Debug enables colored, human-readable logs to be streamed to [standard out](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)>>>). In production, it's recommended to be set to `false`. - -For example, if `true` - -``` -10:37AM INF cmd/pomerium version=v0.0.1-dirty+ede4124 -10:37AM INF proxy: new route from=httpbin.corp.beyondperimeter.com to=https://httpbin.org -10:37AM INF proxy: new route from=ssl.corp.beyondperimeter.com to=http://neverssl.com -10:37AM INF proxy/authenticator: grpc connection OverrideCertificateName= addr=auth.corp.beyondperimeter.com:443 -``` - -If `false` - -``` -{"level":"info","version":"v0.0.1-dirty+ede4124","time":"2019-02-18T10:41:03-08:00","message":"cmd/pomerium"} -{"level":"info","from":"httpbin.corp.beyondperimeter.com","to":"https://httpbin.org","time":"2019-02-18T10:41:03-08:00","message":"proxy: new route"} -{"level":"info","from":"ssl.corp.beyondperimeter.com","to":"http://neverssl.com","time":"2019-02-18T10:41:03-08:00","message":"proxy: new route"} -{"level":"info","OverrideCertificateName":"","addr":"auth.corp.beyondperimeter.com:443","time":"2019-02-18T10:41:03-08:00","message":"proxy/authenticator: grpc connection"} -``` - -### Log Level - -- Environmental Variable: `LOG_LEVEL` -- Config File Key: `log_level` -- Type: `string` -- Options: `debug` `info` `warn` `error` -- Default: `debug` - -Log level sets the global logging level for pomerium. Only logs of the desired level and above will be logged. - -### Proxy Log Level - -- Environmental Variable: `PROXY_LOG_LEVEL` -- Config File Key: `proxy_log_level` -- Type: `string` -- Options: `debug` `info` `warn` `error` -- Default: value of `log_level` or `debug` if both are unset - -Log level sets the logging level for the pomerium proxy service. Only logs of the desired level and above will be logged. - -### Insecure Server - -- Environmental Variable: `INSECURE_SERVER` -- Config File Key: `insecure_server` -- Type: `bool` -- Required if certificates unset - -Turning on insecure server mode will result in pomerium starting, and operating without any protocol encryption in transit. - -This setting can be useful in a situation where you have Pomerium behind a TLS terminating ingress or proxy. However, even in that case, it is highly recommended to use TLS to protect the confidentiality and integrity of service communication even behind the ingress using self-signed certificates or an internal CA. Please see our helm-chart for an example of just that. - -:::warning - -Pomerium should _never_ be exposed to the internet without TLS encryption. - -::: - ### Autocert - Environmental Variable: `AUTOCERT` @@ -218,83 +125,6 @@ certificates: The Client Certificate Authority is the x509 _public-key_ used to validate [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) client certificates. If not set, no client certificate will be required. -### Global Timeouts - -- Environmental Variables: `TIMEOUT_READ` `TIMEOUT_WRITE` `TIMEOUT_IDLE` -- Config File Key: `timeout_read` `timeout_write` `timeout_idle` -- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` -- Example: `TIMEOUT_READ=30s` -- Defaults: `TIMEOUT_READ=30s` `TIMEOUT_WRITE=0` `TIMEOUT_IDLE=5m` - -Timeouts set the global server timeouts. For route-specific timeouts, see [policy](./#policy). - -![cloudflare blog on timeouts](https://blog.cloudflare.com/content/images/2016/06/Timeouts-001.png) - -> For a deep dive on timeout values see [these](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) [two](https://blog.cloudflare.com/exposing-go-on-the-internet/) excellent blog posts. - -### GRPC Options - -These settings control upstream connections to the Authorize service. - -#### GRPC Address - -- Environmental Variable: `GRPC_ADDRESS` -- Config File Key: `grpc_address` -- Type: `string` -- Example: `:443`, `:8443` -- Default: `:443` or `:5443` if in all-in-one mode - -Address specifies the host and port to serve GRPC requests from. Defaults to `:443` (or `:5443` in all in one mode). - -#### GRPC Insecure - -- Environmental Variable: `GRPC_INSECURE` -- Config File Key: `grpc_insecure` -- Type: `bool` -- Default: `:443` (or `:5443` if in all-in-one mode) - -If set, GRPC Insecure disables transport security for communication between the proxy and authorize components. If running in all-in-one mode, defaults to true as communication will run over localhost's own socket. - -#### GRPC Client Timeout - -Maximum time before canceling an upstream RPC request. During transient failures, the proxy will retry upstreams for this duration, if possible. You should leave this high enough to handle backend service restart and rediscovery so that client requests do not fail. - -- Environmental Variable: `GRPC_CLIENT_TIMEOUT` -- Config File Key: `grpc_client_timeout` -- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` -- Default: `10s` - -#### GRPC Client DNS RoundRobin - -Enable grpc DNS based round robin load balancing. This method uses DNS to resolve endpoints and does client side load balancing of _all_ addresses returned by the DNS record. Do not disable unless you have a specific use case. - -- Environmental Variable: `GRPC_CLIENT_DNS_ROUNDROBIN` -- Config File Key: `grpc_client_dns_roundrobin` -- Type: `bool` -- Default: `true` - -#### GRPC Server Max Connection Age - -Set max connection age for GRPC servers. After this interval, servers ask clients to reconnect and perform any rediscovery for new/updated endpoints from DNS. - -See for details - -- Environmental Variable: `GRPC_SERVER_MAX_CONNECTION_AGE` -- Config File Key: `grpc_server_max_connection_age` -- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` -- Default: `5m` - -#### GRPC Server Max Connection Age Grace - -Additive period with `grpc_server_max_connection_age`, after which servers will force connections to close. - -See for details - -- Environmental Variable: `GRPC_SERVER_MAX_CONNECTION_AGE_GRACE` -- Config File Key: `grpc_server_max_connection_age_grace` -- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` -- Default: `5m` - ### Cookie options These settings control the Pomerium session cookies sent to users's browsers. @@ -365,135 +195,38 @@ Setting this to false enables hostile javascript to steal session cookies and im Sets the lifetime of session cookies. After this interval, users will be forced to go through the OAuth login flow again to get a new cookie. -### HTTP Redirect Address +### Debug -- Environmental Variable: `HTTP_REDIRECT_ADDR` -- Config File Key: `http_redirect_addr` -- Type: `string` -- Example: `:80`, `:8080` -- Optional +- Environmental Variable: `POMERIUM_DEBUG` +- Config File Key: `pomerium_debug` +- Type: `bool` +- Default: `false` -If set, the HTTP Redirect Address specifies the host and port to redirect http to https traffic on. If unset, no redirect server is started. +::: danger -### Metrics Address - -- Environmental Variable: `METRICS_ADDRESS` -- Config File Key: `metrics_address` -- Type: `string` -- Example: `:9090`, `127.0.0.1:9090` -- Default: `disabled` -- Optional - -Expose a prometheus format HTTP endpoint on the specified port. Disabled by default. - -:::warning - -**Use with caution:** the endpoint can expose frontend and backend server names or addresses. Do not externally expose the metrics if this is sensitive information. +Enabling the debug flag will result in sensitive information being logged!!! ::: -**Metrics tracked** +By default, JSON encoded logs are produced. Debug enables colored, human-readable logs to be streamed to [standard out](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)>>>). In production, it's recommended to be set to `false`. -Name | Type | Description ---------------------------------------------- | --------- | ----------------------------------------------------------------------- -boltdb_free_alloc_size_bytes | Gauge | Bytes allocated in free pages -boltdb_free_page_n | Gauge | Number of free pages on the freelist -boltdb_freelist_inuse_size_bytes | Gauge | Bytes used by the freelist -boltdb_open_txn | Gauge | number of currently open read transactions -boltdb_pending_page_n | Gauge | Number of pending pages on the freelist -boltdb_txn | Gauge | total number of started read transactions -boltdb_txn_cursor_total | Counter | Total number of cursors created -boltdb_txn_node_deref_total | Counter | Total number of node dereferences -boltdb_txn_node_total | Counter | Total number of node allocations -boltdb_txn_page_alloc_size_bytes_total | Counter | Total bytes allocated -boltdb_txn_page_total | Counter | Total number of page allocations -boltdb_txn_rebalance_duration_ms_total | Counter | Total time spent rebalancing -boltdb_txn_rebalance_total | Counter | Total number of node rebalances -boltdb_txn_spill_duration_ms_total | Counter | Total time spent spilling -boltdb_txn_spill_total | Counter | Total number of nodes spilled -boltdb_txn_split_total | Counter | Total number of nodes split -boltdb_txn_write_duration_ms_total | Counter | Total time spent writing to disk -boltdb_txn_write_total | Counter | Total number of writes performed -groupcache_cache_hits_total | Counter | Total cache hits in local or cluster cache -groupcache_cache_hits_total | Counter | Total cache hits in local or cluster cache -groupcache_gets_total | Counter | Total get request, including from peers -groupcache_loads_deduped_total | Counter | gets without cache hits after duplicate suppression -groupcache_loads_total | Counter | Total gets without cache hits -groupcache_local_load_errs_total | Counter | Total local load errors -groupcache_local_loads_total | Counter | Total good local loads -groupcache_peer_errors_total | Counter | Total errors from peers -groupcache_peer_loads_total | Counter | Total remote loads or cache hits without error -groupcache_server_requests_total | Counter | Total gets from peers -grpc_client_request_duration_ms | Histogram | GRPC client request duration by service -grpc_client_request_size_bytes | Histogram | GRPC client request size by service -grpc_client_requests_total | Counter | Total GRPC client requests made by service -grpc_client_response_size_bytes | Histogram | GRPC client response size by service -grpc_server_request_duration_ms | Histogram | GRPC server request duration by service -grpc_server_request_size_bytes | Histogram | GRPC server request size by service -grpc_server_requests_total | Counter | Total GRPC server requests made by service -grpc_server_response_size_bytes | Histogram | GRPC server response size by service -http_client_request_duration_ms | Histogram | HTTP client request duration by service -http_client_request_size_bytes | Histogram | HTTP client request size by service -http_client_requests_total | Counter | Total HTTP client requests made by service -http_client_response_size_bytes | Histogram | HTTP client response size by service -http_server_request_duration_ms | Histogram | HTTP server request duration by service -http_server_request_size_bytes | Histogram | HTTP server request size by service -http_server_requests_total | Counter | Total HTTP server requests handled by service -http_server_response_size_bytes | Histogram | HTTP server response size by service -pomerium_build_info | Gauge | Pomerium build metadata by git revision, service, version and goversion -pomerium_config_checksum_int64 | Gauge | Currently loaded configuration checksum by service -pomerium_config_last_reload_success | Gauge | Whether the last configuration reload succeeded by service -pomerium_config_last_reload_success_timestamp | Gauge | The timestamp of the last successful configuration reload by service -redis_conns | Gauge | Number of total connections in the pool -redis_hits_total | Counter | Total number of times free connection was found in the pool -redis_idle_conns | Gauge | Number of idle connections in the pool -redis_misses_total | Counter | Total number of times free connection was NOT found in the pool -redis_stale_conns_total | Counter | Total number of stale connections removed from the pool -redis_timeouts_total | Counter | Total number of times a wait timeout occurred +For example, if `true` -### Tracing +``` +10:37AM INF cmd/pomerium version=v0.0.1-dirty+ede4124 +10:37AM INF proxy: new route from=httpbin.corp.beyondperimeter.com to=https://httpbin.org +10:37AM INF proxy: new route from=ssl.corp.beyondperimeter.com to=http://neverssl.com +10:37AM INF proxy/authenticator: grpc connection OverrideCertificateName= addr=auth.corp.beyondperimeter.com:443 +``` -Tracing tracks the progression of a single user request as it is handled by Pomerium. +If `false` -Each unit work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections. - -#### Shared Tracing Settings - -Config Key | Description | Required -:------------------ | :----------------------------------------------------------------------------------- | -------- -tracing_provider | The name of the tracing provider. (e.g. jaeger, zipkin) | ✅ -tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or `.01%` | ❌ - -#### Jaeger (partial) - -**Warning** At this time, Jaeger protocol does not capture spans inside the proxy service. Please use Zipkin protocol with Jaeger for full support. - -[Jaeger](https://www.jaegertracing.io/) is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including: - -- Distributed context propagation -- Distributed transaction monitoring -- Root cause analysis -- Service dependency analysis -- Performance / latency optimization - -Config Key | Description | Required -:-------------------------------- | :------------------------------------------ | -------- -tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector. | ✅ -tracing_jaeger_agent_endpoint | Send spans to jaeger-agent at this address. | ✅ - -#### Zipkin - -Zipkin is an open source distributed tracing system and protocol. - -Many tracing backends support zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol. - -Config Key | Description | Required -:---------------------- | :------------------------------- | -------- -tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅ - -#### Example - -![jaeger example trace](./img/jaeger.png) pomerium_config_last_reload_success_timestamp | Gauge | The timestamp of the last successful configuration reload by service pomerium_build_info | Gauge | Pomerium build metadata by git revision, service, version and goversion +``` +{"level":"info","version":"v0.0.1-dirty+ede4124","time":"2019-02-18T10:41:03-08:00","message":"cmd/pomerium"} +{"level":"info","from":"httpbin.corp.beyondperimeter.com","to":"https://httpbin.org","time":"2019-02-18T10:41:03-08:00","message":"proxy: new route"} +{"level":"info","from":"ssl.corp.beyondperimeter.com","to":"http://neverssl.com","time":"2019-02-18T10:41:03-08:00","message":"proxy: new route"} +{"level":"info","OverrideCertificateName":"","addr":"auth.corp.beyondperimeter.com:443","time":"2019-02-18T10:41:03-08:00","message":"proxy/authenticator: grpc connection"} +``` ### Forward Auth @@ -578,8 +311,293 @@ services: - "traefik.http.routers.httpbin.middlewares=test-auth@docker" ``` +### Global Timeouts + +- Environmental Variables: `TIMEOUT_READ` `TIMEOUT_WRITE` `TIMEOUT_IDLE` +- Config File Key: `timeout_read` `timeout_write` `timeout_idle` +- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` +- Example: `TIMEOUT_READ=30s` +- Defaults: `TIMEOUT_READ=30s` `TIMEOUT_WRITE=0` `TIMEOUT_IDLE=5m` + +Timeouts set the global server timeouts. For route-specific timeouts, see [policy](./#policy). + +![cloudflare blog on timeouts](https://blog.cloudflare.com/content/images/2016/06/Timeouts-001.png) + +> For a deep dive on timeout values see [these](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) [two](https://blog.cloudflare.com/exposing-go-on-the-internet/) excellent blog posts. + +### GRPC Options + +These settings control upstream connections to the Authorize service. + +#### GRPC Address + +- Environmental Variable: `GRPC_ADDRESS` +- Config File Key: `grpc_address` +- Type: `string` +- Example: `:443`, `:8443` +- Default: `:443` or `:5443` if in all-in-one mode + +Address specifies the host and port to serve GRPC requests from. Defaults to `:443` (or `:5443` in all in one mode). + +#### GRPC Insecure + +- Environmental Variable: `GRPC_INSECURE` +- Config File Key: `grpc_insecure` +- Type: `bool` +- Default: `:443` (or `:5443` if in all-in-one mode) + +If set, GRPC Insecure disables transport security for communication between the proxy and authorize components. If running in all-in-one mode, defaults to true as communication will run over localhost's own socket. + +#### GRPC Client Timeout + +Maximum time before canceling an upstream RPC request. During transient failures, the proxy will retry upstreams for this duration, if possible. You should leave this high enough to handle backend service restart and rediscovery so that client requests do not fail. + +- Environmental Variable: `GRPC_CLIENT_TIMEOUT` +- Config File Key: `grpc_client_timeout` +- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` +- Default: `10s` + +#### GRPC Client DNS RoundRobin + +Enable grpc DNS based round robin load balancing. This method uses DNS to resolve endpoints and does client side load balancing of _all_ addresses returned by the DNS record. Do not disable unless you have a specific use case. + +- Environmental Variable: `GRPC_CLIENT_DNS_ROUNDROBIN` +- Config File Key: `grpc_client_dns_roundrobin` +- Type: `bool` +- Default: `true` + +#### GRPC Server Max Connection Age + +Set max connection age for GRPC servers. After this interval, servers ask clients to reconnect and perform any rediscovery for new/updated endpoints from DNS. + +See for details + +- Environmental Variable: `GRPC_SERVER_MAX_CONNECTION_AGE` +- Config File Key: `grpc_server_max_connection_age` +- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` +- Default: `5m` + +#### GRPC Server Max Connection Age Grace + +Additive period with `grpc_server_max_connection_age`, after which servers will force connections to close. + +See for details + +- Environmental Variable: `GRPC_SERVER_MAX_CONNECTION_AGE_GRACE` +- Config File Key: `grpc_server_max_connection_age_grace` +- Type: [Go Duration](https://golang.org/pkg/time/#Duration.String) `string` +- Default: `5m` + +### HTTP Redirect Address + +- Environmental Variable: `HTTP_REDIRECT_ADDR` +- Config File Key: `http_redirect_addr` +- Type: `string` +- Example: `:80`, `:8080` +- Optional + +If set, the HTTP Redirect Address specifies the host and port to redirect http to https traffic on. If unset, no redirect server is started. + +### Insecure Server + +- Environmental Variable: `INSECURE_SERVER` +- Config File Key: `insecure_server` +- Type: `bool` +- Required if certificates unset + +Turning on insecure server mode will result in pomerium starting, and operating without any protocol encryption in transit. + +This setting can be useful in a situation where you have Pomerium behind a TLS terminating ingress or proxy. However, even in that case, it is highly recommended to use TLS to protect the confidentiality and integrity of service communication even behind the ingress using self-signed certificates or an internal CA. Please see our helm-chart for an example of just that. + +:::warning + +Pomerium should _never_ be exposed to the internet without TLS encryption. + +::: + +### Log Level + +- Environmental Variable: `LOG_LEVEL` +- Config File Key: `log_level` +- Type: `string` +- Options: `debug` `info` `warn` `error` +- Default: `debug` + +Log level sets the global logging level for pomerium. Only logs of the desired level and above will be logged. + +### Metrics Address + +- Environmental Variable: `METRICS_ADDRESS` +- Config File Key: `metrics_address` +- Type: `string` +- Example: `:9090`, `127.0.0.1:9090` +- Default: `disabled` +- Optional + +Expose a prometheus format HTTP endpoint on the specified port. Disabled by default. + +:::warning + +**Use with caution:** the endpoint can expose frontend and backend server names or addresses. Do not externally expose the metrics if this is sensitive information. + +::: + +**Metrics tracked** + +Name | Type | Description +--------------------------------------------- | --------- | ----------------------------------------------------------------------- +boltdb_free_alloc_size_bytes | Gauge | Bytes allocated in free pages +boltdb_free_page_n | Gauge | Number of free pages on the freelist +boltdb_freelist_inuse_size_bytes | Gauge | Bytes used by the freelist +boltdb_open_txn | Gauge | number of currently open read transactions +boltdb_pending_page_n | Gauge | Number of pending pages on the freelist +boltdb_txn | Gauge | total number of started read transactions +boltdb_txn_cursor_total | Counter | Total number of cursors created +boltdb_txn_node_deref_total | Counter | Total number of node dereferences +boltdb_txn_node_total | Counter | Total number of node allocations +boltdb_txn_page_alloc_size_bytes_total | Counter | Total bytes allocated +boltdb_txn_page_total | Counter | Total number of page allocations +boltdb_txn_rebalance_duration_ms_total | Counter | Total time spent rebalancing +boltdb_txn_rebalance_total | Counter | Total number of node rebalances +boltdb_txn_spill_duration_ms_total | Counter | Total time spent spilling +boltdb_txn_spill_total | Counter | Total number of nodes spilled +boltdb_txn_split_total | Counter | Total number of nodes split +boltdb_txn_write_duration_ms_total | Counter | Total time spent writing to disk +boltdb_txn_write_total | Counter | Total number of writes performed +groupcache_cache_hits_total | Counter | Total cache hits in local or cluster cache +groupcache_cache_hits_total | Counter | Total cache hits in local or cluster cache +groupcache_gets_total | Counter | Total get request, including from peers +groupcache_loads_deduped_total | Counter | gets without cache hits after duplicate suppression +groupcache_loads_total | Counter | Total gets without cache hits +groupcache_local_load_errs_total | Counter | Total local load errors +groupcache_local_loads_total | Counter | Total good local loads +groupcache_peer_errors_total | Counter | Total errors from peers +groupcache_peer_loads_total | Counter | Total remote loads or cache hits without error +groupcache_server_requests_total | Counter | Total gets from peers +grpc_client_request_duration_ms | Histogram | GRPC client request duration by service +grpc_client_request_size_bytes | Histogram | GRPC client request size by service +grpc_client_requests_total | Counter | Total GRPC client requests made by service +grpc_client_response_size_bytes | Histogram | GRPC client response size by service +grpc_server_request_duration_ms | Histogram | GRPC server request duration by service +grpc_server_request_size_bytes | Histogram | GRPC server request size by service +grpc_server_requests_total | Counter | Total GRPC server requests made by service +grpc_server_response_size_bytes | Histogram | GRPC server response size by service +http_client_request_duration_ms | Histogram | HTTP client request duration by service +http_client_request_size_bytes | Histogram | HTTP client request size by service +http_client_requests_total | Counter | Total HTTP client requests made by service +http_client_response_size_bytes | Histogram | HTTP client response size by service +http_server_request_duration_ms | Histogram | HTTP server request duration by service +http_server_request_size_bytes | Histogram | HTTP server request size by service +http_server_requests_total | Counter | Total HTTP server requests handled by service +http_server_response_size_bytes | Histogram | HTTP server response size by service +pomerium_build_info | Gauge | Pomerium build metadata by git revision, service, version and goversion +pomerium_config_checksum_int64 | Gauge | Currently loaded configuration checksum by service +pomerium_config_last_reload_success | Gauge | Whether the last configuration reload succeeded by service +pomerium_config_last_reload_success_timestamp | Gauge | The timestamp of the last successful configuration reload by service +redis_conns | Gauge | Number of total connections in the pool +redis_hits_total | Counter | Total number of times free connection was found in the pool +redis_idle_conns | Gauge | Number of idle connections in the pool +redis_misses_total | Counter | Total number of times free connection was NOT found in the pool +redis_stale_conns_total | Counter | Total number of stale connections removed from the pool +redis_timeouts_total | Counter | Total number of times a wait timeout occurred + +### Proxy Log Level + +- Environmental Variable: `PROXY_LOG_LEVEL` +- Config File Key: `proxy_log_level` +- Type: `string` +- Options: `debug` `info` `warn` `error` +- Default: value of `log_level` or `debug` if both are unset + +Log level sets the logging level for the pomerium proxy service. Only logs of the desired level and above will be logged. + +### Service Mode + +- Environmental Variable: `SERVICES` +- Config File Key: `services` +- Type: `string` +- Default: `all` +- Options: `all` `authenticate` `authorize` `cache` or `proxy` + +Service mode sets the pomerium service(s) to run. If testing, you may want to set to `all` and run pomerium in "all-in-one mode." In production, you'll likely want to spin up several instances of each service mode for high availability. + +### Shared Secret + +- Environmental Variable: `SHARED_SECRET` +- Config File Key: `shared_secret` +- Type: [base64 encoded] `string` +- Required + +Shared Secret is the base64 encoded 256-bit key used to mutually authenticate requests between services. It's critical that secret keys are random, and stored safely. Use a key management system or `/dev/urandom/` to generate a key. For example: + +``` +head -c32 /dev/urandom | base64 +``` + +### Tracing + +Tracing tracks the progression of a single user request as it is handled by Pomerium. + +Each unit work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections. + +#### Shared Tracing Settings + +Config Key | Description | Required +:------------------ | :----------------------------------------------------------------------------------- | -------- +tracing_provider | The name of the tracing provider. (e.g. jaeger, zipkin) | ✅ +tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or `.01%` | ❌ + +#### Jaeger (partial) + +**Warning** At this time, Jaeger protocol does not capture spans inside the proxy service. Please use Zipkin protocol with Jaeger for full support. + +[Jaeger](https://www.jaegertracing.io/) is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including: + +- Distributed context propagation +- Distributed transaction monitoring +- Root cause analysis +- Service dependency analysis +- Performance / latency optimization + +Config Key | Description | Required +:-------------------------------- | :------------------------------------------ | -------- +tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector. | ✅ +tracing_jaeger_agent_endpoint | Send spans to jaeger-agent at this address. | ✅ + +#### Zipkin + +Zipkin is an open source distributed tracing system and protocol. + +Many tracing backends support zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol. + +Config Key | Description | Required +:---------------------- | :------------------------------- | -------- +tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅ + +#### Example + +![jaeger example trace](./img/jaeger.png) pomerium_config_last_reload_success_timestamp | Gauge | The timestamp of the last successful configuration reload by service pomerium_build_info | Gauge | Pomerium build metadata by git revision, service, version and goversion + ## Authenticate Service +### Authenticate Callback Path + +- Environmental Variable: `AUTHENTICATE_CALLBACK_PATH` +- Config File Key: `authenticate_callback_path` +- Type: `string` +- Default: `/oauth2/callback` +- Optional + +The authenticate callback path is the path/url from the authenticate service that will receive the response from your identity provider. The value must exactly match one of the authorized redirect URIs for the OAuth 2.0 client. + +This value is referred to as the `redirect_url` in the [OpenIDConnect][oidc rfc] and OAuth2 specs. + +See also: + +- [OAuth2 RFC 6749](https://tools.ietf.org/html/rfc6749#section-3.1.2) +- [OIDC Spec][oidc rfc] +- [Google - Setting Redirect URI](https://developers.google.com/identity/protocols/OpenIDConnect#setredirecturi) + ### Authenticate Service URL - Environmental Variable: `AUTHENTICATE_SERVICE_URL` @@ -590,18 +608,6 @@ services: Authenticate Service URL is the externally accessible URL for the authenticate service. -### Identity Provider Name - -- Environmental Variable: `IDP_PROVIDER` -- Config File Key: `idp_provider` -- Type: `string` -- Required -- Options: `azure` `google` `okta` `onelogin` or `oidc` - -Provider is the short-hand name of a built-in OpenID Connect (oidc) identity provider to be used for authentication. To use a generic provider,set to `oidc`. - -See [identity provider] for details. - ### Identity Provider Client ID - Environmental Variable: `IDP_CLIENT_ID` @@ -620,14 +626,17 @@ Client ID is the OAuth 2.0 Client Identifier retrieved from your identity provid Client Secret is the OAuth 2.0 Secret Identifier retrieved from your identity provider. See your identity provider's documentation, and our [identity provider] docs for details. -### Identity Provider URL +### Identity Provider Name -- Environmental Variable: `IDP_PROVIDER_URL` -- Config File Key: `idp_provider_url` +- Environmental Variable: `IDP_PROVIDER` +- Config File Key: `idp_provider` - Type: `string` -- Required, depending on provider +- Required +- Options: `azure` `google` `okta` `onelogin` or `oidc` -Provider URL is the base path to an identity provider's [OpenID connect discovery document](https://openid.net/specs/openid-connect-discovery-1_0.html). For example, google's URL would be `https://accounts.google.com` for [their discover document](https://accounts.google.com/.well-known/openid-configuration). +Provider is the short-hand name of a built-in OpenID Connect (oidc) identity provider to be used for authentication. To use a generic provider,set to `oidc`. + +See [identity provider] for details. ### Identity Provider Scopes @@ -648,23 +657,14 @@ Identity provider scopes correspond to access privilege scopes as defined in Sec Identity Provider Service Account is field used to configure any additional user account or access-token that may be required for querying additional user information during authentication. For a concrete example, Google an additional service account and to make a follow-up request to query a user's group membership. For more information, refer to the [identity provider] docs to see if your provider requires this setting. -### Authenticate Callback Path +### Identity Provider URL -- Environmental Variable: `AUTHENTICATE_CALLBACK_PATH` -- Config File Key: `authenticate_callback_path` +- Environmental Variable: `IDP_PROVIDER_URL` +- Config File Key: `idp_provider_url` - Type: `string` -- Default: `/oauth2/callback` -- Optional +- Required, depending on provider -The authenticate callback path is the path/url from the authenticate service that will receive the response from your identity provider. The value must exactly match one of the authorized redirect URIs for the OAuth 2.0 client. - -This value is referred to as the `redirect_url` in the [OpenIDConnect][oidc rfc] and OAuth2 specs. - -See also: - -- [OAuth2 RFC 6749](https://tools.ietf.org/html/rfc6749#section-3.1.2) -- [OIDC Spec][oidc rfc] -- [Google - Setting Redirect URI](https://developers.google.com/identity/protocols/OpenIDConnect#setredirecturi) +Provider URL is the base path to an identity provider's [OpenID connect discovery document](https://openid.net/specs/openid-connect-discovery-1_0.html). For example, google's URL would be `https://accounts.google.com` for [their discover document](https://accounts.google.com/.well-known/openid-configuration). ## Proxy Service @@ -690,16 +690,6 @@ Authorize Service URL is the location of the internally accessible authorize ser If your load balancer does not support gRPC pass-through you'll need to set this value to an internally routable location (`https://pomerium-authorize-service.default.svc.cluster.local`) instead of an externally routable one (`https://authorize.corp.example.com`). -### Override Certificate Name - -- Environmental Variable: `OVERRIDE_CERTIFICATE_NAME` -- Config File Key: `override_certificate_name` -- Type: `int` -- Optional -- Example: `*.corp.example.com` if wild card or `authenticate.corp.example.com`/`authorize.corp.example.com` - -Secure service communication can fail if the external certificate does not match the internally routed service hostname/[SNI](https://en.wikipedia.org/wiki/Server_Name_Indication). This setting allows you to override that value. - ### Certificate Authority - Environmental Variable: `CERTIFICATE_AUTHORITY` or `CERTIFICATE_AUTHORITY_FILE` @@ -709,6 +699,16 @@ Secure service communication can fail if the external certificate does not match Certificate Authority is set when behind-the-ingress service communication uses self-signed certificates. Be sure to include the intermediary certificate. +### Default Upstream Timeout + +- Environmental Variable: `DEFAULT_UPSTREAM_TIMEOUT` +- Config File Key: `default_upstream_timeout` +- Type: [Duration](https://golang.org/pkg/time/#Duration) `string` +- Example: `10m`, `1h45m` +- Default: `30s` + +Default Upstream Timeout is the default timeout applied to a proxied route when no `timeout` key is specified by the policy. + ### Headers - Environmental Variable: `HEADERS` @@ -742,26 +742,6 @@ By default, conservative [secure HTTP headers](https://www.owasp.org/index.php/O ![pomerium security headers](./img/security-headers.png) -### Refresh Cooldown - -- Environmental Variable: `REFRESH_COOLDOWN` -- Config File Key: `refresh_cooldown` -- Type: [Duration](https://golang.org/pkg/time/#Duration) `string` -- Example: `10m`, `1h45m` -- Default: `5m` - -Refresh cooldown is the minimum amount of time between allowed manually refreshed sessions. - -### Default Upstream Timeout - -- Environmental Variable: `DEFAULT_UPSTREAM_TIMEOUT` -- Config File Key: `default_upstream_timeout` -- Type: [Duration](https://golang.org/pkg/time/#Duration) `string` -- Example: `10m`, `1h45m` -- Default: `30s` - -Default Upstream Timeout is the default timeout applied to a proxied route when no `timeout` key is specified by the policy. - ### JWT Claim Headers - Environmental Variable: `JWT_CLAIMS_HEADERS` @@ -778,6 +758,26 @@ Any claim in the pomerium session JWT can be placed into a corresponding header Use this option if you previously relied on `x-pomerium-authenticated-user-{email|user-id|groups}` for downstream authN/Z. +### Override Certificate Name + +- Environmental Variable: `OVERRIDE_CERTIFICATE_NAME` +- Config File Key: `override_certificate_name` +- Type: `int` +- Optional +- Example: `*.corp.example.com` if wild card or `authenticate.corp.example.com`/`authorize.corp.example.com` + +Secure service communication can fail if the external certificate does not match the internally routed service hostname/[SNI](https://en.wikipedia.org/wiki/Server_Name_Indication). This setting allows you to override that value. + +### Refresh Cooldown + +- Environmental Variable: `REFRESH_COOLDOWN` +- Config File Key: `refresh_cooldown` +- Type: [Duration](https://golang.org/pkg/time/#Duration) `string` +- Example: `10m`, `1h45m` +- Default: `5m` + +Refresh cooldown is the minimum amount of time between allowed manually refreshed sessions. + ## Cache Service The cache service is used for storing user session data. @@ -800,6 +800,21 @@ When deployed in a distributed fashion, autocache uses [gossip](https://github.c Autocache does not require any additional settings but does require that the cache url setting returns name records that correspond to a [list of peers](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services). +### [Bolt](https://godoc.org/go.etcd.io/bbolt/) + +Bolt is a simple, lightweight, low level key value store and is the underlying storage mechanism in projects like [etcd](https://etcd.io/). Bolt persists data to a file, and has no built in eviction mechanism. + +Bolt is suitable for all-in-one deployments that do not require concurrent / distributed writes. + +#### Bolt Path + +- Environmental Variable: `CACHE_STORE_PATH` +- Config File Key: `cache_store_path` +- Type: `string` +- Example: `/etc/bolt.db` + +CacheStorePath is the path to save bolt's database file. + ### [Redis](https://redis.io/) Redis, when used as a [LRU cache](https://redis.io/topics/lru-cache), functions in a very similar way to autocache. Redis store support allows you to leverage existing infrastructure, and to persist session data if that is a requirement. @@ -821,21 +836,6 @@ CacheStoreAddr specifies the host and port on which the cache store should conne CacheStoreAddr is the password used to connect to redis. -### [Bolt](https://godoc.org/go.etcd.io/bbolt/) - -Bolt is a simple, lightweight, low level key value store and is the underlying storage mechanism in projects like [etcd](https://etcd.io/). Bolt persists data to a file, and has no built in eviction mechanism. - -Bolt is suitable for all-in-one deployments that do not require concurrent / distributed writes. - -#### Bolt Path - -- Environmental Variable: `CACHE_STORE_PATH` -- Config File Key: `cache_store_path` -- Type: `string` -- Example: `/etc/bolt.db` - -CacheStorePath is the path to save bolt's database file. - ## Policy - Environmental Variable: `POLICY` @@ -864,59 +864,14 @@ In this example an incoming request with a path prefix of `/admin` would be hand A list of policy configuration variables follows. -### From +### Allowed Domains -- `yaml`/`json` setting: `from` -- Type: `URL` (must contain a scheme and hostname, must not contain a path) -- Required -- Example: `https://httpbin.corp.example.com` - -`From` is externally accessible source of the proxied request. - -### To - -- `yaml`/`json` setting: `to` -- Type: `URL` (must contain a scheme and hostname) -- Required -- Example: `http://httpbin` , `https://192.1.20.12:8080`, `http://neverssl.com` - -`To` is the destination of a proxied request. It can be an internal resource, or an external resource. - -### Prefix - -- `yaml`/`json` setting: `prefix` -- Type: `string` -- Optional -- Example: `/admin` - -If set, the route will only match incoming requests with a path that begins with the specified prefix. - -### Path - -- `yaml`/`json` setting: `path` -- Type: `string` -- Optional -- Example: `/admin/some/exact/path` - -If set, the route will only match incoming requests with a path that is an exact match for the specified path. - -### Regex - -- `yaml`/`json` setting: `regex` -- Type: `string` (containing a regular expression) -- Optional -- Example: `^/(admin|superuser)/.*$` - -If set, the route will only match incoming requests with a path that matches the specified regular expression. The supported syntax is the same as the Go [regexp package](https://golang.org/pkg/regexp/) which is based on [re2](https://github.com/google/re2/wiki/Syntax). - -### Allowed Users - -- `yaml`/`json` setting: `allowed_users` +- `yaml`/`json` setting: `allowed_domains` - Type: collection of `strings` - Required -- Example: `alice@pomerium.io` , `bob@contractor.co` +- Example: `pomerium.io` , `gmail.com` -Allowed users is a collection of whitelisted users to authorize for a given route. +Allowed domains is a collection of whitelisted domains to authorize for a given route. ### Allowed Groups @@ -927,14 +882,14 @@ Allowed users is a collection of whitelisted users to authorize for a given rout Allowed groups is a collection of whitelisted groups to authorize for a given route. -### Allowed Domains +### Allowed Users -- `yaml`/`json` setting: `allowed_domains` +- `yaml`/`json` setting: `allowed_users` - Type: collection of `strings` - Required -- Example: `pomerium.io` , `gmail.com` +- Example: `alice@pomerium.io` , `bob@contractor.co` -Allowed domains is a collection of whitelisted domains to authorize for a given route. +Allowed users is a collection of whitelisted users to authorize for a given route. ### CORS Preflight @@ -945,6 +900,33 @@ Allowed domains is a collection of whitelisted domains to authorize for a given Allow unauthenticated HTTP OPTIONS requests as [per the CORS spec](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#Preflighted_requests). +### From + +- `yaml`/`json` setting: `from` +- Type: `URL` (must contain a scheme and hostname, must not contain a path) +- Required +- Example: `https://httpbin.corp.example.com` + +`From` is externally accessible source of the proxied request. + +### Path + +- `yaml`/`json` setting: `path` +- Type: `string` +- Optional +- Example: `/admin/some/exact/path` + +If set, the route will only match incoming requests with a path that is an exact match for the specified path. + +### Prefix + +- `yaml`/`json` setting: `prefix` +- Type: `string` +- Optional +- Example: `/admin` + +If set, the route will only match incoming requests with a path that begins with the specified prefix. + ### Public Access - `yaml`/`json` setting: `allow_public_unauthenticated_access` @@ -956,6 +938,15 @@ Allow unauthenticated HTTP OPTIONS requests as [per the CORS spec](https://devel If this setting is enabled, no whitelists (e.g. Allowed Users) should be provided in this route. +### Regex + +- `yaml`/`json` setting: `regex` +- Type: `string` (containing a regular expression) +- Optional +- Example: `^/(admin|superuser)/.*$` + +If set, the route will only match incoming requests with a path that matches the specified regular expression. The supported syntax is the same as the Go [regexp package](https://golang.org/pkg/regexp/) which is based on [re2](https://github.com/google/re2/wiki/Syntax). + ### Route Timeout - `yaml`/`json` setting: `timeout` @@ -965,15 +956,45 @@ If this setting is enabled, no whitelists (e.g. Allowed Users) should be provide Policy timeout establishes the per-route timeout value. Cannot exceed global timeout values. -### Websocket Connections +### Preserve Host Header -- Config File Key: `allow_websockets` +- `yaml`/`json` setting: `preserve_host_header` - Type: `bool` +- Optional - Default: `false` -If set, enables proxying of websocket connections. +When enabled, this option will pass the host header from the incoming request to the proxied host, instead of the destination hostname. -**Use with caution:** By definition, websockets are long-lived connections, so [global timeouts](#global-timeouts) are not enforced. Allowing websocket connections to the proxy could result in abuse via [DOS attacks](https://www.cloudflare.com/learning/ddos/ddos-attack-tools/slowloris/). +See [ProxyPreserveHost](http://httpd.apache.org/docs/2.0/mod/mod_proxy.html#proxypreservehost). + +### Set Request Headers + +- Config File Key: `set_request_headers` +- Type: map of `strings` key value pairs +- Optional + +Set Request Headers allows you to set static values for given request headers. This can be useful if you want to pass along additional information to downstream applications as headers, or set authentication header to the request. For example: + +```yaml +- from: https://httpbin.corp.example.com + to: https://httpbin.org + allowed_users: + - bdd@pomerium.io + set_request_headers: + # works auto-magically! + # https://httpbin.corp.example.com/basic-auth/root/hunter42 + Authorization: Basic cm9vdDpodW50ZXI0Mg== + X-Your-favorite-authenticating-Proxy: "Pomerium" +``` + +### To + +- `yaml`/`json` setting: `to` +- Type: `URL` (must contain a scheme and hostname) +- Required +- Example: `http://httpbin` , `https://192.1.20.12:8080`, `http://neverssl.com` + +`To` is the destination of a proxied request. It can be an internal resource, or an external resource. ### TLS Skip Verification @@ -1009,39 +1030,28 @@ Note: This setting will replace (not append) the system's trust store for a give Pomerium supports client certificates which can be used to enforce [mutually authenticated and encrypted TLS connections](https://en.wikipedia.org/wiki/Mutual_authentication) (mTLS). For more details, see our [mTLS example repository](https://github.com/pomerium/examples/tree/master/mutual-tls) and the [certificate docs](../docs/reference/certificates.md). -### Set Request Headers +### Websocket Connections -- Config File Key: `set_request_headers` -- Type: map of `strings` key value pairs -- Optional - -Set Request Headers allows you to set static values for given request headers. This can be useful if you want to pass along additional information to downstream applications as headers, or set authentication header to the request. For example: - -```yaml -- from: https://httpbin.corp.example.com - to: https://httpbin.org - allowed_users: - - bdd@pomerium.io - set_request_headers: - # works auto-magically! - # https://httpbin.corp.example.com/basic-auth/root/hunter42 - Authorization: Basic cm9vdDpodW50ZXI0Mg== - X-Your-favorite-authenticating-Proxy: "Pomerium" -``` - -### Preserve Host Header - -- `yaml`/`json` setting: `preserve_host_header` +- Config File Key: `allow_websockets` - Type: `bool` -- Optional - Default: `false` -When enabled, this option will pass the host header from the incoming request to the proxied host, instead of the destination hostname. +If set, enables proxying of websocket connections. -See [ProxyPreserveHost](http://httpd.apache.org/docs/2.0/mod/mod_proxy.html#proxypreservehost). +**Use with caution:** By definition, websockets are long-lived connections, so [global timeouts](#global-timeouts) are not enforced. Allowing websocket connections to the proxy could result in abuse via [DOS attacks](https://www.cloudflare.com/learning/ddos/ddos-attack-tools/slowloris/). ## Authorize Service +### Authenticate Service URL + +- Environmental Variable: `AUTHENTICATE_SERVICE_URL` +- Config File Key: `authenticate_service_url` +- Type: `URL` +- Required +- Example: `https://authenticate.corp.example.com` + +Authenticate Service URL is the externally accessible URL for the authenticate service. + ### Signing Key - Environmental Variable: `SIGNING_KEY`