docs: tcp support (#1712)

docs/.vuepress/config.js

@@ -122,6 +122,7 @@ module.exports = {
             "topics/kubernetes-integration",
             "topics/production-deployment",
             "topics/programmatic-access",
+            "topics/tcp-support",
             "topics/impersonation",
           ],
         },

docs/docs/topics/tcp-support.md

@@ -0,0 +1,106 @@
+---
+title: TCP Support
+description: >-
+  This article describes how to leverage pomerium for TCP proxying
+meta:
+  - name: keywords
+    content: pomerium pomerium-cli proxy identity-access-proxy ssh tcp postgres database redis mysql application non-http
+---
+
+# TCP Support
+
+## About
+
+Operations and engineering teams frequently require access to lower level administrative and data protocols such as SSH, RDP, Postgres, MySQL, Redis, etc.
+
+In addition to managing HTTP based applications, Pomerium can be used to protect non-HTTP systems with the same consistent authorization policy. This is achieved by tunneling TCP over HTTP with the help of a client side command built into [`pomerium-cli`](/docs/installation.md#pomerium-cli).
+
+
+Internally, Pomerium uses the [`CONNECT` method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT) to establish the TCP tunnel.
+
+::: warning
+To minimize issues with TCP support, Pomerium should not be placed behind another HTTP proxy.  Instead, configure your load balancer in L4 or TCP mode.
+
+Otherwise, the HTTP proxy in front of Pomerium must know how to properly handle the `CONNECT` command and proxy it upstream.  This capability will be specific to each proxy implementation.
+:::
+
+## Configuring
+
+TCP configuration is simple. Just specify the correct scheme and ports in your policy [`to`](/reference/#to) and [`from`](/reference/#from) fields.
+
+Example:
+```yaml
+policy:
+    - from: tcp+https://redis.corp.example.com:6379
+      to: tcp://redis.internal.example.com:6379
+      allowed_users:
+        - contractor@not-example.com
+      allowed_groups:
+        - datascience@example.com
+```
+
+Notes:
+
+* When configuring a TCP route, any HTTP specific settings such as `regex_rewrite_pattern`, or `set_request_headers` have no effect.
+* While data is encrypted from a user system to Pomerium's proxy, the underlying application protocol must also support encryption for data to be fully encrypted end-to-end.
+* The ports in `from` and `to` are independent.  Users only need to know the `from` URL to connect.  The `to` can be changed without end user participation.
+
+## Using
+
+While HTTP routes can be consumed with just a normal browser, `pomerium-cli` must serve as a proxy for TCP routes.  It is [available](/docs/installation.md#pomerium-cli) for a variety of platforms in various formats.
+
+To connect, you normally need just the external hostname and port of your TCP route:
+
+```bash
+% pomerium-cli tcp redis.corp.example.com:6379
+5:57PM INF tcptunnel: listening on 127.0.0.1:52046
+```
+
+By default, `pomerium-cli` will start a listener on loopback on a random port.
+
+On first connection, you will be sent through a standard Pomerium HTTP authentication flow.  After completing this, your TCP connection should be established!
+
+```bash
+% redis-cli -h localhost -p 52046
+localhost:52046> keys *
+(empty array)
+localhost:52046>
+```
+
+## Advanced Usage
+
+### Listen Configuration
+
+You may specify an optional address and port for the `tcp` command to listen on.
+
+`-` specifies that STDIN and STDOUT should be directly attached to the remote TCP connection.  This is useful for [SSH](#ssh-helper-configuration) or for sending data through a shell pipe.
+
+### SSH Helper Configuration
+
+A sample SSH ProxyCommand configuration in `ssh_config`:
+
+```
+Host *.corp.example.com
+    ProxyCommand ~/bin/pomerium-cli tcp --listen - %h:%p
+```
+
+A sample SSH command using `pomerium-cli` as a ProxyCommand:
+
+```bash
+ssh -o ProxyCommand='pomerium-cli tcp --listen - %h:%p' ssh.localhost.pomerium.io
+```
+
+More information on SSH ProxyCommand:
+
+- [https://man.openbsd.org/ssh_config.5#ProxyCommand](https://man.openbsd.org/ssh_config.5#ProxyCommand)
+- [https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump](https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump)
+
+### Custom URL
+
+If the Pomerium proxy is not reachable through port `443` or the route is not in external DNS, a custom URL may be specified:
+
+```bash
+% pomerium-cli tcp --pomerium-url https://pomerium.corp.example.com:8443 redis.corp.example.com:6379
+```
+
+The above command connects to `https://pomerium.corp.example.com:8443` and then requests the TCP route named `redis.corp.example.com:6379`.

docs/reference/readme.md

@@ -503,10 +503,9 @@ Expose a prometheus endpoint on the specified port.
 | pomerium_config_last_reload_success           | Gauge     | Whether the last configuration reload succeeded by service              |
 | pomerium_config_last_reload_success_timestamp | Gauge     | The timestamp of the last successful configuration reload by service    |
 | redis_conns                                   | Gauge     | Number of total connections in the pool                                 |
-| redis_idle_conns                              | Gauge     | Number of idle connections in the pool                                  |
+| redis_idle_conns                              | Gauge     | Total number of times free connection was found in the pool             |
-| redis_stale_conns                             | Gauge     | Number of stale connections in the pool                                 |
+| redis_wait_count_total                        | Counter   | Total number of connections waited for                                  |
-| redis_miss_count_total                        | Counter   | Total number of times a connection was NOT found in the pool            |
+| redis_wait_duration_ms_total                  | Counter   | Total time spent waiting for connections                                |
-| redis_hit_count_total                         | Counter   | Total number of times a connection was found in the pool                |
 | storage_operation_duration_ms                 | Histogram | Storage operation duration by operation, result, backend and service    |
 
 #### Envoy Proxy Metrics
@@ -556,10 +555,10 @@ Each unit work is called a Span in a trace. Spans include metadata about the wor
 
 #### Shared Tracing Settings
 
-Config Key          | Description                                                                          | Required
+| Config Key          | Description                                                                          | Required |
-:------------------ | :----------------------------------------------------------------------------------- | --------
+| :------------------ | :----------------------------------------------------------------------------------- | -------- |
-tracing_provider    | The name of the tracing provider. (e.g. jaeger, zipkin)                              | ✅
+| tracing_provider    | The name of the tracing provider. (e.g. jaeger, zipkin)                              | ✅        |
-tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or `.01%` | ❌
+| tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or `.01%` | ❌        |
 
 #### Jaeger (partial)
 
@@ -573,10 +572,10 @@ tracing_sample_rate | Percentage of requests to sample in decimal notation. Defa
 - Service dependency analysis
 - Performance / latency optimization
 
-Config Key                        | Description                                 | Required
+| Config Key                        | Description                                 | Required |
-:-------------------------------- | :------------------------------------------ | --------
+| :-------------------------------- | :------------------------------------------ | -------- |
-tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector.    | ✅
+| tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector.    | ✅        |
-tracing_jaeger_agent_endpoint     | Send spans to jaeger-agent at this address. | ✅
+| tracing_jaeger_agent_endpoint     | Send spans to jaeger-agent at this address. | ✅        |
 
 #### Zipkin
 
@@ -584,9 +583,9 @@ Zipkin is an open source distributed tracing system and protocol.
 
 Many tracing backends support zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol.
 
-Config Key              | Description                      | Required
+| Config Key              | Description                      | Required |
-:---------------------- | :------------------------------- | --------
+| :---------------------- | :------------------------------- | -------- |
-tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅
+| tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅        |
 
 #### Example
 
@@ -1043,10 +1042,13 @@ Requires setting [Google Cloud Serverless Authentication Service Account](./#goo
 ### From
 - `yaml`/`json` setting: `from`
 - Type: `URL` (must contain a scheme and hostname, must not contain a path)
+- Schemes: `https`, `tcp+https`
 - Required
-- Example: `https://verify.corp.example.com`
+- Example: `https://verify.corp.example.com`, `tcp+https://ssh.corp.example.com:22`
 
-`From` is the externally accessible source of the proxied request.
+`From` is the externally accessible URL for the proxied request.
+
+Specifying `tcp+https` for the scheme enables [TCP proxying](../docs/topics/tcp-support.md) support for the route. You may map more than one port through the same hostname by specifying a different `:port` in the URL.
 
 
 ### Kubernetes Service Account Token
@@ -1233,11 +1235,14 @@ Remove Request Headers allows you to remove given request headers. This can be u
 ### To
 - `yaml`/`json` setting: `to`
 - Type: `URL` (must contain a scheme and hostname)
+- Schemes: `http`, `https`, `tcp`
 - Required
 - Example: `http://verify` , `https://192.1.20.12:8080`, `http://neverssl.com`, `https://verify.org/anything/`
 
 `To` is the destination of a proxied request. It can be an internal resource, or an external resource.
 
+Must be `tcp` if `from` is `tcp+https`.
+
 :::warning
 
 Be careful with trailing slash.

docs/reference/settings.yaml

@@ -1159,10 +1159,13 @@ settings:
         attributes: |
           - `yaml`/`json` setting: `from`
           - Type: `URL` (must contain a scheme and hostname, must not contain a path)
+          - Schemes: `https`, `tcp+https`
           - Required
-          - Example: `https://verify.corp.example.com`
+          - Example: `https://verify.corp.example.com`, `tcp+https://ssh.corp.example.com:22`
         doc: |
-          `From` is the externally accessible source of the proxied request.
+          `From` is the externally accessible URL for the proxied request.
+
+          Specifying `tcp+https` for the scheme enables [TCP proxying](../docs/topics/tcp-support.md) support for the route. You may map more than one port through the same hostname by specifying a different `:port` in the URL.
       - name: "Kubernetes Service Account Token"
         keys:
           [
@@ -1359,11 +1362,14 @@ settings:
         attributes: |
           - `yaml`/`json` setting: `to`
           - Type: `URL` (must contain a scheme and hostname)
+          - Schemes: `http`, `https`, `tcp`
           - Required
           - Example: `http://verify` , `https://192.1.20.12:8080`, `http://neverssl.com`, `https://verify.pomerium.com/anything/`
         doc: |
           `To` is the destination of a proxied request. It can be an internal resource, or an external resource.
 
+          Must be `tcp` if `from` is `tcp+https`.
+
           :::warning
 
           Be careful with trailing slash.

scripts/generate-settings-docs.py

@@ -1,4 +1,4 @@
-#!/bin/env python3
+#!/usr/bin/env python3
 import os.path
 from typing import Any, IO
 import yaml