Clusters in Pomerium Zero

This document provides an overview of Clusters in Pomerium Zero.

Overview

When you install Pomerium Zero, you get your own Cluster. A cluster consists of one or more replicas of Pomerium Core, the primary server component that secures your services. Clusters are deployed locally and managed through Pomerium Zero's hosted control plane.

Why clusters?

A cluster model reduces the complexity of managing Pomerium: All of a cluster's configuration, including settings, routes, policies, and certificates, are managed from a central control plane hosted in the Pomerium Zero cloud. This allows you to manage multiple isolated Pomerium Core deployments, each with its own configuration.

Configuration changes applied in the hosted control plane are communicated over a streaming connection to your local cluster. Each cluster is connected to its own storage backend, which synchronizes state across replicas.

Each replica runs Pomerium Core in an all-in-one mode, meaning the services that comprise Core are deployed together and share session state. Similarly, replicas communicate with each other to synchronize state across a cluster. The interconnectivity between replicas enables you to horizontally scale your cluster to support multiple workloads and higher availability.

This model affords the following benefits:

• Simplified configuration
• Enhanced data privacy
• Improved scalability

Cluster benefits

Simplified configuration

Cluster configuration is fully managed in Pomerium Zero. Configuration changes applied in Pomerium Zero are fetched and applied locally by your cluster.

This model ensures that:

• Your cluster remains synchronized with Pomerium Zero as changes are applied
• You don't have to manage cluster configuration files or environment variables
• Pomerium Zero persists your cluster's last-known configuration

Enhanced data privacy

Pomerium can't monitor your organization's traffic or collect private user data. However, Pomerium does need to store configuration details required to maintain the hosted control plane.

Pomerium also stores deployment and configuration details required to control access to your services, such as routes, policies, TLS certificates, and sensitive encrypted data when appropriate.

Improved scalability

Replicas within a cluster synchronize records over a streaming connection, which keeps the cluster abreast of configuration changes. In this way, you can deploy multiple replicas to meet the needs of your infrastructure and organization.

See Persistence and scalability below for more information.

Deploy your cluster

Cluster token

When you deploy a cluster in Pomerium Zero, a Cluster Token is generated for you. The cluster token environment variable instructs Pomerium to run in a special "Zero-managed" mode, wherein your cluster connects to the Pomerium Zero cloud.

You can generate a new cluster token at any time. See the Rotate cluster token steps below for more information.

Rotate cluster token

A cluster token does not have a set expiration time. However, you may want to rotate the cluster token for security reasons. For example:

• A cluster token was accidentally exposed
• You may lose track of the existing token, and want a new one

caution

Rotating the cluster token invalidates the current token, and shuts your cluster down.

To rotate your cluster token:

Zero Console

1. From the navigation bar, select Manage Clusters
2. Under Actions, select Rotate Token
3. In the confirmation modal, select Confirm
4. Copy the token value and store it somewhere safe

Zero API

Send a POST request to the /token endpoint:

curl 'https://console.pomerium.app/api/v0/organizations/{organizationId}/clusters/{clusterId}/token' \
--header 'Authorization: Bearer {ID-TOKEN}'

You'll receive a response with a new cluster refreshToken:

{
  "refreshToken": "token"
}

After rotating the cluster token, you must pass it into your configuration and run Pomerium. See the steps below for your deployment environment.

Linux

In the shell script below, replace <cluster_token> with your new cluster token before running the command:

curl https://console.pomerium.app/install.bash | env POMERIUM_ZERO_TOKEN=<cluster_token> bash -s install

Docker

1. In your docker-compose.yaml file, replace <CLUSTER_TOKEN> with your new token:

docker-compose.yaml

services:
  pomerium:
    image: pomerium/pomerium:latest
    ports:
      - 443:443
    restart: always
    environment:
      POMERIUM_ZERO_TOKEN: <CLUSTER_TOKEN>
      XDG_CACHE_HOME: /var/cache
    volumes:
      - pomerium-cache:/var/cache
    networks:
      main: {}

networks:
  main: {}

volumes:
  pomerium-cache:

1. Run docker compose up -d

Kubernetes

1. Run the following command, replacing cluster_token with your new cluster token:

$ export POMERIUM_ZERO_TOKEN=<cluster_token>

1. Then, to write the new cluster token to the Kubernetes secret, run:

kubectl -n pomerium patch secret pomerium-zero-token -p "{\"stringData\":{\"pomerium_zero_token\":\"${POMERIUM_ZERO_TOKEN}\"}}"

Detected and Override IP Address

When you first deploy a cluster, it connects itself to the Pomerium Zero cloud, and Pomerium Zero automatically assigns the cluster a unique Starter Domain. During this process, Pomerium Zero attempts to detect the cluster's outbound IP address to connect to it.

Once Pomerium Zero successfully connects to the cluster's Detected IP Address, it sets A (IPv4) or AAAA (IPv6) records for the starter domain. Depending on your environment, the detected IP address may not be reachable by you, the end-user. This could be the case if, for example, you're running a cluster:

• Locally from your laptop behind a NAT-enabled router
• Inside a corporate intranet
• Behind a firewall
• In a Kubernetes server where the ingress load balancer IP address doesn't match the egress IP address

In such cases, you may see one of the following errors:

• ERR_CONNECTION_REFUSED
• DNS_PROBE_FINISHED_NXDOMAIN (when accessing a route)

The ERR_CONNECTION_REFUSED error means that your cluster is running, but Pomerium Zero can't reach it with the detected IP address. The DNS_PROBE_FINISHED_NXDOMAIN error means that there is no DNS record available for the domain. This could mean that the domain name changes haven't finished propagating yet, or that the cluster never connected to Pomerium Zero because the IP address couldn't be detected.

As a workaround, we've provided the Override IP Address field, which allows you to manually set the IP address so that Pomerium Zero can connect to your cluster.

Manage cluster configuration

Starter domain

Pomerium Zero generates and assigns a unique Starter Domain to your cluster. A starter domain is a randomly generated domain name that follows this format:

<CLUSTER-SUBDOMAIN-XXXX>.pomerium.app

For example: voracious-ape-1578.pomerium.app

The starter domain comes with its own DNS records and TLS certificates so you can quickly test Pomerium Zero before adding your own custom domain.

Add Custom Domain

See the Custom Domains page for more information on how to add custom domains to a cluster.

Cluster name

The Cluster Name is a customizable identifier for your cluster. It defaults to the randomly generated subdomain of your cluster domain, but you can change it at any time. Changing the cluster name does not affect your starter domain.

If you want to change the cluster name, you can update it in your cluster settings:

1. From the navigation bar dropdown, select Manage Clusters
2. Under Actions, select the edit icon
3. In the Cluster Name field, enter your preferred cluster name
4. Save your changes and apply the changeset

Routes, policies, and certificates

Routes and policies defined in a cluster are scoped only to that cluster, and are not available in other clusters.

Pomerium Zero automatically provisions and renews wildcard TLS certificates issued by ZeroSSL for each cluster. Like routes and policies, certificates are scoped to a cluster and are not available in other clusters.

info

To learn more about routing, policies, and certificates in Pomerium, see the following docs:

• Routing
• Policy Language
• Certificates

Persistence and scalability

Persistence

By default, Pomerium uses an in-memory storage backend to synchronize a cluster's configuration and session state. This is suitable for test environments, but not for production. In the event of a service outage, your cluster can't fetch its configuration from the cloud, so it can't serve traffic.

To avoid service outages, we recommend that you deploy a PostgreSQL database to persist your cluster's configuration state. Since all replicas share the same storage connection, the database will persist cluster state if a replica restarts due to a service outage.

To configure a database connection in Pomerium Zero:

1. Select Settings
2. Go to the General tab
3. Enter the connection string in the Databroker Storage Connection String field

Replicas

To deploy a replica, run the Pomerium Zero installation script using the same cluster token you used to deploy your cluster. This will generate another replica, which you can verify on the Status page.

note

Replicas left inactive for over 24 hours will not appear on the Status page.

Cluster Status / Troubleshooting

The Status page of the Zero Console displays notifications and alerts regarding the health of a cluster. If you see an alert in the Status page, it means something may be wrong with your cluster and you should take steps to resolve it.

We've addressed all the cluster status alerts below. Refer to the relevant alert to learn how to fix it.

Certificates are expired or invalid

This cluster status alert inspects a cluster's certificates to see if they expire within 10 days. It runs daily, and every time the cluster configuration is updated.

What causes this alert:

• If your cluster uses managed certificates provisioned by Pomerium, then this error is a bug. Please contact support.
• If you manually uploaded your own custom certificate, then you must upload a new certificate.

Steps to resolve: Upload a valid certificate, or allow Pomerium to automate certificate management for you by adding a Custom Domain.

Hostnames do not have matching certificates

This cluster status alert ensures that all the hostnames available in a cluster's configuration have matching certificates. It runs every time the cluster configuration is updated.

What causes this alert: Either the From URL in a route block is incorrect, or the certificate is missing.

Steps to resolve: Make sure all From URLs are correct, and that the hostnames have matching certificates.

info

For more information on certificates in Pomerium, see the following pages:

• Certificates
• Certificates reference

Cluster configuration is not current

This cluster status alert makes sure the cluster is running the latest changeset. It runs daily to check for the latest configuration, or every time the cluster configuration is updated.

What causes this alert: This cluster status alert may indicate cluster connectivity issues. Specifically, Pomerium Zero can't connect to the cluster to apply the latest changeset to its configuration.

Steps to resolve: Restart the cluster. If that doesn't resolve the issue, please contact support.

Cluster version is not current

This cluster status alert makes sure the cluster replica is running the latest release. It runs daily.

What causes this alert: The replica is not running the latest version of Pomerium Zero.

Steps to resolve: Update your Pomerium Zero cluster configuration to run the latest version of Pomerium.

Cluster using an in-memory storage backend

This cluster status alert throws an error if it detects a cluster isn't connected to a persistent storage backend. It runs every time the cluster configuration is updated.

What causes this alert:

By default, a cluster uses an in-memory storage backend to synchronize replicas. This does not persist your configuration. For production environments, we recommend connecting each cluster instance to its own dedicated PostgreSQL database. See the Persistence page for instructions.

Steps to resolve: See the Databroker Storage Connection String reference page to learn how to connect to a database instance in Pomerium.

Incompatible configuration

This cluster status alert looks for configuration incompatibilities in a replica. It runs every time the cluster configuration is updated.

What causes this alert: This is caused by a version incompatibility in the cluster configuration. For example, you somehow apply a configuration change that isn't compatible with your version of Pomerium Zero.

Steps to resolve: If you see this warning, please contact support.

Cluster can't communicate with the storage backend

This cluster status alert monitors the connection to a PostgreSQL database. It runs every time the cluster configuration is updated.

What causes this alert: Pomerium can't connect to your PostgreSQL database.

Steps to resolve: Check the parameters of your Databroker Storage Connection String.

There's an issue with cluster configuration

This cluster status alert tells you if Envoy accepted the cluster configuration or not after a configuration update.

What causes this alert: If you see this warning, please contact support.

There's an issue with the network listener

This cluster status alert makes sure the network listener is configured correctly. It runs every time the cluster configuration is updated.

What causes this alert: Typically, this error is caused when Pomerium's network listener can't bind to the cluster's port. The error may also be the result of a non-root process with insufficient permissions attempting to bind to a privileged port.

Steps to resolve:

• If this is a port binding error, you may need to kill a process running on that port or specify a port to bind to.
• If this is a permissions issue, you may need to add the CAP_NET_BIND_SERVICE capability to the process.

There's an issue with the route configuration

This cluster status alert verifies the route configuration. It runs every time the cluster configuration is updated.

What causes this alert: A route misconfiguration in a cluster.

Steps to resolve: If you see this warning, it's likely a problem with Pomerium. Please contact support.

There's an issue with the configuration

This cluster status alert monitors xDS issues in a replica. It runs every time the cluster configuration is updated.

What causes this alert: If you see this warning, it's likely a problem with Pomerium. Please contact support.

Pomerium can't write some of its cache files

This cluster status alert makes sure Pomerium can persist a replica's bootstrap configuration. It runs when you start a cluster, or every time the cluster configuration is updated.

What causes this alert: Pomerium doesn't have sufficient permissions to read from a replica's configuration file.

Steps to resolve: Check your file permissions to make sure Pomerium has read access.

Cluster connectivity issues with the cloud

This cluster status alert monitors the connection between the Pomerium Zero cloud and a cluster. It maintains a consistent streaming connection.

What causes this alert: Connectivity issues between Pomerium Zero and a cluster.

Steps to resolve: You may experience this issue if you're accessing Pomerium Zero behind a firewall. Check your firewall settings to make sure it allows HTTP and HTTPS traffic.