Securing Kubernetes with Pomerium

Securing access to the Kubernetes Control plane

Currently, many organizations grant access to Kubernetes by having the user tunnel through the network perimeter to access a bastion host, then from there SSH port-forward to the API server (often through kubectl).

Users required to use third-party client for access

Learn More

Coarse-grained access controls applied here on session start

Learn More

Expanded data boundary with third party network

Learn More

The perimeter problem - once a user has access, they can realistically go anywhere.

Learn More

Reverse proxy for reachability, but no access controls

Learn More

Bastion host is unable to limit access to servers behind it. Users have access to both clusters.

Learn More

Reaching API server which controls the cluster

Learn More

The user access flow can be boiled down to this:

1. The User logs in via the VPN client. This is where authentication and authorization is checked.

2. The User now can reach the bastion host inside the VPC. The bastion host acts as a simple reverse proxy with access to the internal IP addresses of the Kubernetes API servers.

   • The bastion host itself has been configured with the necessary roles and permissions to access the Kubernetes API servers.

   • Firewall rules allow users to SSH access to the bastion host, and allow for the bastion host to communicate with the API servers.

The Users can now use SSH port-forwarding through the bastion host to access any reachable Kubernetes API servers

The Kubernetes Role Based Access Control (RBAC) is a key security control mechanism for the Kubernetes API server to provision least privilege access. However, this entire setup before reaching the kubectl — Steps 1 to 3, from the VPN client to the bastion host — exists because integrating with other systems is out of scope for the Kubernetes API server.

Why is the generic bastion host setup “wrong"?

This is a well-known system, but it has certain demerits:

• Overprovisioned access may result in privilege abuse: Organizations often don’t want their users to be unproductive while waiting for access, they tend to shift towards giving more access than necessary. This happens even when the organization originally started off with least privilege access best practices; in the end, productivity has been prioritized and users or roles gain more access over time. Finally, granted permissions are often difficult to remove for various reasons.

• Unfettered lateral movement: This architecture falls firmly into the Perimeter Problem: once a user has tunneled to reach the bastion host, they have unlimited access to any cluster reachable from the bastion host. In this above case, the user would have access to the control planes for both Clusters A and B.

• Lacks continuous verification: User access is session-based, meaning there is little to no security measures in place once a user gains access. While it’s possible to inspect traffic, it is almost never done due to the increased lift and complications of inspecting this traffic. As a result, companies cannot prevent bad actions before they happen.

• Lack of contextual awareness: The RBAC system is identity-aware but not context-aware. The modern threat landscape should assume compromised credentials and malicious insiders, meaning the context surrounding access requests should be considered. This includes details such as the time of day, geo-location, application state, devices, and more, which the built-in Kubernetes access controls do not consider.

This setup exists because the Kubernetes API server doesn’t integrate well with other systems for authenticating human access. We solve this by extending the server’s capabilities with a reverse proxy which can integrate with missing features.