Whether we want it or not adoption of Kubernetes is growing. It can be set up as a managed solution (all major cloud providers provide such products) or we can set it up by yourselves. No matter if we select the former or the latter - we would like to make it as secure as it can be.
One of the solutions to make the Kubernetes cluster more secure is to hide a control plane (to be more specific - kube-apiserver
) behind a firewall.
That means cluster management is not available from the Internet.
That creates problems with accessing it.
We can SSH to a server that is in the same network and run kubectl
commands from there, but this is a nuisance which we want to avoid.
Fortunately, SSH tunnels came with help in that case (not for the first time!) -
we can create a tunnel to the server in the same network and pass all traffic to the cluster through it!
To do it we can run the command:
$ ssh our-gate.example.com -L 16443:10.0.10.2:443
In the command, we’re assuming that:
- our server is available with the domain
our-gate.example.com
, - the cluster is in the same local network as the server and its control plane is available at
10.0.10.2
IP address, - we want the tunnel to be available at port
16443
.
Now we need to replace IP address / hostname of cluster in ~/.kube/config
to point to local tunnel.
To do this we will open aforementioned file and replace put https://127.0.0.1:16443
in clusters.cluster.server
property:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <long string with CA data>
# line below was something like https://our-public-address-of-cluster.example.com or just local IP like https://10.0.10.2
server: https://127.0.0.1:16443
name: my-k8s-cluster
After that we can try to run a command like kubectl cluster-info
:
~ > kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: x509: certificate is valid for our-public-address-of-cluster.example.com, 10.0.10.2, not 127.0.0.1
It fails, but why?
It is because we want to connect to 127.0.0.1
but API (securely server over https) is responding with a certificate issued for different domain / IP address.
Fortunately solution for that is simple which is to put clusters.cluster.tls-server-name
property with one of the valid values from the error message -
in our case it can be 10.0.10.2
.
Our final configuration will look like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <long string with CA data>
server: https://127.0.0.1:16443
tls-server-name: 10.0.10.2
name: my-k8s-cluster
contexts:
- context:
cluster: my-k8s-cluster
user: admin
name: my-k8s-cluster
current-context: my-k8s-cluster
kind: Config
preferences: {}
users:
- name: admin
user:
token: <some-fancy-token>
And now running cluster-info
command will gives us what we expect:
~ > kubectl cluster-info
Kubernetes control plane is running at https://10.0.10.2:16443
GLBCDefaultBackend is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
KubeDNS is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://10.0.10.2:16443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
Yay! Now we can use kubectl
, helm
and other software that uses configuration from ~/.kube/config
from our local machine!
And a small “bonus” (and also describing it to the future myself) - accessing Kubernetes in GCP (GKE) through Identity Aware Proxy:
$ gcloud compute ssh "some-instance" --zone "$ZONE" --project "$PROJECT" --tunnel-through-iap --ssh-flag="-L 16443:10.0.10.2:443"
That is it - the SSH tunnel again saves the day! :-)