# Kubernetes metrics-server with TLS verification

These are my notes on what it took to integrate metrics-server into my sandbox Kubernetes the Hard Way learning cluster. Note my cluster is on VirtualBox locally hence some IP network ranges might look different.

tl;dr - you can skip all this by running with --kubelet-insecure-tls but for something approaching what you’d use in production read on.

I initially installed metrics-server using Helm thinking it would just work out of the box. This is not the case beccause Helm won’t configure the PKI infrastructure necessary to make this work. As I understand it you need a new CA specifically for handling extension apiserver traffic. In this case the metrics-server

## Prerequisites

Obviously a running K8s cluster of any type but I set one up using the above Kubernetes the Hard Way. Actually installing metrics-server can easily be done via Helm but this leaves it in a non-working state; this post attempts to address how to get it from non-working -> working.

You’ll also need:

• A CA certificate + key for the aggregation layer
• A key + cert signed by above CA for the metrics-server

## Terms

• Main apiserver - The core kube-apiserver
• Extension apiserver - The extension api-server, in this case metrics-server

## High level overview

The parameters below are not the only ones you need to get things working; we’re just focusing on the thorniest side of things: PKI based trust.

Overview of where options are used

This is a summary of this detailed image

Main apiserver

The apiserver, as usual, has the cluster CA certificate to verify incoming requests. This is done with --client-ca-file.

The key is to understanding that the main apiserver is acting as a proxy so it become a TLS client of the extension apiserver. Thus we need to inform it of the appropriate CA and cert details with:

• --requestheader-client-ca-file - The is the CA we trust when connecting to the extension apiserver.
• --proxy-client-cert-file - This is used by the main apiserver when calling out to the extension apiserver. Must be signed by a CA the extension trusts, I think it would have to be the same one specified in --requestheader-client-ca-file
• --proxy-client-key-file - Key belonging to above cert

If you’re not running kube-proxy on your masters then you’ll need to set up routing so that your masters can reach the workers. Once that is done add --enable-aggregator-routing=true to the kube-apiserver arguments.

Extension apiserver

The extension apiserver, metrics-server in this case, also needs to trusts the incoming requests proxied by the main apiserver. We provide similar arguments to the main apiserver and some extra:

• --requestheader-client-ca-file - The is the CA we trust when receiving connections from the main apiserver
• --client-ca-file - Once the above requestheader check is OK we take the CN from certificates signed with this CA (in my learning cluster the same CA as --requestheader-client-ca-file) and impersonate that CN instead of the request being from the main apiserver. I think.
• --tls-cert-file - HTTPS cert file signed by the aggregation CA in my case
• --tls-proviate-key-file - Key for above
• --kubelet-certificate-authority - The main apiserver CA certicate so that we can speak to the worker nodes

## References

You can read more below:

## Full argument lists

Your paths and cert names will vary.

Main apiserver

/usr/local/bin/kube-apiserver \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/audit.log \
--authorization-mode=Node,RBAC \
--client-ca-file=/var/lib/kubernetes/ca.pem \
--enable-swagger-ui=true \
--etcd-cafile=/var/lib/kubernetes/ca.pem \
--etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
--etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
--etcd-servers=https://192.168.99.11:2379,https://192.168.99.12:2379,https://192.168.99.13:2379 \
--event-ttl=1h \
--experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
--kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
--kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
--kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
--kubelet-https=true \
--runtime-config=api/all \
--service-account-key-file=/var/lib/kubernetes/service-account.pem \
--service-cluster-ip-range=10.32.0.0/24 \
--service-node-port-range=30000-32767 \
--tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
--tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
--enable-aggregator-routing=true \
--v=2


metrics-server

  /metrics-server
--cert-dir=/var/lib/kubernetes
--kubelet-certificate-authority=/var/lib/kubernetes/ca.pem