I’ve been using and learning k8s for a couple of months now. For a project which involves a huge amount of resource monitoring and planning I needed to record the CPU and memory utilization of every pod. I will write about how I did that using InfluxDB, Telegraf and visualized it Grafana later. Before that I had this (silly) problem with monitoring resource usage in k8s which definitely was rooted in my lack of knowledge. Hopefully this could help other nubs like me.
The first part of the following article talks about metric server in Kubernetes and the second part explains how to alter the resolution (query interval) from the default 1m value.
To collect metrics from different resources in the k8s environment you need metrics server. It talks to Kubelets and provides the metrics through an API. You can also see those metrics using
kubectl top .
If you are using Minikube, you first need to enable this.
minikube addons enable metrics-server
to see a list of addons
minikube addons list
and after a while you can see the metrics for your pods or the nodes.
$kubectl top nodesNAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 4517m 45% 4248Mi 8%$ kubectl top pod
NAME CPU(cores) MEMORY(bytes)
auth-5bb5847cbc-5mpxz 284m 35Mi
auth-5bb5847cbc-5v6xg 144m 51Mi
auth-5bb5847cbc-ktj9s 181m 50Mi
auth-5bb5847cbc-m8bc8 200m 59Mi
books-7b66d7765f-pttq6 143m 55Mi
gateway-6f46df854-kvwq2 280m 40Mi
mongo-0 87m 184Mi
Changing the resolution (interval)
If you run
watch kubectl top pod you notice the values are changed every 1 minute. If you need a lower resolution, let’s say 15 seconds this is how to do it.
Note that the Metrics-server Github page suggests not to use a value less that 15s.
You can get the deployment details of the metrics-server using this command:
kubectl -n kube-system get deployments.apps
and it’s possible to edit the deployment details,
kubectl -n kube-system edit deployments.apps metrics-server
it’s a pretty long file, the part we are interested in is where it runs the command to start metrics-server:
the first item in command section is running the metrics-server and the rest is passing options. So what are the options for metrics-server? You can check all the options here. Between them, there is one which we need:
—- metric-resolution duration The resolution at which heapster will retain metrics. (default 1m0s)
We add this option to the deployment file we just opened for editing:
Note the new
-— metric-resolution=15s. Save the file and exit (:wq if you are in vim). Wait a few seconds so the new deployment is applied and a new pod of metrics-server is running. Then run the
watch kubectl top pod again and you will notice it’s updating every 15 seconds.