Photo by Chris Liverani on Unsplash

Change Metric Resolution Interval on Minikube

I’ve been using and learning k8s for a couple of months now. For a project which involves a huge amount of resource monitoring and planning I needed to record the CPU and memory utilization of every pod. I will write about how I did that using InfluxDB, Telegraf and visualized it Grafana later. Before that I had this (silly) problem with monitoring resource usage in k8s which definitely was rooted in my lack of knowledge. Hopefully this could help other nubs like me.

The first part of the following article talks about metric server in Kubernetes and the second part explains how to alter the resolution (query interval) from the default 1m value.

Metrics Server

To collect metrics from different resources in the k8s environment you need metrics server. It talks to Kubelets and provides the metrics through an API. You can also see those metrics using kubectl top .

If you are using Minikube, you first need to enable this.

minikube addons enable metrics-server

to see a list of addons

minikube addons list

and after a while you can see the metrics for your pods or the nodes.

$kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   4517m        45%    4248Mi          8%
$ kubectl top pod
NAME                    CPU(cores)   MEMORY(bytes)
auth-5bb5847cbc-5mpxz   284m         35Mi
auth-5bb5847cbc-5v6xg   144m         51Mi
auth-5bb5847cbc-ktj9s   181m         50Mi
auth-5bb5847cbc-m8bc8   200m         59Mi
books-7b66d7765f-pttq6  143m         55Mi
gateway-6f46df854-kvwq2 280m         40Mi
mongo-0                 87m          184Mi

Changing the resolution (interval)

If you run watch kubectl top pod you notice the values are changed every 1 minute. If you need a lower resolution, let’s say 15 seconds this is how to do it.

Note that the Metrics-server Github page suggests not to use a value less that 15s.

You can get the deployment details of the metrics-server using this command:

kubectl -n kube-system get deployments.apps

and it’s possible to edit the deployment details,

kubectl -n kube-system edit deployments.apps metrics-server

it’s a pretty long file, the part we are interested in is where it runs the command to start metrics-server:

spec:
      containers:
      - command:
        - /metrics-server
        - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
        image: k8s.gcr.io/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        name: metrics-server

the first item in command section is running the metrics-server and the rest is passing options. So what are the options for metrics-server? You can check all the options here. Between them, there is one which we need:

 —- metric-resolution duration                        The resolution at which heapster will retain metrics. (default 1m0s)

We add this option to the deployment file we just opened for editing:

spec:
      containers:
      - command:
        - /metrics-server
        - --metric-resolution=15s
        - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true
        image: k8s.gcr.io/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        name: metrics-server

Note the new-— metric-resolution=15s. Save the file and exit (:wq if you are in vim). Wait a few seconds so the new deployment is applied and a new pod of metrics-server is running. Then run the watch kubectl top pod again and you will notice it’s updating every 15 seconds.