Monitoring Kubernetes with Graphite

in Kubernetes , Monitoring and Observability

Monitoring Kubernetes with Graphite .png

In this article, we will be covering how to monitor Kubernetes using Graphite, and we’ll do the visualization with Grafana. The focus will be on monitoring and plotting essential metrics for monitoring Kubernetes clusters.


    We will download, implement and monitor with the custom dashboards for Kubernetes that can be downloaded from the Grafana dashboard resources. These monitoring dashboards have variables to allow drilling down into data at granular level.

    To follow along with this blog, sign up for the MetricFire free trial, where you can use Graphite and Grafana directly in our platform. MetricFire is a hosted Graphite, Grafana and Prometheus service, where we do the setup and management of these open-source tools so you don’t have to. 

    ‍ 

    Introduction to Kubernetes

    According to Kubernetes, Kubernetes is a “portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.”

    In today’s industry, monitoring Kubernetes deployments is a challenging but necessary activity. In this article, we are not going to cover Kubernetes basics, as we’ll jump more into how to use the dashboards. For a more detailed introduction to Kubernetes monitoring, refer to our Introduction to Monitoring Kubernetes on the MetricFire blog. 

    Introduction to Graphite

    Graphite is a very popular enterprise monitoring tool, and here at MetricFire we have a Hosted Graphite product that has been developing and improving since 2012. Graphite is a time-series monitoring tool that receives metrics data pushed to it by a collector such as StatsD or collectd, and then monitors the time-series metrics as specified by the user. If you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts and the Installation and Setup of Graphite before reading this article. 

    Setting up Graphite, Grafana and Kubernetes

    For the purpose of this article, we will use a Kubernetes cluster deployed on AWS. Follow the instructions on the Amazon AWS EKS user guide to install AWS CLI, and make sure you also install the kubectl command line tool.

    Once the kubectl is installed, running the command “kubectl cluster-info” should give the following output:

    ~ ./kubectl cluster-info
    Kubernetes master is running at https://92797800687320683968AF0937C2B5D3.yl4.ap-south-1.eks.amazonaws.com
    CoreDNS is running at https://92797800687320683968AF0937C2B5D3.yl4.ap-south-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

    Next, let’s set up Grafana on our kubernetes cluster. Here is the simple configuration file which will create a Grafana Pod and a Service running on Kubernetes:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
     labels:
       app: grafana
     name: grafana
    spec:
     replicas: 1
     selector:
       matchLabels:
         app: grafana
     template:
       metadata:
         labels:
           app: grafana
       spec:
         containers:
         - image: grafana/grafana:5.4.3
           name: grafana
           ports:
           - containerPort: 3000
             name: http
    
           volumeMounts:
             - name: grafana-storage
               mountPath: /var/lib/grafana
         volumes:
           - name: grafana-storage
             persistentVolumeClaim:
               claimName: grafana-storage
         securityContext:
           runAsNonRoot: true
           runAsUser: 65534
           fsGroup: 472
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
     name: grafana-storage
    spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
    ---
    apiVersion: v1
    kind: Service
    metadata:
     name: grafana
     labels:
       app: grafana
    spec:
     type: LoadBalancer
     ports:
     - port: 3000
       protocol: TCP
       targetPort: 3000
     selector:
       app: grafana

    Run, “kubectl create -f grafana-deployment.yml” to create Grafana Pod and Service.

    If we run the command “kubectl get service”, you should get an output similar to below:

    ➜  ~/github/k8-graphite-monitoring (master) kubectl get service
    NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                       AGE
    grafana      LoadBalancer   10.100.252.229   a942a31a4780f11eaa9010a814a720da-1449083553.ap-south-1.elb.amazonaws.com   3000:31159/TCP                3d21h
    kubernetes   ClusterIP      10.100.0.1       <none>                                                                     443/TCP

    ➜ ~/github/k8-graphite-monitoring (master) kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 10.100.252.229 a942a31a4780f11eaa9010a814a720da-1449083553.ap-south-1.elb.amazonaws.com 3000:31159/TCP 3d21h kubernetes ClusterIP 10.100.0.1 <none> 443/TCP

    Grafana.png
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     labels:
       app: graphite
     name: graphite
    spec:
     replicas: 1
     selector:
       matchLabels:
         app: graphite
     template:
       metadata:
         labels:
           app: graphite
       spec:
         containers:
         - image: graphiteapp/graphite-statsd
           name: graphite
           ports:
           - containerPort: 2003
             name: carbon-plain     
           - containerPort: 2004
             name: carbon-pkl   
           - containerPort: 2023
             name: carbon-ag-plain
           - containerPort: 2024
             name: carbon-ag-pkl   
           - containerPort: 8125
             name: statsd   
           - containerPort: 8126
             name: statsd-admin  
           - containerPort: 80
             name: http       
    ---
    apiVersion: v1
    kind: Service
    metadata:
     name: graphite
     labels:
       app: graphite
    spec:
     type: LoadBalancer
     ports:
     - port: 80
       protocol: TCP
       targetPort: 80
       name: http
     - port: 2003
       protocol: TCP
       targetPort: 2003
       name: carbon
     selector:
       app: graphite

    Run, “kubectl create -f graphite-deployment.yml” to create this Pod and Service.

    At this point, both Grafana and Graphite, should be up and running. 

    Run, “kubectl get service” to make sure both the services are up and running.

    ➜  ~/github/k8-graphite-monitoring (master) kubectl get service
    NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                       AGE
    grafana      LoadBalancer   10.100.252.229   a942a31a4780f11eaa9010a814a720da-1449083553.ap-south-1.elb.amazonaws.com   3000:31159/TCP                3d21h
    graphite     LoadBalancer   10.100.216.91    ac0f466207b2211eaa9010a814a720da-687823427.ap-south-1.elb.amazonaws.com    80:32198/TCP,2003:32041/TCP   104s
    kubernetes   ClusterIP      10.100.0.1       <none>                                                                     443/TCP                       3d22h

    Just like Grafana, we can enter http://ac0f466207b2211eaa9010a814a720da-687823427.ap-south-1.elb.amazonaws.com in the browser to open the Graphite web application, as shown below:

    Graphite.png

    Now, we will add Graphite as the data source in Grafana. Browse to the data source section of Grafana and add Graphite as the data source, as shown below:

    Data source.png

    Now, we will run Snap Daemon in our Kubernetes cluster. Snap Daemon is the monitoring daemon which will pull various kubernetes monitoring metrics and push them into Graphite.

    Before we run the Snap Daemon, we need to make one small change inside snap_ds.yml. We will update the hostname and port of Graphite service in the config section as shown below:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
     name: snap
    spec:
     selector:
       matchLabels:
         name: snap
     template:
       metadata:
         name: snap
         labels:
           name: snap
       spec:
         hostPID: true
         hostNetwork: true
         containers:
         - name: snap
           image: raintank/snap_k8s:v4
           volumeMounts:
             - mountPath: /sys/fs/cgroup
               name: cgroup
             - mountPath: /var/run/docker.sock
               name: docker-sock
             - mountPath: /var/lib/docker
               name: fs-stats
             - mountPath: /usr/local/bin/docker
               name: docker
             - mountPath: /proc_host
               name: proc
             - mountPath: /opt/snap/tasks
               name: snap-tasks
           ports:
           - containerPort: 8181
             hostPort: 8181
             name: snap-api
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           env:
             - name: PROCFS_MOUNT
               value: /proc_host
         volumes:
           - name: dev
             hostPath:
               path: /dev
           - name: cgroup
             hostPath:
               path: /sys/fs/cgroup
           - name: docker-sock
             hostPath:
               path: /var/run/docker.sock
           - name: fs-stats
             hostPath:
               path: /var/lib/docker
           - name: docker
             hostPath:
               path: /usr/bin/docker
           - name: proc
             hostPath:
               path: /proc
           - name: snap-tasks
             configMap:
               name: snap-tasks
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
     name: snap-tasks
    data:
     core.json: |-
       {
           "version": 1,
           "schedule": {
               "type": "simple",
               "interval": "10s"
           },
           "workflow": {
               "collect": {
                   "metrics": {
                       "/intel/docker/*":{},
                       "/intel/procfs/cpu/*": {},
                       "/intel/procfs/meminfo/*": {},
                       "/intel/procfs/iface/*": {},
                       "/intel/linux/iostat/*": {},
                       "/intel/procfs/load/*": {}
                   },
                   "config": {
                       "/intel/procfs": {
                           "proc_path": "/proc_host"
                       }
                   },
                   "process": null,
                   "publish": [
                       {
                           "plugin_name": "graphite",                   
                           "config": {
                               "prefix": "snap.dev.&lt;%NODE%&gt;",
                               "server": "ac0f466207b2211eaa9010a814a720da-687823427.ap-south-1.elb.amazonaws.com",
                               "port": 2003
                           }
                       }
                   ]
               }
           }
       }

    Run, “kubectl create -f snap_ds.yml” to create Snap Daemon and get it started. If we now run, “kubectl get pod”, we will see output similar to below:

    NAME                       READY   STATUS    RESTARTS   AGE
    grafana-6f64b8c7f6-tc7qn   1/1     Running   0          3d23h
    graphite-775d8b989-zwp9x   1/1     Running   0          65m
    snap-dblx2                 1/1     Running   0          56m
    snap-g5pzm                 1/1     Running   0          56m
    snap-jdqrg                 1/1     Running   0          56m

    We see one Pod for Grafana, one for Graphite and three for Snap. This is because we have 3 nodes in our Kubernetes cluster and Snap will run as a daemon on each of the nodes to pull its metrics and push it to Graphite.

    ‍ 

    Plotting the monitoring visualization on Grafana

    Grafana comes with lots of pre-built dashboards that you can find on the Grafana dashboards resources site. We are going to utilize one of these dashboards for our Kubernetes monitoring: Kubernetes Container Stats

    On the Kubernetes Container Stats page, click the link “Download JSON” and import it into our Grafana portal. Make sure to choose the Graphite data source appropriately.

    Import.png

    As soon as the dashboard is imported, we should see the metrics being shown in our dashboard, similar to below:

    Grafana Dashboard.png
    Grafana Dashboard 2.png

    Similarly, there is another dashboard Kubernetes Node host metrics. When this is imported, it will show the metrics per host selected in the dashboard. 

    Grafana Dashboard 3.png

    You might want to set up alerts on these dashboards if the values in these dashboards exceed some critical threshold. Check out our article Grafana Dashboards from Basic to Advanced to learn how to set up Grafana alerts, and build custom dashboards.

    You can also create other types of visualizations based on the metrics exposed by Kubernetes. Have a look at the article Our Favorite Grafana Dashboards to create some of the more advanced dashboards.

    Setting up the Monitoring through MetricFire

    The setup which we have done above works for very basic Kubernetes infrastructure which would contain just a few nodes. In order to handle production level load, which would be a few hundred nodes and upwards of a few Mbps network traffic, you would need to scale out both Graphite and Grafana to handle the increasing load. 

    That’s where Hosted Graphite and Hosted Grafana comes into the picture. It allows you to scale for long-term storage, as well as automatically providing redundant storage of data. 

    Hosted Graphite and Hosted Grafana through MetricFire allows for the continuous active deployment of new features, as MetricFire’s products all have their foundations in the ever-growing open source projects. Sign up for the MetricFire free trial here, and start building Kubernetes dashboards within a few minutes.

    Conclusion

    In this article, we have seen how to set up Kubernetes monitoring with Graphite. We have seen some advanced visualizations to monitor Kubernetes using Graphite and Grafana. 

    Sign up here for a free trial of our Hosted Graphite and Grafana offering. Also, if you have any questions about our products, or about how MetricFire can help your company, talk to us directly by booking a demo


    Get similar stories in your inbox weekly, for free



    Share this story with your friends
    metricfire
    MetricFire

    MetricFire provides a complete infrastructure and application monitoring platform from a suite of open source monitoring tools. Depending on your setup, choose Hosted Prometheus or Graphite and view your metrics on beautiful Grafana dashboards in real-time.

    Latest stories


    GitHub Is Making Protocol Security Changes for SSH Users

    A report detailing changes being made by Git systems to the algorithm at GitHub.

    Different Reactions From the Cybersecurity Community Regarding the Ransomware Bill

    Insights into the proposed bill on ransomware attacks and the possible effect.

    WhiteSource’s Analysis to Improve Relations Among Developers and Security Teams

    A report from the insights released by WhiteSource on DevSecOps practices

    Security: Conti Ransomware Gangs Increase Attacks on International Organizations

    News on security agencies' investigation of the rising attacks on organizations by ransomware experts.

    An Insight Into How the Conti Gang Targets Microsoft Exchange Servers

    We look at how attackers can use PowerShell to orchestrate attacks on vulnerable servers and …

    What Developers Think About GitHub Copilot

    GitHub Copilot is a controversial developer assistant introduced to the public in June 2021. It …