How to collect HAProxy metrics

in Visualization , Monitoring and Observability

haproxy.png

This article is a full tutorial on HAProxy monitoring and the best tools to get it done right. We will be looking into how to collect HAProxy metrics using a collectd daemon, push them into Graphite and visualize them in Grafana


    To follow the steps in this blog, sign up for the MetricFire free trial, where you can use Graphite and Grafana directly in our platform. MetricFire is a hosted Graphite, Grafana and Prometheus service, where we do the setup and management of these open-source tools so you don’t have to. 

    Introduction to HAProxy

    HAProxy is one of the most popular load balancers for HTTP and TCP applications. Along with load balancing, it comes with various features such as health checks, Web Application Firewall (WAF) capabilities, bot protection, and DDoS protection. 

    HAProxy is highly performant and hence it’s used as a load balancer for high traffic websites serving millions of requests per second such as Airbnb, Instagram and Reddit. It is imperative that DevOps teams for such websites need highly efficient monitoring infrastructure to monitor the traffic, response times and various other metrics related to HAProxy.

    In this article, we will be running a simple Node.js application, scale it to run multiple instances and finally load balance it behind HAProxy. 

    Enabling HAProxy to emit metrics

    In its default configuration, HAProxy does not emit metrics which can be pushed to Graphite. This is obviously a big blocker for doing HAProxy monitoring. Hence, it's important to configure HAProxy to enable a stats socket through which metrics related to HAProxy can be read by other applications.

    Enabling the stats socket is as simple as adding the following line in the global section of the HAProxy configuration (be default located at /etc/haproxy/haproxy.cfg)

    global
    	stats socket /run/haproxy/haproxy.sock

    If required, we can also enable the stats page through the following configuration. However, keep in mind that the stats page allows the user to disable and enable backends and hence can cause major changes to the infrastructure. Therefore, its access should be highly restricted.

    listen stats :1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /
        stats auth Username:Password
    HAProxy Stats Page HAProxy Stats Page

    Verify HAProxy Socket Metrics

    Once the HAProxy stats socket has been enabled, make sure to reload the configuration using “sudo service haproxy reload”. If you are serving a very high traffic website, we recommend running this command only during planned downtime to avoid any potential disruption to the live traffic. 

    Once the HAProxy configuration has been reloaded, run the following command on the HAProxy server to verify it’s emitting metrics.

    echo "show stat" | nc -U /var/run/haproxy.sock

    It should produce the similar output as below:

    # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,
    stats,FRONTEND,,,0,1,2000,2,4402,261854,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,12,0,1,0,0,,0,5,13,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,2,12,0,0,
    stats,BACKEND,0,0,0,0,200,0,4402,261854,0,0,,0,0,0,0,UP,0,0,0,,0,721,,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,689,,,0,0,0,5,,,,,,,,,,,,,,http,roundrobin,,,,,,,
    http-frontend,FRONTEND,,,0,1,2000,1,0,188,0,0,1,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,1,,,,0,0,0,1,0,0,,0,1,1,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,1,1,0,0,0,
    http-backend,webapp-172-18-0-2,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,721,0,,1,4,1,,0,,2,0,,0,L7OK,200,0,0,0,0,0,0,0,,,,,0,0,,,,,-1,OK,,0,0,0,0,,,,Layer7 check passed,,2,3,4,,,,172.18.0.2:3000,,http,,,,,,,,
    http-backend,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,721,0,,1,4,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,SRV_ID,http,roundrobin,,,,,,,
    default,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,5,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,0,0,0,0,0,
    default,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,721,,,1,5,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,http,roundrobin,,,,,,,

    Introduction to collectd

    Now that we have enabled HAProxy to emit stats, it’s time to read them and push it onto Graphite. For this, we need some sort of agent which polls HAProxy to collect stats and push it to Graphite. For this purpose, we are going to use collectd.

    Collectd is a very light weight daemon written in C which runs on the server, polls the systems such as HAProxy for metrics and pushes it to a custom source such as Graphite. In this case, we will be using collectd to poll the HAProxy metrics and push out to Hosted Graphite instances. 

    Configuring collectd to poll HAProxy 

    Collectd comes out of the box with various plugins to collect metrics from CPU, file system, network interface, memory, etc. In order to poll the metrics from HAProxy, collectd offers custom HAProxy plugins to poll metrics from HAProxy. Setting up this plugin is as simple as defining a Plugin configuration section for collectd and specifying the host and port where HAProxy is emitting it’s stats. 

    <Plugin "python">
    	ModulePath "/usr/share/collectd/collectd-haproxy"
    
    	Import "haproxy"
    
    	<Module "haproxy">
        	Socket "/var/run/haproxy.sock"
        	Interval 10
        	ProxyMonitor "http-backend"
        	ProxyMonitor "backend"
        	EnhancedMetrics 'True'
        	</Module>
    </Plugin>

    Configure collectd to push metrics to Hosted Graphite

    Now that collectd is configured to read metrics from HAProxy, we will configure it to push the same metrics to Hosted Graphite. Collectd comes with an out of box plugin called Write Graphite which allows us to push the metrics to any running Graphite instance including Hosted Graphite. Configuring this plugin is as simple as defining a plugin section with following configuration:

    LoadPlugin write_graphite
    
    <Plugin "write_graphite">
        	<Carbon>
                	Host "carbon.hostedgraphite.com”
                	Port "2003"
                	Prefix "your-api-key.collectd."
        	</Carbon>
    </Plugin>

    Make sure to check out our dedicated collectd documentation page through which you can download a configuration file for collectd, prefilled with your Hosted Graphite account details. 

    At this point, collectd should be successfully sending metrics to Hosted Graphite. You can verify the same by visiting Hosted Graphite and browsing to Metrics->Account Diagnostics page. 

    Hosted Graphite.png

    If you are unable to view these metrics in your case, double check the configuration provided in the collect configuration and make sure that the outbound port 2003 is not blocked by a firewall configuration. If this still doesn’t help, reach out to Hosted Graphite support through the chat bubble in the Hosted Graphite window. 

    Plotting the metrics on Hosted Grafana

    Now let’s plot these metrics into visualizations into our Hosted Grafana application - the final step in getting our HAProxy monitoring set up! Hosted Grafana is built on top of Grafana and provides us the ability to create beautiful graphs and visualizations such as gauges, bar graphs and various other types of visualizations. It can hook into various types of data sources such as Graphite, InfluxDb, MySQL and Elasticsearch. On top of all, it also provides alerting capabilities as well.

    So, let’s login to Hosted Graphite at https://www.hostedgraphite.com/app and browse to Dashboards -> Grafana to open the Hosted Grafana application page.

    Since Hosted Grafana is part of MetricFire’s offering itself, the datasource connection between Grafana and Graphite would have been automatically done for us.

    Grafana comes with a lot of pre-built dashboards as part of its Grafana dashboards library. We can either download one of the prebuilt dashboards for HAProxy monitoring or create one from scratch. In this case, we downloaded a pre-built dashboard and customized according to our needs.

    Here is what it looks like:

    Grafana Dashboard.png

    Hosted Grafana is a very powerful visualization tool built over Grafana. It allows you to create your own visualizations as well as setup alerts whenever a metric value crosses a certain threshold.

    Check out our article Grafana Dashboards from Basic to Advanced to learn how to set up Grafana alerts, and build custom dashboards.

    You can also create other types of visualizations based on the metrics exposed by HAProxy. Have a look at the article Our Favorite Grafana Dashboards to create some of the more advanced dashboards.

    List of Metrics supported by collectd HAProxy Plugin

    Collectd HAProxy supports many different types of metrics. Here is the list of metrics supported:

    • Connection Rate
    • Cumulative Requests
    • Idle Percentage
    • Current Sessions
    • Session Rate
    • Bytes out
    • Bytes In
    • Session Time Average
    • Request Rate
    • Session Rate
    • Response 2xx
    • Response 4xx
    • Response 5xx
    • Error Requests
    • Denied Requests
    • Error Connection
    • Denied Response
    • Current Queue
    • Queue Time Average
    • Response Time Average
    • Error Response
    • Retries
    • Redispatched
    • Max Connections
    • Max connection Rate
    • Max Session Rate
    • Max SSL Connections
    • Max Pipes
    • Run Queue
    • Piped Used
    • Uptime Seconds
    • Current Connections
    • Current SSL Connections
    • Failed Checks
    • Downtime
    • Response 1xx
    • Response 3xx
    • Queue Max
    • Queue Limit
    • Session Rate Limit
    • Session Rate Max
    • Session Total
    • Session Limit
    • Session Max
    • Throttle
    • Connect Time Average
    • Active Servers
    • Backup Servers
    • Health check duration
    • Last Session
    • Connection Rate

    Conclusion

    In this article, we had a full tutorial on HAProxy monitoring. We looked at how we can collect HAProxy metrics using collectd and push them to Hosted Graphite and Hosted Grafana. This allows us to monitor HAProxy load balancer metrics.

    Sign up here for a free trial of our Hosted Graphite and Grafana offering. Also, if you have any questions about our products, or about how MetricFire can help your company, talk to us directly by booking a demo.


    Get similar stories in your inbox weekly, for free



    Share this story with your friends
    metricfire
    MetricFire

    MetricFire provides a complete infrastructure and application monitoring platform from a suite of open source monitoring tools. Depending on your setup, choose Hosted Prometheus or Graphite and view your metrics on beautiful Grafana dashboards in real-time.

    Latest stories


    200 Million Certificates in 24 Hours

    Let's Encrypt has been providing free Certificate Authority (CA) for websites in need of them …

    Gatling VS K6

    Gatling and K6 are performance load testing tools, and they are both open source, easy …

    Red Hat Ansible Platform 1 vs 2; What’s the Difference?

    Red Hat Ansible is a platform used by enterprises to manage, unify and execute infrastructure …

    Domino Data Labs Raised $100 Million in the Latest Funding Round

    Culled from the news released by Domino Data labs on funding and the company's progress …

    New Release: The Microsoft Azure Purview Is Now Available on General Availability

    News report detailing the announcement of the release of Azure purview on GA

    Google Introduces Online Training Program to Improve Cloud Skills

    Google addresses existing cloud personnel deficiency with training programs.