Quantcast Vladimir's Ganglia gmetric repository

Ganglia Gmetric repository


Quick Links



This is my collection of metric collection tools to be used with GangliaGanglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. This is not to say that same scripts cannot be used with other monitoring tools you just need to modify the script to do so.

1. Apache traffic stats (updated 2010-01-14)

Purpose: Collect realtime Apache statistics such as number of successful hits, redirected pages, error pages, etc aka. HTTP code 200, 300, 400.
OS: All OSes

After you start collecting data you should get something like this :-)

Apache Report

Steps to get this going are as follows

  1. Download and install the cool Ganglia-Logtailer contributed graciously to open source by people at Linden Lab.
  2. There is a Linden Lab supplied ApacheLogtailer class you can use however that one involves major changes to the logging format which I didn't like. Therefore I modified the stock Apache LogFormat by appending %D which logs the time to service a request in microseconds. To do so define a new LogFormat in Apache configuration ie.
    LogFormat "%h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" %D" combined-time
    or if you are using a proxy such as Haproxy or Varnish you will want to log the IP that proxy passes in the X-Forwarded-For header.
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" %D" combined-haproxy
  3. Now you have to attach the log format to your log ie. instead of using the TransferLog you would put in
    CustomLog "/var/log/httpd/access_log" combined-haproxy
  4. Restart/Reload Apache so that Apache starts logging in the new format
  5. Download my ApacheLogtailer class or use the one that comes with ganglia-logtailer
  6. Now configure Logtailer to process the logs per ganglia-logtailer documentation ie. I run it out of the cron
    /opt/logtailer/ganglia-logtailer --classname ApacheLogtailer --log_file /var/log/httpd/access_log --mode cron
  7. If everything was set up correctly you should see 7 new metrics ie. apache_hits, apache_200, apache_300, apache_400, apache_500
  8. If you want the consolidated graph such as one above you will need to download apache_report.php and put in your ganglia-web directory under graph.d subdirectory.
  9. You also have to modify the conf.php. Add
    $optional_graphs = array('apache_report');
  10. You also need to modify get_context.php. At the end add
    "apache_report" => 1,
    to the list of report graphs
  11. Now apache_report should be available in the Metric drop down box in the Cluster Report. If you want it added to every host you will need to modify the host_view.tpl template by adding something like this
    <IMG BORDER=0 HEIGHT="154" WIDTH="397" ALT="{cluster_url} Apache Report" SRC="./graph.php?g=apache_report&z=medium&c={cluster_url}&{graphargs}">

Useful tip: Monitor apache_500 metrics. HTTP 500 errors should never happen thus if they start happening there is a problem ie. server was misconfigured.


2. Memcached stats (updated 2009-08-03)

Purpose: Collect realtime Memcached stats such as number of current connections, hit ratio, current number of items, number of evictions (early expiry of items due to cache fullness) etc.
OS: All OSes
Memcached Stats

Steps to get this going are as follows

  1. Install the Cache::Memcached Perl module from CPAN ie.
  2. Download the Ganglia collection script ganglia_memcached.pl
  3. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust
  4. Run the script by hand to make sure it doesn't error out. You should see something like this
  5. Place the script in a cron to run on desired schedule e.g. every minute

3. NFS server/client stats

Purpose: This script will collect NFSd server and/or NFS client stats on the host running the service.
OS: Linux only

Sample NFSd server access

Please download the script ganglia_nfs_stats.pl and install it on a machine running NFSd. Then do following

4. Local Disk Stats (Updated 2009-08-28)

Purpose: This script will collect local disk stats on the host running the service.
OS: Linux only

Example Disk Stats

Please download the script ganglia_disk_stats.pl and install it. Then do following

5. mySQL server stats

Purpose: This script will collect mySQL server stats as reported by extended-status
OS: Cross platform - any platform that has perl support including Windows (cygwin recommended for Windows)

mysqladmin will report extended-status statistics such as these.  Explanation of all metrics can be found at

http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html

| Threads_cached                    | 0          |
| Threads_connected | 5 |
| Threads_created | 14706 |
| Threads_running | 1 |

Unfortunately that doesn't tell us much unless we can graph it. After you run the script you get these pretty graphs :-)

Example mySQL queries

Please download the script ganglia_mysql_stats.pl and install it. Then do following

6. Postfix queue size (updated 2009-08-03)

Purpose: This script will keep track of Postfix queue size ie. number of messages in queue
OS: Any platform that uses Postfix

Steps to get this going are as follows
  1. Download the Ganglia collection script ganglia_postfix_mailqueue.sh
  2. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust.
  3. Run the script by hand to make sure it doesn't error out.
  4. Put it in a cron and run it however often you want.

7. Apache Workers Busy/Idle Stats (updated 2009-08-03)

Purpose: This script will keep track of busy/idle Apache workers
OS: Cross platform - any platform that has perl support including Windows (cygwin recommended for Windows)

Steps to get this going are as follows
  1. You need to configure the Apache mod_status e.g.
    <VirtualHost *:80>
        	ServerName monitoring.site
            <Location /status>
    	  SetHandler server-status
    	  Order deny,allow
    	  Deny from all
    	  Allow from 127.0.0.1
    	</Location>
    </VirtualHost>
    
  2. Make sure you can open up the URL.
  3. Download the Ganglia collection script ganglia_apache_workers.sh
  4. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust.
  5. Adjust the URL of the monitoring site specified above.
  6. Run the script by hand to make sure it doesn't error out.
  7. Put it in a cron and run it however often you want.

8. BIND DNS stats (updated 2009-09-06)

Purpose: This script will provide you DNS statistics such as number of succesful, failed, recursion requests etc.
OS: Any platform

Steps to get this going are as follows

  1. First make sure that the user who will be executing the collection script is able to execute rndc stats command. You could use this document to learn how to set it up
  2. Download the Ganglia collection script ganglia_bind_stats.pl.txt
  3. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust. Also adjust the location of the rndc script if necessary
  4. Last but not least specify the location of the statistics file with $bind_stats variable. This is same as statistics-file definition in the named.conf.
  5. Run the script by hand to make sure it doesn't error out.
  6. Put it in a cron and run it however often you want.

9. mySQL binary log growth rate (updated 2009-10-01)

Purpose: This script will provide with a running average binary log growth rate. We are not so much concerned by intermediate spikes but overall growth so if your average binary log growth double after a software release you may want to investigate.
OS: UNIX

Steps to get this going are as follows

  1. Download the Ganglia collection script ganglia_mysql_logfile_growth_rate.sh.txt
  2. Make it executable and I would recommend putting in on your mySQL db server
  3. Inside the script please adjust variable LOGFILE_BASE which should contain the base file name for your binary logs ie. usually binary logs will be of the format /var/lib/mysql/binlog.000040. Put in only /var/lib/mysql/binlog
  4. Run the script by hand to make sure it doesn't error out.
  5. Put it in a cron and run it however often you want.


Varnish HTTP accelerator stats (Updated: 2009-11-20)

To get real time Varnish stats in Ganglia please download the cool Ganglia-Logtailer. I have contributed code to parse out Varnish logs as produced varnishncsa. Specifically you invoke it via

/opt/logtailer/ganglia-logtailer --classname VarnishLogtailer --log_file /var/log/varnish varnishncsa.log --mode cron
This will generate real-time metrics counting how many HTTP 200, 300, 400 and 500 requests are being processed by varnish.

Varnish Report Snapshot


Author: Vladimir Vuksan
Page Last Change: $Date: 2010-01-14 20:02:39 -0500 (Thu, 14 Jan 2010) $

Go back