Quick Links

• Apache traffic Stats
• Apache workers (busy/idle) Stats
• Memcached Stats
• NFS server/client stats
• Local Disk Stats
• mySQL server stats
• mySQL binary log growth rate
• BIND DNS server stats
• Postfix Queue Size
• Varnish (HTTP accelerator) Traffic stats

1. Apache traffic stats (updated 2010-01-14)

After you start collecting data you should get something like this :-)

Steps to get this going are as follows

1. Download and install the cool Ganglia-Logtailer contributed graciously to open source by people at Linden Lab.
2. There is a Linden Lab supplied ApacheLogtailer class you can use however that one involves major changes to the logging format which I didn't like. Therefore I modified the stock Apache LogFormat by appending %D which logs the time to service a request in microseconds. To do so define a new LogFormat in Apache configuration ie.

   LogFormat "%h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" %D" combined-time

   or if you are using a proxy such as Haproxy or Varnish you will want to log the IP that proxy passes in the X-Forwarded-For header.

   LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\" %D" combined-haproxy

3. Now you have to attach the log format to your log ie. instead of using the TransferLog you would put in

   CustomLog "/var/log/httpd/access_log" combined-haproxy

4. Restart/Reload Apache so that Apache starts logging in the new format
5. Download my ApacheLogtailer class or use the one that comes with ganglia-logtailer
6. Now configure Logtailer to process the logs per ganglia-logtailer documentation ie. I run it out of the cron

   /opt/logtailer/ganglia-logtailer --classname ApacheLogtailer --log_file /var/log/httpd/access_log --mode cron

7. If everything was set up correctly you should see 7 new metrics ie. apache_hits, apache_200, apache_300, apache_400, apache_500
8. If you want the consolidated graph such as one above you will need to download apache_report.php and put in your ganglia-web directory under graph.d subdirectory.
9. You also have to modify the conf.php. Add

   $optional_graphs = array('apache_report');

10. You also need to modify get_context.php. At the end add

    "apache_report" => 1,

    to the list of report graphs
11. Now apache_report should be available in the Metric drop down box in the Cluster Report. If you want it added to every host you will need to modify the host_view.tpl template by adding something like this

    <IMG BORDER=0 HEIGHT="154" WIDTH="397" ALT="{cluster_url} Apache Report" SRC="./graph.php?g=apache_report&z=medium&c={cluster_url}&{graphargs}">

Useful tip: Monitor apache_500 metrics. HTTP 500 errors should never happen thus if they start happening there is a problem ie. server was misconfigured.

2. Memcached stats (updated 2009-08-03)

Steps to get this going are as follows

1. Install the Cache::Memcached Perl module from CPAN ie.
   • perl -MCPAN -e 'install Cache::Memcached'
2. Download the Ganglia collection script ganglia_memcached.pl
3. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust
4. Run the script by hand to make sure it doesn't error out. You should see something like this

memcache_hit_ratio: 74.37%
memcache_fill_ratio: 89.21%
memcache_items: 280366
memcache_curr_conn: 5

5. Place the script in a cron to run on desired schedule e.g. every minute

3. NFS server/client stats

• Check that $gmetric_command variable points to the gmetric binary
• Currently following NFS calls are being collected - getattr, setattr, lookup, access, read, write, create, remove. If you need additional calls collected adjust the @which_metrics array.
• Put it in a cron and run it however often you want. I recommend running it every minute.
• If you don't supply an argument NFSd (server) statistics will be sent
• To collect NFS client statistics supply -c flag
• If you want both run two separate invocations ie. one with the -c argument and one without
• IMPORTANT: This script collects only NFSv3 statistics.

4. Local Disk Stats (Updated 2009-08-28)

• Check that $gmetric_command variable points to gmetric binary
• Currently following metrics are collected - read, write, io wait queue. If you need additional metrics collected adjust the @which_metrics array.
• Put it in a cron and run it however often you want.
• IMPORTANT: You need to supply disk you want stats as the argument e.g. 
• ./ganglia_disk_stats.pl sda
• if you type cat /proc/diskstats you can see list of all devices

5. mySQL server stats

| Threads_cached                    | 0          |
| Threads_connected                 | 5          |
| Threads_created                   | 14706      |
| Threads_running                   | 1          |

• Check that $gmetric_exec and $gmetric_options variable match your environment
• Also check the stats command and make sure you can actually execute mysqladmin with the credentials you specified.
• Currently following metrics are collected
  • "Bytes_received"
  • "Bytes_sent"
  • "Com_delete"
  • "Com_insert"
  • "Com_replace"
  • "Com_select"
  • "Com_update"
  • "Questions"
  • "Connections"
  • "Slow_queries"
  • "Threads_created"
  • "Threads_running"
• If you need additional metrics collected adjust the %counter_metrics or %absolute_metrics array/hash.
• Difference between counter and absolute metrics is that for certain things mySQL keeps a running counter of things ie. how many bytes have been transferred since the stats were last cleared. In that case we need to calculate the delta/difference. Absolute metrics are non-counter values ie. number of threads_running right now.
• Put it in a cron and run it however often you want.

6. Postfix queue size (updated 2009-08-03)

1. Download the Ganglia collection script ganglia_postfix_mailqueue.sh
2. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust.
3. Run the script by hand to make sure it doesn't error out.
4. Put it in a cron and run it however often you want.

7. Apache Workers Busy/Idle Stats (updated 2009-08-03)

1. You need to configure the Apache mod_status e.g.

   <VirtualHost *:80>
       	ServerName monitoring.site
           <Location /status>
   	  SetHandler server-status
   	  Order deny,allow
   	  Deny from all
   	  Allow from 127.0.0.1
   	</Location>
   </VirtualHost>
   

2. Make sure you can open up the URL.
3. Download the Ganglia collection script ganglia_apache_workers.sh
4. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust.
5. Adjust the URL of the monitoring site specified above.
6. Run the script by hand to make sure it doesn't error out.
7. Put it in a cron and run it however often you want.

8. BIND DNS stats (updated 2009-09-06)

Purpose: This script will provide you DNS statistics such as number of succesful, failed, recursion requests etc.
OS: Any platform

Steps to get this going are as follows

1. First make sure that the user who will be executing the collection script is able to execute rndc stats command. You could use this document to learn how to set it up
2. Download the Ganglia collection script ganglia_bind_stats.pl.txt
3. Place it somewhere and make it executable. Script assumes gmetric is in /usr/bin/gmetric. If not please adjust. Also adjust the location of the rndc script if necessary
4. Last but not least specify the location of the statistics file with $bind_stats variable. This is same as statistics-file definition in the named.conf.
5. Run the script by hand to make sure it doesn't error out.
6. Put it in a cron and run it however often you want.

9. mySQL binary log growth rate (updated 2009-10-01)

Purpose: This script will provide with a running average binary log growth rate. We are not so much concerned by intermediate spikes but overall growth so if your average binary log growth double after a software release you may want to investigate.
OS: UNIX

Steps to get this going are as follows

1. Download the Ganglia collection script ganglia_mysql_logfile_growth_rate.sh.txt
2. Make it executable and I would recommend putting in on your mySQL db server
3. Inside the script please adjust variable LOGFILE_BASE which should contain the base file name for your binary logs ie. usually binary logs will be of the format /var/lib/mysql/binlog.000040. Put in only /var/lib/mysql/binlog
4. Run the script by hand to make sure it doesn't error out.
5. Put it in a cron and run it however often you want.

Varnish HTTP accelerator stats (Updated: 2009-11-20)

To get real time Varnish stats in Ganglia please download the cool Ganglia-Logtailer. I have contributed code to parse out Varnish logs as produced varnishncsa. Specifically you invoke it via

/opt/logtailer/ganglia-logtailer --classname VarnishLogtailer --log_file /var/log/varnish varnishncsa.log --mode cron

• If you want a consolidated graph such as one above you will need to download varnish_report.php and put in your ganglia-web directory under graph.d subdirectory.
• You also have to modify the conf.php. Add

  $optional_graphs = array('varnish_report');

• You also need to modify get_context.php. At the end add

  "varnish_report" => 1,

  to the list of report graphs
• Now varnish_report should be available in the Metric drop down box in the Cluster Report. If you want it added to every host you will need to modify the host_view.tpl template by adding something like this

  <IMG BORDER=0 HEIGHT="154" WIDTH="397" ALT="{cluster_url} Varnish Report" SRC="./graph.php?g=varnish_report&z=medium&c={cluster_url}&{graphargs}">