Monitor HBase & Hadoop with Ganglia on EC2

This post is a recipe on setting up Ganglia to monitor an HBase and Hadoop cluster on the Ubuntu OS on Amazon EC2. Ganglia is a monitoring system for grids and clusters consisting of the following 3 components:

gmond
A Ganglia Monitoring Daemon (gmond) runs on each node in the cluster and collects statistics from the node it runs on as well as other nodes in the cluster. Normally it is a multicast system where each gmond node receives data from its peers. However, since Amazon EC2 does not support multicast at this time, you must setup Ganglia Monitoring Daemons in unicast mode where each node in a cluster is configured to send its data to one pre-designated node.

gmetad
A Ganglia Meta Daemon (gmetad) runs for each grid and collects data from the Ganglia Monitoring Daemons, one from each cluster. It stores the data it collects on the file system. We'll only be configuring one grid and therefore one gmetad. We'll be running gmetad from the same node that the PHP Web Front End is installed on.

PHP Web Front End
A PHP application reads the data and provides a UI to visualize the data over time with pretty graphs.  Example from UC Berkeley.

Approach
Our HBase cluster has one Master node (with Hadoop's Name Node and Job Tracker) and the rest Region Server nodes (each with a Hadoop Data Node and Task Tracker).  Since we must use Ganglia in unicast mode on EC2, we need to choose one of these nodes to receive data from each of the other nodes.  We'll configure each Region Server gmond to send data to the gmond running on the Master.  A separate node will be used to run gmetad and the PHP web front end.  The gmetad will be configured to poll the Master's gmond for aggregated cluster data.

Getting the right version of Ganglia is important because of compatibility issues with Hadoop and HBase's Ganglia clients. We're using HBase 0.20.6 and Hadoop 0.20.2. Therefore, we're going to use Ganglia 3.0.7, the latest of 3.0.x at the time of writing.  You could use the latest Ganglia 3.1.x if you're willing to apply the patch found in HADOOP-4675 to your Hadoop installation.

Setting up gmond
Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Generate a default config file.

gmond --default_config > /etc/gmond.conf

Configure /etc/gmond.conf

Modify the globals.

globals {
  user = ganglia
}

Define the cluster.

cluster {
  name = HBase
  owner = "Your Company"
  latlong = "N34.02 W118.45"
  url = "http://yourcompany.com/"
}

Disable multicast and define the host, the HBase master, where nodes in the cluster send data.

udp_send_channel {
  # mcast_join = 239.2.11.71
  host = master.yourcompany.com
  port = 8649
  ttl = 1
}

udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  # bind = 239.2.11.71
}

Run gmond:

gmond

Note: As far as I know, there is no way to stop gmond other than killing the process.

You can test each gmond node to make sure it is working. From each node:

telnet localhost 8649

You should see XML output.

Once gmond is installed and running on each node, move on to setup gmetad.

Setting up gmetad
We're going to setup a new EC2 instance to run gmetad and the PHP web front end.  A t1.micro should be sufficient for modest clusters.  gmetad will be configured to poll the gmond running on the HBase master for aggregated cluster data.

Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Copy the default config file.

cp /opt/ganglia-3.0.7/gmetad/gmetad.conf /etc/gmetad.conf

Configure /etc/gmetad.conf

setuid_username "ganglia"
data_source "HBase" master.yourcompany.com
gridname "YourCompany"

Create directories where Ganglia will save its data.

mkdir /var/lib/ganglia
mkdir /var/lib/ganglia/rrds/
chown -R ganglia:ganglia /var/lib/ganglia/

Run gmetad:

gmetad

For troubleshooting, it might be helpful not to daemonize gmetad:

gmetad -d 1

Note: As far as I know, there is no way to stop gmetad other than killing the process.

Setting up the PHP Web Front End
Install Apache and PHP libraries:

apt-get -y install rrdtool apache2 php5-mysql libapache2-mod-php5 php5-gd

Copy PHP web app into Apache's HTML directory:

cp -r /opt/ganglia-3.0.7/web /var/www/ganglia

Restart Apache:

/etc/init.d/apache2 restart

You should now be able to visit the Ganglia console on http://ganglia.yourcompany.com/ganglia and see pretty graphs and charts for machine metrics like load, memory, disk use, etc.

Configuring Hadoop and HBase
The next step is to configure Hadoop and HBase to start sending metrics to the master gmond. Detailed instructions are given here and here for Hadoop and HBase respectively.

Here's what we end up with inside HBase's conf/hadoop-metrics.properties:

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.period=10
hbase.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=master.yourcompany.com:8649

And inside Hadoop's conf/hadoop-metrics.properties:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=master.yourcompany.com:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

Once the hadoop-metrics.properties files are modified on all nodes in your cluster, you'll need to restart HBase and Hadoop.

Metrics
After restarting, you should see the following metrics show up in Ganglia for your HBase and Hadoop cluster:

hbase.regionserver.blockCacheCount
hbase.regionserver.blockCacheFree
hbase.regionserver.blockCacheHitRatio
hbase.regionserver.blockCacheSize
hbase.regionserver.compactionQueueSize
hbase.regionserver.fsReadLatency_avg_time
hbase.regionserver.fsReadLatency_num_ops
hbase.regionserver.fsSyncLatency_avg_time
hbase.regionserver.fsSyncLatency_num_ops
hbase.regionserver.fsWriteLatency_avg_time
hbase.regionserver.fsWriteLatency_num_ops
hbase.regionserver.memstoreSizeMB
hbase.regionserver.regions
hbase.regionserver.requests
hbase.regionserver.storefileIndexSizeMB
hbase.regionserver.storefiles
hbase.regionserver.stores

rpc.metrics.RpcProcessingTime_avg_time
rpc.metrics.RpcProcessingTime_num_ops
rpc.metrics.RpcQueueTime_avg_time
rpc.metrics.RpcQueueTime_num_ops
rpc.metrics.addColumn_avg_time
rpc.metrics.addColumn_num_ops
rpc.metrics.checkAndPut_avg_time
rpc.metrics.checkAndPut_num_ops
rpc.metrics.close_avg_time
rpc.metrics.close_num_ops
rpc.metrics.createTable_avg_time
rpc.metrics.createTable_num_ops
rpc.metrics.deleteColumn_avg_time
rpc.metrics.deleteColumn_num_ops
rpc.metrics.deleteTable_avg_time
rpc.metrics.deleteTable_num_ops
rpc.metrics.delete_avg_time
rpc.metrics.delete_num_ops
rpc.metrics.disableTable_avg_time
rpc.metrics.disableTable_num_ops
rpc.metrics.enableTable_avg_time
rpc.metrics.enableTable_num_ops
rpc.metrics.exists_avg_time
rpc.metrics.exists_num_ops
rpc.metrics.getClosestRowBefore_avg_time
rpc.metrics.getClosestRowBefore_num_ops
rpc.metrics.getClusterStatus_avg_time
rpc.metrics.getClusterStatus_num_ops
rpc.metrics.getHServerInfo_avg_time
rpc.metrics.getHServerInfo_num_ops
rpc.metrics.getOnlineRegionsAsArray_avg_time
rpc.metrics.getOnlineRegionsAsArray_num_ops
rpc.metrics.getProtocolVersion_avg_time
rpc.metrics.getProtocolVersion_num_ops
rpc.metrics.getRegionInfo_avg_time
rpc.metrics.getRegionInfo_num_ops
rpc.metrics.getRegionsAssignment_avg_time
rpc.metrics.getRegionsAssignment_num_ops
rpc.metrics.get_avg_time
rpc.metrics.get_num_ops
rpc.metrics.incrementColumnValue_avg_time
rpc.metrics.incrementColumnValue_num_ops
rpc.metrics.isMasterRunning_avg_time
rpc.metrics.isMasterRunning_num_ops
rpc.metrics.lockRow_avg_time
rpc.metrics.lockRow_num_ops
rpc.metrics.modifyColumn_avg_time
rpc.metrics.modifyColumn_num_ops
rpc.metrics.modifyTable_avg_time
rpc.metrics.modifyTable_num_ops
rpc.metrics.next_avg_time
rpc.metrics.next_num_ops
rpc.metrics.openScanner_avg_time
rpc.metrics.openScanner_num_ops
rpc.metrics.put_avg_time
rpc.metrics.put_num_ops
rpc.metrics.regionServerReport_avg_time
rpc.metrics.regionServerReport_num_ops
rpc.metrics.regionServerStartup_avg_time
rpc.metrics.regionServerStartup_num_ops
rpc.metrics.shutdown_avg_time
rpc.metrics.shutdown_num_ops
rpc.metrics.unlockRow_avg_time
rpc.metrics.unlockRow_num_ops

dfs.datanode.blockChecksumOp_avg_time
dfs.datanode.blockChecksumOp_num_ops
dfs.datanode.blockReports_avg_time
dfs.datanode.blockReports_num_ops
dfs.datanode.block_verification_failures
dfs.datanode.blocks_read
dfs.datanode.blocks_removed
dfs.datanode.blocks_replicated
dfs.datanode.blocks_verified
dfs.datanode.blocks_written
dfs.datanode.bytes_read
dfs.datanode.bytes_written
dfs.datanode.copyBlockOp_avg_time
dfs.datanode.copyBlockOp_num_ops
dfs.datanode.heartBeats_avg_time
dfs.datanode.heartBeats_num_ops
dfs.datanode.readBlockOp_avg_time
dfs.datanode.readBlockOp_num_ops
dfs.datanode.readMetadataOp_avg_time
dfs.datanode.readMetadataOp_num_ops
dfs.datanode.reads_from_local_client
dfs.datanode.reads_from_remote_client
dfs.datanode.replaceBlockOp_avg_time
dfs.datanode.replaceBlockOp_num_ops
dfs.datanode.writeBlockOp_avg_time
dfs.datanode.writeBlockOp_num_ops
dfs.datanode.writes_from_local_client
dfs.datanode.writes_from_remote_client

jvm.metrics.gcCount
jvm.metrics.gcTimeMillis
jvm.metrics.logError
jvm.metrics.logFatal
jvm.metrics.logInfo
jvm.metrics.logWarn
jvm.metrics.memHeapCommittedM
jvm.metrics.memHeapUsedM
jvm.metrics.memNonHeapCommittedM
jvm.metrics.memNonHeapUsedM
jvm.metrics.threadsBlocked
jvm.metrics.threadsNew
jvm.metrics.threadsRunnable
jvm.metrics.threadsTerminated
jvm.metrics.threadsTimedWaiting
jvm.metrics.threadsWaiting

As you can see, the metrics are incredibly useful for understanding the health of your HBase and Hadoop cluster over time.

Swivel4J - A New Java Client for Swivel

At work, we were looking for a simple way to visualize business data without having to build and host it ourselves. This led us to Swivel (the new Swivel as they call it which is geared towards private data shared within an organization). With Swivel, we can upload our data, create pretty charts and graphs, and share them privately within our company.

Swivel has a REST API for managing the charts. Though not difficult, we found it a little unwieldy to have to use an HTTP client to setup each API call. This naturally led to a small library that wrapped the REST API calls in a native Java API.

We decided to make this library available to everyone as Swivel4J which is open source and hosted on Google Code. We hope this makes it easier for Java, Groovy, and Scala developers to get started with Swivel and promotes the use of Swivel in general.

Please try it out and give feedback, submit patches, etc.

Amazon ELB - Capturing Client IP Address

If you're using Amazon EC2's Elastic Load Balancer (ELB) for load balancing web applications, you may have noticed that in your web access logs, the remote host IP address is the same for every request. The IP address you see is the private IP address of the load balancer.

If you want to see the IP address of the client (called remote host in access log documentation), you'll need to look at the value of the X-Forwarded-For request header which ELB populates when it forwards the request.

This can be achieved in an Apache access log by using the syntax:

%{X-Forwarded-For}i

You Apache log format would then look something like this:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-elb
CustomLog log/acces_log combined-elb

See Apache Custom Log Formats

If you're using a Tomcat application server, you could define an access log Valve like this:

<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs" prefix="access_log." suffix=".txt"
    pattern="%{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
    resolveHosts="false"/>

See Tomcat 6 Valve Configuration Reference.

If you're trying to capture the client IP address within your application code, simply use whatever API you have to read the X-Forwarded-For request header. For example in Java use the HttpServletRequest:

String clientIpAddress = request.getHeader("X-Forwarded-For");

instead of

String clientIpAddress = request.getRemoteAddr();

Serving Gzipped Javascript Files from Amazon CloudFront

It is well-known that using a content delivery network (CDN) and compressing an HTTP response with gzip can significantly improve the performance and reduce the cost of a web site. If your CDN is Amazon's CloudFront, you'll face difficulties serving gzipped content to browsers that support it. Most web servers are able to examine the HTTP request headers sent by the browser and dynamically choose whether to deliver compressed or uncompressed content. CloudFront, however, does not yet offer this feature.

People have overcome this problem by manually inspecting the Accept-Encoding request header for a dynamically served page and writing references to either original or compressed files accordingly using a naming convention like myscript.js and myscript.js.gz for regular and compressed files respectively.

If all your pages are statically served from CloudFront, however, there is no opportunity to inspect the request headers. Javascript running on a statically served page has no ability to ask the browser if it supports gzipped content. We found ourselves in this situation and resorted to the following solution to determine if the browser supports gzip:

1) Create a small gzipped file, gzipcheck.js.jgz, and make it available in CloudFront. This file should contain one line of code:

gzipEnabled = true;

2) Use the following code to attempt to load and run this file. You'll probably want to put it in the HTML HEAD section before any other Javascript code.

<script type="text/javascript" src="gzipcheck.js.jgz">
</script>

If the file loads, it sets a flag, gzipEnabled, that indicates whether or not the browser supports gzip.

Use the result to drive a file naming convention for references to other static files. For example, you can upload and reference each compressed Javascript file with an additional .jgz extension. Why .jgz instead of .gz? Because of an annoying limitation of Safari.

When you upload your files to S3/CloudFront, make sure to set the proper HTTP response headers on your files that end in .jgz:

Content-Encoding = gzip
Content-Type = application/x-javascript

And if you want browsers to cache your files forever (almost):

Cache-Control = max-age=315360000
Expires = Tue, 31 Dec 2019 20:00:00 GMT

Thanks to Ricardo Rangel who helped design and code this solution.

Update

As of December 20, 2015, Amazon Web Services announced support for Gzipped CloudFront files!  See Serving Compressed Files from their developer guide.

Ubuntu License Plate on the Santa Monica Freeway

I came across this car while driving eastbound on the Santa Monica Freeway this past Saturday. Do you think the license plate is referring to the Ubuntu Linux OS or just the Ubuntu philosophy?

It also had some funny stickers and decals:
* Evolve FISH Logo Decorative Silver Car Emblem
* Goodnight Bush: A Parody
* Got Hope Bumper Sticker