Giant Ampersand - Washington/Centinela Redevelopment Part 6

It's been almost 5 years since my last post about the corner lots of Washington and Centinela which are supposedly destined for redevelopment. That's because pretty much nothing has happened on those lots - until this past week.

All of the sudden a 8-foot tall white wooden ampersand appeared in the middle of the lot on the north east corner.  I had no idea what it meant, but I was really happy to see it.  To me it meant that someone out there is thinking about this vacant plot of land.

A little Googling produced an answer.  The mysterious ampersand is a public project by Los Angeles-based artist Shannon Ebner.  According to the LAXART web site:

The ampersand is a conjoiner; it signals an incomplete thought and acknowledges that there is more to be said.

This is part of a Shannon Ebner exhibition at the Hammer Museum which started July 16, 2011.  Apparently this isn't the first time she's used an ampersand in her work.  See this photo from a year ago in Milan.

I've been waiting for the redevelopment on this corner to get underway since I moved to this Del Rey neighborhood 11 years ago.  I really hope, as the giant ampersand acknowledges, that there is more to be said...

Using a Canon Pixma MX882 All-In-One with Ubuntu 10.10

Are you thinking about buying a Canon Pixma MX882 Wireless Office All-In-One Printer and wondering if it will work with your Ubuntu computer?  Although Canon does not make this clear in any way, the answer is YES!

+

Instead of using software that comes on a CD with the printer, you'll have to download and install drivers yourself.  They are not easy to find if you live in the US since they're only available on European and Asian Canon websites.  I'll tell you how to download and install them.

Printing

Finding and installing the drivers

1. Visit the Linux IJ Printer Driver page on the Canon Europe web site.
2. Click the Linux IJ Printer Driver MX880 series.tar link in the Downloads section to download the 7299.5 kb tarball.
3. Inside the tarball, find and unpack cnijfilter-mx880series-3.50-1-deb.tar.gz.
4. Inside the resulting cnijfilter-mx880series-3.50-1-deb/packages directory, locate and double click cnijfilter-common_3.50-1_i386.deb and  cnijfilter-mx880series_3.50-1_i386.deb in that order.  Ubuntu will guide you through the installation. 

Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Adding the printer

1. From Ubuntu's menu, select: System,  Administration, then Printing.
2. From Printing window that opens, select Add.
3. After a few seconds,  you should see a Canon MX880 printer show up under Network Printer.
4. Select it and click Forward.
5. Finish entering your printer Name, Description, and Location.

Scanning

Finding and installing the drivers

1. Visit the Linux IJ Scanner Driver page on the Canon Europe web site.
2. Click the Linux IJ Scanner Driver MX880 series.tar link in the Downloads section to download the 3072 kb tarball.
3. Inside the tarball, find and unpack scangearmp-mx880series-1.70-1-deb.tar.gz.
4. Inside the resulting scangearmp-mx880series-1.70-1-deb/packages directory, locate and double click scangearmp-common_1.70-1_i386.deb and  scangearmp-mx880series_1.70-1_i386.deb in that order.  Ubuntu will guide you through the installation. 

Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Scan Gear MP

With the drivers, an app called Scan Gear MP is also installed.  This app serves as an interface to your scanner and has a Basic Mode:

And an Advanced Mode:

You can launch the app from the terminal by typing scangearmp and it is also available within Gimp by selecting File, Create, Scan Gear MP...


I have not yet figure out a way to use the scanner via XSane.  I'd love to hear comments about whether this is possible and how to do it.  Without XSane support, I can't use familiar programs such as SimpleScan and gscan2pdf.

VueScan

One more thing to mention.  Before I figured out Canon made drivers for this scanner, I came across a program called VueScan which claimed to support MX880 series printers on Linux via the network without installing any drivers at all.  It took seconds to download, launch, and see that this program could communicate with my MX882 scanner with ease!  It may be worth the $79.95 if you can't get Canon's drivers to work.

Happy printing, scanning, copying, and faxing!

Edit Remote EC2 Text Files with TextWrangler

This post explains how to use TextWrangler on a Mac to edit text files that live on a remote Amazon EC2 instance with Ubuntu.

I assume you already know how to SSH to Amazon using an Identity File:

ssh -i ~ec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

You can use a similar command to connect to your EC2 instance via SFTP:

sftp -o IdentityFile=eec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

TextWrangler allows you to open a file from an SFTP server, but it doesn't allow you to configure an Identity File as you did when SSHing or SFTPing from the command line. To work around this limitation, you simply have to configure your SSH client so that it knows the Identity File you want to use. Create ~/.ssh/config if it doesn't already exist and add a line specifying your Identity File location:

echo 'IdentityFile ~ec2-keys/id_your-keypair' >> ~/.ssh/config

Now you're ready to use TextWrangler to connect to your server and browse files. Choose: File -> Open from FTP/SFTP Server... Then enter your server publisher DNS name, check the SFTP box and enter  ubuntu as the User name.  You can leave the Password: field blank.

After you hit connect,  you TextWrangler will allow you to browse the file system of your EC2 instance, choose a file, and edit it.  When you save the file, it will update directly on the remote EC2 instance.  That's all there is to it!

Monitor HBase & Hadoop with Ganglia on EC2

This post is a recipe on setting up Ganglia to monitor an HBase and Hadoop cluster on the Ubuntu OS on Amazon EC2. Ganglia is a monitoring system for grids and clusters consisting of the following 3 components:

gmond
A Ganglia Monitoring Daemon (gmond) runs on each node in the cluster and collects statistics from the node it runs on as well as other nodes in the cluster. Normally it is a multicast system where each gmond node receives data from its peers. However, since Amazon EC2 does not support multicast at this time, you must setup Ganglia Monitoring Daemons in unicast mode where each node in a cluster is configured to send its data to one pre-designated node.

gmetad
A Ganglia Meta Daemon (gmetad) runs for each grid and collects data from the Ganglia Monitoring Daemons, one from each cluster. It stores the data it collects on the file system. We'll only be configuring one grid and therefore one gmetad. We'll be running gmetad from the same node that the PHP Web Front End is installed on.

PHP Web Front End
A PHP application reads the data and provides a UI to visualize the data over time with pretty graphs.  Example from UC Berkeley.

Approach
Our HBase cluster has one Master node (with Hadoop's Name Node and Job Tracker) and the rest Region Server nodes (each with a Hadoop Data Node and Task Tracker).  Since we must use Ganglia in unicast mode on EC2, we need to choose one of these nodes to receive data from each of the other nodes.  We'll configure each Region Server gmond to send data to the gmond running on the Master.  A separate node will be used to run gmetad and the PHP web front end.  The gmetad will be configured to poll the Master's gmond for aggregated cluster data.

Getting the right version of Ganglia is important because of compatibility issues with Hadoop and HBase's Ganglia clients. We're using HBase 0.20.6 and Hadoop 0.20.2. Therefore, we're going to use Ganglia 3.0.7, the latest of 3.0.x at the time of writing.  You could use the latest Ganglia 3.1.x if you're willing to apply the patch found in HADOOP-4675 to your Hadoop installation.

Setting up gmond
Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Generate a default config file.

gmond --default_config > /etc/gmond.conf

Configure /etc/gmond.conf

Modify the globals.

globals {
  user = ganglia
}

Define the cluster.

cluster {
  name = HBase
  owner = "Your Company"
  latlong = "N34.02 W118.45"
  url = "http://yourcompany.com/"
}

Disable multicast and define the host, the HBase master, where nodes in the cluster send data.

udp_send_channel {
  # mcast_join = 239.2.11.71
  host = master.yourcompany.com
  port = 8649
  ttl = 1
}

udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  # bind = 239.2.11.71
}

Run gmond:

gmond

Note: As far as I know, there is no way to stop gmond other than killing the process.

You can test each gmond node to make sure it is working. From each node:

telnet localhost 8649

You should see XML output.

Once gmond is installed and running on each node, move on to setup gmetad.

Setting up gmetad
We're going to setup a new EC2 instance to run gmetad and the PHP web front end.  A t1.micro should be sufficient for modest clusters.  gmetad will be configured to poll the gmond running on the HBase master for aggregated cluster data.

Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Copy the default config file.

cp /opt/ganglia-3.0.7/gmetad/gmetad.conf /etc/gmetad.conf

Configure /etc/gmetad.conf

setuid_username "ganglia"
data_source "HBase" master.yourcompany.com
gridname "YourCompany"

Create directories where Ganglia will save its data.

mkdir /var/lib/ganglia
mkdir /var/lib/ganglia/rrds/
chown -R ganglia:ganglia /var/lib/ganglia/

Run gmetad:

gmetad

For troubleshooting, it might be helpful not to daemonize gmetad:

gmetad -d 1

Note: As far as I know, there is no way to stop gmetad other than killing the process.

Setting up the PHP Web Front End
Install Apache and PHP libraries:

apt-get -y install rrdtool apache2 php5-mysql libapache2-mod-php5 php5-gd

Copy PHP web app into Apache's HTML directory:

cp -r /opt/ganglia-3.0.7/web /var/www/ganglia

Restart Apache:

/etc/init.d/apache2 restart

You should now be able to visit the Ganglia console on http://ganglia.yourcompany.com/ganglia and see pretty graphs and charts for machine metrics like load, memory, disk use, etc.

Configuring Hadoop and HBase
The next step is to configure Hadoop and HBase to start sending metrics to the master gmond. Detailed instructions are given here and here for Hadoop and HBase respectively.

Here's what we end up with inside HBase's conf/hadoop-metrics.properties:

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.period=10
hbase.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=master.yourcompany.com:8649

And inside Hadoop's conf/hadoop-metrics.properties:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=master.yourcompany.com:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

Once the hadoop-metrics.properties files are modified on all nodes in your cluster, you'll need to restart HBase and Hadoop.

Metrics
After restarting, you should see the following metrics show up in Ganglia for your HBase and Hadoop cluster:

hbase.regionserver.blockCacheCount
hbase.regionserver.blockCacheFree
hbase.regionserver.blockCacheHitRatio
hbase.regionserver.blockCacheSize
hbase.regionserver.compactionQueueSize
hbase.regionserver.fsReadLatency_avg_time
hbase.regionserver.fsReadLatency_num_ops
hbase.regionserver.fsSyncLatency_avg_time
hbase.regionserver.fsSyncLatency_num_ops
hbase.regionserver.fsWriteLatency_avg_time
hbase.regionserver.fsWriteLatency_num_ops
hbase.regionserver.memstoreSizeMB
hbase.regionserver.regions
hbase.regionserver.requests
hbase.regionserver.storefileIndexSizeMB
hbase.regionserver.storefiles
hbase.regionserver.stores

rpc.metrics.RpcProcessingTime_avg_time
rpc.metrics.RpcProcessingTime_num_ops
rpc.metrics.RpcQueueTime_avg_time
rpc.metrics.RpcQueueTime_num_ops
rpc.metrics.addColumn_avg_time
rpc.metrics.addColumn_num_ops
rpc.metrics.checkAndPut_avg_time
rpc.metrics.checkAndPut_num_ops
rpc.metrics.close_avg_time
rpc.metrics.close_num_ops
rpc.metrics.createTable_avg_time
rpc.metrics.createTable_num_ops
rpc.metrics.deleteColumn_avg_time
rpc.metrics.deleteColumn_num_ops
rpc.metrics.deleteTable_avg_time
rpc.metrics.deleteTable_num_ops
rpc.metrics.delete_avg_time
rpc.metrics.delete_num_ops
rpc.metrics.disableTable_avg_time
rpc.metrics.disableTable_num_ops
rpc.metrics.enableTable_avg_time
rpc.metrics.enableTable_num_ops
rpc.metrics.exists_avg_time
rpc.metrics.exists_num_ops
rpc.metrics.getClosestRowBefore_avg_time
rpc.metrics.getClosestRowBefore_num_ops
rpc.metrics.getClusterStatus_avg_time
rpc.metrics.getClusterStatus_num_ops
rpc.metrics.getHServerInfo_avg_time
rpc.metrics.getHServerInfo_num_ops
rpc.metrics.getOnlineRegionsAsArray_avg_time
rpc.metrics.getOnlineRegionsAsArray_num_ops
rpc.metrics.getProtocolVersion_avg_time
rpc.metrics.getProtocolVersion_num_ops
rpc.metrics.getRegionInfo_avg_time
rpc.metrics.getRegionInfo_num_ops
rpc.metrics.getRegionsAssignment_avg_time
rpc.metrics.getRegionsAssignment_num_ops
rpc.metrics.get_avg_time
rpc.metrics.get_num_ops
rpc.metrics.incrementColumnValue_avg_time
rpc.metrics.incrementColumnValue_num_ops
rpc.metrics.isMasterRunning_avg_time
rpc.metrics.isMasterRunning_num_ops
rpc.metrics.lockRow_avg_time
rpc.metrics.lockRow_num_ops
rpc.metrics.modifyColumn_avg_time
rpc.metrics.modifyColumn_num_ops
rpc.metrics.modifyTable_avg_time
rpc.metrics.modifyTable_num_ops
rpc.metrics.next_avg_time
rpc.metrics.next_num_ops
rpc.metrics.openScanner_avg_time
rpc.metrics.openScanner_num_ops
rpc.metrics.put_avg_time
rpc.metrics.put_num_ops
rpc.metrics.regionServerReport_avg_time
rpc.metrics.regionServerReport_num_ops
rpc.metrics.regionServerStartup_avg_time
rpc.metrics.regionServerStartup_num_ops
rpc.metrics.shutdown_avg_time
rpc.metrics.shutdown_num_ops
rpc.metrics.unlockRow_avg_time
rpc.metrics.unlockRow_num_ops

dfs.datanode.blockChecksumOp_avg_time
dfs.datanode.blockChecksumOp_num_ops
dfs.datanode.blockReports_avg_time
dfs.datanode.blockReports_num_ops
dfs.datanode.block_verification_failures
dfs.datanode.blocks_read
dfs.datanode.blocks_removed
dfs.datanode.blocks_replicated
dfs.datanode.blocks_verified
dfs.datanode.blocks_written
dfs.datanode.bytes_read
dfs.datanode.bytes_written
dfs.datanode.copyBlockOp_avg_time
dfs.datanode.copyBlockOp_num_ops
dfs.datanode.heartBeats_avg_time
dfs.datanode.heartBeats_num_ops
dfs.datanode.readBlockOp_avg_time
dfs.datanode.readBlockOp_num_ops
dfs.datanode.readMetadataOp_avg_time
dfs.datanode.readMetadataOp_num_ops
dfs.datanode.reads_from_local_client
dfs.datanode.reads_from_remote_client
dfs.datanode.replaceBlockOp_avg_time
dfs.datanode.replaceBlockOp_num_ops
dfs.datanode.writeBlockOp_avg_time
dfs.datanode.writeBlockOp_num_ops
dfs.datanode.writes_from_local_client
dfs.datanode.writes_from_remote_client

jvm.metrics.gcCount
jvm.metrics.gcTimeMillis
jvm.metrics.logError
jvm.metrics.logFatal
jvm.metrics.logInfo
jvm.metrics.logWarn
jvm.metrics.memHeapCommittedM
jvm.metrics.memHeapUsedM
jvm.metrics.memNonHeapCommittedM
jvm.metrics.memNonHeapUsedM
jvm.metrics.threadsBlocked
jvm.metrics.threadsNew
jvm.metrics.threadsRunnable
jvm.metrics.threadsTerminated
jvm.metrics.threadsTimedWaiting
jvm.metrics.threadsWaiting

As you can see, the metrics are incredibly useful for understanding the health of your HBase and Hadoop cluster over time.

Swivel4J - A New Java Client for Swivel

At work, we were looking for a simple way to visualize business data without having to build and host it ourselves. This led us to Swivel (the new Swivel as they call it which is geared towards private data shared within an organization). With Swivel, we can upload our data, create pretty charts and graphs, and share them privately within our company.

Swivel has a REST API for managing the charts. Though not difficult, we found it a little unwieldy to have to use an HTTP client to setup each API call. This naturally led to a small library that wrapped the REST API calls in a native Java API.

We decided to make this library available to everyone as Swivel4J which is open source and hosted on Google Code. We hope this makes it easier for Java, Groovy, and Scala developers to get started with Swivel and promotes the use of Swivel in general.

Please try it out and give feedback, submit patches, etc.