Sia Not a Fit for Personal Backups

In this post, I will describe my experience trying to use Sia as personal backup service. Sia is a decentralized cloud storage platform.  I heard about Sia back in late 2017 when just about everyone was talking about cryptocurrencies, altcoins, and blockchains.  It appears to still be active 3 years later, so I took that as a good sign.  Spoiler alert - I got everything working, but learned that Sia is not a good fit for a personal backup service.

Alternative to CrashPlan

Ever since CrashPlan changed their focus to enterprise users and raise their price to a minimum of $10/month, I have been looking for a lower-cost file backup alternative for my home computer.  I finally decided to give Sia a try. If it worked out, I could reduce my annual backup expense from $120 to just $5!

Before I get into my Sia experience, here is an overview of all the steps I will attempt:

1. Read about Sia (30 minutes)
2. Download Sia UI (5 minutes)
3. Wait for blockchain syncing to complete (8 hours)
4. Purchase Siacoin in Bittrex (30 minutes).  Note, if you don't already have an account with a crypto exchange that supports Siacoin, then this step might take up to 24 hours to get set up.
5. Transfer Siacoin to Sia UI wallet (60 minutes)
6. Download SiaSync (10 minutes)
7. Sync files (3 hours)
8. Verify ability to restore on a different computer (4 hours)

Sia Terminology

I did a little reading on Sia's website to get familiar with their terminology.  I am a Renter that will be uploading files to other computers owned by other people called Hosts.  For that privilege, I will be configuring an Allowance of Siacoin (SIA), which is how much money I'm willing to spend.

The Sia UI

My home computer is an Ubuntu 18.04 desktop.  I downloaded the SIA UI Linux application from their Getting Started page. Double clicking it didn't do anything.  I made it executable and then launched it from the command line:

chmod +x Sia-UI-v1.5.3.AppImage

./Sia-UI-v1.5.3.AppImage

This brought up the SIA UI.  It assigned me a Sia Seed which I saved as a secure LastPass note.   The app displayed a message saying it was syncing and I could see the Block Height continue to increase, but I didn't know what that meant.  I selected Rent from the menu and then Setup Allowance.  This launched a calculator where I plugged in my estimated initial usage of 0.1 TB (100 GB).  It said the target price was 1250 SC which is the cost of a TB per month.  I wasn't sure if that is the current market rate or some kind of bid that I will be placing, but I kept it as-is, and continued.  Next I was asked if I am sure I want to set an allowance of 250 SC to rent storage on Sia.  I confirmed and was then given an error message saying that I have to wait for Sia to finish syncing before I can set an allowance.  I actually tried this several times because at first it seemed like nothing happened.  Later I found out that all of these contracts were queuing up and would go into effect as soon as my SC arrived in my wallet.  I wish the Sia UI had shown me these pending contracts so I wouldn't have generated so many!  My Block Height had reached over 40,000 by this point, but there was no indication of how long I'd have to wait for syncing to complete.

The Big Wait

I found this support post Why does Sia take so long to sync? which explained that before getting started, Sia would have to download an entire copy of its blockchain (over 22 GB) which could take several hours to over a day!  They should really tell you this when you first open the UI.  I decided to leave this to finish syncing over night and come back in the morning.  According to SiaStats, the Block Height needed to get to 292,652.  Sure enough, it had completed syncing by the morning.  The Dashboard showed 292,652 Block Height, 9 Connected Peers, and 203 Active Hosts.

With the Sync completed, I selected Rent in the UI and tried to set up an Allowance. I received a message saying Allowance successfully updated.  However, nothing in the Rent or Wallet section updated, so I have a feeling nothing actually happened since I don't have any Siacoin yet.

Getting Siacoin

The next step was to obtain some Siacoin (SC) I could use to purchase a contract.  Luckily I already had a Bittrex account which is one of the exchanges where you can trade for Siacoin.  Siacoin was currently trading at 1 SC for 0.00000013 BTC.  I tried to buy 1,500 SC for 0.00019568 BTC ($5.30 USD) which I estimated would last me a whole year.  Bittrex warned me that the minimum trade was 0.0005 BTC.  So I increased to 4,500 SC for 0.00058704 BTC ($15.91 USD) and tried again.

Bittrex Limit Order of SC

Transferring Siacoin

Once the order went through, I had to transfer the SC out of Bittrex and into my Sia UI Wallet.  Within the Sia UI, I went to Wallet, then Receive, then Generate New Address.  I initiated a Withdrawal within Bittrex, and entered the wallet address that I just generated from Sia UI.  The withdrawal showed as Completed in Bittrex pretty quickly.  However, I had to wait about 7 minutes before it showed up in the Sia UI.  I was relieved it showed up, though, as cryptocurrency transfers generally make me nervous.  The transaction status showed as 0/6, and I had to wait another 10 minutes before transaction fully completed with status of 6/6.  I transferred 1,500 SC, but was charged a 0.1 SC transaction fee, so 1499.9 SC ended up in my Sia UI wallet.

Contracts

Looks like 50 Contracts got created and and 143.52 SC were allocated when I attempted to set up an Allowance several times earlier.  I still don't really understand Contracts well enough to know how many I should have.  For now I'm going to assume that I have more than what I need and move on.

Syncing Files

To set up file syncing between my desktop and the Sia network, I turned to Siasync.  As new files are created or removed from a source folder, it will add or remove them from the Sia network accordingly.  This replicates the behavior I had with CrashPlan and what we're all used to with services like Dropbox.  Following their instructions, I downloaded the Siasync binaries for the latest version, 1.0.1 beta 1 and made the file executable from the command line:

chmod +x Siasync-linux-amd64

I was hoping I could make a single folder containing symlinks to other folders I wanted to sync.  I tried it, but Siasync wasn't recognizing any symlinked folders.  To sync multiple folders as subfolders to a single backup folder on the Sia network, I used the following separate commands:

./Siasync-linux-amd64 -sync-only -size-only -subfolder backups/Desktop ~/Desktop

./Siasync-linux-amd64 -sync-only -size-only -subfolder backups/Music ~/Music

./Siasync-linux-amd64 -sync-only -size-only -subfolder backups/Pictures ~/Pictures

./Siasync-linux-amd64 -sync-only -size-only -debug -subfolder backups/Videos ~/Videos

This worked, and all my files got backed up on the Sia network!

While searching for solutions on syncing using symlinks, I came across this Sia Cloud Storage Extension for GNOME which promises to take the place of Siasync with support for symlinks that you can put in a synced ~/Sia folder.  I made a note to try that later. 

Sanity Check

As a last step, I wanted to check if I could access my files from another computer. To do this, I downloaded Sia on a Macbook Pro. To try to save some time, I copied the consensus.db file (all 22 GB of it) from my Ubuntu desktop (192.168.0.51) so Sia UI wouldn't have to download it again:

scp ken@192.168.0.51:.config/Sia-UI/sia/consensus/consensus.db ~/Library/Application Support/Sia-UI/sia/consensus/.

This only took a few minutes, but Sia UI was still taking hours to finish configuring itself.

While waiting, I checked back with the Sia UI on my Ubuntu desktop and saw that the file uploads appeared to be stuck.   The health of each file was still at 0% and the number of items uploading wasn't changing.  I read lots of Sia documentation, but I couldn't figure out if something was wrong or if I just had to wait longer.  I found that Sia had a Discord server, so I signed on there to try to get help from the community.  I posted about my problem in the #help channel, and a nice person named Covalent helped me out.  He asked me to show him the output of siac renter -v.  After seeing that he concluded that my allowance was way too small.  He also explained to me that each file would take up a minimum of 40 MB!  With over 35,000 files, I was going to have to pay for over 1.4 TB of data!  That would make my backups cost even more than I was paying for CrashPlan.  Sia docs mention this 40 MB atomic file size here, but I didn't see anything about this in the Getting Started docs.  They should really make this clear for people up front.  If I knew that going into it, I wouldn't have done any of this.

Final Thoughts

It's been a long 24 hours learning about Sia and getting it set up as a personal backup system. In the end, it just doesn't seem well-suited for this use case, at least not in an affordable way.  I might try zipping up my directories and storing them on Sia as fewer, but larger compressed files. If that proves to be too convenient, I'll be sticking with CrashPlan for the time being.

Using a Brother MFC-J450DW All-In-One with Ubuntu

Are you thinking about buying a Brother MFC-J450DW Wireless Office All-In-One Printer/Scanner/Copier/Fax Machine and wondering if it will work with your Ubuntu computer? The answer is YES! Brother actually includes Linux on their Supported OS Page.  I was able to get things working on Ubuntu 14.04 LTS "Trusty Tahr".

+

Brother Software Installation

Driver Downloads

The CD provided with the printer contains Drivers, Utilities, and Users Manuals, but only for Windows and Macintosh computers.  To get the Linux Drivers, you must visit the MFC-J450DW Downloads Page.  Select Linux as the OS and Linux (deb) as the OS Version or go directly to this page.  Follow directions to download and install the Driver Install Tool.

From a terminal:

gunzip linux-brprinter-installer-*.*.*-*.gz
sudo su -
bash linux-brprinter-installer-*.*.*-* MFC-J450DW
Will you specify the DeviceURI? Y, 14 (Auto)

After all this, you should be able to see your new printer:

System Settings -> Printers -> MFCJ450DW

You should also be able to scan with the application Simple Scan which comes with Ubuntu.

No Brother Software Installation

If you would rather not install any software on your computer whatsoever, the MFC-J450DW makes this possible.  I actually prefer this because it makes it easy to print and scan to and from any device you might have including Macs and iOS devices.

Google Cloud Print

Another option is to setup Google Cloud Print which lets you print from any device via your Google Account.  Brother's Google Cloud Print Instruction Page just tells you to find the right manual for your model.  If you visit the Brother MFC-J450DW Manuals Page, you will see the Brother Google Cloud Print Guide Page which let's you download the actual Brother Google Cloud Print Guide [PDF].  To save time, just make sure you have a Google Account and then go directly to your printer's admin web interface which is accessible via a web browser at http://[PRINTER_IP_ADDRESS]/ where PRINTER_IP_ADDRESS is the address of your printer on the local network.  I was able to obtain this IP address by looking at my router's list of connected clients.  You can also find out via the printer's control panel by selecting Menu -> Network -> WLAN -> TCP/IP -> IP Address.

The printer's Google Cloud Print admin page, at http://[PRINTER_IP_ADDRESS]/net/net/gcp.html?pageid=1084, only let's you associate the printer with one Google account.  However Google let's you share the printer with other people very easily.

Scan to the Web

Another option for scanning is to use Brother Web Connect.  Instructions can be found at the Brother Web Connect Page which lets you download the Brother Web Connect Guide [PDF].  I found it very easy to setup Web Connect for Google Drive and Dropbox.  Other options available that I haven't tried yet are Picassa, Flickr, Facebook, Evernote, Onedrive, and Box.  After you get it setup, you simply use the printer's control panel with WEB -> [Service] -> [Configured Service].  After scanning, the file will show up in your Configured Service in a folder called From_BrotherDevice.  So easy!

Please comment if you have any problems or other tips for people trying to get the Brother MFC-J450DW (or similar printer) working with Ubuntu.

Using a Canon Pixma MX882 All-In-One with Ubuntu 10.10

Are you thinking about buying a Canon Pixma MX882 Wireless Office All-In-One Printer and wondering if it will work with your Ubuntu computer?  Although Canon does not make this clear in any way, the answer is YES!

+

Instead of using software that comes on a CD with the printer, you'll have to download and install drivers yourself.  They are not easy to find if you live in the US since they're only available on European and Asian Canon websites.  I'll tell you how to download and install them.

Printing

Finding and installing the drivers

1. Visit the Linux IJ Printer Driver page on the Canon Europe web site.
2. Click the Linux IJ Printer Driver MX880 series.tar link in the Downloads section to download the 7299.5 kb tarball.
3. Inside the tarball, find and unpack cnijfilter-mx880series-3.50-1-deb.tar.gz.
4. Inside the resulting cnijfilter-mx880series-3.50-1-deb/packages directory, locate and double click cnijfilter-common_3.50-1_i386.deb and  cnijfilter-mx880series_3.50-1_i386.deb in that order.  Ubuntu will guide you through the installation. 

Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Adding the printer

1. From Ubuntu's menu, select: System,  Administration, then Printing.
2. From Printing window that opens, select Add.
3. After a few seconds,  you should see a Canon MX880 printer show up under Network Printer.
4. Select it and click Forward.
5. Finish entering your printer Name, Description, and Location.

Scanning

Finding and installing the drivers

1. Visit the Linux IJ Scanner Driver page on the Canon Europe web site.
2. Click the Linux IJ Scanner Driver MX880 series.tar link in the Downloads section to download the 3072 kb tarball.
3. Inside the tarball, find and unpack scangearmp-mx880series-1.70-1-deb.tar.gz.
4. Inside the resulting scangearmp-mx880series-1.70-1-deb/packages directory, locate and double click scangearmp-common_1.70-1_i386.deb and  scangearmp-mx880series_1.70-1_i386.deb in that order.  Ubuntu will guide you through the installation. 

Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Scan Gear MP

With the drivers, an app called Scan Gear MP is also installed.  This app serves as an interface to your scanner and has a Basic Mode:

And an Advanced Mode:

You can launch the app from the terminal by typing scangearmp and it is also available within Gimp by selecting File, Create, Scan Gear MP...


I have not yet figure out a way to use the scanner via XSane.  I'd love to hear comments about whether this is possible and how to do it.  Without XSane support, I can't use familiar programs such as SimpleScan and gscan2pdf.

VueScan

One more thing to mention.  Before I figured out Canon made drivers for this scanner, I came across a program called VueScan which claimed to support MX880 series printers on Linux via the network without installing any drivers at all.  It took seconds to download, launch, and see that this program could communicate with my MX882 scanner with ease!  It may be worth the $79.95 if you can't get Canon's drivers to work.

Happy printing, scanning, copying, and faxing!

Edit Remote EC2 Text Files with TextWrangler

This post explains how to use TextWrangler on a Mac to edit text files that live on a remote Amazon EC2 instance with Ubuntu.

I assume you already know how to SSH to Amazon using an Identity File:

ssh -i ~ec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

You can use a similar command to connect to your EC2 instance via SFTP:

sftp -o IdentityFile=eec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

TextWrangler allows you to open a file from an SFTP server, but it doesn't allow you to configure an Identity File as you did when SSHing or SFTPing from the command line. To work around this limitation, you simply have to configure your SSH client so that it knows the Identity File you want to use. Create ~/.ssh/config if it doesn't already exist and add a line specifying your Identity File location:

echo 'IdentityFile ~ec2-keys/id_your-keypair' >> ~/.ssh/config

Now you're ready to use TextWrangler to connect to your server and browse files. Choose: File -> Open from FTP/SFTP Server... Then enter your server publisher DNS name, check the SFTP box and enter  ubuntu as the User name.  You can leave the Password: field blank.

After you hit connect,  you TextWrangler will allow you to browse the file system of your EC2 instance, choose a file, and edit it.  When you save the file, it will update directly on the remote EC2 instance.  That's all there is to it!

Monitor HBase & Hadoop with Ganglia on EC2

This post is a recipe on setting up Ganglia to monitor an HBase and Hadoop cluster on the Ubuntu OS on Amazon EC2. Ganglia is a monitoring system for grids and clusters consisting of the following 3 components:

gmond
A Ganglia Monitoring Daemon (gmond) runs on each node in the cluster and collects statistics from the node it runs on as well as other nodes in the cluster. Normally it is a multicast system where each gmond node receives data from its peers. However, since Amazon EC2 does not support multicast at this time, you must setup Ganglia Monitoring Daemons in unicast mode where each node in a cluster is configured to send its data to one pre-designated node.

gmetad
A Ganglia Meta Daemon (gmetad) runs for each grid and collects data from the Ganglia Monitoring Daemons, one from each cluster. It stores the data it collects on the file system. We'll only be configuring one grid and therefore one gmetad. We'll be running gmetad from the same node that the PHP Web Front End is installed on.

PHP Web Front End
A PHP application reads the data and provides a UI to visualize the data over time with pretty graphs.  Example from UC Berkeley.

Approach
Our HBase cluster has one Master node (with Hadoop's Name Node and Job Tracker) and the rest Region Server nodes (each with a Hadoop Data Node and Task Tracker).  Since we must use Ganglia in unicast mode on EC2, we need to choose one of these nodes to receive data from each of the other nodes.  We'll configure each Region Server gmond to send data to the gmond running on the Master.  A separate node will be used to run gmetad and the PHP web front end.  The gmetad will be configured to poll the Master's gmond for aggregated cluster data.

Getting the right version of Ganglia is important because of compatibility issues with Hadoop and HBase's Ganglia clients. We're using HBase 0.20.6 and Hadoop 0.20.2. Therefore, we're going to use Ganglia 3.0.7, the latest of 3.0.x at the time of writing.  You could use the latest Ganglia 3.1.x if you're willing to apply the patch found in HADOOP-4675 to your Hadoop installation.

Setting up gmond
Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Generate a default config file.

gmond --default_config > /etc/gmond.conf

Configure /etc/gmond.conf

Modify the globals.

globals {
  user = ganglia
}

Define the cluster.

cluster {
  name = HBase
  owner = "Your Company"
  latlong = "N34.02 W118.45"
  url = "http://yourcompany.com/"
}

Disable multicast and define the host, the HBase master, where nodes in the cluster send data.

udp_send_channel {
  # mcast_join = 239.2.11.71
  host = master.yourcompany.com
  port = 8649
  ttl = 1
}

udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  # bind = 239.2.11.71
}

Run gmond:

gmond

Note: As far as I know, there is no way to stop gmond other than killing the process.

You can test each gmond node to make sure it is working. From each node:

telnet localhost 8649

You should see XML output.

Once gmond is installed and running on each node, move on to setup gmetad.

Setting up gmetad
We're going to setup a new EC2 instance to run gmetad and the PHP web front end.  A t1.micro should be sufficient for modest clusters.  gmetad will be configured to poll the gmond running on the HBase master for aggregated cluster data.

Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Copy the default config file.

cp /opt/ganglia-3.0.7/gmetad/gmetad.conf /etc/gmetad.conf

Configure /etc/gmetad.conf

setuid_username "ganglia"
data_source "HBase" master.yourcompany.com
gridname "YourCompany"

Create directories where Ganglia will save its data.

mkdir /var/lib/ganglia
mkdir /var/lib/ganglia/rrds/
chown -R ganglia:ganglia /var/lib/ganglia/

Run gmetad:

gmetad

For troubleshooting, it might be helpful not to daemonize gmetad:

gmetad -d 1

Note: As far as I know, there is no way to stop gmetad other than killing the process.

Setting up the PHP Web Front End
Install Apache and PHP libraries:

apt-get -y install rrdtool apache2 php5-mysql libapache2-mod-php5 php5-gd

Copy PHP web app into Apache's HTML directory:

cp -r /opt/ganglia-3.0.7/web /var/www/ganglia

Restart Apache:

/etc/init.d/apache2 restart

You should now be able to visit the Ganglia console on http://ganglia.yourcompany.com/ganglia and see pretty graphs and charts for machine metrics like load, memory, disk use, etc.

Configuring Hadoop and HBase
The next step is to configure Hadoop and HBase to start sending metrics to the master gmond. Detailed instructions are given here and here for Hadoop and HBase respectively.

Here's what we end up with inside HBase's conf/hadoop-metrics.properties:

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.period=10
hbase.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=master.yourcompany.com:8649

And inside Hadoop's conf/hadoop-metrics.properties:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=master.yourcompany.com:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

Once the hadoop-metrics.properties files are modified on all nodes in your cluster, you'll need to restart HBase and Hadoop.

Metrics
After restarting, you should see the following metrics show up in Ganglia for your HBase and Hadoop cluster:

hbase.regionserver.blockCacheCount
hbase.regionserver.blockCacheFree
hbase.regionserver.blockCacheHitRatio
hbase.regionserver.blockCacheSize
hbase.regionserver.compactionQueueSize
hbase.regionserver.fsReadLatency_avg_time
hbase.regionserver.fsReadLatency_num_ops
hbase.regionserver.fsSyncLatency_avg_time
hbase.regionserver.fsSyncLatency_num_ops
hbase.regionserver.fsWriteLatency_avg_time
hbase.regionserver.fsWriteLatency_num_ops
hbase.regionserver.memstoreSizeMB
hbase.regionserver.regions
hbase.regionserver.requests
hbase.regionserver.storefileIndexSizeMB
hbase.regionserver.storefiles
hbase.regionserver.stores

rpc.metrics.RpcProcessingTime_avg_time
rpc.metrics.RpcProcessingTime_num_ops
rpc.metrics.RpcQueueTime_avg_time
rpc.metrics.RpcQueueTime_num_ops
rpc.metrics.addColumn_avg_time
rpc.metrics.addColumn_num_ops
rpc.metrics.checkAndPut_avg_time
rpc.metrics.checkAndPut_num_ops
rpc.metrics.close_avg_time
rpc.metrics.close_num_ops
rpc.metrics.createTable_avg_time
rpc.metrics.createTable_num_ops
rpc.metrics.deleteColumn_avg_time
rpc.metrics.deleteColumn_num_ops
rpc.metrics.deleteTable_avg_time
rpc.metrics.deleteTable_num_ops
rpc.metrics.delete_avg_time
rpc.metrics.delete_num_ops
rpc.metrics.disableTable_avg_time
rpc.metrics.disableTable_num_ops
rpc.metrics.enableTable_avg_time
rpc.metrics.enableTable_num_ops
rpc.metrics.exists_avg_time
rpc.metrics.exists_num_ops
rpc.metrics.getClosestRowBefore_avg_time
rpc.metrics.getClosestRowBefore_num_ops
rpc.metrics.getClusterStatus_avg_time
rpc.metrics.getClusterStatus_num_ops
rpc.metrics.getHServerInfo_avg_time
rpc.metrics.getHServerInfo_num_ops
rpc.metrics.getOnlineRegionsAsArray_avg_time
rpc.metrics.getOnlineRegionsAsArray_num_ops
rpc.metrics.getProtocolVersion_avg_time
rpc.metrics.getProtocolVersion_num_ops
rpc.metrics.getRegionInfo_avg_time
rpc.metrics.getRegionInfo_num_ops
rpc.metrics.getRegionsAssignment_avg_time
rpc.metrics.getRegionsAssignment_num_ops
rpc.metrics.get_avg_time
rpc.metrics.get_num_ops
rpc.metrics.incrementColumnValue_avg_time
rpc.metrics.incrementColumnValue_num_ops
rpc.metrics.isMasterRunning_avg_time
rpc.metrics.isMasterRunning_num_ops
rpc.metrics.lockRow_avg_time
rpc.metrics.lockRow_num_ops
rpc.metrics.modifyColumn_avg_time
rpc.metrics.modifyColumn_num_ops
rpc.metrics.modifyTable_avg_time
rpc.metrics.modifyTable_num_ops
rpc.metrics.next_avg_time
rpc.metrics.next_num_ops
rpc.metrics.openScanner_avg_time
rpc.metrics.openScanner_num_ops
rpc.metrics.put_avg_time
rpc.metrics.put_num_ops
rpc.metrics.regionServerReport_avg_time
rpc.metrics.regionServerReport_num_ops
rpc.metrics.regionServerStartup_avg_time
rpc.metrics.regionServerStartup_num_ops
rpc.metrics.shutdown_avg_time
rpc.metrics.shutdown_num_ops
rpc.metrics.unlockRow_avg_time
rpc.metrics.unlockRow_num_ops

dfs.datanode.blockChecksumOp_avg_time
dfs.datanode.blockChecksumOp_num_ops
dfs.datanode.blockReports_avg_time
dfs.datanode.blockReports_num_ops
dfs.datanode.block_verification_failures
dfs.datanode.blocks_read
dfs.datanode.blocks_removed
dfs.datanode.blocks_replicated
dfs.datanode.blocks_verified
dfs.datanode.blocks_written
dfs.datanode.bytes_read
dfs.datanode.bytes_written
dfs.datanode.copyBlockOp_avg_time
dfs.datanode.copyBlockOp_num_ops
dfs.datanode.heartBeats_avg_time
dfs.datanode.heartBeats_num_ops
dfs.datanode.readBlockOp_avg_time
dfs.datanode.readBlockOp_num_ops
dfs.datanode.readMetadataOp_avg_time
dfs.datanode.readMetadataOp_num_ops
dfs.datanode.reads_from_local_client
dfs.datanode.reads_from_remote_client
dfs.datanode.replaceBlockOp_avg_time
dfs.datanode.replaceBlockOp_num_ops
dfs.datanode.writeBlockOp_avg_time
dfs.datanode.writeBlockOp_num_ops
dfs.datanode.writes_from_local_client
dfs.datanode.writes_from_remote_client

jvm.metrics.gcCount
jvm.metrics.gcTimeMillis
jvm.metrics.logError
jvm.metrics.logFatal
jvm.metrics.logInfo
jvm.metrics.logWarn
jvm.metrics.memHeapCommittedM
jvm.metrics.memHeapUsedM
jvm.metrics.memNonHeapCommittedM
jvm.metrics.memNonHeapUsedM
jvm.metrics.threadsBlocked
jvm.metrics.threadsNew
jvm.metrics.threadsRunnable
jvm.metrics.threadsTerminated
jvm.metrics.threadsTimedWaiting
jvm.metrics.threadsWaiting

As you can see, the metrics are incredibly useful for understanding the health of your HBase and Hadoop cluster over time.

Ubuntu License Plate on the Santa Monica Freeway

I came across this car while driving eastbound on the Santa Monica Freeway this past Saturday. Do you think the license plate is referring to the Ubuntu Linux OS or just the Ubuntu philosophy?

It also had some funny stickers and decals:
* Evolve FISH Logo Decorative Silver Car Emblem
* Goodnight Bush: A Parody
* Got Hope Bumper Sticker

ejabberd on Amazon EC2 Ubuntu AMI

It turned out to be harder that I expected to setup a Jabber (XMPP) server on an Ubuntu virtual machine (ami-ce44a1a7) within Amazon EC2. I chose to setup ejabberd since it was an easy to install via apt-get. I was a little nervous about working with a server built with Erlang since I knew nothing about it, but it was a reputable server and I was counting on not needing to have any Erlang knowledge to work with ejabberd. That was a correct assumption for the most part.

First, I logged into the VM and installed ejabberd:

apt-get install ejabberd

Next I edited the ejabberd config file, /etc/ejabberd/ejabberd.cfg, making kweiner@jabber.pop140.com an admin user and setting the hostname as jabber.pop140.com:

%% Admin user
{acl, admin, {user, "kweiner", "jabber.pop140.com"}}.

%% Hostname
{hosts, ["jabber.pop140.com"]}.

Next, I restarted the server and registered kweiner as a new user:

/etc/init.d/ejabberd restart
ejabberdctl register kweiner jabber.pop140.com mypasswd

Then I authorized traffic on ports 5222 (Jabber), 5223 (Jabber encrypted for old clients), 5269 (Other Jabber servers), and 5280 (Jabber web admin tool):

ec2-authorize default -p 5222
ec2-authorize default -p 5223
ec2-authorize default -p 5269
ec2-authorize default -p 5280

I could now hit the admin screen on which I logged in as kweiner@jabber.pop140.com:

http://jabber.pop140.com:5280/admin/

From here it was possible to add new users and browse server statistics. I experimented with adding users and using various Jabber clients like Gajim to send messages from one user to another.

My first problem came when I tried to communicate with users registered in other Jabber servers like Jabber.org and Google Talk. I struggled for hours trying to figure out why users on my server couldn't communicate with users from these other servers. Thankfully James Murty gave me a bit of help on this Jabber on EC2 message board thread. It turned out that I needed to configure SRV records in my DNS settings.

I logged into GoDaddy where my domain is registered and configured SRV records as follows:


After this, I was able to use nslookup to verify that the SRV records were setup properly:

kweiner~$ nslookup
> set type=srv
> _xmpp-server._tcp.pop140.com
Server:         66.75.160.63
Address:        66.75.160.63#53

Non-authoritative answer:
_xmpp-server._tcp.pop140.com    service = 10 10 5269 jabber.pop140.com.

That did it! My jabber server was finally federating with other jabber servers and my users could talk to their users.

I encountered my next big problem when I tried to use jabber again after terminating and relaunching my AMI. ejabberd failed to start and I found the following error message in the /var/log/ejabberd/ejabberd.log:

application: ejabberd
exited: {bad_return,{{ejabberd_app,start,[normal,[]]},
           {'EXIT',{{badmatch,{aborted,{no_exists,config}}},
                    [{ejabberd_config,set_opts,1},
                     {ejabberd_app,start,2},
                     {application_master,start_it_old,4}]}}}}

After some googling, I found that ejabberd associates itself with an Erlang node name, a concept I don't really understand that well. By default the node name was dynamically set based on the hostname for the machine. It looked something like this: ejabberd@domU-12-31-38-00-9D-63. This node name is somehow linked to the Mnesia database stored as files within /var/lib/ejabberd. The problem is that the hostname and therefore the node name changes everytime the AMI is relaunched which confuses ejabberd.

One solution I found is to explicitly set the node name. I did this by modifying /etc/default/ejabberd adding the line:

export ERLANG_NODE=ejabberd@jabber

This requires adding jabber as a host name inside /etc/hosts:

127.0.0.1 localhost.localdomain localhost jabber

I made those changes, removed all the database files from /var/lib/ejabberd, and restarted ejabberd. That did it! The node name was the same regardless of the hostname associated with the particular AMI instance.

This was a lot of effort, but it probably would have been easier if I had been familiar with Erlang applications and SRV DNS settings. I hope this post helps someone else struggling to setup ejabberd on EC2 as I did.

Jungle Disk on Linux

I first heard about Jungle Disk on the Linux Action Show. It is a client that manages files on Amazon's Simple Storage Service (S3). The client is cross-platform (Linux, Windows, Mac OS) and costs only a one-time fee of $20 plus $1 a month if you want the optional "extra" services including a web interface to your files.

With Jungle Disk, you pay as you go for file storage: 15 cents/gig/month for storage and a similar low fee for data transfer. You can use S3 as a virtual drive or "briefcase" with unlimited capacity, or you can use it as a backup location for automated or manual backups from various computers. I use it for both.

So far I've setup my Ubuntu desktop and Windows laptop from work as well as my Ubuntu desktop at home. I have backups configured for important files from each of those. From my eee PC, I passed on installing Jungle Disk and instead relied on the web interface that came with the Jungle Disk extras. I don't store any files on my eee, so I don't need a client to back anything up.

So far I'm very pleased with Jungle Disk. It will get pricey as I start to backup everything including the 50+ gigs of photos, videos, and music files I have. 50 gigs would be $90 a year. However, this is well worth it if I ever need to recover my precious files in the event of a fire or multi-hard drive crash. Photos and videos especially are irreplaceable.

I took some notes on setting up Jungle Disk on Ubuntu:

* Sign up for Amazon S3.

* Download and unzip Jungle Disk (jungledisk.tar.gz) in /opt.

* Run Jungle Disk Monitor:
./junglediskmonitor

* Install FUSE and davfs:
apt-get install fuse-utils
apt-get install davfs2

* Add your user to the fuse group:
sudo adduser kweiner fuse

Create and mount a directory to access Jungle Disk on the file system:
sudo mkdir /mnt/jungledisk sudo chgrp fuse /mnt/jungledisk sudo chmod g+w /mnt/jungledisk sudo mount.davfs http://localhost:2667 /mnt/jungledisk -o nolocks

Facebook Has No Love for Linux

ABC News and Facebook recently added the US Politics application which tracks what's happening right now in the world of politics. As the 2008 presentational race is heating up, there are daily polls for capturing the views of Facebook users.

On my computer and web browser (Firefox 2 on Ubuntu Linux), the user interface for these polls is so messed up that I cannot vote at all. Determined to participate in the polls, I emailed Facebook support to let them know that they needed to fix the application for Linux users. This was their response:

Hi Ken,

Unfortunately, Facebook is not meant to work with Linux. Please switch to a different operating system for best results on this website. Let me know if you have any questions, and I'd be happy to help.

Thanks for contacting Facebook,

Maude
User Operations
Facebook

This is pathetic for a website like Facebook that would like to, and probably will, sign up everyone in the world eventually. There are millions of Linux users that they are leaving out by having this policy. Also, support for Linux wouldn't be that difficult to achieve. In fact, I have been using Facebook heavily on Linux for the past few months and this has been the very first problem I've experienced.

I wrote another email to Facebook urging them to support Linux users. If you care about the future of Facebook and other large web sites working well on Linux operating systems, I urge you to do the same.

Ubuntu Baby

max-week2-019
Originally uploaded by kweiner

From early on, Max is learning to love Ubuntu!

A good friend of mine made this onesie with his computer and Iron On printer paper. It was definitely the most original baby gift we received.

When Max is old enough to start using a computer, he will start off with Edubuntu, a version of Ubuntu geared towards kids. The Edubuntu website calls it " Linux for Young Human Beings".

Audio CD Creation on Ubuntu

My wife asked me to make a compilation of R.E.M. songs for her father and burn them to a CD that he could play at home or in his car. Believe it or not, I have never tried to make an audio CD before. I hardly knew where to start.

The first thing I checked was what programs came with Ubuntu that might help me. In the Applications --> Sound & Video menu, I found Sound Juicer CD Extractor and Serpentine Audio CD Creator.

Sound Juicer pops up when I put an audio CD in my computer. It was very easy to see how to rip the music off the CD, but I didn't know what format I should rip it in so that it could be burned onto a CD-R and played in a normal CD player. I figured that if I created MP3 or OGG files, only special, modern CD players could read the data. After some Googling, I found that I was right - I would need to rip the music as .wav (WAVE) files if I wanted to support normal CD players.

Unfortunately, Sound Juicer did not contain a gstreamer audio profile for the lossless .wav format. I had to add one using these instructions. After that, I was able to easily rip music into the .wav format and then setup a playlist of music to burn onto a CD using Serpentine. With Serpentine, it was as easy as opening the audio files, ordering them, and clicking one button to burn them to a CD. Serpentine also kept track of the time taken up by the playlist, letting me know how much more room was left on the CD. Easy as pie.

The resulting CD, however, was not perfect. It plays fine on Ubuntu Linux, Windows XP, and in my new car CD player, but it fails to play in my portable Sony CD player and an old Toshiba DVD player.

I assume there is something special about newer CD players and computers that give them the capability to play an audio CD created on a computer, but I have no idea what it is. I am wondering if there was some different CD-R type or some way I could have burned the music so that it would play on the oldest of CD players. If you know anything about this, please share.

Easy Upgrade to Ubuntu Feisty Fawn

Last Thursday, April 19, Ubuntu released its latest version, 7.04 (Feisty Fawn). The announcement is here. I decided to wait a few days before upgrading my operating system at home from 6.10 (Edgy Eft) to 7.04 just in case there were any bugs in the upgrade process that would get fixed immediately after the release. Even though a fair amount of people were reporting various problems with the upgrade process on the Ubuntu Forums, I decided to give it a go.

I didn't customize my installation very much so I figured the upgrade process had a good chance of succeeding. Ubuntu's upgrade instructions recommend upgrading via the network, but since many people were reporting that the upgrade servers were getting hammered due to the recent release, I decided to upgrade using the "Alternate CD". I downloaded the ISO using Azureus, my favorite BitTorrent client. It took a while to download, so I just let it go overnight. Then I burned it to a CD and popped it it. Ubuntu immediately popped up a window asking me if I wanted to upgrade to 7.04. I clicked "Upgrade", followed the instructions, and everything went smoothly! In less than an hour later, I was fully upgraded to Feisty Fawn.

I haven't really noticed any major differences between Edgy and Feisty yet other than the fact that the "Hibernate" feature of my computer is working now and it never worked with Edgy. That is pretty exciting for me actually, since I had been completely shutting down the computer before, and now I can let it go to sleep. I think it works now because Feisty has a new version of the Kernel, 2.6.20, which probably contains recently added support for newer hardware. The rest of the new features in Feisty are listed here. There are also two new games included with Feisty: Sudoku and Chess.

I can't decide whether or not to try the upgrade at work. I can't afford any downtime at work, so if it doesn't go as smoothly, I would be screwed. I will probably wait until things at work slow down a bit, and then try the upgrade on a Friday. That way, if I end up having to reinstall, I can do it over the weekend.

Intel to nVidia

My Dell Dimension 9200c came with a G965 chipset and an Intel GMA X3000 on-board video chip. After installing Ubuntu 6.10 (Edgy Eft), I was never able to set the resolution to 1680 x 1050 which is the proper setting for my Dell 2007WFP monitor. I posted a couple of times (1, 2) to the Ubuntu Forums, but no one was able to help me.

I thought I would wait it out until the i810 driver or some other Linux driver supported the X3000 properly. I was hoping that I'd have better luck with Ubuntu 7.04 (Feisty Fawn). I tried both the Herd-1 and Herd-2 pre-releases of Feisty and still had no luck. I was still stuck at 1280 x 1024.

Finally, I got sick of looking at the distorted images (stretched horizontally) and decided to get another video card to put in my computer's empty PCI Express x16 slot. A Dell tech support guy told me that if I pop in a new video card, the on-board chip would automatically shut off. I figured I would buy an nVidia-based card because nVidia drivers are known to work well on Linux.

After a lot of research online, I went to buy an eVGA GeForce 7300 GS at Circuit City. To my dismay, when I attempted to install it, it didn't fit! I have one of those mini computer chassis, and I learned that I needed a half height (sometimes called low profile) video card. While researching, I only knew to look for a PCI Express x16 card - I didn't know they came in more than one height.

So I returned the eVGA card to Circuit City and began to research cards again, looking for low profile PCI Express cards. I couldn't find a single store in my area that sold one. Finally, I found one on Amazon.com: the Geforce 7300GS 256MB DDR2 Pcie. I ordered this along with a Low Profile Bracket Kit which had to be purchased separately.

This video card installed successfully! I was then able to run sudo dpkg-reconfigure xserver-xorg which reconfigured my xorg.conf with the open source nv driver. I changed this to the proprietary nVidia driver using these instructions. After this, everything worked! My resolution went to 1680 x 1050, I could run glxgears, and games like Second Life looked soooooooooooo much better.

The only drawback is that the new video card has its own fan which is annoyingly loud. Because of this, I am thinking about moving my computer into a cabinet or under the desk.