AutoScaling Amazon SQS Queue Processors

One of my favorite things about running servers in Amazon EC2 is the ability to use AutoScaling to automatically add and remove nodes as web traffic increases and decreases.  Not only does this generally save money, but it also helps prepare a system to handle traffic spikes.

Recently I had some fun setting up AutoScaling for a different use case -- a cluster of machines processing messages on an Amazon SQS queue.  The idea was to add and remove nodes as the number of visible messages on the queue fluctuated.  Again, this keeps costs lower by only running as many nodes as are necessary to process the current workload and handles workload spikes.

Here are sample steps and commands that you can use for setting up AutoScaling for SQS queue processors:

Create an AMI for a node that will process SQS messages.  The node that boots from this AMI should automatically launch one or more queue processing processes.  A user-data script may be useful for this.

Create a Launch Configuration for the queue processor node:

as-create-launch-config MyQueueConfig --image-id [INSERT YOUR AMI ID HERE] --instance-type c1.medium --key [INSERT YOUR KEYNAME HERE] --user-data [INSERT YOUR USER DATA SCRIPT HERE]

Create an AutoScaling group for queue processors:

as-create-auto-scaling-group MyQueueGroup --launch-configuration MyQueueConfig --availability-zones us-east-1b --min-size 1 --max-size 10

Create Policies that add/remove 1 node to/from the cluster.  They will be invoked when the number of messages on the queue grows excessively high or decreases to an acceptable level:

as-put-scaling-policy MyScaleUpPolicy -g MyQueueGroup --adjustment=1 --type ChangeInCapacity

as-put-scaling-policy MyScaleDownPolicy -g MyQueueGroup --adjustment=-1 --type ChangeInCapacity

Create Alarms to scale up/down when the number of messages on the queue grows excessively high or decreases to an acceptable level.  Use the Policy ARN's returned by the previous as-put-scaling-policy commands.

mon-put-metric-alarm --alarm-name MyHighMessagesAlarm --alarm-description "Scale up when number of messages on queue is high" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 1000 --comparison-operator GreaterThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleUpPolicy ARN HERE]

mon-put-metric-alarm --alarm-name MyLowMessagesAlarm --alarm-description "Scale down when number of messages on queue is low" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 100 --comparison-operator LessThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleDownPolicy ARN HERE]

In this example the Alarms cause the cluster to scale up when the number of visible messages on the queue remains above 1000 for 10 consecutive minutes and scale down when the number of visible messages falls below 100 for 10 consecutive minutes.

The Policies above adjust the amount of nodes in the cluster by a specific amount, but it is also possible to specify the adjustment in terms of percentages.  Using --adjustment 10 --type PercentChangeInCapacity would adjust the amount of nodes by 10 percent.

It also would have been possible to base scaling activities on other AWS/SQS metrics such as:

• NumberOfMessagesSent
• NumberOfMessagesReceived
• NumberOfMessagesDeleted
• NumberOfEmptyReceives
• ApproximateNumberOfMessagesVisible
• ApproximateNumberOfMessagesNotVisible
• ApproximateNumberOfMessagesDelayed
• SentMessageSize

Here are a few online references relevant to AutoScaling SQS queue processors:

• Auto Scaling Amazon EC2 with Amazon SQS
• Auto Scaling with Alarms and Load Balancing
• Auto Scaling API Reference

Happy SQS AutoScaling!

Edit Remote EC2 Text Files with TextWrangler

This post explains how to use TextWrangler on a Mac to edit text files that live on a remote Amazon EC2 instance with Ubuntu.

I assume you already know how to SSH to Amazon using an Identity File:

ssh -i ~ec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

You can use a similar command to connect to your EC2 instance via SFTP:

sftp -o IdentityFile=eec2-keys/id_your-keypair ubuntu@ec2-public-dns-name.amazonaws.com

TextWrangler allows you to open a file from an SFTP server, but it doesn't allow you to configure an Identity File as you did when SSHing or SFTPing from the command line. To work around this limitation, you simply have to configure your SSH client so that it knows the Identity File you want to use. Create ~/.ssh/config if it doesn't already exist and add a line specifying your Identity File location:

echo 'IdentityFile ~ec2-keys/id_your-keypair' >> ~/.ssh/config

Now you're ready to use TextWrangler to connect to your server and browse files. Choose: File -> Open from FTP/SFTP Server... Then enter your server publisher DNS name, check the SFTP box and enter  ubuntu as the User name.  You can leave the Password: field blank.

After you hit connect,  you TextWrangler will allow you to browse the file system of your EC2 instance, choose a file, and edit it.  When you save the file, it will update directly on the remote EC2 instance.  That's all there is to it!

Monitor HBase & Hadoop with Ganglia on EC2

This post is a recipe on setting up Ganglia to monitor an HBase and Hadoop cluster on the Ubuntu OS on Amazon EC2. Ganglia is a monitoring system for grids and clusters consisting of the following 3 components:

gmond
A Ganglia Monitoring Daemon (gmond) runs on each node in the cluster and collects statistics from the node it runs on as well as other nodes in the cluster. Normally it is a multicast system where each gmond node receives data from its peers. However, since Amazon EC2 does not support multicast at this time, you must setup Ganglia Monitoring Daemons in unicast mode where each node in a cluster is configured to send its data to one pre-designated node.

gmetad
A Ganglia Meta Daemon (gmetad) runs for each grid and collects data from the Ganglia Monitoring Daemons, one from each cluster. It stores the data it collects on the file system. We'll only be configuring one grid and therefore one gmetad. We'll be running gmetad from the same node that the PHP Web Front End is installed on.

PHP Web Front End
A PHP application reads the data and provides a UI to visualize the data over time with pretty graphs.  Example from UC Berkeley.

Approach
Our HBase cluster has one Master node (with Hadoop's Name Node and Job Tracker) and the rest Region Server nodes (each with a Hadoop Data Node and Task Tracker).  Since we must use Ganglia in unicast mode on EC2, we need to choose one of these nodes to receive data from each of the other nodes.  We'll configure each Region Server gmond to send data to the gmond running on the Master.  A separate node will be used to run gmetad and the PHP web front end.  The gmetad will be configured to poll the Master's gmond for aggregated cluster data.

Getting the right version of Ganglia is important because of compatibility issues with Hadoop and HBase's Ganglia clients. We're using HBase 0.20.6 and Hadoop 0.20.2. Therefore, we're going to use Ganglia 3.0.7, the latest of 3.0.x at the time of writing.  You could use the latest Ganglia 3.1.x if you're willing to apply the patch found in HADOOP-4675 to your Hadoop installation.

Setting up gmond
Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Generate a default config file.

gmond --default_config > /etc/gmond.conf

Configure /etc/gmond.conf

Modify the globals.

globals {
  user = ganglia
}

Define the cluster.

cluster {
  name = HBase
  owner = "Your Company"
  latlong = "N34.02 W118.45"
  url = "http://yourcompany.com/"
}

Disable multicast and define the host, the HBase master, where nodes in the cluster send data.

udp_send_channel {
  # mcast_join = 239.2.11.71
  host = master.yourcompany.com
  port = 8649
  ttl = 1
}

udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  # bind = 239.2.11.71
}

Run gmond:

gmond

Note: As far as I know, there is no way to stop gmond other than killing the process.

You can test each gmond node to make sure it is working. From each node:

telnet localhost 8649

You should see XML output.

Once gmond is installed and running on each node, move on to setup gmetad.

Setting up gmetad
We're going to setup a new EC2 instance to run gmetad and the PHP web front end.  A t1.micro should be sufficient for modest clusters.  gmetad will be configured to poll the gmond running on the HBase master for aggregated cluster data.

Add a ganglia user.

adduser --disabled-login --no-create-home ganglia

Download and unpack Ganglia 3.0.7.

wget http://downloads.sourceforge.net/project/ganglia/ganglia%20monitoring%20core/3.0.7%20%28Fossett%29/ganglia-3.0.7.tar.gz
tar -xzvf ganglia-3.0.7.tar.gz -C /opt
rm ganglia-3.0.7.tar.gz

Install dependencies

apt-get -y install build-essential libapr1-dev libconfuse-dev libexpat1-dev python-dev librrd2-dev

Compile and install.

cd /opt/ganglia-3.0.7
./configure
make && make install

Copy the default config file.

cp /opt/ganglia-3.0.7/gmetad/gmetad.conf /etc/gmetad.conf

Configure /etc/gmetad.conf

setuid_username "ganglia"
data_source "HBase" master.yourcompany.com
gridname "YourCompany"

Create directories where Ganglia will save its data.

mkdir /var/lib/ganglia
mkdir /var/lib/ganglia/rrds/
chown -R ganglia:ganglia /var/lib/ganglia/

Run gmetad:

gmetad

For troubleshooting, it might be helpful not to daemonize gmetad:

gmetad -d 1

Note: As far as I know, there is no way to stop gmetad other than killing the process.

Setting up the PHP Web Front End
Install Apache and PHP libraries:

apt-get -y install rrdtool apache2 php5-mysql libapache2-mod-php5 php5-gd

Copy PHP web app into Apache's HTML directory:

cp -r /opt/ganglia-3.0.7/web /var/www/ganglia

Restart Apache:

/etc/init.d/apache2 restart

You should now be able to visit the Ganglia console on http://ganglia.yourcompany.com/ganglia and see pretty graphs and charts for machine metrics like load, memory, disk use, etc.

Configuring Hadoop and HBase
The next step is to configure Hadoop and HBase to start sending metrics to the master gmond. Detailed instructions are given here and here for Hadoop and HBase respectively.

Here's what we end up with inside HBase's conf/hadoop-metrics.properties:

hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.period=10
hbase.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=master.yourcompany.com:8649

And inside Hadoop's conf/hadoop-metrics.properties:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=master.yourcompany.com:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=master.yourcompany.com:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=master.yourcompany.com:8649

Once the hadoop-metrics.properties files are modified on all nodes in your cluster, you'll need to restart HBase and Hadoop.

Metrics
After restarting, you should see the following metrics show up in Ganglia for your HBase and Hadoop cluster:

hbase.regionserver.blockCacheCount
hbase.regionserver.blockCacheFree
hbase.regionserver.blockCacheHitRatio
hbase.regionserver.blockCacheSize
hbase.regionserver.compactionQueueSize
hbase.regionserver.fsReadLatency_avg_time
hbase.regionserver.fsReadLatency_num_ops
hbase.regionserver.fsSyncLatency_avg_time
hbase.regionserver.fsSyncLatency_num_ops
hbase.regionserver.fsWriteLatency_avg_time
hbase.regionserver.fsWriteLatency_num_ops
hbase.regionserver.memstoreSizeMB
hbase.regionserver.regions
hbase.regionserver.requests
hbase.regionserver.storefileIndexSizeMB
hbase.regionserver.storefiles
hbase.regionserver.stores

rpc.metrics.RpcProcessingTime_avg_time
rpc.metrics.RpcProcessingTime_num_ops
rpc.metrics.RpcQueueTime_avg_time
rpc.metrics.RpcQueueTime_num_ops
rpc.metrics.addColumn_avg_time
rpc.metrics.addColumn_num_ops
rpc.metrics.checkAndPut_avg_time
rpc.metrics.checkAndPut_num_ops
rpc.metrics.close_avg_time
rpc.metrics.close_num_ops
rpc.metrics.createTable_avg_time
rpc.metrics.createTable_num_ops
rpc.metrics.deleteColumn_avg_time
rpc.metrics.deleteColumn_num_ops
rpc.metrics.deleteTable_avg_time
rpc.metrics.deleteTable_num_ops
rpc.metrics.delete_avg_time
rpc.metrics.delete_num_ops
rpc.metrics.disableTable_avg_time
rpc.metrics.disableTable_num_ops
rpc.metrics.enableTable_avg_time
rpc.metrics.enableTable_num_ops
rpc.metrics.exists_avg_time
rpc.metrics.exists_num_ops
rpc.metrics.getClosestRowBefore_avg_time
rpc.metrics.getClosestRowBefore_num_ops
rpc.metrics.getClusterStatus_avg_time
rpc.metrics.getClusterStatus_num_ops
rpc.metrics.getHServerInfo_avg_time
rpc.metrics.getHServerInfo_num_ops
rpc.metrics.getOnlineRegionsAsArray_avg_time
rpc.metrics.getOnlineRegionsAsArray_num_ops
rpc.metrics.getProtocolVersion_avg_time
rpc.metrics.getProtocolVersion_num_ops
rpc.metrics.getRegionInfo_avg_time
rpc.metrics.getRegionInfo_num_ops
rpc.metrics.getRegionsAssignment_avg_time
rpc.metrics.getRegionsAssignment_num_ops
rpc.metrics.get_avg_time
rpc.metrics.get_num_ops
rpc.metrics.incrementColumnValue_avg_time
rpc.metrics.incrementColumnValue_num_ops
rpc.metrics.isMasterRunning_avg_time
rpc.metrics.isMasterRunning_num_ops
rpc.metrics.lockRow_avg_time
rpc.metrics.lockRow_num_ops
rpc.metrics.modifyColumn_avg_time
rpc.metrics.modifyColumn_num_ops
rpc.metrics.modifyTable_avg_time
rpc.metrics.modifyTable_num_ops
rpc.metrics.next_avg_time
rpc.metrics.next_num_ops
rpc.metrics.openScanner_avg_time
rpc.metrics.openScanner_num_ops
rpc.metrics.put_avg_time
rpc.metrics.put_num_ops
rpc.metrics.regionServerReport_avg_time
rpc.metrics.regionServerReport_num_ops
rpc.metrics.regionServerStartup_avg_time
rpc.metrics.regionServerStartup_num_ops
rpc.metrics.shutdown_avg_time
rpc.metrics.shutdown_num_ops
rpc.metrics.unlockRow_avg_time
rpc.metrics.unlockRow_num_ops

dfs.datanode.blockChecksumOp_avg_time
dfs.datanode.blockChecksumOp_num_ops
dfs.datanode.blockReports_avg_time
dfs.datanode.blockReports_num_ops
dfs.datanode.block_verification_failures
dfs.datanode.blocks_read
dfs.datanode.blocks_removed
dfs.datanode.blocks_replicated
dfs.datanode.blocks_verified
dfs.datanode.blocks_written
dfs.datanode.bytes_read
dfs.datanode.bytes_written
dfs.datanode.copyBlockOp_avg_time
dfs.datanode.copyBlockOp_num_ops
dfs.datanode.heartBeats_avg_time
dfs.datanode.heartBeats_num_ops
dfs.datanode.readBlockOp_avg_time
dfs.datanode.readBlockOp_num_ops
dfs.datanode.readMetadataOp_avg_time
dfs.datanode.readMetadataOp_num_ops
dfs.datanode.reads_from_local_client
dfs.datanode.reads_from_remote_client
dfs.datanode.replaceBlockOp_avg_time
dfs.datanode.replaceBlockOp_num_ops
dfs.datanode.writeBlockOp_avg_time
dfs.datanode.writeBlockOp_num_ops
dfs.datanode.writes_from_local_client
dfs.datanode.writes_from_remote_client

jvm.metrics.gcCount
jvm.metrics.gcTimeMillis
jvm.metrics.logError
jvm.metrics.logFatal
jvm.metrics.logInfo
jvm.metrics.logWarn
jvm.metrics.memHeapCommittedM
jvm.metrics.memHeapUsedM
jvm.metrics.memNonHeapCommittedM
jvm.metrics.memNonHeapUsedM
jvm.metrics.threadsBlocked
jvm.metrics.threadsNew
jvm.metrics.threadsRunnable
jvm.metrics.threadsTerminated
jvm.metrics.threadsTimedWaiting
jvm.metrics.threadsWaiting

As you can see, the metrics are incredibly useful for understanding the health of your HBase and Hadoop cluster over time.

Amazon ELB - Capturing Client IP Address

If you're using Amazon EC2's Elastic Load Balancer (ELB) for load balancing web applications, you may have noticed that in your web access logs, the remote host IP address is the same for every request. The IP address you see is the private IP address of the load balancer.

If you want to see the IP address of the client (called remote host in access log documentation), you'll need to look at the value of the X-Forwarded-For request header which ELB populates when it forwards the request.

This can be achieved in an Apache access log by using the syntax:

%{X-Forwarded-For}i

You Apache log format would then look something like this:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-elb
CustomLog log/acces_log combined-elb

See Apache Custom Log Formats

If you're using a Tomcat application server, you could define an access log Valve like this:

<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs" prefix="access_log." suffix=".txt"
    pattern="%{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
    resolveHosts="false"/>

See Tomcat 6 Valve Configuration Reference.

If you're trying to capture the client IP address within your application code, simply use whatever API you have to read the X-Forwarded-For request header. For example in Java use the HttpServletRequest:

String clientIpAddress = request.getHeader("X-Forwarded-For");

instead of

String clientIpAddress = request.getRemoteAddr();

Serving Gzipped Javascript Files from Amazon CloudFront

It is well-known that using a content delivery network (CDN) and compressing an HTTP response with gzip can significantly improve the performance and reduce the cost of a web site. If your CDN is Amazon's CloudFront, you'll face difficulties serving gzipped content to browsers that support it. Most web servers are able to examine the HTTP request headers sent by the browser and dynamically choose whether to deliver compressed or uncompressed content. CloudFront, however, does not yet offer this feature.

People have overcome this problem by manually inspecting the Accept-Encoding request header for a dynamically served page and writing references to either original or compressed files accordingly using a naming convention like myscript.js and myscript.js.gz for regular and compressed files respectively.

If all your pages are statically served from CloudFront, however, there is no opportunity to inspect the request headers. Javascript running on a statically served page has no ability to ask the browser if it supports gzipped content. We found ourselves in this situation and resorted to the following solution to determine if the browser supports gzip:

1) Create a small gzipped file, gzipcheck.js.jgz, and make it available in CloudFront. This file should contain one line of code:

gzipEnabled = true;

2) Use the following code to attempt to load and run this file. You'll probably want to put it in the HTML HEAD section before any other Javascript code.

<script type="text/javascript" src="gzipcheck.js.jgz">
</script>

If the file loads, it sets a flag, gzipEnabled, that indicates whether or not the browser supports gzip.

Use the result to drive a file naming convention for references to other static files. For example, you can upload and reference each compressed Javascript file with an additional .jgz extension. Why .jgz instead of .gz? Because of an annoying limitation of Safari.

When you upload your files to S3/CloudFront, make sure to set the proper HTTP response headers on your files that end in .jgz:

Content-Encoding = gzip
Content-Type = application/x-javascript

And if you want browsers to cache your files forever (almost):

Cache-Control = max-age=315360000
Expires = Tue, 31 Dec 2019 20:00:00 GMT

Thanks to Ricardo Rangel who helped design and code this solution.

Update

As of December 20, 2015, Amazon Web Services announced support for Gzipped CloudFront files!  See Serving Compressed Files from their developer guide.

Cloud Studio Released

Cloud Services just launched Cloud Studio which is a java-based desktop application for managing images and instances in Amazon's EC2. I've yet to try it, but from the screenshots, it looks really nice. I'm not sure, though, if it does anything more than the very capable Elasticfox which runs right in your Firefox browser. Cloud Studio is also available as an Eclipse plugin which is really convenient for Java developers and users of the popular open source IDE.

GUI's like Elasticfox and Cloud Studio make working with Amazon EC2 really easy. Now I wish someone would create similar management tools for Amazon's SQS and SimpleDB. I'd like to be able to list, see stats for, and manage my SQS queues and SimpleDB domains, items, and attributes without having to run command line scripts. Hint, hint, Cloud Services!

ejabberd on Amazon EC2 Ubuntu AMI

It turned out to be harder that I expected to setup a Jabber (XMPP) server on an Ubuntu virtual machine (ami-ce44a1a7) within Amazon EC2. I chose to setup ejabberd since it was an easy to install via apt-get. I was a little nervous about working with a server built with Erlang since I knew nothing about it, but it was a reputable server and I was counting on not needing to have any Erlang knowledge to work with ejabberd. That was a correct assumption for the most part.

First, I logged into the VM and installed ejabberd:

apt-get install ejabberd

Next I edited the ejabberd config file, /etc/ejabberd/ejabberd.cfg, making kweiner@jabber.pop140.com an admin user and setting the hostname as jabber.pop140.com:

%% Admin user
{acl, admin, {user, "kweiner", "jabber.pop140.com"}}.

%% Hostname
{hosts, ["jabber.pop140.com"]}.

Next, I restarted the server and registered kweiner as a new user:

/etc/init.d/ejabberd restart
ejabberdctl register kweiner jabber.pop140.com mypasswd

Then I authorized traffic on ports 5222 (Jabber), 5223 (Jabber encrypted for old clients), 5269 (Other Jabber servers), and 5280 (Jabber web admin tool):

ec2-authorize default -p 5222
ec2-authorize default -p 5223
ec2-authorize default -p 5269
ec2-authorize default -p 5280

I could now hit the admin screen on which I logged in as kweiner@jabber.pop140.com:

http://jabber.pop140.com:5280/admin/

From here it was possible to add new users and browse server statistics. I experimented with adding users and using various Jabber clients like Gajim to send messages from one user to another.

My first problem came when I tried to communicate with users registered in other Jabber servers like Jabber.org and Google Talk. I struggled for hours trying to figure out why users on my server couldn't communicate with users from these other servers. Thankfully James Murty gave me a bit of help on this Jabber on EC2 message board thread. It turned out that I needed to configure SRV records in my DNS settings.

I logged into GoDaddy where my domain is registered and configured SRV records as follows:


After this, I was able to use nslookup to verify that the SRV records were setup properly:

kweiner~$ nslookup
> set type=srv
> _xmpp-server._tcp.pop140.com
Server:         66.75.160.63
Address:        66.75.160.63#53

Non-authoritative answer:
_xmpp-server._tcp.pop140.com    service = 10 10 5269 jabber.pop140.com.

That did it! My jabber server was finally federating with other jabber servers and my users could talk to their users.

I encountered my next big problem when I tried to use jabber again after terminating and relaunching my AMI. ejabberd failed to start and I found the following error message in the /var/log/ejabberd/ejabberd.log:

application: ejabberd
exited: {bad_return,{{ejabberd_app,start,[normal,[]]},
           {'EXIT',{{badmatch,{aborted,{no_exists,config}}},
                    [{ejabberd_config,set_opts,1},
                     {ejabberd_app,start,2},
                     {application_master,start_it_old,4}]}}}}

After some googling, I found that ejabberd associates itself with an Erlang node name, a concept I don't really understand that well. By default the node name was dynamically set based on the hostname for the machine. It looked something like this: ejabberd@domU-12-31-38-00-9D-63. This node name is somehow linked to the Mnesia database stored as files within /var/lib/ejabberd. The problem is that the hostname and therefore the node name changes everytime the AMI is relaunched which confuses ejabberd.

One solution I found is to explicitly set the node name. I did this by modifying /etc/default/ejabberd adding the line:

export ERLANG_NODE=ejabberd@jabber

This requires adding jabber as a host name inside /etc/hosts:

127.0.0.1 localhost.localdomain localhost jabber

I made those changes, removed all the database files from /var/lib/ejabberd, and restarted ejabberd. That did it! The node name was the same regardless of the hostname associated with the particular AMI instance.

This was a lot of effort, but it probably would have been easier if I had been familiar with Erlang applications and SRV DNS settings. I hope this post helps someone else struggling to setup ejabberd on EC2 as I did.

Amazon EC2 - My First Step into the Cloud

Ever since returning from the Web 2.0 Expo in San Francisco last month, I have been excited about learning how to setup a machine in the Amazon Elastic Compute Cloud (EC2). My first goal was just to get a Linux machine running in the cloud on which later I will try to install enough software to get a basic web application running.

To learn what to do, I consulted a the Programming Amazon Web Services book which is fortunately available on Safari Online. In parallel I read the Getting Started Guide on Amazon's website. The steps to get a virtual machine running were roughly the following:

• Register for Amazon EC2.
• Download a X.509 certificate private key and public key pair and store them in a ~/.ec2 directory on my computer.
• Take note of my AWS account number.
• Download the Amazon EC2 Command-Line Tools
• Set environment variables: EC2_HOME, EC2_PRIVATE_KEY, EC2_CERT
• Run ec2-describe-images -o self -o amazon to search for images.
• Generate keypair using ec2-add-keypair gsg-keypair
• Fired up the Getting Started AMI with ec2-run-instances ami-2bb65342 -k gsg-keypair
• Open SSH and HTTP ports: ec2-authorize default -p 22, ec2-authorize default -p 80
• SSH into the instance: ssh -i id_rsa_gsg_keypair root@ec2-75-101-209-13.compute-1.amazonaws.com
• Access the instance's web server: http://ec2-75-101-209-13.compute-1.amazonaws.com/
• Shutdown the instance: ec2-terminate-instances i-db6ea2b2
  

That's it! Everything worked as advertised and I got through all of that in less than an hour. This whole procedure cost me about 10 cents. I am looking forward to my next goal which is to find an Ubuntu server image, install a database with data, and learn how to preserve the data so that it survives an image restart.