Wednesday, December 14, 2011

AutoScaling Amazon SQS Queue Processors

One of my favorite things about running servers in Amazon EC2 is the ability to use AutoScaling to automatically add and remove nodes as web traffic increases and decreases.  Not only does this generally save money, but it also helps prepare a system to handle traffic spikes.

Recently I had some fun setting up AutoScaling for a different use case -- a cluster of machines processing messages on an Amazon SQS queue.  The idea was to add and remove nodes as the number of visible messages on the queue fluctuated.  Again, this keeps costs lower by only running as many nodes as are necessary to process the current workload and handles workload spikes.

Here are sample steps and commands that you can use for setting up AutoScaling for SQS queue processors:

Create an AMI for a node that will process SQS messages.  The node that boots from this AMI should automatically launch one or more queue processing processes.  A user-data script may be useful for this.

Create a Launch Configuration for the queue processor node:

as-create-launch-config MyQueueConfig --image-id [INSERT YOUR AMI ID HERE] --instance-type c1.medium --key [INSERT YOUR KEYNAME HERE] --user-data [INSERT YOUR USER DATA SCRIPT HERE]

Create an AutoScaling group for queue processors:

as-create-auto-scaling-group MyQueueGroup --launch-configuration MyQueueConfig --availability-zones us-east-1b --min-size 1 --max-size 10

Create Policies that add/remove 1 node to/from the cluster.  They will be invoked when the number of messages on the queue grows excessively high or decreases to an acceptable level:

as-put-scaling-policy MyScaleUpPolicy -g MyQueueGroup --adjustment=1 --type ChangeInCapacity

as-put-scaling-policy MyScaleDownPolicy -g MyQueueGroup --adjustment=-1 --type ChangeInCapacity

Create Alarms to scale up/down when the number of messages on the queue grows excessively high or decreases to an acceptable level.  Use the Policy ARN's returned by the previous as-put-scaling-policy commands.

mon-put-metric-alarm --alarm-name MyHighMessagesAlarm --alarm-description "Scale up when number of messages on queue is high" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 1000 --comparison-operator GreaterThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleUpPolicy ARN HERE]

mon-put-metric-alarm --alarm-name MyLowMessagesAlarm --alarm-description "Scale down when number of messages on queue is low" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 100 --comparison-operator LessThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleDownPolicy ARN HERE]

In this example the Alarms cause the cluster to scale up when the number of visible messages on the queue remains above 1000 for 10 consecutive minutes and scale down when the number of visible messages falls below 100 for 10 consecutive minutes.

The Policies above adjust the amount of nodes in the cluster by a specific amount, but it is also possible to specify the adjustment in terms of percentages.  Using --adjustment 10 --type PercentChangeInCapacity would adjust the amount of nodes by 10 percent.

It also would have been possible to base scaling activities on other AWS/SQS metrics such as:
  • NumberOfMessagesSent
  • NumberOfMessagesReceived
  • NumberOfMessagesDeleted
  • NumberOfEmptyReceives
  • ApproximateNumberOfMessagesVisible
  • ApproximateNumberOfMessagesNotVisible
  • ApproximateNumberOfMessagesDelayed
  • SentMessageSize
Here are a few online references relevant to AutoScaling SQS queue processors:

Thursday, October 20, 2011

Online Advertising Formula Reference

Occasionally I need to manipulate online advertising formulas to make a quick calculation.  The following serves a quick cheat sheet for this purpose. Note that the term Cost is from the perspective of the Advertiser. The web site or publisher can translate that into Revenue.

CPM Campaigns

Cost = Impressions / 1000 * CPM

CPC Campaigns

Cost = Impressions * CTR * CPC
CPM = CTR * CPC * 1000

CPA Campaigns

Cost = Impressions * CTR * AR * CPA
CPM = CTR * AR * CPA * 1000

CPMCost Per 1000 Impressions
CPCCost Per Click
CPACost Per Action
CTRClick Thru Rate (Clicks / Impressions)
ARAction Rate (Actions / Clicks)

Monday, July 18, 2011

Giant Ampersand - Washington/Centinela Redevelopment Part 6

It's been almost 5 years since my last post about the corner lots of Washington and Centinela which are supposedly destined for redevelopment. That's because pretty much nothing has happened on those lots - until this past week.

All of the sudden a 8-foot tall white wooden ampersand appeared in the middle of the lot on the north east corner.  I had no idea what it meant, but I was really happy to see it.  To me it meant that someone out there is thinking about this vacant plot of land.

A little Googling produced an answer.  The mysterious ampersand is a public project by Los Angeles-based artist Shannon Ebner.  According to the LAXART web site:

The ampersand is a conjoiner; it signals an incomplete thought and acknowledges that there is more to be said.

This is part of a Shannon Ebner exhibition at the Hammer Museum which started July 16, 2011.  Apparently this isn't the first time she's used an ampersand in her work.  See this photo from a year ago in Milan.

I've been waiting for the redevelopment on this corner to get underway since I moved to this Del Rey neighborhood 11 years ago.  I really hope, as the giant ampersand acknowledges, that there is more to be said...

Tuesday, April 26, 2011

Using a Canon Pixma MX882 All-In-One with Ubuntu 10.10

Are you thinking about buying a Canon Pixma MX882 Wireless Office All-In-One Printer and wondering if it will work with your Ubuntu computer?  Although Canon does not make this clear in any way, the answer is YES!


Instead of using software that comes on a CD with the printer, you'll have to download and install drivers yourself.  They are not easy to find if you live in the US since they're only available on European and Asian Canon websites.  I'll tell you how to download and install them.


Finding and installing the drivers

  1. Visit the Linux IJ Printer Driver page on the Canon Europe web site.
  2. Click the Linux IJ Printer Driver MX880 series.tar link in the Downloads section to download the 7299.5 kb tarball.
  3. Inside the tarball, find and unpack cnijfilter-mx880series-3.50-1-deb.tar.gz.
  4. Inside the resulting cnijfilter-mx880series-3.50-1-deb/packages directory, locate and double click cnijfilter-common_3.50-1_i386.deb and  cnijfilter-mx880series_3.50-1_i386.deb in that order.  Ubuntu will guide you through the installation. 
Note: The guide that came with this driver asks you to run the program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Adding the printer

  1. From Ubuntu's menu, select: System,  Administration, then Printing.
  2. From Printing window that opens, select Add.
  3. After a few seconds,  you should see a Canon MX880 printer show up under Network Printer.
  4. Select it and click Forward.
  5. Finish entering your printer Name, Description, and Location.


Finding and installing the drivers

  1. Visit the Linux IJ Scanner Driver page on the Canon Europe web site.
  2. Click the Linux IJ Scanner Driver MX880 series.tar link in the Downloads section to download the 3072 kb tarball.
  3. Inside the tarball, find and unpack scangearmp-mx880series-1.70-1-deb.tar.gz.
  4. Inside the resulting scangearmp-mx880series-1.70-1-deb/packages directory, locate and double click scangearmp-common_1.70-1_i386.deb and  scangearmp-mx880series_1.70-1_i386.deb in that order.  Ubuntu will guide you through the installation. 
Note: The guide that came with this driver asks you to run the program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

    Scan Gear MP

    With the drivers, an app called Scan Gear MP is also installed.  This app serves as an interface to your scanner and has a Basic Mode:

    And an Advanced Mode:

    You can launch the app from the terminal by typing scangearmp and it is also available within Gimp by selecting FileCreate, Scan Gear MP...

    I have not yet figure out a way to use the scanner via XSane.  I'd love to hear comments about whether this is possible and how to do it.  Without XSane support, I can't use familiar programs such as SimpleScan and gscan2pdf.


    One more thing to mention.  Before I figured out Canon made drivers for this scanner, I came across a program called VueScan which claimed to support MX880 series printers on Linux via the network without installing any drivers at all.  It took seconds to download, launch, and see that this program could communicate with my MX882 scanner with ease!  It may be worth the $79.95 if you can't get Canon's drivers to work.

    Happy printing, scanning, copying, and faxing!

    Thursday, February 03, 2011

    Edit Remote EC2 Text Files with TextWrangler

    This post explains how to use TextWrangler on a Mac to edit text files that live on a remote Amazon EC2 instance with Ubuntu.

    I assume you already know how to SSH to Amazon using an Identity File:

    ssh -i ~ec2-keys/id_your-keypair

    You can use a similar command to connect to your EC2 instance via SFTP:

    sftp -o IdentityFile=eec2-keys/id_your-keypair

    TextWrangler allows you to open a file from an SFTP server, but it doesn't allow you to configure an Identity File as you did when SSHing or SFTPing from the command line. To work around this limitation, you simply have to configure your SSH client so that it knows the Identity File you want to use. Create ~/.ssh/config if it doesn't already exist and add a line specifying your Identity File location:

    echo 'IdentityFile ~ec2-keys/id_your-keypair' >> ~/.ssh/config

    Now you're ready to use TextWrangler to connect to your server and browse files. Choose: File -> Open from FTP/SFTP Server... Then enter your server publisher DNS name, check the SFTP box and enter  ubuntu as the User name.  You can leave the Password: field blank.

    After you hit connect,  you TextWrangler will allow you to browse the file system of your EC2 instance, choose a file, and edit it.  When you save the file, it will update directly on the remote EC2 instance.  That's all there is to it!