Tuesday, April 17, 2012

Avoiding Spam Emails with Google Apps

Employees at my startup have started complaining that their outgoing emails were landing in their recipient's Spam folders.  It seems to happen more often with recipients that use Outlook. After a little research I discovered that our Google Apps and GoDaddy DNS setup was missing some key configurations: an SPF Record and DKIM Record.

After making the changes described below and waiting a few hours for DNS changes to propagate, the Spam problem has been resolved!

Sender Policy Framework (SPF) Records

An SPF record is a type of DNS record that identifies which mail servers are permitted to send email on behalf of your domain.   If an email message comes from a server other than the Google Apps mail servers listed in the SPF record, the recipient's mail server can reject it as spam.  More info.

Google gives these instructions for creating an SPF record. I followed those instructions which resulted in me adding a new TXT DNS record in GoDaddy:


TXT Value:
v=spf1 include:_spf.google.com ~all

DomainKeys Identified Mail (DKIM) Standard

You can add a digital "signature" to the header of mail messages sent from your domain.  Recipients can check the domain signature to verify that the message really comes from your domain and that it has not been changed along the way.   Google gives these instructions for getting your emails signed and verified.  I followed those instructions which resulted in me adding another TXT DNS record:


TXT Value:
v=DKIM1; k=rsa; p=CIGfMA0GCSqGSIb2DQEBAQAAA4GNADCBiQHKgQCj+tnMQMGMn8NfHnpDmgPa7ICUKdXdyzTlkBglZKRfEtF9msn1v/TmHZEvWFFp3KiaL2Igs7K57l+n/QJlk8Aj9C9nTGmXnzm9BL2zOQQL/zxJh9qh22bnO8uf7tM7sGHxr3z7yIkpXzA96G0inqmNb2XztXKseV4dp5jXbow4+QIDAQAB

I used the following services to test if the above configurations were working properly:


The OpenSPF site contains detailed instructions.  You basically send an email to spf-test@openspf.org. Your email will bounce but the bounce message will contain diagnostic information.  Before I made any changes, the bounced email contained:

SPF Tests: Mail-From Result="none": Mail From="ken@mystartup.com" HELO name="mail-vb0-f45.google.com" HELO Result="none" Remote IP=""

After adding an SPF record, I got:

SPF Tests: Mail-From Result="pass": Mail From="ken@mystartup.com" HELO name="mail-ob0-f173.google.com" HELO Result="none" Remote IP=""

After setting up DKIM, a new section appeared:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
       d=mystartup.com; s=google;


For this one, you send an email to check-auth@verifier.port25.com.  If you SPF record and DKIM is setup properly, you should get a reply containing this summary:

Summary of Results
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham


This is similar to the service from Port25.com.  You send an email to check@isnotspam.com and get a similar reply.

Wednesday, December 14, 2011

AutoScaling Amazon SQS Queue Processors

One of my favorite things about running servers in Amazon EC2 is the ability to use AutoScaling to automatically add and remove nodes as web traffic increases and decreases.  Not only does this generally save money, but it also helps prepare a system to handle traffic spikes.

Recently I had some fun setting up AutoScaling for a different use case -- a cluster of machines processing messages on an Amazon SQS queue.  The idea was to add and remove nodes as the number of visible messages on the queue fluctuated.  Again, this keeps costs lower by only running as many nodes as are necessary to process the current workload and handles workload spikes.

Here are sample steps and commands that you can use for setting up AutoScaling for SQS queue processors:

Create an AMI for a node that will process SQS messages.  The node that boots from this AMI should automatically launch one or more queue processing processes.  A user-data script may be useful for this.

Create a Launch Configuration for the queue processor node:

as-create-launch-config MyQueueConfig --image-id [INSERT YOUR AMI ID HERE] --instance-type c1.medium --key [INSERT YOUR KEYNAME HERE] --user-data [INSERT YOUR USER DATA SCRIPT HERE]

Create an AutoScaling group for queue processors:

as-create-auto-scaling-group MyQueueGroup --launch-configuration MyQueueConfig --availability-zones us-east-1b --min-size 1 --max-size 10

Create Policies that add/remove 1 node to/from the cluster.  They will be invoked when the number of messages on the queue grows excessively high or decreases to an acceptable level:

as-put-scaling-policy MyScaleUpPolicy -g MyQueueGroup --adjustment=1 --type ChangeInCapacity

as-put-scaling-policy MyScaleDownPolicy -g MyQueueGroup --adjustment=-1 --type ChangeInCapacity

Create Alarms to scale up/down when the number of messages on the queue grows excessively high or decreases to an acceptable level.  Use the Policy ARN's returned by the previous as-put-scaling-policy commands.

mon-put-metric-alarm --alarm-name MyHighMessagesAlarm --alarm-description "Scale up when number of messages on queue is high" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 1000 --comparison-operator GreaterThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleUpPolicy ARN HERE]

mon-put-metric-alarm --alarm-name MyLowMessagesAlarm --alarm-description "Scale down when number of messages on queue is low" --metric-name ApproximateNumberOfMessagesVisible --namespace AWS/SQS --statistic Average --period 60 --threshold 100 --comparison-operator LessThanThreshold --dimensions QueueName=MyQueue --evaluation-periods 10 --alarm-actions [INSERT MyScaleDownPolicy ARN HERE]

In this example the Alarms cause the cluster to scale up when the number of visible messages on the queue remains above 1000 for 10 consecutive minutes and scale down when the number of visible messages falls below 100 for 10 consecutive minutes.

The Policies above adjust the amount of nodes in the cluster by a specific amount, but it is also possible to specify the adjustment in terms of percentages.  Using --adjustment 10 --type PercentChangeInCapacity would adjust the amount of nodes by 10 percent.

It also would have been possible to base scaling activities on other AWS/SQS metrics such as:
  • NumberOfMessagesSent
  • NumberOfMessagesReceived
  • NumberOfMessagesDeleted
  • NumberOfEmptyReceives
  • ApproximateNumberOfMessagesVisible
  • ApproximateNumberOfMessagesNotVisible
  • ApproximateNumberOfMessagesDelayed
  • SentMessageSize
Here are a few online references relevant to AutoScaling SQS queue processors:

Thursday, October 20, 2011

Online Advertising Formula Reference

Occasionally I need to manipulate online advertising formulas to make a quick calculation.  The following serves a quick cheat sheet for this purpose. Note that the term Cost is from the perspective of the Advertiser. The web site or publisher can translate that into Revenue.

CPM Campaigns

Cost = Impressions / 1000 * CPM

CPC Campaigns

Cost = Impressions * CTR * CPC
CPM = CTR * CPC * 1000

CPA Campaigns

Cost = Impressions * CTR * AR * CPA
CPM = CTR * AR * CPA * 1000

CPMCost Per 1000 Impressions
CPCCost Per Click
CPACost Per Action
CTRClick Thru Rate (Clicks / Impressions)
ARAction Rate (Actions / Clicks)

Monday, July 18, 2011

Giant Ampersand - Washington/Centinela Redevelopment Part 6

It's been almost 5 years since my last post about the corner lots of Washington and Centinela which are supposedly destined for redevelopment. That's because pretty much nothing has happened on those lots - until this past week.

All of the sudden a 8-foot tall white wooden ampersand appeared in the middle of the lot on the north east corner.  I had no idea what it meant, but I was really happy to see it.  To me it meant that someone out there is thinking about this vacant plot of land.

A little Googling produced an answer.  The mysterious ampersand is a public project by Los Angeles-based artist Shannon Ebner.  According to the LAXART web site:

The ampersand is a conjoiner; it signals an incomplete thought and acknowledges that there is more to be said.

This is part of a Shannon Ebner exhibition at the Hammer Museum which started July 16, 2011.  Apparently this isn't the first time she's used an ampersand in her work.  See this photo from a year ago in Milan.

I've been waiting for the redevelopment on this corner to get underway since I moved to this Del Rey neighborhood 11 years ago.  I really hope, as the giant ampersand acknowledges, that there is more to be said...

Tuesday, April 26, 2011

Using a Canon Pixma MX882 All-In-One with Ubuntu 10.10

Are you thinking about buying a Canon Pixma MX882 Wireless Office All-In-One Printer and wondering if it will work with your Ubuntu computer?  Although Canon does not make this clear in any way, the answer is YES!


Instead of using software that comes on a CD with the printer, you'll have to download and install drivers yourself.  They are not easy to find if you live in the US since they're only available on European and Asian Canon websites.  I'll tell you how to download and install them.


Finding and installing the drivers

  1. Visit the Linux IJ Printer Driver page on the Canon Europe web site.
  2. Click the Linux IJ Printer Driver MX880 series.tar link in the Downloads section to download the 7299.5 kb tarball.
  3. Inside the tarball, find and unpack cnijfilter-mx880series-3.50-1-deb.tar.gz.
  4. Inside the resulting cnijfilter-mx880series-3.50-1-deb/packages directory, locate and double click cnijfilter-common_3.50-1_i386.deb and  cnijfilter-mx880series_3.50-1_i386.deb in that order.  Ubuntu will guide you through the installation. 
Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

Adding the printer

  1. From Ubuntu's menu, select: System,  Administration, then Printing.
  2. From Printing window that opens, select Add.
  3. After a few seconds,  you should see a Canon MX880 printer show up under Network Printer.
  4. Select it and click Forward.
  5. Finish entering your printer Name, Description, and Location.


Finding and installing the drivers

  1. Visit the Linux IJ Scanner Driver page on the Canon Europe web site.
  2. Click the Linux IJ Scanner Driver MX880 series.tar link in the Downloads section to download the 3072 kb tarball.
  3. Inside the tarball, find and unpack scangearmp-mx880series-1.70-1-deb.tar.gz.
  4. Inside the resulting scangearmp-mx880series-1.70-1-deb/packages directory, locate and double click scangearmp-common_1.70-1_i386.deb and  scangearmp-mx880series_1.70-1_i386.deb in that order.  Ubuntu will guide you through the installation. 
Note: The guide that came with this driver asks you to run the install.sh program, but it didn't work for me.  When I ran it, I got an error: An error occurred. The package management system cannot be identified.

    Scan Gear MP

    With the drivers, an app called Scan Gear MP is also installed.  This app serves as an interface to your scanner and has a Basic Mode:

    And an Advanced Mode:

    You can launch the app from the terminal by typing scangearmp and it is also available within Gimp by selecting FileCreate, Scan Gear MP...

    I have not yet figure out a way to use the scanner via XSane.  I'd love to hear comments about whether this is possible and how to do it.  Without XSane support, I can't use familiar programs such as SimpleScan and gscan2pdf.


    One more thing to mention.  Before I figured out Canon made drivers for this scanner, I came across a program called VueScan which claimed to support MX880 series printers on Linux via the network without installing any drivers at all.  It took seconds to download, launch, and see that this program could communicate with my MX882 scanner with ease!  It may be worth the $79.95 if you can't get Canon's drivers to work.

    Happy printing, scanning, copying, and faxing!