Amazon ELB - Capturing Client IP Address

If you're using Amazon EC2's Elastic Load Balancer (ELB) for load balancing web applications, you may have noticed that in your web access logs, the remote host IP address is the same for every request. The IP address you see is the private IP address of the load balancer.

If you want to see the IP address of the client (called remote host in access log documentation), you'll need to look at the value of the X-Forwarded-For request header which ELB populates when it forwards the request.

This can be achieved in an Apache access log by using the syntax:

%{X-Forwarded-For}i

You Apache log format would then look something like this:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined-elb
CustomLog log/acces_log combined-elb

See Apache Custom Log Formats

If you're using a Tomcat application server, you could define an access log Valve like this:

<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs" prefix="access_log." suffix=".txt"
    pattern="%{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
    resolveHosts="false"/>

See Tomcat 6 Valve Configuration Reference.

If you're trying to capture the client IP address within your application code, simply use whatever API you have to read the X-Forwarded-For request header. For example in Java use the HttpServletRequest:

String clientIpAddress = request.getHeader("X-Forwarded-For");

instead of

String clientIpAddress = request.getRemoteAddr();

Serving Gzipped Javascript Files from Amazon CloudFront

It is well-known that using a content delivery network (CDN) and compressing an HTTP response with gzip can significantly improve the performance and reduce the cost of a web site. If your CDN is Amazon's CloudFront, you'll face difficulties serving gzipped content to browsers that support it. Most web servers are able to examine the HTTP request headers sent by the browser and dynamically choose whether to deliver compressed or uncompressed content. CloudFront, however, does not yet offer this feature.

People have overcome this problem by manually inspecting the Accept-Encoding request header for a dynamically served page and writing references to either original or compressed files accordingly using a naming convention like myscript.js and myscript.js.gz for regular and compressed files respectively.

If all your pages are statically served from CloudFront, however, there is no opportunity to inspect the request headers. Javascript running on a statically served page has no ability to ask the browser if it supports gzipped content. We found ourselves in this situation and resorted to the following solution to determine if the browser supports gzip:

1) Create a small gzipped file, gzipcheck.js.jgz, and make it available in CloudFront. This file should contain one line of code:

gzipEnabled = true;

2) Use the following code to attempt to load and run this file. You'll probably want to put it in the HTML HEAD section before any other Javascript code.

<script type="text/javascript" src="gzipcheck.js.jgz">
</script>

If the file loads, it sets a flag, gzipEnabled, that indicates whether or not the browser supports gzip.

Use the result to drive a file naming convention for references to other static files. For example, you can upload and reference each compressed Javascript file with an additional .jgz extension. Why .jgz instead of .gz? Because of an annoying limitation of Safari.

When you upload your files to S3/CloudFront, make sure to set the proper HTTP response headers on your files that end in .jgz:

Content-Encoding = gzip
Content-Type = application/x-javascript

And if you want browsers to cache your files forever (almost):

Cache-Control = max-age=315360000
Expires = Tue, 31 Dec 2019 20:00:00 GMT

Thanks to Ricardo Rangel who helped design and code this solution.

Update

As of December 20, 2015, Amazon Web Services announced support for Gzipped CloudFront files!  See Serving Compressed Files from their developer guide.

Ubuntu License Plate on the Santa Monica Freeway

I came across this car while driving eastbound on the Santa Monica Freeway this past Saturday. Do you think the license plate is referring to the Ubuntu Linux OS or just the Ubuntu philosophy?

It also had some funny stickers and decals:
* Evolve FISH Logo Decorative Silver Car Emblem
* Goodnight Bush: A Parody
* Got Hope Bumper Sticker

Products I Can't Live Without - 2009

Inspired by this post from Michael Arrington of TechCrunch, I thought it would be interesting to make my own annual list of products and services that I can't live without. Here we go for 2009 (in the order that they came to mind):

Gmail, Google Calendar, Remember The Milk, iPhone, F-Spot, Quicken, Google Reader, Facebook, Google Sites, Flickr, iTunes, Twitter, Jungle Disk, Dropbox, Delicious

Gmail
I use this every minute every day for both personal and work email (my company uses Google Apps for email and calendar). I just love the user interface and ease of accessing it on the web and the iPhone. Beats the hell out of Outlook for work email.

Google Calendar
Again, this is how I keep track of temporal events in my life both at home and at work. I love that I can get text messages with alerts for calendar events.

Remember The Milk
Ever since reading David Allan's Getting Things Done, I was in search of a tool to put his ideas to work. About a year ago, I settled with Remember The Milk (RTM) and have used it ever since for task management. The user interface is outstanding and I can access it from the web and iPhone. Perhaps the best feature is that it integrates perfectly with Gmail as a "Gadget".

iPhone
I don't identify with being an Apple person, but I couldn't resist the lure of the iPhone. It really adds a lot of value to my life as I can manage email, read blogs, listen to Podcasts, use Google Maps, locate GeoCaches, and even SSH to a server at work. Oh, and visual voice mail rocks!

F-Spot
This GNOME desktop application is what I use to manage photos. It stores photo metadata in a SQLite database which I can easily access if I ever want to export the data somewhere else. It also makes it easy to upload photos to Flickr.

Quicken
I have been using Quicken for years to manage my personal finances. I upgrade it to the latest version every even year, so right now I have Quicken 2008. Honestly, I would love to stop using Quicken and switch to a web-based solution, but I can't find one that is fully-featured enough for me. I just have to have the ability to create arbitrary asset accounts for my wallet (cash), reimbursements, etc. Most web-based services don't offer this yet. I've got my eye on Mint which, I have a feeling, will get good enough for me to make the switch one day.

Google Reader
I read a lot of blogs. I couldn't possibly keep up if I didn't use a blog reader. In late 2008, I switched from Bloglines to Google Reader because the user interface for Google Reader just kept getting better and eventually it won me over. A big part of this was the excellent user interface of Google Reader on the iPhone.

Facebook
This barely made it to my list. Maybe I could live without it, but I do find myself checking it almost daily because so many of my friends use it and somehow it makes me feel like I'm in touch with all of them without ever really talking to them :). Also, Facebook is the platform I used to develop and launch my book sharing application We Read which continues to grow slowly, but steadily each day and recently passed 1000 users.

Google Sites
This past year I finally committed to using a wiki for my family. I have been using wikis for years for work and open source software projects, and always thought it would be useful for keeping track of personal things like lists of gifts to get people, account information for utilities, financial services, and contractors, etc, etc. I tried setting up and hosting wikis like XWiki and Confluence, but ultimately went with the free, online Google Sites which was formerly JotSpot.

Flickr
I purchased the Pro Account and now use Flickr for all my online photo sharing needs. The pro account allows me to upload an unlimited amount of photos for about $25/year. My family depends on this to see the many photos I take of my son as he grows up.

iTunes
I hate the fact that iTunes doesn't run on Linux machines, but I can't resist it because it is really the only good option for managing music and videos on an iPod. I use it daily to load up my iPhone with podcasts.

Twitter
I use Twitter a lot. I blame Twitter for killing my urge to blog. It is just so much easier to share things 140 characters at a time. I have Twitter linked with Facebook so that each Tweet I send out ends up updating my Facebook status.

Jungle Disk
This might be one of the most important services I use. It keeps my music, videos, photos, and other important files backed up on Amazon's S3 network. At only $.15/GB, it ends up being pretty cheap to store a lot of data. This way, if my house ever burns down or someone steals my computers, or I experience a hard disk failure, I will always be able to restore my precious data.

Dropbox
I have started storing all files that I access often in a special directory on each computer (home, work, laptop, etc) called the Dropbox. Each directory gets automatically synchronized so I have easy, local, access to the files I need wherever I am. It also makes it extremely easy to share files with other people that use Dropbox.

Delicious
I store all my bookmarks in Delicious and take advantage of its Firefox plugin to easily tag sites and recall sites I've already tagged. I don't do a lot with the social features of Delicious like bookmark sharing, though. Maybe it's just because I don't know which of my friends use Delicious too.

Well, there you have it! I can't wait to see how this list compares to the one I'll make in January 2010.

Groovy Best Fit Line

An engineer on my team at work asked if I could help with coding an algorithm for calculating a line that best fits a given set of data points. I thought, "what a great excuse to practice some Groovy", and got started right away.

A few Google searches led me to this 10-yr-old page on Regression Functions by Stefan Waner from Hofstra University. Stefan outlined the exact algorithm I was looking for.

Here is my implementation in Groovy:

// Prints the start/end points of a line that best fits 4 sample data points
pts = [[1, 1.5], [2, 1.6], [3, 2.1], [4, 3.0]]
println bestFit(pts)

/**
* Given a set of points, uses a linear regression algorithm to find the start and end points
* of a line that best fits the set of points.
* Returns the two points as [[x1, y1], [x2, y2]].
* The algorithm is from
* http://people.hofstra.edu/stefan_waner/realworld/calctopic1/regression.html.
*/
def bestFit(pts) {
  // Find sums of x, y, xy, x^2
  n = pts.size()
  xSum = pts.collect() {p -> p[0]}.sum()
  ySum = pts.collect() {p -> p[1]}.sum()
  xySum = pts.collect() {p -> p[0]*p[1]}.sum()
  xSqSum = pts.collect() {p -> p[0]*p[0]}.sum()

  // Find m and b such that y = mx + b
  // m is the slope of the line and b is the y-intercept
  m = (n*xySum - xSum*ySum) / (n*xSqSum - xSum*xSum)
  b = (ySum - m*xSum) / n

  // Find start and end points based on the left-most and right-most points
  x1 = pts.collect() {p -> p[0]}.min()
  y1 = m*x1 + b
  x2 = pts.collect() {p -> p[0]}.max()
  y2 = m*x2 + b

  [[x1, y1], [x2, y2]]
}

Running this script prints the following:

[[1, 1.3], [4, 2.8]]

You gotta love Groovy!