What a long battle with Cisco IOS this has been, but after quite a bit of tinkering I've gotten things working the way that I would like. Here's a technical description of the details in hope that this helps someone else.

The Setup

• Load balancers with private IP address like 172.16.0.10 on a /24, running example.com
• Cisco ASA Firewalls running 7.2(1) or newer, that map public IP addresses (I'll use 192.168.0.193 on a /24 here instead of a real public IP)
• Internal DNS servers that map loadbalancer.private to 172.16.0.10
• External DNS servers that map example.com to 192.168.0.193
• Random application server behind the firewall with no public IP address and a private IP of 172.16.0.20

The Problem

Applications behind the firewall need to access other applications behind the firewall using the public DNS name (example.com) instead of the private one (loadbalancer.private).

Some possible solutions

As an easy-to-set-up solution, we currently have the internal dns servers set up to map example.com to 172.16.0.10 which works fine, except it requires updating DNS records in multiple places. Our naming scheme slowly got a bit more complex, and I've had to add explicit relay rules to our DNS server configuration files to relay certain lookups from the internal DNS servers to the external DNS server's internal IP address. Sending it to the DNS server's external IP address doesn't work because the Cisco ASA will not send traffic back out on the same interface that it came in on, even after network translations have been done. (For a different portion of our external IP space, I added some static routes to the core router but when we move those IPs behind this firewall, this ASA feature will break those routes as well)

The current mapping of public IPs to private IPs looks like:

static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255

One feature that Cisco suggests to solve our problem is using "DNS Doctoring" which is just simply adding the 'dns' keyword to the end of the mapping like:

static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 dns

which modifies DNS queries going through the firewall from the inside interface to change the IP from 192.168.0.193 to 172.16.0.10. This would great, if your DNS server is outside of the firewall, which ours is not. Our internal DNS queries never travel through the ASA so this didn't do anything for us.

Up next was trying out

same-security-traffic permit intra-interface

which "permits communication in and out of the same interface" which sounds like it's the exact right solution for the problem because that was the limitation that broke things. However, adding this in didn't seem to change anything and traffic still was not permitted in and out the same interface.

The Solution

After a lot of troubleshooting, which involves an ASA 5510 and a 3524-XL on the floor under my desk, downloading and installing new versions of IOS, a lot of Googling, a lot of cursing, and a lot of sketching possible things out on paper, I finally figured out the missing piece: Hairpinning which is "the process by which traffic is sent back out the same interface on which it arrived." Here is the configuration that finally got traffic flowing from 172.16.0.10 to 192.168.0.193 on the ASA back out to 172.16.0.10 on the same interface it started on:

!--- Output suppressed.
!
interface Ethernet0/0
 nameif outside
 security-level 0
 ip address 192.168.0.192 255.255.255.0 
!
interface Ethernet0/1
 nameif inside
 security-level 100
 ip address 172.16.0.1 255.255.0.0 
! 
!--- Output suppressed.
!
same-security-traffic permit intra-interface
access-list outside_in extended permit icmp any any 
access-list outside_in extended permit tcp any any 
!
!--- Output suppressed.
!
global (outside) 1 interface
nat (inside) 1 172.16.0.0 255.255.0.0
alias (inside) 192.168.0.193 172.16.0.10 255.255.255.255
alias (inside) 10.0.0.20 172.16.0.20 255.255.255.255
static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 
access-group outside_in in interface outside
!
!--- Output suppressed.

The trick here was, combined with "same-security-traffic permit intra-interface" to add the alias lines, the first one:

alias (VLAN100) 192.168.0.193 172.16.0.10 255.255.255.255

does something sensible and aliases 192.168.0.193 to 172.16.0.10 on the inside interface so any time traffic comes in here matching that IP, it gets rewritten. The second line is also required but doesn't make as much sense:

alias (inside) 10.0.0.20 172.16.0.20 255.255.255.255

This line is telling the ASA to take any traffic coming in destined to 10.0.0.20 and map it to 172.16.0.20, however, we don't have any devices on 10.0.0.0/8 and there are no routes for this, so there will never be any traffic coming in to 10.0.0.20. That said, this line has to exist so that there is a mapping back to 172.16.0.20 in the alias table so that the ASA knows it's alright to send traffic to it. Using a "real" public IP here would both use up our public IPs and perhaps pose some security risk, so it's safer to use these non-public IPs and add a rule to prevent incoming traffic from the outside from reaching them. If the alias command would work for an IP range instead of one host, this would be pretty much perfect.

The result

Things finally work! Here is a trace of a ping from 172.16.0.20 to 192.168.0.193 (which works now!):

ICMP echo request from VLAN100:172.16.0.20 to VLAN100:192.168.0.193 ID=12034 seq=0 len=56
ICMP echo request translating VLAN100:172.16.0.20 to VLAN100:10.0.0.20
ICMP echo request untranslating VLAN100:192.168.0.193 to VLAN100:172.16.0.10

So the ASA is doing the translating the proper way and not doing anything with 10.0.0.20. This is good news because it means that our naming and routing architecture can be greatly simplififed:

• All relay rules for external facing domains that have previously required this "split-horizion" DNS can be removed, returning the DNS server configurations to a generic state
• All crazy static routes for external IP addresses can be removed from our core router
• All external facing domain zones can be removed from the internal DNS servers, and updates when things are moved only have to be done in one place

The only penalty for this is adding in the alias lines to our ASA configuration for each existing static mapping that we have, as well as adding an alias line for each server that needs to communicate with the external IP addresses of things behind the same ASA which should be limited to the internal DNS servers and a few application servers.

References

• Configuring Interfaces for the Cisco ASA 5505 Adaptive Security Appliance - cisco.com
• same-security-traffic through show asdm sessions Commands - cisco.com
• PIX/ASA: Perform DNS Doctoring with the static Command and Two NAT Interfaces Configuration Example - cisco.com

EDIT: Another way to do this

After sharing this with some coworkers, it turns out that 'hairpinning' is definitely the key word and one of them stumbled across this article:

Setup U-Turn (Hairpinning) on Cisco ASA

It solves the same issue with a slightly more graceful solution because no alias entries are needed for non-public services, in fact, no aliases are needed at all. To have the exact same functionality as above, here is the working configuration for the problem above with this new methodology:

!--- Output suppressed.
!
interface Ethernet0/0
 nameif outside
 security-level 0
 ip address 192.168.0.192 255.255.255.0 
!
interface Ethernet0/1
 nameif inside
 security-level 100
 ip address 172.16.0.1 255.255.0.0 
! 
!--- Output suppressed.
!
same-security-traffic permit intra-interface
access-list outside_in extended permit icmp any any 
access-list outside_in extended permit tcp any any 
!
!--- Output suppressed.
!
global (outside) 1 interface
global (inside) 1 interface
nat (inside) 1 172.16.0.0 255.255.0.0
static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 
static (inside,inside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 
access-group outside_in in interface outside
!
!--- Output suppressed.

This past Monday, Seth and I headed to California for the week to get some work done. We didn't have hotel reservations that we knew of, and had a mess of things to clean up in the datacenter, so we drove the rental car straight from the airport to the office at around 11am on Monday and got started.

Monday

We spent Monday in the office, getting some face time with the new office IT guy ("chicks" is his username which is the source of much hilarity) and meeting with some people that we have ongoing projects with. Lunch was at Dittmer's Gourmet Meats and dinner was 4x4s at In-N-Out. We ended up crashing at Jesse's house after sitting in his hottub drinking Micky's, and watching Apocalypse Briggs (part 1 here, additional parts in "related videos"). It's nice sharing rooms with Seth because he likes sleeping on the floor which means no complicated figuring out beds/couches/etc. A pillow and a blanket, and he's set!

Tuesday

Pretty early on Tuesday, we headed directly to the datacenter, stopping at Le Boulanger on the way for tasty breakfast sandwiches. After getting our hands added to the biometrics system, we began sorting spare parts, getting rid of trash and server packaging, and removing wires that weren't plugged in to anything. 2 people from Virident Systems showed up with a box for us to install that we're doing some experimenting with, and things are looking pretty good so far with that. They took us out to eat at a Malaysian place that was pretty good, and our afternoon in the datacenter was more cleaning up. We drove to Thee Parkside in the city for beer and $2 tacos with some of the Gallery crew, and headed over to Digg with Robert for a few more beers. Afterwards, Seth and I drove Bharat back home and slept at his brand new house in Menlo Park. Digg HQ:

Wednesday

We started off Wednesday morning dropping Bharat off at work at Google, and getting a quick tour of Google HQ for Seth. After that was another datacenter day, interrupted with a trip to the office for some Japanese food for lunch. The grand total of trash we cleaned up filled up a 48 gallon plastic bin, and we began fixing labels on machines, noting rack locations in our ZenOSS installation, and properly labeling all the outlets on our PDUs and what they are connected to. Aside from everything looking a _lot_ better, highlights of the day included finding a machine we didn't know about with 32G of RAM (now a OpenVZ box doing a lot of things). For the evening we headed up to Lila's place in the hills of Los Gatos where the SugarCRM IT crew enjoyed beers and pork ribs, and Seth and I slept in a spare room there after staying up long past the always amazing sunsets:

Thursday

After the crazy drive back down from Lila's, we headed to the datacenter for the morning. It took us about 4 hours to finish things up including rewiring all the cat5 in one rack and mostly wiring up a new rack of machines (still waiting on the switch and PDUs before that will be done). Back at the office, we had a very late lunch of more 4x4s at In-N-Out because they apparently couldn't make us 5x5s. I spent the rest of the afternoon catching up on some of the ticket backlog assigned to me since we'd been busy all week doing other things, and around 6:30 we drove up to Igor's place and got to see "mini beast", Igor's newborn. Several other SugarCRM people met up with us to head to Whiskey Thieves for some whiskey sampling. At some point, Julian and I put a few dollars into the Area51 machine there and ended up with 5th and 6th place on the high score list, and he told us "The Japanese Fan Story" which you should get him to share if you haven't heard it yet. Afterwards, we stopped by The Owl Tree and ended up at Cocobang for some super spicy Korean BBQ chicken to finish off the week. A week's work:

Friday

Friday morning was back to the airport to fly home. It was another crazy exhausting week in California and while we got a lot done, I'm definitely glad to be home. Delta helped us out because both our flight our and our flight back took ~45 minutes less than expected. All meals not described above were either not eaten, or consisted of cherry coke and taquitos from 711. Now that I'm home, it's time to hunt down some people to pay their hosting bills (Eldon- While biking today I saw you on your bike so I know you are alive!) and mow the grass. Pictures from the week are at http://ckdake.com/gallery/2009/june-california/.

SugarCON was last week, so I was in San Francisco, California to interact with some of SugarCRM's customers and other employees. We did a good bit of socializing which is partly captured in this photo album of pictures from my little camera, but I took a lot of pictures with the big camera and some things are worth pointing out here. Tuesday evening was the SugarCON boat cruise around the bay, which had some awesome views including:

Wednesday after SugarCON ended, a few of us went to a shooting range for about an hour and took a few pictures inside. (I took this 365 picture on the way there.) San showed up on Thursday and did a bit of San Francisco exploring on her own, and after a boot-camp with some SugarCRM partners, the IT team and most of our significant others headed to Lake Tahoe for a few nights away from the city. The lake was a short walk through the woods from our cabin, and of the Lake Tahoe pictures I uploaded, these capture it pretty well:

We didn't do any winter sports other than throwing a few snowballs, but it was nice to have casual conversations with work people over beers or in the hot tub instead of on IRC. (Also, FYI, when you see "tonemapped" in a URL of a picture here, its a tonemapped HDR image resulting from a combination of 3 images that gives more details in highlights and shadows than a single image would, sometimes more accurately producing the look of a scene and sometimes just making it look awesome.) On Sunday, Jesse, Seth, San and I headed back towards the cost with the hope of reaching Point Reyes before sunset. On the way, we stopped a few times. First for a bathroom break and some pictures including:

Then, at a Point Reyes visitor center on the San Andreas Fault to walk along the "earthquake trail" and break open our 1.5lb bacon and cheese sandwich on the fault line:

Even though we made it to Point Reyes lighthouse before sundown, it was after 4:30pm so the visitor center and lighthouse were closed. Regardless, some great views:

We arrived back in San Francisco under the cover of darkness, and got in some exploring the next day before heading home including Twin Peaks, Fort Point NHS, and the Golden Gate Botanical Gardens:

More pictures from everything are in these albums:

• SugarCON Boat Trip
• Shooting in CA
• Lake Tahoe on Saturday
• Point Reyes
• Exploring San Francisco
• SugarCON etc

All in all, it was a pretty good trip but after 10 days away from home, I was glad to be back!