HOWTO
Software RAID Drive Replacement
Submitted by ckdake on Sat, 2010-08-21 12:37Software RAID is such a finicky thing. I've blogged in the past on how to grow a software RAID array in Linux, but have since switched to using hardware RAID in all of my servers and a Drobo at home (which I wrote a little about here). My Drobo just got 2 new 2TB drives and all that took was sliding the drives in, but one of my old servers lost a software-RAID drive last week.
This crashed the server for some reason, and a matching replacement drive for the Hitachi Ultra320 SCSI 73GB drives was $400 or so, so I bought an IBM Ultra320 SCSI 73GB drive online for $50 shipped. It arrived this week and I headed to the datacenter today to install it. What should be a stupid-simple process like it is with Drobo was a lot more involved. This was made even more painful because while the labeled size of the drives was identically, the new one had a few less actual blocks on it than the old one. Should this happen to you with an ext2/3/4 data volume, (or me again in the future) these are the steps to take:
- Shut down the system
- Replace the failed drive
- Boot up from a recovery CD (I used a Gentoo install CD)
- Use fdisk to partition the new drive with as close of a partition layout as you can to the old drive. Here are my two drives:
Disk /dev/sda: 73.4 GB, 73407868928 bytes 255 heads, 63 sectors/track, 8924 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xc36bc36b Device Boot Start End Blocks Id System /dev/sda1 * 1 5 40131 fd Linux raid autodetect /dev/sda2 6 249 1959930 82 Linux swap / Solaris /dev/sda3 250 8924 69681937+ fd Linux raid autodetect
Disk /dev/sdb: 72.9 GB, 72892735488 bytes 255 heads, 63 sectors/track, 8862 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xf496138a Device Boot Start End Blocks Id System /dev/sdb1 * 1 5 40131 fd Linux raid autodetect /dev/sdb2 6 249 1959930 82 Linux swap / Solaris /dev/sdb3 250 8862 69183922+ fd Linux raid autodetect
- Note any block sizes that are different. In my case, the new sdb3 is 69183922 which is smaller than 69681937 on the old drive
- Use resize2fs to resize the existing partition on the good device to the size of the new partition on the new device. Yes, you are modifying it directly instead of going through the software RAID (one of the few nice things about software RAID):
resize2fs -f /dev/sda3 69183922
- Use mdadm to shrink the raid device to the size of the new partition on the new device:
mdadm /dev/md127 --grow --size=69183922
- Just to be safe, run a filesystem check on the new RAID volume:
e2fsck -y /dev/md127
- Add in the new drive to the RAID array:
mdadm /dev/md127 --add /dev/sdb3
- And wait for it to finish resyncing:
watch cat /proc/mdstat
If you run into device busy errors, you may need reboots, stopping the raid device (mdadm --stop /dev/md127), etc. And if you screw up the block sizes, all kinds of bad things happen. Also, if you just shrink the RAID volume but don't shrink the filesystem, or you shrink the RAID volume first, be prepared to spend far to much time trying to fix things! Sometimes it's faster (if your volume isn't close to full) to resize2fs down to a very small number, shring the raid volume to a very small number, let all the synchronizing happen, and then grow the raid volume with "--grow --size=max" and then resize your filesystem up to the new size of the RAID volume.
There are worse things to spend part of a saturday afternoon on, but I'd rather be outside!
My Dive Into Rails Development
Submitted by ckdake on Mon, 2010-08-16 21:59Being new to working in a software development environment, last week was a bit of a crash course in getting together all the pieces. Here are some notes, mostly as a future reference for me but hopefully useful to someone else out there getting started writing software. Everyone seems to use Mac OS X these days so this is a bit skewed towards Macs, but most things apply anywhere.
TextMate
Rails developers love them some TextMate. It's a clean text editor for Mac OS X that mostly stays out of the way and has helpful bundles that do things like auto-completion. If you want to use it for things like git commit messages, add these to your ~/.profile:
export EDITOR="mate -w" export VISUAL="mate -w"
I'm a Vim person myself, but I've been using TextMate at work to get a feel for the workflow that most people seem to use.
Pig
I played with Pig a bit at SugarCRM, but never on a Mac. It's one of the best ways to ramp up number crunching and file processing. If you want to use it on a Mac, you'll need to set your JAVA_HOME environment variable. I did this by adding the following to my ~/.profile:
export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Home"
Homebrew
It used to be Fink, then DarwinPorts, then MacPorts. Now it seems "Homebrew" is the popular way to get some things up and running that aren't that fun to get up and running on Macs. Example:
brew install mysql
There you go! MySQL is now installed and running. Watch the end of the output for instructions on starting it on boot using launchd, etc. Rails apps tend to use sqlite for development but sometimes you might want to use MySQL.
Ruby Gems
All the functionality you need has probably been written by someone else and stuck in a Ruby Gem. For example:
sudo gem install clarity
Will install a fancy web interface for looking at log files in real time. To prevent all the documentation from being installed on your machine, set up a ~/.gemrc file with the following contents:
gem: --no-ri --no-rdoc
Some people like putting this in /etc/gemrc, but it's always better to put user specific things in your home directory!
It is recommended that you always install gems, including rubygems itself, from source and not from your operating systems package manager. Additionally, once installed everything will probably need to be updated with something like "gem update --system".
RVM
RMV is the Ruby Version Manager. This lets you have multiple versions of Ruby installed side by side, as well as create sets of Gems that can be used in different applications. For example, once set up you can put a ".rvmrc" file anywhere in your directory hierarcy, with contents like "rvm 1.8.7@mygemset". Whenever you cd into this directory, rvm will set up your envrionment to use Ruby 1.8.7 and the "mygemset" gemset, and can update your prompt to remind you what is active. Getting this bit to work requires adding a few things to your ~/.profile:
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" export PS1="\$(~/.rvm/bin/rvm-prompt) $PS1"
GIT and Github
Git is the source code management tool of choice for a lot of people. It's fully distributed which enables some interesting workflows, and the Github website has done an awesome job of gluing together lots of pieces that enable effortless collaboration. The Git Reference covers (very well!) most of what you need to know about Git, but here are a few particularily useful things I've found.
Need to delete a remote branch? Delete the branch locally, then push 'nothing' to the remote branch by using the push command and leaving the front of the colon blank. Example:
git push origin :deletedbranch
Need to undo something? 'git revert' doesn't do what you think. Re-checkout the file(s) instead.
To get color in some of the command output, add the following to your ~/.gitconfig:
[color]
diff = yes
status = yes
branch = yes
For auto completion and a more helpful prompt and tab completion of branch names and more, clone a copy of .git_completion.sh into your home folder, and put the following in your ~/.profile:
source ~/.git_completion.sh
function parse_git_dirty {
[[ $(git status 2> /dev/null | tail -n1) != "nothing to commit (working directory clean)" ]] && echo "*"
}
function parse_git_branch {
git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e "s/* \(.*\)/[\1$(parse_git_dirty)]/"
}
export PS1='\h:\W$(__git_ps1 "[\[\e[0;32m\]%s\[\e[0m\]\[\e[0;33m\]$(parse_git_dirty)\[\e[0m\]]")$ '
And read about using "git stash". It is your friend, especially when you forget to create branches to work on different features and ideas, which you _should_ be doing!
Rails
Ah Rails, I wouldn't be here without you. There is plenty of documentation out there on Rails, but a few points of interest:
- Models are named like "Thing" but in the database the table will be called "things". I'm used to naming my tables "thing" and both case and plurality matter here so thats taken some habit adjusting
- script/server starts up your rails app in webrick, while script/console sets up your app in an interactive ruby shell that you can use to poke around.
- If you need to debug something, add breakpoints to your code with "debugger" function calls, start script/server with "-u", and when your app hits the debugger call the script/server will drop you into a debugging session. It works a lot like GDB and is pretty handy for tracking things down.
- Making a new {anything}? There is probably a generator for that. Running "script/generate" will tell you about all your options
- "Rake" is "Make" for rails. "rake -T" will list all the available tasks. Looking at how requests are routed? Check out the output of "rake routes".
Rails does a lot of Magic and it can be a bit overwhelming, but overall it's been pretty friendly to me so far. Still need to know more? Watch all of the Railscasts and search the documentation at: http://api.rubyonrails.org/.
Capistrano
Capistrano is the most popular tool at the moment for deploying rails apps. A deploy to a remote server is as simple as "cap deoply" from your development machine, and if you're doing anything more involved than that for new software releases, you're doing it wrong. "cap -T" shows available capistrano tasks. That said, if you're doing more than just deploying a new version of an app...
Chef
Chef is the new Puppet. I used Puppet a lot at SugarCRM, and figured that I need to learn more about Chef so that I can pick the best tool to use, so I've spent some time with Chef over the last week. The "Opscode Platform" is a hosted Chef server that gives you a few things for free without having to set up the whole stack, and I recommend using that for at least your first round of testing. Also worth grabbing into your setup is a copy of Opscode's Chef Cookbooks that are very helpful for setting up pretty common things. Some pages worth reading:
- Chef Quick Start - start here
- Launch Cloud Instances with Knife - Use "Knife" which comes with Chef for firing up Amazon EC2 instances
- Deploy Resource - Capistrano, meet your New Fancyness
The end result of this tinkering, Add your Amazon EC2 ssh key to your ssh-agent by putting this in your ~/.profile:
ssh-add ~/.ssh/id_knife-test 2> /dev/null
Run a command like:
knife ec2 server create "role[${MYAPP}]" -i ${MYAMI} -G default -x ubuntu -S knife-testAnd in a few seconds, MYAPP will be up and running on a new Amazon EC2 instance. Pretty sweet! Assuming of course that you got Chef working and set up a Recipe for your app similar to this one I wrote for deploying ithought.org, recoded as a Rails app, inside of Apache and mod_passenger: https://gist.github.com/aef17730ef8514799dc1.
Thats it for week one!
I've left out a lot, and I have a long way to go, but hopefully some of those snippets will be useful to someone. A few of them took me a little while to figure out and searching the internet for things about "knife" and "make those things stop doing that thing" can be tricky!
Debugging a (particular) failing boot service on Linux
Submitted by ckdake on Thu, 2010-01-21 19:51At work I recently rolled out a newer version of the Dell OpenManage tools which included for the first time a build of Openwsman. We didn't specifically need this functionality, but it's good to stay current with the OpenManage tools. To load in the (unrelated) new kernel on a test machine, I rebooted the machine using Cobbler's power management functionality on our administrative system, but after 5 minutes the machine was still not responding to pings so something was broken. I used remote desktop to hop on our one Windows server in the datacenter which we use to get at the interactive consoles of our servers (Thankfully the new DRAC6 card's have console applets that work on Macs!), and pulled up the console for this machine.
The boot process was hung on "Starting openwsman" and didn't seem to be doing anything. Doh!
I restarted the machine again, and at the grub boot menu added a "S" to the boot string to start up the system in single user mode, and booted things up. "chkconfig openwsman off" to disable the service, and another reboot to get the machine back up and running to let me troubleshoot a little better. I took a look in /etc/init.d/openwsman to see what might be hanging, and nothing immediately looked suspicious. It was a pretty standard init script, with the extra feature of generating OpenSSL certificates if they didn't exist already:
if [ ! -f "/etc/openwsman/serverkey.pem" ]; then
if [ -f "/etc/ssl/servercerts/servercert.pem" \
-a -f "/etc/ssl/servercerts/serverkey.pem" ]; then
echo "Using common server certificate /etc/ssl/servercerts/servercert.pem"
ln -s /etc/ssl/servercerts/server{cert,key}.pem /etc/openwsman/
else
echo "Generating Openwsman server public certificate and private key"
FQDN=`hostname --fqdn`
if [ "x${FQDN}" = "x" ]; then
FQDN=localhost.localdomain
fi
cat << EOF | sh /etc/openwsman/owsmangencert.sh > /dev/null 2>&1
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
${FQDN}
root@${FQDN}
EOF
fi
fi
It's a little strange, but not unheard of practice to do this, and shouldn't cause any problems. (Puppet and Func, two other systems tools we use, generate their certs in the application which is a lot more common.)
I extracted the only possible culprit from the owsmangencert.sh script and tried running the openssl command manually:
openssl req -days 365 $@ -config /etc/openwsman/ssleay.cnf \ -new -x509 -nodes -out cert.out \ -keyout key.out
and it seemed that this was indeed the problem. It just sat there and didn't complete with the speediness I expect from OpenSSL. Time for strace!
cat << EOF | strace openssl req -days 365 -config ./ssleay.cnf.2 -new -x509 -nodes -out cert.out -keyout key.out > -- > SomeState > SomeCity > SomeOrganization > SomeOrganizationalUnit > test > root@test > EOF
This ended up doing a long read with output like:
open("/dev/random", O_RDONLY) = 3
read(3, "\323K\372u_ya'\27\266\320\25\22\373\240\330~'\224\310\243\356\225\350.\245\362\3058\230Zb"..., 1024) = 128
read(3, "K\7:\273Zdr\274\25\227\263\366\260U\337Owp\6y\2333c\361\322\334\217\370.k\375]"..., 896) = 128
read(3, "dH\375V\327\230Bi\221\342\326\26R\301v^Qv5f\347\303g7\2747\345\360\207A!\227"..., 768) = 128
read(3, "X&\254r\331\353<:\36!\333\340\353", 640) = 13
read(3, "\357F\27\347\372atf", 627) = 8
read(3, "\231\347\232\362\345\215n\227", 619) = 8
read(3, "\324\304\323\30\325\10G\332", 611) = 8
Looks like /dev/random wasn't returning random data nearly fast enough, which makes a whole lot of sense! /dev/random is "good" random data because it is based on environmental entropy and the entropy data is only used once, but on a modern multi-core systems doing lots of things, there usually isn't much entropy available. That means that while this command would eventually finish, it could take a very long time.
The fix: using /dev/urandom instead. It is "not quite as good" random data because the output may have less entropy than /dev/random, and it uses internal entropy bits multiple times to generate it's output, but it's "good enough" for generating cryptographic keys. And, it is non blocking which means that a caller will never have to wait inane amounts of time for enough "random" data. (See http://en.wikipedia.org/wiki//dev/random for a longer explanation.
I replaced the two occurrences of /dev/random, one in /etc/openwsman/ssleay.cnf and one in /etc/openwsman/owsmangencert.sh, and initial startup of openwsman (including key generation) became pretty instantaneous. "chkconfig --levels 2345 openwsmand on" to turn it back on, and a reboot (after removing the generated keys and certs) to confirm, and the machine booted up as expected. To make this work everywhere, I customized those two config files and added them to our Puppet system so that all Dell servers would get Openwsman set up properly when the update is run globally:
file {
"ssleay.cnf":
path => "/etc/openwsman/ssleay.cnf",
source => "puppet://$server/dell/ssleay.cnf",
}
file {
"owsmangencert.sh":
path => "/etc/openwsman/owsmangencert.sh",
source => "puppet://$server/dell/owsmangencert.sh",
}
Problem solved and all machines will automatically get the correct fix, so the next time a machine won't finish starting up, it will be a new and different problem to debug.
Monitoring Nginx with ZenOSS
Submitted by ckdake on Tue, 2009-11-24 10:41If you're using ZenOSS for network monitoring, and you have a few loadbalancers (or servers of some sort) running Nginx, chances are pretty good that you want to see what your load balancers are up to inside of ZenOSS. It requires gluing some things together, but once you know what kind of glue to use, it's a pretty straightforward process. First, you need to add status pages for Nginx. This means that it must be compiled with "--with-http_stub_status_module", but if you're using Nginx from a package provider like EPEL, it already has this included. Add nginx_status to your configuration by adding something like:
location /nginx_status {
stub_status on;
access_log off;
allow IP.OF.MONITORING.SERVER;
deny all;
}
to your first server {} block. Reload nginx and visit http://IP.OF.NGINX.SERVER/nginx_status and you should get some stats like:
Active connections: 8 server accepts handled requests 455010 455010 781977 Reading: 0 Writing: 2 Waiting: 6
Up next is getting this information into ZenOSS. We'll do this using a nagios plugin grabbed from here: check_nginx. This plugin does most of the work, but it's not quite enough to use here because it bases the numbers it provides on the difference in the accepts/connects over 2 runs 1 second apart instead of real numbers, and the format doesn't work with ZenOSS's input parser, so you'll need to apply this patch:
--- check_nginx.sh 2009-11-24 07:31:35.000000000 -0800
+++ check_nginx.sh.1 2009-11-24 06:45:56.000000000 -0800
@@ -181,17 +181,13 @@
if [ "$secure" = 1 ]
then
wget_opts="-O- -q -t 3 -T 3 --no-check-certificate"
- out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
- sleep 1
- out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
+ out=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
else
wget_opts="-O- -q -t 3 -T 3"
- out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
- sleep 1
- out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
+ out=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
fi
- if [ -z "$out1" -o -z "$out2" ]
+ if [ -z "$out" ]
then
echo "UNKNOWN - Local copy/copies of $status_page is empty."
exit $ST_UK
@@ -199,13 +195,9 @@
}
get_vals() {
- tmp1_reqpsec=`echo ${out1}|awk '{print $10}'`
- tmp2_reqpsec=`echo ${out2}|awk '{print $10}'`
- reqpsec=`expr $tmp2_reqpsec - $tmp1_reqpsec`
-
- tmp1_conpsec=`echo ${out1}|awk '{print $9}'`
- tmp2_conpsec=`echo ${out2}|awk '{print $9}'`
- conpsec=`expr $tmp2_conpsec - $tmp1_conpsec`
+ reqpsec=`echo ${out}|awk '{print $10}'`
+
+ conpsec=`echo ${out}|awk '{print $9}'`
reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
if [ "$reqpcon" = ".99" ]
@@ -220,7 +212,7 @@
}
do_perfdata() {
- perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
+ perfdata="reqpsec=$reqpsec conpsec=$conpsec conpreq=$reqpcon"
}
# Here we go!
@@ -247,17 +239,17 @@
then
if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
then
- echo "WARNING - ${output} | ${perfdata}"
+ echo "WARNING - ${output} | ${perfdata};"
exit $ST_WR
elif [ "$reqpsec" -ge "$critical" ]
then
- echo "CRITICAL - ${output} | ${perfdata}"
+ echo "CRITICAL - ${output} | ${perfdata};"
exit $ST_CR
else
- echo "OK - ${output} | ${perfdata} ]"
+ echo "OK - ${output} | ${perfdata}; ]"
exit $ST_OK
fi
else
- echo "OK - ${output} | ${perfdata}"
+ echo "OK - ${output} | ${perfdata};"
exit $ST_OK
fi
by saving that to "check_nginx.patch" and running "patch -p0 < check_nginx.path" from the folder where you have the check_nginx script saved. Make sure that the ZenOSS user can run the script, and make sure it works. The output should look like:
OK - nginx is running. 782995 requests per second, 455656 connections per second (1.71 requests per connection) | reqpsec=782995 conpsec=455656 conpreq=1.71;
With all this working, now you'll just need to create a new Template in ZenOSS, add a COMMAND data source to it with a command like:
/home/zenoss/scripts/check_nginx_ng.sh -N -H ${dev/manageIp}
Then add two datapoints, reqpsec and conpsec. Make sure to set both to the "COUNTER" type because the patched check_nginx script reports constantly increasing numbers instead of the GAUGE that it was before! Bind this new template to any devices or device classes where servers are running Nginx, and create any graphs you like. I have a graph on each device that shows reqpsec and conpsec, as well as a report that shows the aggregate reqpsec and conpsec for all of the loadbalancers. If your command is named "check_nginx" in the template, you can use the variables in any report by adding "checK_nginx_reqpsec" as a data point.
Without too much trouble, you now have fancy graphs in ZenOSS for your Nginx statistics, and you can set thresholds for these if you have conditions that should send out alarms.
Hairpinning with a Cisco ASA
Submitted by ckdake on Fri, 2009-11-13 12:01What a long battle with Cisco IOS this has been, but after quite a bit of tinkering I've gotten things working the way that I would like. Here's a technical description of the details in hope that this helps someone else.
The Setup
- Load balancers with private IP address like 172.16.0.10 on a /24, running example.com
- Cisco ASA Firewalls running 7.2(1) or newer, that map public IP addresses (I'll use 192.168.0.193 on a /24 here instead of a real public IP)
- Internal DNS servers that map loadbalancer.private to 172.16.0.10
- External DNS servers that map example.com to 192.168.0.193
- Random application server behind the firewall with no public IP address and a private IP of 172.16.0.20
The Problem
Applications behind the firewall need to access other applications behind the firewall using the public DNS name (example.com) instead of the private one (loadbalancer.private).
Some possible solutions
As an easy-to-set-up solution, we currently have the internal dns servers set up to map example.com to 172.16.0.10 which works fine, except it requires updating DNS records in multiple places. Our naming scheme slowly got a bit more complex, and I've had to add explicit relay rules to our DNS server configuration files to relay certain lookups from the internal DNS servers to the external DNS server's internal IP address. Sending it to the DNS server's external IP address doesn't work because the Cisco ASA will not send traffic back out on the same interface that it came in on, even after network translations have been done. (For a different portion of our external IP space, I added some static routes to the core router but when we move those IPs behind this firewall, this ASA feature will break those routes as well)
The current mapping of public IPs to private IPs looks like:
static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255
One feature that Cisco suggests to solve our problem is using "DNS Doctoring" which is just simply adding the 'dns' keyword to the end of the mapping like:
static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 dns
which modifies DNS queries going through the firewall from the inside interface to change the IP from 192.168.0.193 to 172.16.0.10. This would great, if your DNS server is outside of the firewall, which ours is not. Our internal DNS queries never travel through the ASA so this didn't do anything for us.
Up next was trying out
same-security-traffic permit intra-interface
which "permits communication in and out of the same interface" which sounds like it's the exact right solution for the problem because that was the limitation that broke things. However, adding this in didn't seem to change anything and traffic still was not permitted in and out the same interface.
The Solution
After a lot of troubleshooting, which involves an ASA 5510 and a 3524-XL on the floor under my desk, downloading and installing new versions of IOS, a lot of Googling, a lot of cursing, and a lot of sketching possible things out on paper, I finally figured out the missing piece: Hairpinning which is "the process by which traffic is sent back out the same interface on which it arrived." Here is the configuration that finally got traffic flowing from 172.16.0.10 to 192.168.0.193 on the ASA back out to 172.16.0.10 on the same interface it started on:
!--- Output suppressed. ! interface Ethernet0/0 nameif outside security-level 0 ip address 192.168.0.192 255.255.255.0 ! interface Ethernet0/1 nameif inside security-level 100 ip address 172.16.0.1 255.255.0.0 ! !--- Output suppressed. ! same-security-traffic permit intra-interface access-list outside_in extended permit icmp any any access-list outside_in extended permit tcp any any ! !--- Output suppressed. ! global (outside) 1 interface nat (inside) 1 172.16.0.0 255.255.0.0 alias (inside) 192.168.0.193 172.16.0.10 255.255.255.255 alias (inside) 10.0.0.20 172.16.0.20 255.255.255.255 static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 access-group outside_in in interface outside ! !--- Output suppressed.
The trick here was, combined with "same-security-traffic permit intra-interface" to add the alias lines, the first one:
alias (VLAN100) 192.168.0.193 172.16.0.10 255.255.255.255
does something sensible and aliases 192.168.0.193 to 172.16.0.10 on the inside interface so any time traffic comes in here matching that IP, it gets rewritten. The second line is also required but doesn't make as much sense:
alias (inside) 10.0.0.20 172.16.0.20 255.255.255.255
This line is telling the ASA to take any traffic coming in destined to 10.0.0.20 and map it to 172.16.0.20, however, we don't have any devices on 10.0.0.0/8 and there are no routes for this, so there will never be any traffic coming in to 10.0.0.20. That said, this line has to exist so that there is a mapping back to 172.16.0.20 in the alias table so that the ASA knows it's alright to send traffic to it. Using a "real" public IP here would both use up our public IPs and perhaps pose some security risk, so it's safer to use these non-public IPs and add a rule to prevent incoming traffic from the outside from reaching them. If the alias command would work for an IP range instead of one host, this would be pretty much perfect.
The result
Things finally work! Here is a trace of a ping from 172.16.0.20 to 192.168.0.193 (which works now!):
ICMP echo request from VLAN100:172.16.0.20 to VLAN100:192.168.0.193 ID=12034 seq=0 len=56 ICMP echo request translating VLAN100:172.16.0.20 to VLAN100:10.0.0.20 ICMP echo request untranslating VLAN100:192.168.0.193 to VLAN100:172.16.0.10
So the ASA is doing the translating the proper way and not doing anything with 10.0.0.20. This is good news because it means that our naming and routing architecture can be greatly simplififed:
- All relay rules for external facing domains that have previously required this "split-horizion" DNS can be removed, returning the DNS server configurations to a generic state
- All crazy static routes for external IP addresses can be removed from our core router
- All external facing domain zones can be removed from the internal DNS servers, and updates when things are moved only have to be done in one place
The only penalty for this is adding in the alias lines to our ASA configuration for each existing static mapping that we have, as well as adding an alias line for each server that needs to communicate with the external IP addresses of things behind the same ASA which should be limited to the internal DNS servers and a few application servers.
References
- Configuring Interfaces for the Cisco ASA 5505 Adaptive Security Appliance - cisco.com
- same-security-traffic through show asdm sessions Commands - cisco.com
- PIX/ASA: Perform DNS Doctoring with the static Command and Two NAT Interfaces Configuration Example - cisco.com
EDIT: Another way to do this
After sharing this with some coworkers, it turns out that 'hairpinning' is definitely the key word and one of them stumbled across this article:
Setup U-Turn (Hairpinning) on Cisco ASA
It solves the same issue with a slightly more graceful solution because no alias entries are needed for non-public services, in fact, no aliases are needed at all. To have the exact same functionality as above, here is the working configuration for the problem above with this new methodology:
!--- Output suppressed. ! interface Ethernet0/0 nameif outside security-level 0 ip address 192.168.0.192 255.255.255.0 ! interface Ethernet0/1 nameif inside security-level 100 ip address 172.16.0.1 255.255.0.0 ! !--- Output suppressed. ! same-security-traffic permit intra-interface access-list outside_in extended permit icmp any any access-list outside_in extended permit tcp any any ! !--- Output suppressed. ! global (outside) 1 interface global (inside) 1 interface nat (inside) 1 172.16.0.0 255.255.0.0 static (inside,outside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 static (inside,inside) 192.168.0.193 172.16.0.10 netmask 255.255.255.255 access-group outside_in in interface outside ! !--- Output suppressed.
Windows 7 64-bit on a MacBook Pro
Submitted by ckdake on Mon, 2009-10-19 13:02I have a MacBook Pro for work, and have Windows on it in a separate partition for the occasional thing that requires it. I had to get it's logic board replaced and for various reasons it made more sense to reinstall OS X and Windows once everything was replaced and back to me. Mac OS X was easy, but Windows was a bit more of a pain because while I have several legal copies of XP, i've lost track of which license keys go with which ISOs and Microsoft wants to call me to activate things. Thankfully, this is a work computer and work as a Microsoft volume license plan so I grabbed a 64-bit Windows 7 Professional ISO and Key, and gave that a shot. First problem, the DVD wouldn't boot: it instead presented "Select CD-Rom Boot Type" and 2 options, and the keyboard was unresponsive.
After a bit of Googling around, the following is the way to get around this, assuming you have another Windows system or VM available:
- Grab a copy of oscdimg.exe (heres one) to somewhere like C:\
- Put the DVD you already burned into your Windows box
- Run: oscdimg -n -m -bd:\boot\etfsboot.com d:\ c:\win7×64.iso
- Burn this ISO to DVD
- If Needed, use Boot Camp Assistant in Mac OS X to partition your disk for Windows
- Boot up with this DVD (hold down C when booting, use the Boot Camp Assistant to start it up, etc) and Windows 7 will install!
Once it's up and running and you try and install the BootCamp drivers from a Snow Leopard DVD, you'll run into some more things that try and stop you. The fix for them is to manually run the 64-bit driver installer in Administrator mode. One way to do this:
- Open up an Admin mode console by right-clicking on "Command Prompt" in start -> all programs -> accessories, and clicking "Run as administrator"
- type in "cd d:/Drivers/Apple" to change to the Drivers/Apple folder on the DVD
- type in "msi.exe BootCamp64.msi" to launch the 64-bit installer, and it will install and do it's thing
After those few steps and not too much time, you'll be up and running in Windows 7 on your shiny Mac laptop. Some things I've noticed:
- It's faster feeling than Windows XP
- It's prettier than Windows XP
- It feels like it wakes and sleeps faster than OS X
- Steam and Team Fortress 2 work and seem faster than Windows XP
- The Internet works.
