ithought. now with a website!

I've finally gotten some things racked at a new datacenter and thought this might be a good time to launch the new www.ithought.org. Everything that used to be at /hosting here has moved there, and theres a lot more new stuff! If you have hosting with me and have questions, or want to refer anyone to me for hosting or consulting stuff, thats the place to send them! If you have hosting with me and want to link to this site instead of ckdake.com, there are some images at the bottom of the new support page.

More updates to come including a far superior status page.

Bacon Pie

Ever tried to figure out how you could use more bacon and more cheese and less everything else in a meal? Then bacon pies are for you! This recipe is guaranteed to fill your house, clothes, pets, and carpet with the flavorful aroma of bacon and cheese for weeks. The end result is surprisingly easy to eat and goes great with very very dry bread and high-gravity beer. When cooked properly, the cheese will be fluffy with a consistency similar to scrambled eggs. Read on for the recipe!

Disable Gateway Smart Packet Detection

Do you have Comcast business cable internet? Do you occasionally have crazy random problems connecting to remote machines/websites and/or do you notices an very unusual number of TCP retransmits when looking at packet traces that go through your Comcast provided SMC Networks cable modem? You are not alone! And there is one checkbox to fix all of this!

Head in to your modem admin page, go to Firewall Settings and check the box next to "Disable Gateway Smart Packet Detection" and all of your problems will be solved. (No guarantees, but seriously, it should work)

How did I find this out? I was getting a lot of things delivered via UPS this week and was unable to get to their website after a few successful attempts. Things still worked through a proxy server at work, so I thought it must be them blocking my IP address due to the number of requests I made (possibly violating some AUP with them) and tcp dump showed my packets going to their webserver (on Akamai) but nothing coming back after my SYN packets went out. After 15 minutes on the phone with Comcast support, they escalated me to second level support which meant a callback in 2 days. The second level guy confirmed my ISP and told me to change this setting because they'd had a lot of problems with Comcast customers and, working with Comcast, they came up with this solution.

Um ok. Whats going on here? The best part about this is that nobody really knows! Searching around the internet about "Gateway Smart Packet Detection" doesn't lead to any documentation or any "good" answers, just lots of people having problems and this checkbox fixing all of them. I've gathered that it is some kind of Anti-DOS feature for blocking multiple attempts at something, but chances are you are better off just turning it off. Hope this helps someone as the problems that checking this box have solved for me have been frustrating me for months!

Scaling with PowerDNS and EveryDNS

Ah DNS, the often overlooked aspect of running websites. Many people I've spoken to bought a domain from Network Solutions, then one from GoDaddy, and maybe one or two from their web-hosting provider. Settings are all over the place, and they use the tools provided by each registrar to manage the DNS for domains purchased there. While this certainly works, it can become quite a hassle to change things around especially if you want an overview of all of your domains or need to change the IP address of a server.

Several years ago, I found out about EveryDNS which is a great free DNS hosting service. They've been very solid and while they have been down a few times from DDOS attacks at 50Mbps+, they definitely can scale better than my little rack of servers. I donated some money to them and currently have about 60 domains with ~600 DNS entries total hosted with them. With EveryDNS, all of my DNS entries are in the same place and when someone purchases hosting from me, I have them set the authoritative nameservers for their domain to the EveryDNS nameservers. This means that I don't need access to their account information, but I can have quick and easy access to DNS entries if I need to move anything around.

I'm preparing to make some big changes to my servers and the hassle of the point-and-click interface becomes a bit to much. ~1800 clicks or so is a lot more complicated than it needs to be! Additionally, for almost everything else I do on the internet, I prefer to own the hardware and software that my information lives on. To address both of these, I installed PowerDNS with a MySQL backend on a server, and then set up DNS replication to EveryDNS (docs on this). PowerDNS with MySQL let's me change the IP address of a server with one SQL statement instead of lots of mouse clicking, regardless of how many domains I have. This setup also allows me to include DNS configuration as part of my web hosting provisioning scripts which greatly simplifies the process of adding a new website to one of my servers. My DNS server is not listed in the authoritative servers list for domains, so the only queries that it responds to are the AXFR queries from EveryDNS. The only negative of this is that EveryDNS only checks once an hour so I can't do any tinkering with short TTLs, but thats a price I'm willing to pay for now! Hopefully they will enable DNS Notify support in the future which would allow for instantaneous updates, and if my hosting operation gets big enough, I'll just roll my own live DNS servers.

Chandler and the Internets

I always have my eyes open for new and better ways to do things, get things done, and share information. I chronicled some of this over a year ago in Web 2.0 and More web apps, and things have changed a bit since then! I'm still using Delicious to manage my bookmarks and Amazon added the ability to add anything to your wishlist from other websites, but those items aren't piped out via their API so thats no fun for my wantlist (which I've been updating for Christmas!).

Plaxo never really became useful to me, nor did 30 Boxes or Basecamp. In the last year, I've been trying to build a more cohesive online presence that doesn't depend so much on Facebook. Enter FriendFeed! All my things from all over the internet pipe into my friendfeed which then gets piped into sites that consume this information like the "other sites" bar here on the left, and my Facebook wall thing. I also signed up for twitter finally and got that tied in to everything.

However, the biggest change in the last few months has been moving from Remember The Milk and Google Calendar to Chandler. Not only is Chandler amazing, it allows me to own my data by running Chandler Server on my own servers so I get simple backups, no unexpected downtime, and the peace of mind that I own the whole stack that contains my information.

Several years ago, I was looking for a good to-do list manager and found out about Chandler but it was in it's infancy and definitely not ready for prime time. I moved around between iCal, sticky notes, paper, text files, and a few other things but none really fit quite right. On August 8, Chandler 1.0 was announced and I decided to give it a shot. First I just used it for my personal tasks related to Gallery, which I stored in Chandler Hub, but in less than a week I realized how powerful of a tool Chandler really was. Next, setting up my own Chandler Server was a breeze. In perhaps an hour I got Chandler Server up and running listening on a local port, set up MySQL as the backend so my existing backup scripts would automatically take care of it, and set up my webserver to do the logging and SSL magic. I stopped using my paid Remember The Milk account and Google Calendar, and over the next few days moved everything into Chandler.

Now, every day after my morning bike ride, I filter through my email (I use the Inbox:0 strategy) either deleting things, responding to quick things, or putting actionable things that take some time into Chandler's desktop client. Once that's empty, I head into Chandler and start down the task list for the day. I have a separate collection for each of the kinds of projects I'm working on (Hosting, House, etc) but usually I hang out in the "Dashboard" view which combines most of my collections. Collections like "Recipies" and ones that don't have actionable items are hidden from my dashboard. This has significantly cut down the amount of time I spend managing myself, and once a week I do a "weekly review" to scan through items marked as "Later" and see if any of them need to be changed. (Right now I have 12 "Now" items, 109 "later" items, and 93 "done" items in my personal collections.) Also, all of my collections synchronize between my desktop and laptop, feed into iCal which syncs to my iPhone, etc, so my information is where I need it when I need it, and I can use Chandler's web interface to get at things if I'm on someone else's computer.

Just using this for my personal things was nice, but my two biggest projects are working on the operations team at SugarCRM and acting as a project manager for Gallery. I set up a Chandler Server at work and we have 5 people sharing a collection called "Sugar Operations" which includes:

  • Tasks for people which get assigned by adding something like @chris in the title
  • Tnternal HOWTOs (like our machine bootstrapping process) that are still in flux. These get moved to a wiki once they stabilize
  • Changing system configuration information
  • Meetings and company events that multiple people are attending

It takes a little while to get new people using the shared collection, but once they're in it's an amazingly effective way to work together as a team. We no longer need weekly "On Tap" emails to figure out what is going on because it's all in Chandler, and we can make changes to information without having to log into a wiki each time or email changes around.

Several weeks ago at the Gallery Sprint, everyone else told me to do what I thought was best for managing project overhead so we're now using a shared collection hosted on Chandler Hub to manage the project and the development of Gallery 3. You can see that collection in your web browser here: chandler.galleryproject.org. Again, it's taken a little bit of effort to get everyone using it the same way but the results have been an even bigger deal here. Developers working on open source projects don't really like doing things other than making decisions and developing, and Chandler is a very low overhead way for developers to keep track of what they are doing in such a way that anyone can easily get the big picture in one place. Chandler has gotten rid of the need for weekly status reports to our mailing list, going through lists of action items in weekly meetings, and figuring out who committed to doing what. Everyone knows who is responsible for things and I can finally spend a lot less time harassing people to do things that they committed to doing already. This frees up a lot of time so that developers can get back to developing and our meetings are back to lively discussion and decision making instead of boring project overhead.

Gallery is still using SourceForge to manage feature requests, bugs, and our code repository, as these aren't really things that Chandler is up to at this point. We wish there was a way to report on things in Chandler and associate them with releases, but it would be a lot of work to make this happen. Trac does a great job of this so theres a possibility of switching to Trac in the future which could replace both Chandler and the SourceForge, but I get the feeling that the developer overhead is a lot higher which might cut down on how useful it is.

I can't express how great it is to have everything that I need in one application on my desktop, and I'm looking forward to the day that I can ditch my e-mail client as well!

VLANs in OpenVZ

OpenVZ seems to be the hot open source container based virtualization tool these days. Instead of tools like VMWare and Xen which virtualize the hardware and allow each guest operating system to run their own kernel, OpenVZ uses operating system level virtualization. While less flexible and less "secure" in some instances, this allows for better performance of the guests due to lower overhead.

I've been tinkering with using OpenVZ for a project to provide rapidly deployable emergency copies of infrastructure for situations where the primary and secondary hardware go down (DNS servers, LDAP servers, etc). OpenVZ meets the need here because it has command line management tools, is low overhead, and these kinds of services don't depend on a specific kernel or hardware stack as much as some others might. The tricky part for me is that some of these services live on separate VLANs.

In this setup, each machine (including the OpenVZ host) has two Gigabit Ethernet interfaces bonded together to two ports on separate switches that are stacked together. This provides higher throughput and prevents interruption of service if a switch, cable, or interface fails. The hosts typically don't know about VLANs and the interfaces on the switches are in access mode which automatically tags all traffic to the proper vlan. However, the OpenVZ host will need access to multiple VLANs so that it's guest machines can get to the right places on the network, so some things need to change. It will need it's own VLAN as well as the VLAN for each guest machine.

Firstly, the switchports are configured to trunk the right VLANs to both of the ports that the OpenVZ host is plugged into. Note that if you do this, you'll loose access to the machine so make sure you're connected out-of-band to the console! On the switch in a config shell (Cisco IOS example):

# interface Gi1/0/1
# switchport trunk encapsulation dot1q
# switchport trunk allowed vlan 10,20-30
# switchport mode trunk
# interface Gi2/0/1
# switchport trunk encapsulation dot1q
# switchport trunk allowed vlan 10,20-30
# switchport mode trunk

Then the OpenVZ machine is configured to support VLANs by adding kernel modules and creating a new interface. Note that these instructions are for RHEL5/CentOS5:

  1. Add "modprobe 8021q" and "modprobe vzethdev" to /etc/rc.modules
  2. chmod +x /etc/rc.modules
  3. Manually run /etc/rc.modules. It will be automatically run when the system boots
  4. reconfigure the /etc/sysconfig/network-scripts/ifcfg-bond0 to have no IP or BOOTPROTO information, "ONBOOT=yes" and "MODE=trunk"
  5. create /etc/sysconfig/network-scripts/ifcfg-bond0.10 like the following:
    DEVICE=bond0.10
      IPADDR=10.0.10.2
      NETMASK=255.255.255.0
      GATEWAY=10.0.10.1
      NETWORK=10.0.10.0
      BROADCAST=10.0.10.255
      ONBOOT=yes
      BOOTPROTO=none
      USERCTL=no
      VLAN=yes
      PHYSDEV=bond0
    
  6. and load the interface with "ifcfg bond0.10 && ifup bond0.10"
  7. make sure that proxy_arp and forwarding are enabled for bond0.10 in /proc/sys/net/ipv4/conf/bond0.10/. If not, you should reconfigure your system to set these by default. Consult your operating system documentation for instructions on this.

Once this is done, you should be able to use this host on the network (on VLAN 10) like nothing changed! If not, make sure routes are set up right, ifconfig looks right, etc. Assuming it works, you're halfway there! Up next is creating an interface for each vlan you want mapped. Here's an example for /etc/sysconfig/network-scripts/ifcfg-bond0.20 on VLAN 20:

DEVICE=bond0.20
  ONBOOT=yes
  BOOTPROTO=none
  USERCTL=no
  VLAN=yes
  PHYSDEV=bond0

Note that it doesn't have any IP information. We'll specify this inside of the OpenVZ instance. Next, we actually create a blank OpenVZ instance (you can use an existing one, but this is provided for completeness sake) and give it an eth0 interface. I'm using 20 as the ID here because this instance will be on VLAN 20, but this is not a requirement.

vzctl create 20 --ostemplate centos-5-x86_64-default-5.2-20081013 --config vps.basic
vzctl set 20 --onboot no --save
vzctl set 20 --hostname vlan20host.local --save
vzctl set 20 --numothersock 120 --save
vzctl set 20 --nameserver 10.0.10.1 --save
vzctl start 20
vzctl set 20 --netif_add eth0 --save

On each host, use "eth0" as the name of the interface. OpenVZ will automatically create the eth0 interface in the guest and an interface like "veth20.0" on the host where "20" in the name represents the guest ID and the .0 indicates that this is the default interface for the guest. You could add an eth0.21 interface to the guest with vzctl if you wanted VLAN 21 also piped into the guest, which would create a eth0.21 on the guest and veth20.21 on the host.

Now that it has the interface, enter the instance with "vzctl enter 20" and set up it's networking by creating /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
  IPADDR=10.0.20.1
  NETMASK=255.255.255.0
  GATEWAY=10.0.20.1
  NETWORK=10.0.20.0
  BROADCAST=10.0.20.255
  ONBOOT=yes
  BOOTPROTO=none
  USERCTL=no

Then "ifcfg eth0 && ifup eth0". You won't be able to send traffic yet, but configuration inside of the guest here is done. Head back to the OpenVZ host and set up a bridge to connect things. Here, we name the bridge so that it's recognizable as this VLAN and add the VLAN interface and OpenVZ host interface to it. Making one bridge per VLAN is probably the right thing to do, and if multiple guests are on the same VLAN, just add their host interfaces to the same bridge.

brctl addbr vzbr20
brctl addif vzbr20 bond0.20
brctl addif vzbr20 veth20.0
ifup vzbr20 0

Again, make sure that forwarding and arp_proxy are enabled for the bridge and the veth20.0 interface (created by adding eth0 to the guest). And that's it! You should be able to ping the guest's gateway from the guest. If you can't, run a ping from the guest and run tcpdump from the host to see where packets stop, looking at interfaces in this order:

veth20.0
vzbr20
bond0.20
bond0

Whichever one it stops on, make sure you have forwarding and proxy_arp enabled, and make sure that the bridge has the two things it should as members with "brctl show"

Each time the vm is rebooted, you'll need to add it's host interface to the bridge again and may need to re-enable forwarding and proxy_arp depending on how your OS is configured. If you are using a newer version of OpenVZ (>3.0.22) there is a easy workaround for this on the Veth wiki page. There is a more complex workaround for older OpenVZ versions there as well, but I should be using the newest version of OpenVZ when it's time to deploy this thing so I'm waiting!

Hopefully you find this useful, please let me know with a comment on the original article at ckdake.com or via an email if you have any suggestions, comments, or corrections!

Helpful References:
http://wiki.openvz.org/Veth
http://wiki.openvz.org/Venet
http://www.howtoforge.com/installing-and-using-openvz-on-centos5.2-p2