I just realized that my yahoo profile is now over 10 years old! I apparently created it on February 12, 1998, and while I know that I had AOL at home before that for perhaps around a year, I can't find any indication that they provide account creation dates anywhere in their system. (And back before AIM was properly integrated, I had to switch from ckdake to theckdake when we canceled AOL and didn't get to switch back to ckdake until perhaps college?)

Before AOL, I got online once or twice at a friends house, but I know my first experience online was at the 99X booth at some olympic experience thing at the 1996 Atlanta Olympics. They had a web browser, I typed in "games" in the address bar and alas, couldn't get to any games. Needless to say I didn't realize that the internet was good to have until later.

In 1999, I purchased my first domain name: ithought.org (for $70 a year or something stupid expensive from Network Solutions) and it's still the one I use for all my servers. ckdake.com finally showed up in 2004.

Things sure have come a long way in ~12 years!

It's been a little over a month since I broke my collarbone, things are slowly getting back to normal. I still haven't ridden a bike outside yet, but I've been on the trainer almost every day and when it's all said and done will have put in ~1000 miles on it. The first 30 days I been worked on my cadence and can do ~125RPMs for as long as I feel like it (on the lowest resistance setting on the trainer) and for the last few days (and next 20 or so) I'm cranking up the resistance and seeing how big I can make the sweat puddle. I'm thinking I should get a heart rate monitor to figure out how much work I'm actually doing. I'm back to using two hands to type pretty well but I'm still mousing with my left hand which is less weird than a month ago but still kinda weird, and it's still not a good idea for me to do things like open bags of chips. More X-Rays in a few weeks and we'll see how it's doing!

I also finally got around to taking some pictures of my house. I bought it in December but have been waiting for things to get a bit more moved in. Theres still no dining room table or bed, but I'm not going to be doing any heavy lifting soon so figure it was time to share some pictures of it as is. Nothing has broken yet, Greg has been awesome by coming over to mow since I'm busted, and thanks to my parents I have some trees and bushes planted in the back yard. If you're in the neighborhood and need to watch a HD movie or play some super smash brothers on a big TV, or have a beer on the back porch, just let me know!

In other news:

• My brother got married and is off to New Zealand for his honeymoon!
• San and I have our plane tickets and hotel reservations for this years Gallery Developer Conference, Amsterdam and Paris in July!
• They finally put no parking signs on my street on one side. Now it will be possible for things like the Garbage and UPS trucks to make it down the street since cars will only be on one side. Getting out of the driveway will be a lot easier too!

Yeah, I broke my collarbone this weekend. I'm going to the effort of typing this up here, so that when people want the whole story i don't have to laboriously type a shorter version on AIM or something.

Saturday morning I got up around 6:30 to head up to North Georgia with Christopher, Kurt, and a pack of people I didn't know. It was pretty overcast with a chance of rain, but we felt chance was in our favor. We met up in a Waffle House parking lot, and the caravan of ~6 cars full of bikes and bikers headed up to Ellijay, GA. As we got closer it started raining, and by the time we got to the trailhead, it was absolutely pouring. Usually, we don't ride in wet conditions because it damages the trails, but the day's ride consisted of gravel fire roads and rock trails with stream crossings, so we wouldn't really be doing any damage.

It took a while for everyone to get suited up, but we finally got going: fire roads and trails uphill for miles. Several of us, me included, thought the pace of the leaders was a bit fast for what was going to be a 4 hour ride, but we all stayed fairly close and I don't think anyone was going slower than they wanted too (and a few people had a race to compete in the next day). It was very wet and while we were completely soaked before we started, somehow we got even wetter. This was my first mountain ride in the rain, and while wet, it was actually pretty nice. On a sunny day, I'm usually pretty hesitant to go through mud or make a stream crossing, but once I'm gross, it's a lot easier to do those things without thinking about it.

Halfway up the elevation gain, there was a great view of the low clouds in the mountains, but unfortunately I left my cameraphone in the car due to the weather (and my weatherproof GPS does't have a camera). The 3 of us in the back told everyone else to go ahead, and we started up the second half of the climb about 5 minutes after them. And what a climb it was! See the profile below:

While going uphill was very hard, the downhill was actually worse. It was still raining, and that combined with the mud my front tire was throwing up made it very hard to see. The 3 of us were flying down a gravel road when I noticed that I was catching up to them. This seemed like a good thing, but it turned out that I was going way too fast. I barely saw the outline of a sharp right turn in front of me and as I started to brake, I realized that there was no way I was going to be able to stop or make the turn, and was faced with an easy decision: drop the bike or launch myself off a cliff. While both tires were locked up and skidding, i kicked my back wheel to the left and dropped my bike on the right side. It stopped me pretty quickly and hurt extremely bad. While just glad to not be off the cliff, this still wasn't fun. I could feel the scuff marks on my back from the gravel, but after sitting still for a few seconds to catch my breath, I realized my collarbone was broken. I gave it a good whack to try and align it properly while the adrenaline was in full force and before the pain or reality really set in, and we then started to worry about me going into shock.

Doh! We were on the top of a mountain in the middle of the woods in the cold rain with no way to get out and no way to stay warm. Thankfully, a lost car pulled up a few minutes later. The guy I was with stashed my bike in the woods and the helpful family up there for some fishing helped me into their car. They were lost as well, but I pulled out my GPS and it gave us directions to the closest emergency room, the North Georgia Medical Center. Over an hour passed and we made it there. They dropped me off and went to fix a flat tire on their car and that's the last I saw of them. So there I was, alone in the ER, wearing spandex, soaking wet, and covered in dirt, without a wallet or phone. They took some x-rays, got my arm in a sling, and suggested I see an Orthopedic surgeon in the next few days, noting that it's a simple fracture that didn't separate. Not 20 minutes later, Kurt and Christopher picked me up at the ER after finishing their ride and we headed back to Atlanta.

After further inspection, the bite valve of my camelback is gone, and my helmet has a huge split in it, so it possibly saved my life. (Wear your helmets kids!) All in all, this is really annoying because I can't really ride for 2 months, it's hard to sleep, and really only being able to use my left hand is not fun, but it could have been a lot worse! While I missed out on the last 2/3 of the ride (which apparently is all awesome, technical, downhill with lots of stream crossings), I'm relatively well off with such a simple injury. Much worse could have happened if I hadn't been thinking, and of the very large number of my friends that have broken a bone biking, many of them needed surgery and metal plates or screws in their ankles, wrists, or collarbones. Health insurance is a good thing (16 pain pills were $3), and hopefully my bike will find it's way home! (I think the guy that hid it drove up the mountain to pick it up after he finished riding.)

So much thanks to everyone that helped, especially the many of you that I don't know. It's going to be quiet on here for a few weeks and I won't be e-mailing much, but I'll be back typing and riding as soon as possible!

As I've noticed from watching hits on my site here, many of you have read my page on PHP security using mod_fastcgi and suexec. The logic on that page still holds, but Gentoo decided to make the switch from mod_fastcgi to mod_fcgid and it broke all sorts of things for me. I got things scratched back together without any security on my old server, and with the installation of my new server a few weeks ago, I set things up more securely again. I still think this way is the way to go for a server where many of the virtual hosts will seldomly see traffic, but if you're running lots of high traffic sites and have a little bit of RAM overhead, you might want to check out this article on mpm-peruser.

For this setup, I decided to stick to some standards. This means no more changing the suxec directory, using /data/, or anything like that. Other than that, the key differences from last time:

• All configuration is now done with with a setup script instead of using a mysql database. There was not really any point for the host names to be in a database, and it makes setup/teardown scripts easier to write as just a bash script.
• Some hosts have PHP, some don't, so no point in setting up all the overhead if a host isn't going to use PHP.
• Most hosts won't have any interest in having their own logs. Statistics can be done using client side things such as Google Analytics, and Apache is happier writing all the logs to 1 place instead of hundreds. I also have split-logs running when logs are rotated, so logs can easily be gathered per-site as needed, just not real time by one of my hosting customers. I've never known of one of my customers using live access to their logs.
• php.ini files are now stored with the wrapper script in the site's cgi-bin directory and file system extended attributes are used to protect it. This means no separate home for php.ini files, and it's easier for users to see what their PHP confguration is.

The script isn't quite ready for sharing yet, but here's what you can do to get a setup like this:

1. on Gentoo, make sure your USE contains: suexec, apache2, cgi, fastcgi, session.
2. on Gentoo, "emerge apache php mod_fcgid". On other platforms, consult your docs (or just download mod_fcgid and use apxs to install it. it should be pretty seamless)
3. Set up your global configuration. On Gentoo, this is done for you, but make sure this gets loaded into your global apache configuration:
   

   LoadModule fcgid_module modules/mod_fcgid.so
   SocketPath /var/run/fcgidsock
   SharememPath /var/run/fcgid_shm
   
   <Location /fcgid>
   SetHandler fcgid-script
   Options ExecCGI
   allow from all
   </Location>
   

4. Add a user and group for your first virtual host, test.example.com. call em "example" if you like
5. Set up the directory tree for the virtual host:
   

   /var/www/test.example.com/
   /var/www/test.example.com/tmp
   /var/www/test.example.com/htdocs
   /var/www/test.example.com/htdocs/cgi-bin
   

6. Make some files:
   

   <!-- /var/www/test.example.com/test.html -->
   hello HTML world!
   

   <? 
   /* /var/www/test.example.com/test.php  */
   print("hello PHP world!");
   ?>
   

   #!/bin/sh
   # /var/www/test.example.com/htdocs/fcgi
   PHPRC=/var/www/test.example.com/htdocs/cgi-bin/
   export PHPRC
   PHP_FCGI_CHILDREN=2
   export PHP_FCGI_CHILDREN
   PHP_FCGI_MAX_REQUESTS=25000
   export PHP_FCGI_MAX_REQUESTS
   exec /usr/bin/php-cgi
   

7. Copy your php.ini to /var/www/test.example.com/htdocs/fcgi and edit it so that directories are right. All you'll likely need to change is upload.tmp_dir and session.save_path, but you may want to change others.
8. Set fcgi to be executable, and make sure permissions are set on it so that it is owned by your test user/group and other users can't mess with it. If things don't work later, this is a frequent culprit
9. Set the immutable bit on php.ini and fcgi (you'll need to be using extended file system attributes on your filesystem to do this, check your OS documentation for details) by running 'chattr +i /var/www/test.example.com/htdocs/*'. You'll need to undo this with chattr -i if you want to change these files in the future.
10. Set up this host's configuration:
    

    <VirtualHost *:80>
            DocumentRoot /var/www/test.example.com/htdocs/
            ServerName test.example.com
            SuexecUserGroup example example
            <Directory /var/www/test.example.com/htdocs/>
                    Options +SymLinksIfOwnerMatch
                    AllowOverride All
                    Order allow,deny
                    Allow from all
                    DirectoryIndex index.html index.php
                    AddType application/x-httpd-fastphp .php
                    Action application/x-httpd-fastphp /cgi-bin/fphp
            </Directory>
    
            <Directory /var/www/test.example.com/htdocs/cgi-bin/>
                    SetHandler fcgid-script
                    FCGIWrapper /var/www/test.example.com/htdocs/cgi-bin/fphp .php
                    Options +ExecCGI -Includes
                    allow from all
            </Directory>
    
    </VirtualHost>
    
    <VirtualHost *:80>
            ServerName aerospace.com
            Redirect Permanent / http://test.example.com/
    </VirtualHost>
    

11. Give apache a restart and that should be it!

Check out the processes running on your server, and after you hit test.php you should see a php-cgi process running as the example user. If you have problems, error_log and suexec_log in /var/log/apache2/ (or /var/log/httpd/) tend to tell you everything you need to know.

An oh yeah, want to use APC to speed up your PHP applications significantly under this setup? Just install APC, then add the configuration for it to the bottom of the php.ini for any hosts that you want to enable this on. Given that APC isn't 100% perfect and crashes sometimes, the beauty of the fcgid setup is that it will take out the php-cgi process and the fcgid manager will just start a new one like nothing happened.

Adobe Fast Web View is a very lightly documented but seemingly often used feature in Adobe Acrobat Reader. From the users point of view, it does what it says and makes pages of a PDF show up in their browser before the entire PDF is completly downloaded, but it's a bit more complicated from a server operators point of view. And, it is enabled by default when installing Adobe Acrobat Reader.

We recently moved sugarcrm.com and some other web properties from stand-alone web servers to a clustered solution involving NFS, load balancers, database replication, etc. It was a pretty complex migration and we're pretty sure that we're running a handful of applications on this cluster that nobody has every clustered before, so needless to say we ran into our share of gotchas. (Other than one web server that seems to be cursed..) One of them was very strange, and involved PDFs: Everything worked fine in all browsers on all platforms until Adobe Acrobat Reader entered the picture. Some number of PDFs would lock up the browser and never load, but only when the PDFs were served from the cluster through the load balancer. When served from one of the cluster web servers but not through the load balancer, everything would work perfectly! Also, with the Adobe plugin disabled, the PDFs would save perfectly and be viewable every time.

Using livehttpheaders it was apparently that 2 HTTP requests were being made so my guess was that the browser would do a GET for the PDF, but when the Adobe plugin took over, it was sending a new HTTP request (all about HTTP). This shouldn't be an issue, but things weren't working! I installed Wireshark on my Windows test installation and dug deeper. Immediately, I noticed that all the data packets in the response coming from the server were fragmented. This typically means that there is an MTU somewhere. However, with some Googling around for PDF files I noticed the same behavior on 50% of the sites I hit, and those PDFs were working fine in the Adobe plugin. Regardless, Igor and I set out to tinker with the MTUs on the web servers and load balancer. Changing the MTU from 1500 down to 1400 did change which PDFs would load in the plugin, but not all of them. Strange!

Again looking in the Wireshark traces, we saw what looked like a TCP reset loop (read all the details about TCP here). After the first part of data came through successfully, every packet from the server was a RST and the Adobe plugin just sat there waiting for data that was never going to arrive. We poked around the load balancer looking for anything that could cause this but no luck. Googling around for this PDF problem, the only solutions we found were recommendations to disable "Fast Web View." What's that? This gave us another thing to search for and led us to a server-side solution in this forum topic. For whatever reason, the load balancer was breaking HTTP requests with a "Request-Range" header, and Adobe Acrobat Reader was using this to attempt to make the PDF load faster. In retrospect, this makes sense, but it sure was a time consuming thing to discover! If you run into this, the solution is to add the following to your Apache configuration file (or something equivalent if you use lighttpd or something else, we found examples of this happening with other server software):

LoadModule headers_module modules/mod_headers.so
...
<FilesMatch "\.(mp3|zip|pdf)$">
    Header unset Accept-Ranges
    RequestHeader unset Range
    RequestHeader unset Unless-Modified-Since
    RequestHeader unset If-Range
</FilesMatch>

Consider yourself warned! It's Apple policy to phase out older routers in such a way that they are no longer usable. Case in point: I have two of the second generation Apple AirPort base stations, called "Dual Ethernet" in some places and "Snow" in others. (See the wikipedia page for more details). They work great, have wireless and wired access, let you use a dial-up connection or dial into them from your remote network, and do 802.11b perfectly. One was at my parents house for a while, another at my girlfriends.

Both are now at my house and I wanted to set them up for other things. First, I found this article on their factory default settings which led me to the one on how to reset the things so that I could reload the software. All seemed to make sense and sounded pretty straightforward. My only Mac is a MacBook Pro for work with the newest version of Mac OS X and AirPort utilities, so I seemed to meet all the requirements. However, the routers were not showing up in the Airport Admin Utility. Weird.

I search around about this problem and found many people with the same issue but no solution, and after digging around on Apple's website some more, I found this Airport Software Compatibility Table. So the "Dual Ethernet" router isn't supported in 10.4 or newer? That seemed strange. I downloaded the AirPort 4.2 for Mac installer, but running on Mac OS X 10.5 it claimed that it required "10.3.3 or newer." Last I checked, 10.5 was newer than 10.3 but *shrug*. I gave Apple tech support a call and after explaining the issue, I was told "Your cheapest option is going to just be buying new routers." WHAT? I have two perfectly fine routers that each cost $300 new, and it turns out Apple decided to just remove support for them from the configuration tool? Wow.

So I dug around some more. I remembered using Windows to configure them at one point so I booted up Windows XP on the MacBook Pro and searched out the older software. It is still available here: AirPort 4.1 Download for Windows and will allow you to configure your older Apple routers. However, this isn't feasible for me because I don't typically have a Windows installation that I can get to for the planned uses of these routers, so they're now for sale. Check Craigslist, Facebook, or event comment here if you want them. Sold to any reasonable offer. These things work great and are only limited by Apple removing functionality from new versions of their software and not providing a downgrade path (Older versions of OS X won't install on these x86 laptops). Even though I recently got a new AirPort 802.11n router (that also works great) I will not be buying another piece of Apple networking hardware.

(Also, I had lots of phone line problems and ended up switching from Speakeasy Business DSL to Comcast Business Cable. I've had to deal with Comcast on the phone once already and their support system is a complete mess, but for the same $ I'm getting my static IP and 15Mb download/2Mb upload instead of the 1.5Mb download/384kb upload.)

As many of you know, I'm the project manager for Gallery. We host all of our bugs, tasks, source code, and mailing lists in our project on SourceForge. Over the last year or so, we've been paying for external security audits of our entire codebase which has been very helpful in identifying potential security problems in Gallery. The last round of audits from Gotham Digital Science were very thorough and gave us a good list of things to improve. SourceForge supports marking tracker items as "private" so that only members of the team can see the issues. We decided to use this for the security fixes because having everything in a central location in a way that team members can see what issues are open, pick ones to work on, and communicate their progress to others, helps us get things fixed much more quickly than just using email or a mailing list.

As project manager, I have a paid subscription to SourceForge which allows me to "monitor" projects, and I monitor both Gallery and gallery-contrib (a separate project we manage to allow anyone to develop for Gallery using SourceForge with no prerequisites). Monitoring a project means that I get an email for every single action that happens on the site: updates to bugs, new feature requests, etc. I received e-mail notifications of all the private security related bugs and didn't think much of it, but at some point it hit me: what if anyone with a paid SourceForge account could monitor a project and thus get notifications of private items?

Andy (another Gallery developer) and I set out to test this out and verified it on gallery-contrib. I removed him from the project, he monitored it, I created private items, and he was still getting the e-mail notifications! Not good! All of our security issues (which costs us a significant chunk of cash to find out about) in the bug tracker on SourceForge were essentially published. Perhaps noone was monitoring our project so it all could still be secure, but wow. I submitted a private bug with SourceForge (fortunately, they do not allow people to monitor their bug list :) ) and it went into the queue. Bharat, the founder of Gallery, used to work at Sourceforge and forwarded the issue along to some of his contacts there, and it was resolved a few days later. We verified that it works the right way now, but it leaves me feeling pretty nervous.

SourceForge is great because they provide all these things for Open Source projects for free including, perhaps most importantly, the ~6Mb/s of traffic our downloads generate. (In my current hosting setup, that would cost me over $400 a month.) However, like any Software-as-a-service, you can never be sure that a "private" checkbox works the way that you expect, and if you do find problems, there is no way to fix them on your own. Private tracker items are now truly private, but this could always be accidentally changed in the future without our knowledge, and we certainly can't keep testing this with the regularly that the severity of our security issues warrants.

While this was very surprising in a very negative way, we won't be switching away from SourceForge any time soon. They do a good job, are very responsive to requests, have the functionality we need, and host more Open Source projects than other things such as Google Code. We'll see what the future holds, perhaps one day we will have the resources to run things on our own!

I ordered a new server today to take over as a primary web hosting server so I can clean some things up on the other two boxes. Initially I was going to use Dell but I had some issues with a recent order with them however they resolved those (more on that later) and when I was specing out things today, they had a great coupon: Buy a $2800+ server and get $850 off. Cool! My budget was $3k for this thing, and this deal enabled me to get faster CPUs for significantly less money. So Dell it is. (I'm a fan of having 1 phone number to resolve warranty issues with for 3 years at a reputable company, which Dell gives me. One of my servers is from Monarch Computer which went out of business 9 months after purchasing and it's having a problem with it's RAM or the motherboard :/)

A Dual QUad Core Intel Xeon E5410 2x6MB Cache at 2.33GHz with a 1333Mhz FSB with 8GB of 667Mhz 2GB Dual Ranked DIMMs, a PERC 6i Serial-Attach SCSI PCIe RAID controller with 256MB Cache, 2 146GB 15K RPM Serial-Attach SCSI 3.5in drives, and mounting rails is on the way! (or will be soon atleast.)

When this shows up, all my websites will move onto it (~40 at last count), and the stuff I outlined in my last post can start to happen. Exciting!