This past weekend was the first Dick Lane Velodrome Pro Race Series event of 2010. I'm now racing in the B category (categories), so I was able to race on both Friday and Saturday for the first time.

Friday Night Sprints

I've done the friday night sprints in the past, and this year my 200m time was 13.18s which is pretty average for me and as usual, did not seed me in the sprint tournament. (I was 18th of 28.) David Espinoza and Andy Lakatosh tied at 11.03s for a new track record. Crazy Fast! Not being elgible or the sprint tournament meant a few mass-start B races which is where things came together. In a small field of 6 including two of my FM:Race teammates: Justin Barber and Jason Atwood, we got to play a little bit with team tactics. The second race, a 10-lap scratch race came together perfectly and landed me a 1st place, Justin 2nd, and Jason 4th. This was the first cash win for Justin and I in the Bs, and we promptly spent the $25 + $15 on pizza and beer at the end of the evening to fuel up for Saturdays racing. I'm looking forward to a season of team tactics in Wednesday night races! My Results. Twotone's Photos.

Saturday Pro Racing

Photo courtesy of twotone, thats me in the red and white in the bottom right.

Saturday was something new, both me racing in the Bs at one of these events and C racers having a few races. Charles Earl, another FM:Race member, snagged 8th in the first C race of the afternoon, only to be run into the boards by another beginner. He managed alright with a bent up bike and a few stitches in his right hand at the emergency room, so hopefully he'll be back on the bike soon. After he left there were a few other crashes, but none that required a trip to the ER.

The B field had 15 people in it, including the faster of the people that show up for Wednesday racing and two fast people from out of town, and we had 5 races:

• 10 Lap Scratch - 8th place. No teamwork and some poor decisions!
• 15 Lap Snowball - On the first lap, I sprinted off the front and managed to win the 4st 4 laps for a total of 10 points (1+2+3+4), but couldn't hold out for lap number 5 and I barely hung on to the rest of the field for the rest of the race. 5 other people got points on later laps, so I ended up in 6th place. If I'd won the 5th lap, I could have snagged 5th or 4th place. Next time!
• Unknown Distance - 8th place. The bell rings and it's the last lap. Stuck in the pack.
• 40 Lap Points Race - A very long and pretty fast race, I hung with the pack for the first 30 laps. There were several breakways that all got caught, and around lap 35 I sprinted from the middle of the pack to grab a few points and sling off the front to try and catch Darren Cormer who was off the front. In about a lap I caught him and we did half-lap pulls for a few laps but were both completely empty and the field caught us with a few laps to go. I crossed the finish line in 5th place for 1 point, and ended up in 7th place.
• Miss-n-out - 15th place. Didn't have anything left in me, last place after getting boxed in and passed on both sides.

I had a lot of fun racing, as well as having a few beers and taking a few photos while watching the pros ride in "The Keirin". Twotone's Photos. The next Pro Race Series event is July 9th and 10th!

I finally got a chance to try out recording some video with my helmetcam and newest light. The results are pretty good, and with a little more tinkering with camera angles, I think the results of night riding will be pretty watchable!

Blankets Creek at Night from Chris Kelly on Vimeo.

FM:Race team members Bob, Jim, Jason, Adrian, and I ride for ~2 hours on Friday night at Blankets Creek in Woodstock, GA. Jason's light died, Bob's pants died, and good times were had.

Music: Silversun Pickups - Substitution

Filmed with a VIO POV1.5 lit by a NiteRider SlickRock 900 and a NiteRider MiNewt x.2 Dual. Next time I'll aim the camera a little higher, but this is a definite improvement from just the NiteRider TrailRat 10W: http://vimeo.com/3394871

At SugarCRM we use OpenVZ on some beefy boxes to host a number of containers that take care of some things that we don't want to dedicate hardware to. An article I wrote in 2008 describes our OpenVZ VLANd network setup.

I recently provisioned a new OpenVZ server but it seemed to be possessed by some sort of evil spirit because networking just wouldn't work quite right on the containers hosted on it all the time. Most of the time things would be fine, but occasionally things would just stop working. I re-imaged the physical server several times which didn't help at all. Today, we used up capacity on all the other OpenVZ servers and needed this one, so I set out to expunge the demons. I set up a test vm and conveniently, networking was completely broken on it, so I set out to work.

Up first were the basics:

• the bridge, vzbr256 was up and running and tcpdump showed traffic in it
• the interface on the host, bond0.256 was up and running and tcpdump showed traffic in it
• the parent interface for the container, veth137.0, was up and running and tcpdump showed traffic
• iptables configuration vm2 was identical to all other OpenVZ servers

When vzbr256 was in discovery mode and passing all traffic, veth137.0 could see traffic between other systems on it's subnet. Once the bridge figured out the topology and switched to forwarding mode, the only traffic goign to veth137.0 was the arp requests from the container side with no arp replies coming from the bridge. This was very strange because the bridge was still receiving the arp traffic and they weren't tagged with vlans or anything funky. Full packet dumps of the arp replies on vm2 showed that they were functionally identical to the ones on the other OpenVZ servers.

Basically the container was asking "where can I find my gateway?" and the gateway was responding with a MAC address. This reply made it all the way back to the bridge, but not back to the container for some reason. If I pinged the container from some other system on the vlan, the container would see that arp request and send a response, and all traffic would start to flow normally.

This sounded a little similar to this Problem with bonding, vlan, bridge, veth but the patch that fixes this was already applied to the Linux kernel that we're running. From this thread on Complex routing and bridging with OpenVZ it looked like some sysctl changes might be useful:

echo 0 > /proc/sys/net/bridge/bridge-nf-call-arptables
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 0 > /proc/sys/net/bridge/bridge-nf-filter-vlan-tagged

A wiki article "Bridge doesn't forward packets" suggested that this might help, and changing 'bridge-nf-filter-vlan-tagged' from 1 to 0 seemed to let that first arp packet through, but resetting everything with that set to 0 from system boot led to the same symptoms and toggling it (and the others) around in different settings didn't help. Perhaps the first toggle was timed just right so it let through that one arp packet.

I compared the configuration of bond0/eth0/eth1 on vm2 with other vm servers and noticed that the MAC addresses of all 3 were the same. On other vm servers, bon0 and eth0 were the same but eth1 was different. Perhaps this was the cause! Our provisioning system using Cobbler continues to evolve and perhaps something changed. I removed the MAC lines from /etc/sysconfig/conf.d/net.* files so that they now look like:

#eth0
DEVICE=eth0
ONBOOT=yes
SLAVE=yes
MASTER=bond0
HOTPLUG=no
BOOTPROTO=none

#eth1
DEVICE=eth1
ONBOOT=yes
SLAVE=yes
MASTER=bond0
HOTPLUG=no
BOOTPROTO=none

#bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MODE=trunk

and reset everything, but the mac addresses still showed up as the same. Given all the information above, and that others were having problems combing bonding ("port channeling") with vlans with bridging, I decided to investigate how our bonding interfaces were set up. vm2's interfaces on the switch were still set to 'spanning-tree portfast' for some reason so I disabled that which had no effect. Digging a little deeper, the options that the bonding module was loaded into the kernel with in /etc/modprobe.conf were the culprit:

alias bond0 bonding
options bond0 mode=1

vm2 was using mode 1 bonding (balance-rr) which does simple round-robin load balancing instead of mode 5 (balance-tlb) which does adaptive transmit load balancing. I changed this on vm2 to:

alias bond0 bonding
options bond0 miimon=80 mode=5

and rebooted. Problem solved!

I can't be sure of the exact root cause here without understand a bit more about kernel level networking than I do, but it seems that linux bridging doesn't know what to do with arp reply packets that arrive in a vlan on a bonding interface via a physical interface other than the one the arp request was sent from. In mode 1 bonding, both physical interfaces get the same MAC address so there is no way to ensure that replies will come in on the same interface they are sent on, but with mode 5 bonding, each card uses its own MAC and replies always come to the same physical interface that the request was sent from.

After fixing this, I updated the provisioning system to configure all new systems with mode 5 bonding, and will be updating all existing systems to use mode 5 bonding the next time they reboot. Thankfully, our longest running system (at 1057 days and counting, last rebooted in mid 2007 before I was hired!) was set up manually before the provisioning system existed and was already using mode 5.

Long story short, if you're having networking problems with OpenVZ on top of a VLAN+bonding infrastructure, check out your bonding mode first and pick one that allows distinct MAC addresses for each physical interface and meets whatever needs you have for using bonding in the first place.

Last year, I told my friend Jim that I would join him for a bike event in Athens, GA as part of Twilight Weekend 2009. A friend from college came in town that weekend so it threw a wrench in the plans and while I got some awesome photos of the races, my bike unfortunately stayed in Atlanta.

Given that Twilight Weekend happens every year, I promised to make it this year and things ended up working out that way! A few months ago, while perusing athenstwilight.com I noticed the "Twiathlon": a 5K run, followed by a 100K bike ride, followed by a tour of the Terrapin brewery and some free beverages. Jim and I have a habit of suggesting ridiculous bike related things and then following through on them, so we signed up for the Twiathlon and Jim reserved a hotel room in Athens for Saturday evening. We also conned our friend and fellow FM:Race team member Bob into joining us (and driving us there!).

Early Saturday morning we strapped our road bikes to the top of Bob's car, hopped in, and drove through the rain to Athens. This was Not Good. The weather forecasts called for rain all day, and we discussed abort conditions as we pulled into Athens. After parking in my terrible "secret" parking place and registering, we moved to FM:Race Dan's order-of-magnitude-better "secret" parking place ~100ft from the finish line and geared up to run. The rain subsided a bit, and we stood in the drizzle and laughed as other runners tried to find cover as if they would somehow remain dry throughout the event.

At 8:30am the 5K Race began, and Jim, Bob, and I stayed together at a sub-7-minute pace as Jason Spruill sprinted off in the distance to try and win something. Jim and I were shooting for sub-25 and we told Bob that he had to do the same thing, so off we ran. Bob slowed down after the first mile but managed to come in at 24 minutes and change, and I stayed with (IRONMAN) Jim until the last little hill where he started to inch away for his 22m32s finish, a personal best for him. I came in at 22m44s which was a personal best for me because it was the first timed 5k I've done, and I thankfully didn't feel the need to quickly empty my stomach like the guy right in front of me wearing a grass skirt. Here's my watches view of the race. It was raining for most of the run, and after the finish we walked through some lighter rain to the cat-5 crit course to see our teammate Justin after he managed to stay with the pack in his first crit. Go Justin! Rumor has it our friend Jon got 4th place in what was also his first crit, but the official results aren't up yet.

Back up at the secret parking lot, we changed from wet running clothes into soon-to-bet wet bike riding clothes and assembled our crew. Me, Bob, and Jim in FM:Race kits on road bikes, Dan in a FM:Race kit on a tandem with one of his kids, and Jason along to keep us honest about keeping the pace up. We barely made the start at 10am and rolled out under cloudy but dry skies on a very very wet 100k worth of road. Our primary tactical focus was beating Dan's wife and 2 younger kids that were doing the 50k, but Jason quickly learned that he could attack on every hill and Jim would try and stay on his wheel but wasn't quite able to keep up. Bob and I had never run before biking and our legs were very confused, but we all managed to hold together as somewhat of a group for most of the ride, including pacelining at 30mph in pouring rain. Terrible, and not something that I'd willingly do in the future. Here were are at the driest point of the ride somewhere around the middle:

This event, the "Terrapin Wake N' Ride" wasn't a race (and while we beat Dan's family, they had to bail a few miles before the end and get picked up due to general terribleness of the conditions), but there are some super sweet prizes and Bob may have won something very fancy. Everyone has the option of taking a 50k or 100k route, and along the route are several checkpoints. Each checkpoint gets you a poker card (we got bonus cards for running the 5k), and at the end of the ride, they give out prizes to people with the best hands. I ended up with 2 pairs, but Bob managed to get 4 7s. Hopefully he'll get a phone call this week once they tally everything up! Here's how my watch saw things.

By the end, we had been riding in pouring rain for 45 minutes (after ~3 hours of slightly less wet riding) and were all pretty wet and miserable, but Bob and I somehow convinced Jim to let us enjoy one free beer at Terrapin in the colder but slightly less wet inside. Then it was back to Athens (riding bikes in the rain slightly buzzed with empty beer glasses in our jersey pockets) for some hot showers and learning about mining silver and copper on the History Channel. To prevent Jim from killing and eating us, we suited back up in our matching FM:Race work shirts (see Bob's), found our way to the race course, accidentally ran into our friend Kazz, and acquired some burritos. The only thing left to do was to find a place to enjoy the races from 5pm-10pm, and a $5 donation to charity got us primo seats (actual seats!) on the inside of the second section of turn 3. Given the weather and potential for beers, I left my nice camera at home but the point-n-shoot shows what our view was like for the evening:

Highlights included not seeing any crashes but seeing the coordinated look of shock from people on the other side of the turn that saw a crash outside of our field of view, having Kazz yell at us as she rode in the pace car during one of the races, and the general awesomeness of drinking beers while watching bike racing. Shortly before the end of the pro mens race, we walked back to the hotel, absolutely destroyed a few Dominos' pizzas, and slept up for the early Sunday morning drive back to Atlanta. Here's the rest of the photos. Jim had fun.

Watching crits in the rain isn't as fun as when it's dry, and I don't plan on riding that much in the rain again, but running in the rain was a lot less terrible than expected. The only things that would have made this better is if it didn't rain the entire time and some of our teammates were in the evening crits, but hopefully both of those will happen next year!

At my day job, I'm responsible for everything that is Operations at SugarCRM including our OnDemand environment where we host SugarCRM for a large number of our customers. The web clusters are a not too surprising Open Source stack including nginx, wackamole, Apache, PHP, MySQL, CentOS, memcache, etc, but because of how SugarCRM works, we do run into some interesting challenges from time to time. Our engineering team develops a single product that can be deployed on customer servers, in our ondemand environment, and anywhere in the cloud, which means that in our environment each customer instance of SugarCRM lives in it's own silo. This puts our own unique spin on the common SaaS challenge of scaling up while keeping response times as low as possible.

One of the key components in keeping response times low is PHP opcode caching. When I started at SugarCRM a few years ago, we were using Zend Platform for opcode caching, session clustering, and a few other things. It did a fine job of the opcode caching but one of our "standard procedures" was to restart the Zend session clustering daemon whenever a customer reported certain kinds of problems. Not good! To permanently resolve this, help move us towards a full Open Source stack (since we are an Open Source company!), and to simplify our architecture, we moved to using APC for opcode caching (and memcached for session clustering).

The APC performance gains were similar to using Zend, but we didn't have any issues with APC requiring service restarts for close to 2 years. Over these two years we have continued to add web servers to our OnDemand cluster as the number of customers increased, and a few weeks ago Apache started (seemingly randomly) getting backed up with all of it's slots in use "Sending Reply". A simple `apachectl reload` would fix this, but it was eerily similar to the days of Zend. Due to the number of other ongoing projects, I didn't have time to investigate this much so I set up an alarm in our monitoring system so that we would get alarms before the web server got completely backed up and could proactively fix the issue.

Unrelated, yesterday I was looking at some of our metrics and noticed that our APC cache hit rate was a lot lower than it should be due to a high "Cache Full Count" (The Cache Full Count number was 10% of the size of the Cache Hit Count). We weren't directly monitoring this and I remember making sure the cache was big enough when this was initially set up, but that was from when we had a much smaller OnDemand customer base. To fix this, I bumped up the cache size from 128M to 512M and our configuration management system slowly started pushing out the changes. Cache Full Count was reset to 0 and stayed at 0, and the hit rate went from ~70% to ~90% across the cluster. Problem Solved! Or so I thought.

A little later on in the day, the web servers started getting backed up, much more quickly than before and all at the same time, and it was starting to cause timeouts for some customers. The proverbial monkeys in our application had gotten mad at the fan.

With a much bigger problem, I did some deeper debugging using strace and the "Sending Reply" apache processes all seemed to be spinlocked on a futex(FUTEX_WAKE) call. Could this be APCs fault? I disabled APC across the cluster and the problem went away, but had to figure out a way to fix this because no APC means a significant increase in the minimum load time to every single page load in our system. Digging through source code and Googling around, I came across How to Dismantle an APC Bomb which explains a set of symptoms that seemed very familiar. The cause was explained as contention caused by apc_store calls which is in the User Cache portion of APC andnot the opcode cache, which gave me an idea: just disable the user cache.

With SugarCRM, this can be done by setting 'external_cache_disabled' or 'external_cache_disabled_apc' in config.php, but I didn't really want to touch thousands of config.php files, so I looked in our code and found:

elseif(function_exists("apc_store") && empty($GLOBALS['sugar_config']['external_cache_disabled_apc']))'

SugarCRM checks if apc_store exists to see if it can use APC, so I simply added apc_store to disable_functions in our php.ini template, re-enabled APC, tested out the changes, and pushed this out to the cluster.

This ended up being the problem and the solution. It turns out that increasing the memory available to APC meant that things were staying in the cache longer which made the contention issue even worse. SugarCRM's use of a user cache simply doesn't scale well in a massive clustered environment, and with it disabled, the opcode cache can do a better job. Hit rates are up to 92% to 99% across the cluster, and response times are down. Below is a sanitized graph of response times to a test instance designed to represent the worst response times across the OnDemand cluster:

• APC's memory was increased from 128M to 512M a little after the dotted red line at 12:00. This improved consistency of 'worst case' response times significantly
• Sometime between 1pm and 2pm, things started to get Very Bad and while response times were still down, web servers were crashing left and right
• At 2:12, APC was disabled completely. Minimum response times went up but things got suspiciously consistent
• After over an hour of investigation and testing, at around 3:20, APC was re-enabled with apc_store disabled, response times went back down, and the response time consistency remained.

I'm glad this issue happened after peak hours on a Friday, and I'm looking forward to seeing how this solution affects response times during peak load periods next week. If you're running SugarCRM at any kind of scale, this may be something to consider.

Very early this morning, I volunteered with Emory Rehabilitation to help support disabled athletes as they competed in the 2010 ING Atlanta Half Marathon in wheelchairs. ~40 or so other volunteers and I rode bikes during the event to provide immediate technical and medical assistance (or call for help), and to direct the racers through the course. Because of how close to the ground they are, it's tough for them to see potholes and other obstacles, and every aspect of the course is more challenging than running or riding a bike: uphills, downhills, and turns. We rode the course several times as a group of the past few weeks to get familiar with the tricky spots so that we could warn the racers ahead of time. I used a Niterider SlickRock 900 on my helmet and the dual-light version of the Niteride MiNewt X2 on my handlebars to both light my way and provide enough light to for my VIO POV 1.5 to capture some video footage (helmet setup photo). Here's the result:

2010 ING Atlanta Wheelchair Half Marathon from Chris Kelly on Vimeo.

Want to help me help some more people by riding bikes without even having to stand up? Consider donating money to the American Diabetes Society through my Tour de Cure page. I'll be riding 100 miles to raise money for Diabetes research.