Competition is good, and with the huge popularity of Apple’s iOS devices and devices running Android, Microsoft is the underdog and being forced to try quite a bit harder to get to the top than they’re used to. In November last year, I got a Microsoft Surface tablet (Blog: So I Got A Surface) which wasn’t really that amazing because I don’t really use a tablet for anything.
In July this year, after having used iPhones for a long time, I switched from an iPhone 4S running iOS (which I sold on eBay for $300+) to a Nokia Lumia 1020 phone running Windows Phone 8. This is the flagship Windows phone which you’ve probably seen advertised all over the place with it’s 41MP camera, and this writeup has been in the works for a few months.
Note: I paid for this phone, pay my monthly phone bill, and in no way is anyone giving me anything in return for talking about this phone.
First up are some notes from initial setup:
- There are several ‘Authenticator’ apps with the same name, and it was tough to tell which one is which. I accidentally started out with the wrong one which was unable to support scrolling, and had to go through the 2-factor setup with all of the services that I use again with the official Microsoft app. The official one is great: Authenticator
- Nokia Account? Windows Account? Login with Facebook? What? I had to set each of these up to get full functionality, but I only had to set up each one once. Compared to how many times you have to type in your Apple account password when setting up an iPhone, this was actually pretty nice.
- Windows phone supports magical contact syncing between all Windows devices, Live.com, Google contacts, LinkedIn, Facebook, etc. That said, there wasn’t a good way to get contacts out of Apple’s Contacts.app into Live.com. I did something terrible and un-repeatable with CSV files and reformatting that I hope to never have to do again. Again, once that was in this works really well.
- This phone is bigger than my iPhone was, not a big deal, but I did have to get a new waterproof case for it to keep it dry on bike rides. E-Case eSeries is the way to go.
- The headphone jack doesn’t support microphones, so it’s a bluetooth headset or using the phone like an old-school phone and holding it to your head. I’ve ended up using speakerphone a lot more than before which has worked fine.
- There isn’t a way to disable screen rotation, which makes using the mail client while laying down problematic. Supposedly this is addressed in Windows Phone 8.1.
- After getting used to Windows 8 RT, the combination of similarities and differences made learning how to use Windows Phone 8 a little frustrating. Apple definitely has them beat in feature/usability parity between phones and tables. Again, this should be a lot better when Windows 8.1 is everywhere.
- The e-mail client doesn’t support setting custom folders for sent/deleted/drafts of e-mails. This makes it hard to play nice with other e-mail clients.
- NCF seems cool! I touched a piece of paper and got a free 20” x 30” photo print from my phone, but I haven’t found any other uses for it yet. There were NFC enabled signs in Japan, but there didn’t seem to be any apps in the Windows Phone Store that could handle their content.
- Speaking of the store, it is accessible both on the phone and via any web browser. iTunes is not required, nor is the completely optional ‘Windows Phone’ Mac app.
- Surprisingly, most of the apps I use are available for Windows Phone. A few were missing, specifically most of the banks, but this is slowly getting better. More on those in a minute.
After using my phone for a few months, it’s show to be a nice little gadget that stays out of the way. I’ve been pleasantly surprised, and encourage other people to give a Windows Phone a try. Some more ‘experience’ based comments:
- The camera is a little gimmicky, but it does take great low light pictures and the animated GIF generating app is a lot of fun.
- SMS and Facebook chat are combined in the same app with seamless conversation history, which is the best way I’ve found to use Facebook chat.
- I haven’t had to do any more tinkering with account credentials. Everything just works.
- Contact syncing with Live.com just works.
- SkyDrive just works, photos I take on my phone show up on my computer, no opening iPhoto or anything else required.
- Chase shipped a Windows Phone app: Chase that supports check scanning, so I opened an account with them and it’s worked flawlessly. (Simple.com and Paypal will hopefully get their acts together soon!)
- I used the phone in wifi-only mode for 2 weeks in Japan. The Maps experience was terrible and I had to use maps.google.com in the web browser most of the time which worked fine, but wow iOS maps is better in other countries. In the US, the built-in maps and navigation maps are pretty great.
- I also switched from Things.app to Remember The Milk for TODO list managing as it’s nice and syncs across all platforms, not just Apple ones.
- With a little help getting started from a coworker that builds Windows Phone apps, I was able to get an app working that unlocks the doors at our office and load it onto my phone. Cool!
So it’s been great (which is good because I’m ‘stuck’ with this phone for 1.5 more years), and will be even better when a few more apps appear. The following are apps that don’t exist yet that I would definitely use:
- Strava cycling and running
- Weather Underground
- Dark Sky
- Yahoo Weather
- Square Wallet
- Loaf Deals
- Amazon Cloud Player
- Amazon Instant Video
- TV Guide
Rumor has it that if I had an Xbox and ran Windows on my desktop (which I may do in the near future) my ‘media’ experience would be better, but I don’t really do much music listening or video watching on my phone. There isn’t exact parity with games, but there are enough quality tower defense games to keep my idle time filled. Since switching, I’ve found myself ‘needing’ to use my phone less which is a good thing. Maybe it’s less notifications, maybe it’s because there aren’t apps that talk to everything and add overhead in combination with convenience, or maybe it’s because there is no ‘hey check out this app’ because I don’t really know anyone else with Windows phones. I’m pretty content! Questions?
Another year, another week long mountain biking trip in the books. Last year we Rode in Sun Valley, Idaho. This year, we headed to Park City, Utah. Park City is the only IMBA Gold Ride Center, and has over 400 miles of bike-legal singletrack within riding distance from downtown.
Day 1 - Masonic/Solamere/Gambel Oak - GPS trace - No Photos - Chris/Paul/Marc/Brian
To get warmed up we rode pretty close to our rental house, realizing in person just how many intersections there would be and how handy having routes in mind, maps in hand, and GPS devices and compasses to show where we were would be. The forecast called for a bit of rain so I left the camera at home, but we ended up not getting too wet. The temperature did drop on us as the day continued from ~50F down to ~38F, and several people stopped by the bike shop at the end of the day to get some warmer clothes. Afterwards was our priciest meal of the trip, meat and whiskey at the High West Distillery in town.
With Bob and Jim in town, we headed off in the snow to ride the Mid-Mountain trail, one of the 'must-do' trails in Park City. Once at elevation, it stays about halfway up the mountains, weaving through all of the ski resorts on very well manicured trail. This was an even colder day, hovering in the mid-30s to low-40s, but the views and trails made up for it. Things got a little sloppy towards the end, requiring the first of several bike washes of the trip.
We took it a little easy agin on Day 3, riding some close-to-home stuff and staying away from the upper elevations. After a short day on bikes we headed down the road to the closest movie theatre to see 'Rush'. Turns out it was a movie about car racing and not a band. Nice!
Our longest initial climb too us 3000+ vertical feet up to the top of the mountain and the start of the Wasatch Crest trail, another must-do trail. The last 1/3 of the climb was in a significant amount of snow on the ground, and was pretty slow going. The view from the top was pretty great, and we broke into small groups to being our slushy descent. Parts of the downhill were pretty nice, but others were in muddy snowmelt with no other way to get down. At the last major turn of the day, Jim/Bob made a slight miscalculation and ended up in Salt Lake City, then a shuttle, then the hot tub.
A long ride to the trailhead via bike paths was a great warmup for perfect weather and a climb to the top of the Flying Dog loop. Not as remote or crazy as some of the further out trails, the views were not as epic but the trails were in much better shape. This day was the only day with weather the way we had expected it to be, so it was a good ride to bring the trip to a close. We ate a pizza at a bustling pizza place at the trail head, and set off to return our bicycles.
Next year, Canada!
I ride bicycles a lot. Not winning-the-tour-de-france a lot, but some-people-think-its-all-I-do a lot. The bikes page on here says more, but to keep a long story short, people that don't have bicycles tend to as me what kind of bike they should get and where they should get it from. The TL;DR version: Pick how much you want to spend, go somewhere where people treat you nicely and tell them exactly what kind of riding you want to do, and buy what they recommend. Here are the most common questions and the answers I've given over and over.
What kind of bicycle should I get?
This depends on the type of riding you will be doing most of the time. Not every type of riding you have seen on youtube and think might be cool. This also depends on how much time you want to put in and how fast you want to go. Brands like Surly and Salsa makes bicycles as specific as
"Moonlander, like its name implies, is meant to go beyond where normal bikes, even normal fat bikes, can go. It is designed from the ground up to ride where there are no roads, no trails, no people."
"The Colossal Ti is for riders that like going fast, but don't believe fast is everything. That dream of riding through the Alps, but recognize that beauty can be found by riding right from their own front door. That enjoy pro racing, but aren't disillusioned enough to believe their bike is keeping them from joining those ranks. That understand a heart rate monitor and power meter will never be as valuable as the discoveries made from turning onto an unknown road, and seeing where it leads."
so there is a bike out there for exactly what you plan on doing. Personally, I'm eyeing one of those Salsa Colossal Tis, but if you'll always be riding the rough streets of Atlanta 4 miles each way to work, you should ask yourself if you'll ride in all weather (if the rain is ok you'll want a bike with fenders), if you don't mind a backpack (racks and waterproof panniers), where you'll be storing it (more expensive bikes will be safer indoors), etc.
Don't just go buy a full-suspension mountain bike. Do you even like mountain biking? Not everyone does! Have you ever ridden a cheap full-suspension mountain bike on the road? They are slow! Will you ever even go mountain biking? That said, if this bike is just going to be used on technical downhills in the mountains with a mix of pedal-up and shuttle-up, it may be a good fit. If you'll always be shuttling, or if you'll really only be doing flow trails, there are different kinds of mountain bikes to get instead.
Don't just go buy a fancy drop-bar road racing bike. Have you ever ridden in drop handlebars? Most non-serious-cyclists people stay on the 'flats' (Hint: they make flat-bar road bikes without the drops). There are parts that require maintenance, they can be harsh on bumpy roads due to their geometry and design, and aren't engineered to be comfortable. If you're looking to get faster and faster and find the biggest hills to climb, a speedy road bike may be for you. Depending on the roads that you'll ride on, the company you'll keep, and the hills you'll climb, you now get to decide between 'compact' and 'road' cranks, and 'roubaix' or 'endurance' or 'race' geometry.
Don't buy a 'track bike' unless you are racing at a velodrome. Want a simple bike for getting around town on? You can still get a 'fixie' that is a lot more comfortable and safe to ride that one that is meant for racing on a track. If you're actually going to a track though, you may want to try that out before buying a bike.
The place you buy your bicycle can help you decide!
What about High Modulus Carbon Frames? Campagnolo EPS Components? Blended Compound tires?
See "Where should I buy a bicycle? What brand should I buy?" and "How much should I spend?". Anything on a bike from a local bike shop that meets your price range is going to be the same as anything else in your price range from any local bike shop. Enjoy!
Where should I buy a bicycle? What brand should I buy?
Step 1: Go to the closest bike shop to your home/office/school.
Step 2: Do they treat you in a way that makes you feel comfortable? If yes: buy one of the brands they sell from them! If no: cross it off the list and repeat Step 1.
Local bike shops have people working there that are knowledgeable, and they provide service, often for free with a new bicycle. Walmart/Target don't count because they don't have knowledgable staff or provide service, and I'll leave comparing the 'Big Box' brands like Performance Bike to local bike shops like Loose Nuts Cycles to you based on how you feel about supporting local small businesses (you should probably support local small businesses).
Sure you can buy a bike from bikesdirect.com but you're on the hook for picking the right one (there are 1000s of choices) and putting it together. If you are looking for a cheap non-name carbon race frame from Taiwan that you'll race for a year, you probably haven't made it this far through this post and bought it on e-bay already. You can also get a gently or harshly used bike from Craigslist and there are good deals if you know exactly what you are looking for. But if you knew that, why are you reading this again?
Your local bike shop will carry a small variety of brands, and can help you pick a bike. Any brand they sell you will be fine, and the people you buy the bicycle from matter far more than the stickers on the frame. They should give you a free water bottle too! Tell them I sent you.
How much should I spend?
First, know that a few things are not included:
- You'll need a helmet. $30 can get you one, but it seems like most are in the $50-$100 range these days.
- A basic flat-fixing-kit and multitool will run another $30 or so.
- Riding at night on dark streets? You'll want to spend $100 on a light to 'see' with. The $20 lights that let you 'be seen' won't show you the pothole you are about to run into.
- Riding to get to work? You may want some bike-specific clothes to wear on your commute, $50 for socks/shorts/shirt is probably as cheap as you can go.
- Riding 'seriously'? Those fancy spandex clothes sure look weird, but once you wear them you'll find they make riding long distances a lot more comfortable. There are backpacks that hold water for mountain biking, color-changing sunglasses, fancy lights, fancy shoes and pedals, etc. Helmet-cameras are pretty cool too. Don't get any of those yet, you'll know when you need to. Everyone gets one "showing up for a ride completely unprepared" ride, and usually (if you talk beforehand) the people you're with have things you can use a time or few to see if something is your style.
And now the easy part. Just pick how much you want to spend.
- Don't spend less than $500 on a new bike. Anything less than this will be frustrating. (Unless this not something you need to depend on. $150 on a every-other-week-coffee-shop run is just fine)
- $500 will get you a bike that lets you find out if you like the kind of bike riding you are doing (commuting, fitness riding, exploring, etc). If you do like it, you'll want a new bike next year.
- $1000 will get you a nice bike. If you like it, you'll be happy with it for a year or two and may start to think about upgrading parts.
- $1500-$2500 will get you a pretty nice bike that can last quite a while. Bicycles in this range can go 10s of 1000s of miles before 'wearing' out assuming you take them by the local bike shop you bought them from for a free tune-up every now and then.
- $3000 and up will buy you something fancy that while possibly impractical, may be the lightest/quietest/fastest/etc of the bunch. It's easy to find $14,000 bikes. Ouch.
So pick how much you want to spend, go somewhere where people treat you nicely and tell them exactly what kind of riding you want to do, and buy what they recommend.
Singletracks.com has a good How To Choose Your First Mountain Bike article too.
It's time to get this blog thing going again! I have a queue of ideas pooled up that I've put a lot of thought into and seem to discuss over and over with people in person, so here is to the first one: some free financial advice for everyone. I am not an expert by any means, and while I'd love to not think about my money at all, the thinking and executing that I've done has given me a pretty decent plan that requires little ongoing involvement on my part and has been called things like 'actually pretty ok' by trained and certified financial planners. Follow these steps and you may eventually find yourself in better financial shape than you are currently. They are roughly ordered in order of priority.
Note: I am not trained or certified in giving any kind of advice, and certainly not financial advice. Everyone's situation is different and while what I describe below works for me, you should talk with someone that is trained and certified before making any financial decisions that you're not completely comfortable with, and probably before you make any financial decisions at all.
Spend less than you earn
Pay your credit cards in full each month, and don't buy things that you don't have cash in your account to cover. If you don't do anything else on this list, you should be doing this one and it'll all be ok. The less you spend, the more you can save, and the quicker you can 'retire' if you are into that kind of thing. Everyone has differing opinions on how much to save but given 'the market', any kind of long-term planning based on exact percentages is going to be no better than make believe. You can't go wrong with this one. Apply this to all of the 'saving' things below by counting them as 'spending', and you'll end up with extra cash for buying fun things and a slowly growing 'nest egg'.
Live Debt Free
Spending less than you earn should cover this, but if you already have debt, do whatever it takes to pay that debt off. Pay off the debt with highest interest rates first, and do your own math if it makes sense to pay off low-interest-rate debt with money that you could be investing and getting better returns on. If you can conservatively get 6% in the stock market, you probably shouldn't pay off your 2% student debt right away, but that 18% credit card bill needs to go. If you aren't prioritizing this way, you are throwing money away. I'd even suggest paying off debt when you can afford it even if the market might do better, because the market is variable and your debt is not. Not ever having to even think about making a student loan payment again is far better than maybe having a few extra dollars in 20 years in my mind.
Look at the Big Picture
Mint is good for this. It can show your net-worth go up each month if you're doing the first two things. This is good. It's also nice for watching your spending categories change over time and identifying things that you're spending a lot more on than you thought.
Build and keep an Emergency Fund
3-6 months of your after-tax income is probably plenty. 3 months of your 'non-savings' expenses each month is a good minimum. This gives you flexibility between jobs, the ability to pay for a car with cash, the ability to handle unexpected medical expenses, etc. Some people say not to use your Emergency fund for things like a car, but if you need a car and you can tap your fund to avoid getting a monthly car payment with interest, do it! I keep mine in an American Express Personal Savings account which pays only about 1% interest, which is low but better than nothing, and I won't loose it if the market crashes. Which it will. Remember that as you pull from your Emergency Fund and as your income and expenses change, you'll need to put money back in. This should be somewhere that is risk-free, where you can access it relatively easily, and definitely not in a retirement account.
Max out 401(k) matching, and possibly the federal limit if you can
If your employer matches up to 5% of your salary, put at least 5% of your salary into your 401(k). If you do not do this, you are throwing away free money. Your employer will have a bunch of plans available, but pick a target-date retirement fund and let it do it's thing. Statistically, no mix of funds that you pick will do better than a target-date fund and they will all take more of your time to manage. If you can afford it (and probably after you've maxed out your ROTH IRA as described below), contribute up to the federal limit of $17,500 for 2013 (IRS details). This is tax-deferred which means that you'll pay taxes on it when you withdraw for retirement, but you'll be paying less taxes then so this is ok. The deduction now means that your taxable income for this year will be lower. Note, this money is for retiring so don't put money in your 401(k) that you'll need to spend before then or you'll pay a lot of penalties if you need to use it.
Use a bank that isn't a pain to deal with
Simple has been my primary bank for over a year. I have a few invites, let me know if you'd like one! If you're paying account management or ATM fees, or you don't have a grasp on how much money is in your account or how much you can spend at any particular time, it's time for a change. If you pay overdraft fees because a deposit hadn't cleared yet and you didn't know, you are throwing away money. Good customer service is always nice too, you'll need it sometimes.
Use the best credit card that you don't have to think about
You could spend all the time in the world picking which credit card to use for every transaction you make, but I've limited myself to 2. For me, first is an American Express Blue Preferred card because it's benefits give me the most bang for my buck with my spending habits (6% on groceries is really good) and American Express have the best extra features and customer service so my groceries and big purchases go there. Second is a Chase Freedom VISA that offers 5% back on some subset of things depending on the month. Thinking about which of the 2 cards to use when is a little overhead for me, but 5% back is a lot compared to the 1% on all purchases that most cards give. These may not be the best cards for you, but search the internet for comparison tools (Mint.com has a decent one here) and pick one or two. Shred the rest and forget about them, and feel free to cancel one of them every now and then. This may hurt your credit score a little bit, but you don't need them and you usually don't need your credit score. Remember that not every place takes American Express because they charge higher merchant fees, so have at least one VISA in the mix.
Pay the littlest amount you can when you borrow money
If you have to borrow money, don't borrow from your retirement accounts. Borrow from the cheapest place possible. This probably means going to your family or taking out a home-equity loan instead of using a credit card. Refinance your mortgage if you can go a full percentage point lower in interest rates (that generally means the refinancing will be 'worth it'), and consolidate any debt to the lowest interest rate possible. If you don't do this, you are throwing money away. LendingClub is one place to borrow money to consolidate. I may even invest in your loan if you go there! If you don't pick the cheapest way to borrow, you're throwing away money.
Max out your ROTH IRA every year if you can and invest it
The most you can put into a ROTH IRA is $5500 a year in 2013 (details from the IRS). You pay taxes on this money before you invest, but it grows tax free so whenever you withdraw for retirement, you won't pay any taxes. This is a good thing and is basically free money compared to just using a brokerage account where you pay taxes on all of the gains, or a tax-deferred account like a 401(k) where you pay taxes eventually. I use Wealthfront for my ROTH because they are low-fee and have several fancy features that put them ahead of most managed investing platforms. I make one yearly contribution to my ROTH from my Emergency Fund (you can also pay monthly or at any other frequency) and so far mine has generally followed the market without any work on my part. Betterment is a similar tool worth investigating. Note, this money is for retiring so don't put money in your ROTH that you'll need to spend before then or you'll pay a lot of penalties and taxes if you need to use it.
You can try and time the market and buy-low, sell-high, but generally you're never going to beat the market. As long as you continually invest and follow the advice above, and as long as you slowly change your asset mix over time to less risky things (All the services and funds mentioned above do this), you'll do ok! I additionally invest in Betterment and LendingClub ever month. LendingClub is like a CD with higher interest rates and some risk. I'm averaging ~10% annual returns at both. This is 'extra' money that you could use in the future for anything you like, but that it'll be ok for you to loose (or for the value to fluctuate with the market). Think about using this money to fund a big trip, pitch in to a down payment on a house, or loan to a friend or family member if they need it.
Lastly, your time is valuable. Automate everything you can including bill payments and investments. Find out the amount you need in your checking account to cover all of your recurring expenses (On Simple.com set up a goal for this amount that you never pull from) and make sure you always have that much available. If you pull it off and automate everything, this entire 'plan' shouldn't take more than a few hours of your time each year.
If you've read other financial advice, you'll notice that a few things are missing from this list, but they're excluded for good reasons.
I don't believe in budgets as firm rules, and they cause far more stress (e.g. spending $5 too much one month on Coffee) than the benefits that they provide. If you're doing everything I described above, you don't need a budget. Spend whatever you like on shiny things or organic Cheetos as long as you're doing everything above.
Stocks/Mutual Funds/ETFs/Investment Categories
Again, none of these matter. As long as your asset mix slowly moves from risky to less risky over time (Which target-date retirement funds, Wealthfront, Betterment, etc will all do for you), nothing you can pick will do better than the market and they will all take more time and effort. Sure, occasionally someone will 'hit it big' on a pick of some kind, but those are the exception and not something you should count on. Your core competency is something other than playing the stock market, so don't bother trying.
Insurance is an expense, not an investment. Only you can judge what kind of insurance you need. If you have $10 million in the bank you probably don't need to pay for anything more than the legally requirement minimums for car insurance because you can just buy a new car and in a bad at-fault accident you'll be writing a check regardless when they take you to court. If you have a mortgage, your lender will require homeowners insurance to cover their investment, and if you have more debts and guaranteed future expenses that won't be covered by your assets (big mortgage, a child, etc) you'll probably want some kind of life and long-term disability insurance. The cash value of a whole life insurance policy can be used as a non-risky asset in your diversification plan, but your retirement and investment accounts can match this on their own.
Plenty of people suggest that you use cash to manage your budgets and spending because it feels more 'real' than using a credit card. Get over it, credit cards give you free money for using them. You can loose cash, and it's a hassle to keep up with.
Paying yourself first
I'm not even sure what this one means! It's important to spend money on food, housing, and on things that bring you happiness, but 'paying yourself' something before you max out your 401(k) matching with your employer would be stupid (again, throwing away free money). Your future self will thank you. The Wikipedia page on Present Value has some things to say about this.
You obviously want to make enough money to meet your current and future needs, but it's pretty independent from everything above. It's probably a good topic for a separate post!
Questions? Ask away in the comments field below!
Several years ago, before GoPro cameras were of acceptable quality, I purchased a VIO POV 1.5 and recorded a good bit of mountain biking with it. Late last year it finally died, and looking for a replacement the GoPro HERO3: Black Edition was recently released and had some amazing looking video samples. The entire camera is bigger than the recording part of the POV 1.5 so there is more to attach to my helmet, but there is no bulky battery/storage unit, it's waterproof, and there are many more mounting options. But what really matters is the video quality. The HERO3 performs amazingly well for low-light things like night mountain biking, works pretty well during the day, and works underwater. I've shot a few things in 4k video but without a way to edit or view those, I've mostly stuck to 30fps 1080p. Below are the results! Subscribe to my my Vimeo account for future ones.
First up, testing out mounting options while mountain biking. Helmet is great if it's aimed right (which can be tricky with no viewfinder/display), Handlebars are surprisingly good but the mount slides around unless you wedge something 'grippy' in it (they have fixed this on a newer version of the clamp mount which was just released), and on the seatpost provides a neat perspective but on my bike I had to rotate the video 180 degrees which seems to confuse iMovie and caused some weird cropping.
Then it was on to the night! Both of these were shot with a ~900 lumen HID light on my helmet and the camera mounted on my handlebars. I've traded-up lights to a ~1800 lumen LED, check back for video from that soon.
Up next was capturing the Thursday Night #fmride. This was helmet mounted and with the higher speeds of a road bike, I could feel the wind drag on my helmet. Not a problem, but it could be on longer rides.
I thought the waterproof housing would be nice for sweat and the occasional rain while riding bikes, but it does a pretty great job at the pool as well:
Lastly was some 'proper' mountain biking with Mike at the Olympic MTB course in Conyers, GA:
If you're looking to get an 'action camera', I recommend getting this a GoPro HERO3: Black Edition. Sure, the battery only lasts about an hour, the mounts take some patience to figure out, and the 2-button UI take some getting used to, but the quality of the video you'll get is completely worth it.
TL;DR: Install linux-firmware and uninstall pecl-apc. scroll down for a graph that shows why.
The backlog of new packages to install was big enough that it was time for me to bite the bullet and upgrade the kernel and udev on all of my Gentoo servers for ithought.org They were a mix of 2.6.*/3* kernels and old unsupported udev.
I documented all the hardware, wrote out compilation plans for kernels for all of them including all the Gentoo, hardware, and software specific configuration options and built new kernels. 3.6.11 for the amd64 boxes and 3.5.7 for the x86 ones. Userland things upgraded successfully, and things were good to go for rebooting.
Just to play it safe (and to replace a server belonging to one of my colocaiton customers) I scheduled some routine maintenance to reboot all of the servers for 9pm on Wednesday February 6th.
The Not So Great Reboot
Rebooting the servers had mixed results. Several needed their root volume in grub.conf updated due to kernel changes in the way that volume names are presented, and I left out a few things that required booting with a boot cd to fix (software raid support on an old amd64 scsi box, and the Fusion MPT SAS driver on the box connected to the Dell MD1000 storage system) and thought I was in the clear, with just a few minutes of downtime for each server. Everything went as planned.
However, the x86/2.5.7 boxes had a problem. None of them could connect to the Internet because they couldn't see their eth* devices. This thread is a similar debugging experience to mine: http://forums.gentoo.org/viewtopic-t-948718-start-25.html. It seemed to be a problem with udev, which was a big problem because due to the new kernels combined with the age of all the other software and what was available in portage, I didn't have many options.
The newest server was new enough that I could use package.mask to rollback to an older udev that worked with the kernel on that system (2.6.36) but the other servers had kernels so old that they didn't work with the oldest versions of udev available in portage, which meant no downgrade was available.
This troubleshooting lasted from around 9:30pm until past 1:30am in the morning, trying kernel reconfiguration, customizing udev's rule files. With some help from a few people in irc.freenode.net#lugatgt that know far more about udev than I, we confirmed that everything in my kernel and udev setups looked correct. All net device starting yielded a 'SIOCADDRT no such process' that was not helpful to any of us for debugging, but a conviently timed glance at the output of `dmesg` yielded the problem, a missing file that is part of the 'linux-firmware' package. I installed linux-firmware via a USB drive on the broken servers, rebooted, and everything was finally working again. Success!
Nagios alarms all cleared, sites were back up, and I headed home to sleep.
The Next Battle
I woke up early to check on things, and one of the servers was down. The graphs indicated a combination of massive iowait processes and RAM utilization, but no swap usage. Interesting.
A few hours later, a stop by #lugatgt and asking some knowledgable coworkers, and I'd gotten acquainted with `iotop`, `iostat`, and others. I'd moved the MySQL datadirectory between RAID devices which only made things worse (and was undone), and narrowed things down to 6-12Mbps of write IO to the / volume. This was very strange because everything in /var (mysql, apache, vhost roots, logs, etc) are all on a separate RAID volume on a separate RAID controller.
The only file marked as open for writing in the php-cgi process's using 99.99% of their time waiting on IO were in /var/log, but no log files were growing, and disk utilization wasn't climbing. This indicated writing to unattached inodes, and it turned out that there is some conflict with the way APC (the opcode cache for php) was configured to use shm (shared memory in kernel land). I disabled apc, bounced apache, and the high-write load on / disappeared and conditions improved.
The impact of disabling APC is that PHP has to do a little more work on each http request which slightly lowers http respons times. It has doubled 'user' CPU load from ~15% to ~30% of one CPU, but has lowered the iowait from tanking the system after this update. That said, pre-update iowait hovered around ~50% and it is now down to ~5% which has some nice performance impacts for the rest of the system and may serve to actually decrease average response time.
It's frustrating when tools that are designed to enhance performance cause things to implode, but it's nice to clean up the stack a little and APC doesn't have the reputation of being the most stable thing out there anyways. Time will tell what impact this has on response code breakdown. Here's graph of CPU usage on this box during the events in the last 24 hours. From descriptions above you can probably see what looks pretty clear in hindsight.
One more server needs a reboot with the newest kernel and udev userland, but given what I know now I'm confident that it will go well (almost confident enough to do remotely).
If you're a hosting customer of mine or a user of one of the sites that I host, I apologize for the extended outage last night and the slowness this morning as I worked out the IO issues, but know that these upgrades include security updates and having consistently configured systems makes testing out upgrades like this possible in the future. I'm on a monthly rotation now for software updates, and everything can now be tested on a backup server before applying it to any of the primary servers. Thanks for your business!