After 2.5 years at SugarCRM, it's time for me to move on. Sugar was a great first job out of graduate school, and my experiences there have been invaluable. It took me from the custom tools I wrote at Georgia Tech for system management and monitoring (for CPR), to in-depth experience based knowledge about much better tools other people have written like Puppet, ZenOSS, Cobbler, and more. What began is a small confusing mess where everyone had root on all the servers and no idea what other people were doing grew into a managed, monitored, and automated environment that made everything easy so that people could focus on getting real work done instead of figuring out things like what changes need to be made to php.ini and who might change them to something else. Reimaging a cluster of web servers, database servers, mongodb servers, memcache servers, nginx servers, and mysql servers used to take quite a bit of time, but can now be done in an hour or so.

Each trip out to Silicon Valley was a new and exciting set of experiences, from Velocity last month, to views from the CTO's porch in the hills (like this one), to having beers at with friends old and new that work at places like Yahoo, Facebook, Google, Digg, Meebo, etc.

SugarCRM is a great product, has a great future, and I look forward to seeing what happens next in the super-competitive world of CRM. If our growth numbers of requests/second and OnDemand customers over the past few years is any indication, the sky is the limit! It's been great working with everyone at SugarCRM, and there are lots of aspects of my job and the people at SugarCRM that I'll miss.

All that said, it's time for me to move on. Working on a remote team makes some things pretty difficult, travelling across the country all the time has lost its novelty, and the interrupt-driven world of operations prevents the kind of focus needed to solve the kinds of problems I would like to solve, so next up for me is a complete change of pace. Tomorrow, July 30th, is my last day at SugarCRM.

On Monday, August 9th I'm starting at Highgroove in Atlanta to try my hand at Ruby on Rails in a software engineering envrionment, working in-person at an office a bit more, and being able to manage my time a bit better without the constant flow of tickets and alarms.

Ruby is new to me but I've done plenty of things in Perl/PHP/Python, and Rails is new to me but I've worked on web applications using various frameworks for almost 10 years so that shouldn't be too much trouble either. The skills I pick up will also help me put a dent in the backlog of personal project ideas I have. I have no idea where the future leads, but regardless of where I end up, the "sweet spot" for me has always been the overlap between different areas of computing (whether its operations and research, or mobile devices and web services) and Ruby+Rails will be a great thing to have in my toolbox to glue things together. This should be fun!

After being out of town for a week, this past Tuesday and Wednesday was the Working Man's Madison at the Dick Lane Velodrome. Jason and I teamed up to represent Faster Mustache's race team, while Jim and Justin from FM:Race wore different jerseys and raced as Man Hands. (Each team had to wear different jersey's to avoid confusion in the Madison races.) Read about what a Madison is over on wikipedia.

Photo of Jason and I about to do an exchange in the 50Lap Madison by Jon Woodroof who, as usual, has lots of photos and a broader writeup: here.

The field for The Madison was somewhat ridiculous: 4 super fast A riders, 2 solid B teams, both from FM, and 2 teams consisting of an A rider and a junior rider. The As, particularly Joey and Oscar racing for Mountain Khakis has no problem lapping the entire field and as things got spread out it got pretty tricky to keep track of where the lead group was, and what placing on the sprint laps (every 10 laps, the field is scored 5,3,2,1) ended up being based on who they were passing at the time. Jason and I also both managed to slip and fall, me in the first race and him in the second, and while only minimal blood was lost it meant the racing rider doing an extra lap while the relief rider that had slipped and fallen at 0MPH at the boards had to pick themselves back up and get their acts back together. My fall early in the first race left my shoulder feeling pretty achy, and we had a few hand-sling transitions that we completely blew. Even after that, we ended up 4th place for the night behind the two A teams and Man Hands, 4 laps behind with 10 points. (results)

On wednesday, Neither Jason nor I placed in the first non-madison race, and our 23.81s one-lap madison-exchange time trial placed us in 4th place (with the entire field placing as the same order as the racing on Tuesday). Last up was the main event, a 50-lap Madison. We were feeling pretty good, and the first 19 laps were super solid. Jason pulled ahead on lap 9 and slung me in to seal a first place in the first sprint, and we stayed with the pack until the second sprint when Joey/Oscar turned up the gas and started lapping the field. Everyone got pretty spread out, and while we both rode solidly and had a lot of great exchanges, we were riding solo for most of the rest of the race. The day ended with us 5th overall in the combined standings for the two days, 7 laps behind the lead with 21 total sprint points. (results).

The only team doing any lapping were the two A teams that grabbed 1st and 2nd, and even though no other teams changed positions, we were all 4,6,7 and 9 laps behind due to timing. I'm happy with our 21 sprint points and hopefully enough teams in the future will do the Madison that we can have separate A and B races with a smaller spread of abilities so the final standings aren't as crazy! Next weekend at the track is "The Omnium", see you there!

What a crazy week last week was! A lot happened in Silicon Valley at Velocity and DevOpsDay, and it was all pretty awesome.

I flew in Monday and picked up a rental car (my first week in California with a car, and I've spent 7+ months of my life there) and headed to Facebook for lunch with my friend Don from Gallery and some informal conversations about Facebook's operations with two people that manage things there. Dinner and some long conversations about technical teams followed at Bharat's (also from Gallery) place in Menlo Park, and I was off to a super late check in at the Hotel.

Tuesday morning, Velocity 2010 began. Seth, Wyatt and I from SugarCRM were a small part of the 1200 person sold-out crowd of performance and operations engineers from around the world. Including:

• Everyone that wrote all the books (e.x. Web Operations co-authored by John Allspaw of Flickr and now Etsy. He signed the pre-release copy I got for free!)
• Everyone that wrote all the tools (e.x. Adam Jacob of Chef, and Luke Kanies of Puppet.)
• "The Greatest SysAdmin in the world" , Theo Schlossnagle, founder of OmniTI. A quote from him: "Ops super powers come from developing a uniform hatred and mistrust for all tech while maintaining a positive outlook."
• The kind of people that are as "graph crazy" as I am and use this sticker in their presentations.
• The developers behind web development and performance tools, including a guy I did some grad school projects with named Jaime that works for Google and did a presentation on SpeedTracer.

Tuesday, Wednesday and Thursday were full of entertaining and informative sessions, and I mostly stuck to the operational track. There is far too much for me to fit in here, but if you'd like to know more about the content of the conference, Ernst at The Agile Admin did great writeups of the sessions he went to which mostly cover the ones I was in, Royans posted a pretty consise topic and idea summary and Velocity has the slides and videos available. If you only have time for a few or don't know where to begin, start with these:

• Scalable Internet Architectures - Theo Schlossnagle's opening session that set the tone for the entire conference.
• Infrastructure Automation with Chef - I learned a lot about Chef from this talk and talked to Adam Jacob (presenter and Chef author) afterwards about some aspects of how Chef works and some problems I have with Puppet. He was super helful about both and once I look at switching out mod_passenger for unicorn for use in our Puppetmaster, I'll be giving Chef a shot.
• Ops Meta-Metrics - John Allspaw's great talk about correlating data related to change and incidents with more standard monitoring data.
• Always Ship Trunk: Managing Change In Complex Websites (PDF) - The way that development should be done and code should be deployed.
• Facebook Operations – A Day In The Life - This was the only standing room only talk I was in, and Tom from Facebook (one of the guys I talked to Monday) explained all kinds of interesting behind-the-scenes details.

Some of the most fun sessions I went to were the evening activities on Tuesday and Wednesday. Tuesday was Ignite Talks which is an hour of presentations, each consisting of 20 slides that automatically advance every 15 seconds forcing the presenter to be quick and concise. This was the first one of these I have been to, and the topics covered ranged from completing a triathlon to cheap scalable storage using laptop drives. Ernst at The Agile Admin did a great writeup of Velocity 2010: Ignite.

Wednesday evening was "Birds of a Feather" sessions which are small conversations around a table between people interested in the same things. Up first for me was a Demo by ZenOSS/Opscode/Dyn about "Geographically Based Cloud Scaling". They demoed using ZenOSS to detect failure of servers and datacenters, which then updated DNS hosted at Dyninc using their API so traffic was only sent to "live" servers. Once enough things failed that there wasn't enough capacity, ZenOSS used Chef (by Opscode) to spin up Amazon EC2 instances and add them to DNS to help handle the load. Very Cool demo! I pointed out that ZenOSS was still a single point of failure, and we discussed strategies for using Chef to ensure that a ZenOSS install is always working properly somewhere. update: ZenOSS Blog post about the demo.

After cloud scaling, I switched rooms to talk "Load Balancing Tips and Tricks" with another small room of people. It turns out that the vast majority of people use NetScalers for load balancing (We at SugarCRM used to use them but switched last December to nginx+wackamole on Dell hardware), and most of them have the same issues we experienced. It's pretty common to not use any of the advanced features in the Citrix load balancers and to do things like session handling, redirects, and serving error pages from upstream servers. In the early days of Youtube, they actually would have to reboot the entire site due to a "feature" in the NetScalers that couldn't be disabled: An "overflow" queue would accept more connections than upstream had resources to handle with the goal of smoothing out small load spikes, but it ended up backing up everything and the only way to clear out the load and get response times back down was to restart all the web servers which gave all open connections error 500s. Youtube doesn't serve video content through their NetScalers and neither should you!

To close out wednesday evening, I headed to Dyntini at the hotel bar: free drinks on DynInc's tab! It's a bunch of nice people, and I'm slowly moving over ithought.org DNS to their infrastructure. Once SugarCRM is ready to do some global load balancing, we'll probably make the switch as well.

Some other interesting talks I have specific thoughts on:

• Grendel is really neat. It's a HTTP based document store with strong server-side PGP encryption: if someone gets a copy of your DB, still don't have data.
• Hidden Scalability Gotchas in Memcached and Friends Neil Gunther comes from an academic background and talked about performance and modeling: "Models are from God, data is from the devil." This talk had a bit of a confrontational response from the audience because ops people in the room want to use models to predict performance and capacity characteristics, but he claims that models are more useful to explain the data. The particular model he presented had things like a variable representing "contention factor," and the value of this variable on a curve fit tells you if cpu/disk/ram/lock/etc contention is limiting the performance of your application. This was used to show some changes that they made to Memcached to improve scalability on systems with large numbers of CPUs
• Drizzle - The momentum in Drizzle sounds really promissing. It's based on MySQL but is getting a lot of features ripped out and/or converted to plugins, and comes with sane defaults. There isn't a reason to switch to using it yet, but if at some point it doesn't become feasable to use MySQL, this will probably be the way to go.
• Choose Your Own Adventure - Adam from Opscode gave a wonderful presentation that was a great close to Velocity. You had to be there, but he's a great public speaker and everything is either a magic unicorn or it isn't. UPDATE: videos of this are now online!

DevOpsDay was a completely different yet still pretty similar experience. A bunch of us met up at LinkedIn on Friday morning to listen to a round of panels discuss DevOps ideas and culture. The entrance was somewhat sneaky.

All of the panels were full with great speakers, including ones that stood out as being great during their presentations at Velocity and some great people that didnt' speak at Velocity like Gene Kim (wrote first version of tripwire), Michael Stahnke (created EPEL), and Israel Gat.

"DevOps" is sort of like the new "Cloud". In 3 years companies may be asking "What should or DevOps strategy be?" which seems pretty funny now, but "The Cloud" was the same way several years ago. It's not a technology, but a culture and processes that revolve around people and technologies. Terms like DevOps and "The Cloud" are bad terms but they're nice because business people see them in magazines and online, so are more receptive when technical people bring things up. One of the panels was on DevOps culture outside of Web Oberations, and I hope to be able to take it's content to heart: DevOps is about removing team barries (Us vs Them) including those with Engineering and those with "them" being "the business". Everything needs to be "we" and companies that get this working correctly see 4x gains in efficiency and productivity.

Adam from Opscode, Theo from OmniTI, Luke from Puppet, and Eric from rPath were in a fantastic panel that ended up focusing much on the "sharing" aspects of the rising world of system automation. Puppet just launched Puppet Forge and Chef has their Cookbooks Library, but at some point configurations are going to be so specific to an enterprise that they'll need to be developed inside of the company. This panel had a lot of back and forth between the panelists, and a lot of audience interaction, with no real conclusion on how much things can truly be automated. Food and drinks at the end of the day with other "DevOps" people led to some great conversations about automation and scalability, and Saturday morning I flew back to Atlanta.

Based on conversations and presentations over the week, here are a few books that are worth checking out:

• Good to Great
• The Visible Ops Handbook
• Continuous Delivery (preorder)
• Leading Geeks
• Web Operations

And lastly, some quotes and concepts to close out with:

• "Hiring too many people to do things that should be automated gets you a "Meat Cloud". This is bad" (Adam Jacob?) - Focus should be on hiring people that can do the automation instead.
• "us ops guys are like cicadas. Every 17 years we get to come out" - Velocity is the only real gathering like this, and each year it grealy increases in size. There was a similar sort of thing in the early years of computing, but for some time the sysadmin has been mostly forgotten.
• "cloud security: Its always going to suck, but we'll always have jobs!"(Ward Spangenberg) - Running your code on someone elses infrastructure is always going to have risks, and smart people will always be needed to make sure everything is functional and secure.
• "The limiting factor for any business is time it takes to restore from app data, code repo backup, and bare metal." (Theo Schlossnagle) Operationally, a business can only recover form disaster as quickly as the application data can be restored. If there are other things that make recovery time longer, they need to be fixed and optimized out so that the restoration of data is the only limiting factor.
• "The purpose of QA is to ensure quality." (John Allswaw) - So why do business have QA departments? Shouldn't engineering and operations be responsible for ensuring quality? A lot of the big-name businesses (Facebook for example) don't have QA departments any more and it's working out alright.

That's it for the 2010 version, I'm looking forward to continuing to interact with all the people I met at Velocity and DevOpsDay, and next year should be even bigger and better.

Getting web applications performing optimally is a never-ending full-time job for a lot of people, including me! I manage the OnDemand environment for thousands of customers at SugarCRM. This post will walk through a few pretty simple steps that can be taken to improve the performance of SugarCRM on a Linux server with MySQL, but they mostly apply to any PHP application running on a LAMP stack.

Tuning a web application is a type of engineering, and as Rico Mariani at Microsoft puts it:

If you are not measuring, you are not engineering.

Monitoring is the key in all of the tips that follow because if you don't know how bad things are, you won't know which of these fixes help you out. Depending on the details of your situation, some can actually do more harm than good so it's important to go through these one by one, understand the implications of them, and compare you data from befor and after you make any changes.

MySQL Performance

Getting things right in MySQL is another full-time job, but there are a few quick things you can do. First of all, make sure that all of your tables are the same character set. If they are not, this can greatly slow down queries that join between tables of different character sets. Take a look in the output of

SHOW TABLE STATUS FROM sugarcrm;

and make sure that everything is the same in your Collation column. If you are using utf8_general_ci in most places, and the table my_custom is a different type, just run:

ALTER TABLE my_custom CONVERT TO CHARACTER SET utf8;

Next up is making sure that you are properly utilizing the MySQL Query Cache. This cache saves the restlts of select queries so that if the same query comes in again before changes to the table, the results can be returned much more quickly. To see the configuration for this, run:

SHOW VARIABLES LIKE "query_cache%";

If query_cache_type is set to something other than "ON" or query_cache_size is set to 0, you will want to change these! 32MB is a fine start, but depending on your workload you may need (a lot) more to get the full benifit from this:

SET GLOBAL query_cache_type = 'ON';
SET GLOBAL query_cache_size = 32000000;

This one change can make a significant difference in performance. To tune your numbers, check the output of:

SHOW STATUS LIKE "Qcache%";

After some time running with the Query Cache on, you should see hits go up and lowmem_prunes should stay pretty low with free_memory not reaching 0. If free_memory seems to run out, or lowmem_prunes continue to climb, increase the size of the query_cache_size. There will continue to be lots of inserts and not_cached due to constantly changing tables and queries other than SELECT queries, so don't worry about those numbers.

Generally, using the InnoDB storage engine instead of the default MyISAM storage engine will help out as well, but any tables that you want to be able to do full-text search on cannot be InnoDB, and there is a lot more to take into consideration here than any quick fix. If you are using all MYISAM tables, this would be a good thing to investigate. Switching to InnoDB and tuning your MySQL storage engine based on your workload are a lot more than I can fit here today, but there is a lot of information out there about these. Head over to the MySQL Performance Blog for more information than you ever wanted to know about MySQL performance (including an excelent article on the MySQL Query Cache).

All of your MySQL STATUS values should be included in your monitoring system, as graphs of these are very helpful over time for diagnosing problems and tuning cache sizes! Also, remember that the Query Cache uses up RAM so keep an eye on that as well. In this case, it will use 32MB and as you increase the query_cache_size, you are directly increasing RAM usage.

PHP Performance

Perhaps the single biggest perfomance enhancement that you can make for SugarCRM and many other PHP applications is OpCode Caching. I've written about some of the complexity involving this before at SugarCRM and Caching with APC, but the simple version is that OpCode caching will improve the performance of your PHP applications. At SugarCRM, we use APC which is pretty straightforward to install. On CentOS, and other rpm/yum Linux distributions simply:

yum install php-pecl-apc

or

pecl install apc

Then, enable apc by adding it's configuration to /etc/php.ini or /etc/php.d/apc.ini, and make sure that apc.enabled is set to 1 and apc.shm_size is set to something reasonable. For one instance of SugarCRM, the default of 32 (for 32MB) is generally plenty but if you have other PHP things running on the server, a bigger number might help.

Copy /usr/share/pear/apc.php to somewhere in your webroot and visit it in your browser, and after a bit of usage of SugarCRM, you should see the "Hits" percentage climb while the "Misses" percentage gets smaller and smaller. Your hit ratio should be higher than 98%. If the "Cache full count" number is more than a very very small percentage of your hit counter, you should increase the size of memory available to APC. On some Linux systems, shm segment size is limited to 32M so instead of increasing shm_size, you will need to increase shm_segments.

Like MySQL, these numbers should be part of your monitoring system so that you know when it's time to change things. Also like MySQL, this directly uses RAM. This example will use 32MB, and increasing it will use more.

Client-Side Performance

Why bother serving files to clients that never (or seldomly) change? A common industy best practice is to set a Cache-Control header for static content to 10 days in the future. Tools like YSlow will let you know if you are not doing this. (They will also give you a lot of other pointers on perfromance! We have internal bugs open at SugarCRM for most of the client side performance issues that tools like YSlow have identified.)

We use apache on our application servers, and to enable these cache-control headers we added this:

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf|xml|txt)$">  
Header set Cache-Control max-age=36000,public
</FilesMatch>

For SugarCRM OnDemand, this cut our number of requests-per-second in half and made a several-hundred-millisecond improvement in the time it takes to load pages from SugarCRM OnDemand.

For this one, monitor your req/s and check your PHP Applications after any changes using a tool like YSlow.

SugarCRM Specific

Lastly, a few specific changes just for SugarCRM:

• Add 'disable_count_query' => true to config.php. This replaces the "1-5 of 200" with a "1-5 of 6+" and makes a big difference when you have a lot of records. This configuration setting should be a default in a future release of SugarCRM.
• Disable explicit caching in the app by setting 'external_cache_disabled' => true in config.php. While SugarCRM can use external caches like APC's user cache and Memcached for some things, this shouldn't be enabled unless you have set up the infrastructure to handle these. This configuration setting be a default in a future release of SugarCRM.
• If developer mode is enabled and you are not using it, turn this off! In SugarCRM 6.0, this is under Adminstration -> System Settings -> Advanced. This is off by default, and if it gets turned on and accidentlally left on, it causes severe performance degradation.
• Turn loging down to the minimum that you need. If your SugarCRM installation is on a NFS based document root with multiple servers, this becomes even more important. This is under Administration -> System Settings -> Logger Settings. "Error" or "Fatal" are good options here.
• Turn off tracker features you are not using. This is in Admin -> Tracker. Each enabled piece of functionality causes things to be written to the database, and will slow down page loads. Leave on only what you need for reports that you have written.

Thats a start!

With those changes, you should be off to a pretty good start at improving the performance of your installation of SugarCRM! These things help us keep SugarCRM performing quickly in the SugarCRM OnDemand environment with a pretty high ratio of active users to servers, and a recent application of these tips that I did to a site that handles 200k+ unique users per month lead to a 65% decrease in system utilization and much improved response times.

[Cross posted on developers.sugarcrm.com.]

Last year my bike trip was to Santa Fe to ride mountain bikes. This year, I flew to Portland, OR for a long weekend of road biking with 2 friends from Atlanta that have moved there: Paul and Austin. I flew out from Atlanta last Friday evening, and the bike crazyness began! First thing Saturday morning Paul and I picked up a rental Fuji Roubaix ACR 2.0 from the super nice guys at Waterfront Bikes. The bike was brand new and fit pretty well, and had pretty white bar tape that I knew wouldn't stay that way for long.

Saturday: Forest Park

From Paul's house, the three of us rode west out of Portland and zigzagged across Forest Park. Part of the loop involved crossing 2 pretty big bridges with traffic, and they were by far the biggest bridges like that I've ridden across. (The bridges in NYC have separated bike lanes.) We got some great views of the city, 2 big climbs, and 2 huge descents, and didn't work too terribly hard. Afterwards, we grabbed tacos from one of the many cheap and delicious food carts in Portland.

• 34 miles, 2500 ft, 2.5 hours
• GPS Trace: Forest Park

Sunday: Hood River

Sunday morning is was raining pretty badly in Portland. We found a route out of Hood River, OR on mapmyride.com and loaded up the car. Once there, we got a small break in the rain as setup for the last stage of the Mt Hood Cycling Classic was getting set up. Our plan was to get back and see the last 30 minutes or so of the pro men's circuit, but we ended up missing it by a few minutes. This ended up being a pretty awesome route, but it would have been more fun if we weren't getting rained on half the time. The long slow uphill for the first half was followed by a long slow downhill for the second half, and both had great scenic views of rolling hills covered in nurseries and orchards. Mt. Hood would have been visible the whole time if we weren't buried in the clouds. For surviving, we rewarded ourselves with food and beer at the Full Sail Brewing Pub in downtown Hood River, OR.

• 50 miles, 3800ft, 3.5 hours
• GPS Trace: Hood River

Monday: Sandy River

Paul had to work on Monday, and I was pretty beat from Sunday's ride, so Austin and I set out on a longer but easier ride out Portland to the East on bike paths. The ride was nice and pretty flat until we got out to Sandy River, where there were a few short climbs and a few great windy downhills. There were no real views, but lots of miles and lots of changes in scenery: from toothless meth heads on the bike path to Alpaca farms in the hills. We got a bit turned around riding back into town but with some pointers from a local on a bike, we made it to downtown Portland for massive burritos from another food cart.

• 70 miles, 2600ft, 4.5 hours
• GPS Trace: Sandy River

Tuesday: Larch Mountain

On Tuesday, Austin was back at work but Paul was able to sneak out for a ride, and a ride it was! We drove east to a home depot parking lot around 250th street, hopped on our bikes, and headed up the road to Larch Mountain. This was a continual climb up to the overlook at 4056ft, which took us 2.5 hours. The skies were clear and after a short hike-a-bike to the top, the view was great: we could see all of the big peaks in the area: Hood, Jefferson, Rainer, St. Helens, and Adams. After a handful of photos, we headed down the mountain at ~30MPH, getting back to where we started in under an hour. More burritos afterwards, and my legs informed me that they did not want to ride any more bikes.

• 47 miles, 4400ft, 3.5 hours
• GPS Trace: Larch Mountain

The End

What an awesome trip! 200+ miles, 13,000+ feet of climbing, and over 15 hours in the saddle including biking to dinner a few nights. I had a great time hanging out with Paul, Austin, and friends, and it was nice to be away from the internet and on a bike instead for 4 days. Based on the weekend, I liked Portland better than San Francisco, but Paul thinks the winter weather in Portland makes it a bit less desirable (weather this weekend was pretty close to perfect). There's more photos here: Road Biking in Portland, OR, though unfortunately I didn't snag any of the great views on Day 1. Waterfront bikes was surprisingly glad to see the bike covered in mud splatter and "well used", and Tuesday night I hopped on a red-eye back to Atlanta. Up next will be a mountain biking trip somewhere new and exciting. Track Racing tonight anyone? I kinda hope it gets rained out...

This past weekend, San and I took our first "real" vacation since our Europe trip in 2008. Friday afternoon we drove down to Destin, FL to spend some time with some of San's extended family. Much food was eaten and many movies were seen. On a boat tour in Destin we got up close and personal with a blue crab and a pufferfish, and we saw bottle nose dolphins and stingrays in the wild from the deck.

Once Sunday afternoon rolled around, we headed over to Seaside for a few relaxing days at the beach. Our room was a tight squeeze on the 3rd floor of a tower with a great view:

We had a great time walking around, making pyramids and sphinxes at the beach, riding beach cruisers around, playing in the ocean, swimming in the pool, and enjoying happy hour with a view of the ocean.

I left my camera in the room for most of the trip so hopefully San will get all the photos she took posted, but I did take mine to the beach Tuesday evening and got some pretty awesome shots thanks to sunset, the moon, low tide, and a f/2.8 L lens. Here's a favorite:

Here's the rest of the photos from the weekend: Seaside 2010. Who wants to go in on a big house there at the end of the summer if it's not consumed by the oil spill?