At SugarCRM we use OpenVZ on some beefy boxes to host a number of containers that take care of some things that we don't want to dedicate hardware to. An article I wrote in 2008 describes our OpenVZ VLANd network setup.

I recently provisioned a new OpenVZ server but it seemed to be possessed by some sort of evil spirit because networking just wouldn't work quite right on the containers hosted on it all the time. Most of the time things would be fine, but occasionally things would just stop working. I re-imaged the physical server several times which didn't help at all. Today, we used up capacity on all the other OpenVZ servers and needed this one, so I set out to expunge the demons. I set up a test vm and conveniently, networking was completely broken on it, so I set out to work.

Up first were the basics:

• the bridge, vzbr256 was up and running and tcpdump showed traffic in it
• the interface on the host, bond0.256 was up and running and tcpdump showed traffic in it
• the parent interface for the container, veth137.0, was up and running and tcpdump showed traffic
• iptables configuration vm2 was identical to all other OpenVZ servers

When vzbr256 was in discovery mode and passing all traffic, veth137.0 could see traffic between other systems on it's subnet. Once the bridge figured out the topology and switched to forwarding mode, the only traffic goign to veth137.0 was the arp requests from the container side with no arp replies coming from the bridge. This was very strange because the bridge was still receiving the arp traffic and they weren't tagged with vlans or anything funky. Full packet dumps of the arp replies on vm2 showed that they were functionally identical to the ones on the other OpenVZ servers.

Basically the container was asking "where can I find my gateway?" and the gateway was responding with a MAC address. This reply made it all the way back to the bridge, but not back to the container for some reason. If I pinged the container from some other system on the vlan, the container would see that arp request and send a response, and all traffic would start to flow normally.

This sounded a little similar to this Problem with bonding, vlan, bridge, veth but the patch that fixes this was already applied to the Linux kernel that we're running. From this thread on Complex routing and bridging with OpenVZ it looked like some sysctl changes might be useful:

echo 0 > /proc/sys/net/bridge/bridge-nf-call-arptables
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 0 > /proc/sys/net/bridge/bridge-nf-filter-vlan-tagged

A wiki article "Bridge doesn't forward packets" suggested that this might help, and changing 'bridge-nf-filter-vlan-tagged' from 1 to 0 seemed to let that first arp packet through, but resetting everything with that set to 0 from system boot led to the same symptoms and toggling it (and the others) around in different settings didn't help. Perhaps the first toggle was timed just right so it let through that one arp packet.

I compared the configuration of bond0/eth0/eth1 on vm2 with other vm servers and noticed that the MAC addresses of all 3 were the same. On other vm servers, bon0 and eth0 were the same but eth1 was different. Perhaps this was the cause! Our provisioning system using Cobbler continues to evolve and perhaps something changed. I removed the MAC lines from /etc/sysconfig/conf.d/net.* files so that they now look like:

#eth0
DEVICE=eth0
ONBOOT=yes
SLAVE=yes
MASTER=bond0
HOTPLUG=no
BOOTPROTO=none

#eth1
DEVICE=eth1
ONBOOT=yes
SLAVE=yes
MASTER=bond0
HOTPLUG=no
BOOTPROTO=none

#bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MODE=trunk

and reset everything, but the mac addresses still showed up as the same. Given all the information above, and that others were having problems combing bonding ("port channeling") with vlans with bridging, I decided to investigate how our bonding interfaces were set up. vm2's interfaces on the switch were still set to 'spanning-tree portfast' for some reason so I disabled that which had no effect. Digging a little deeper, the options that the bonding module was loaded into the kernel with in /etc/modprobe.conf were the culprit:

alias bond0 bonding
options bond0 mode=1

vm2 was using mode 1 bonding (balance-rr) which does simple round-robin load balancing instead of mode 5 (balance-tlb) which does adaptive transmit load balancing. I changed this on vm2 to:

alias bond0 bonding
options bond0 miimon=80 mode=5

and rebooted. Problem solved!

I can't be sure of the exact root cause here without understand a bit more about kernel level networking than I do, but it seems that linux bridging doesn't know what to do with arp reply packets that arrive in a vlan on a bonding interface via a physical interface other than the one the arp request was sent from. In mode 1 bonding, both physical interfaces get the same MAC address so there is no way to ensure that replies will come in on the same interface they are sent on, but with mode 5 bonding, each card uses its own MAC and replies always come to the same physical interface that the request was sent from.

After fixing this, I updated the provisioning system to configure all new systems with mode 5 bonding, and will be updating all existing systems to use mode 5 bonding the next time they reboot. Thankfully, our longest running system (at 1057 days and counting, last rebooted in mid 2007 before I was hired!) was set up manually before the provisioning system existed and was already using mode 5.

Long story short, if you're having networking problems with OpenVZ on top of a VLAN+bonding infrastructure, check out your bonding mode first and pick one that allows distinct MAC addresses for each physical interface and meets whatever needs you have for using bonding in the first place.

At work I recently rolled out a newer version of the Dell OpenManage tools which included for the first time a build of Openwsman. We didn't specifically need this functionality, but it's good to stay current with the OpenManage tools. To load in the (unrelated) new kernel on a test machine, I rebooted the machine using Cobbler's power management functionality on our administrative system, but after 5 minutes the machine was still not responding to pings so something was broken. I used remote desktop to hop on our one Windows server in the datacenter which we use to get at the interactive consoles of our servers (Thankfully the new DRAC6 card's have console applets that work on Macs!), and pulled up the console for this machine.

The boot process was hung on "Starting openwsman" and didn't seem to be doing anything. Doh!

I restarted the machine again, and at the grub boot menu added a "S" to the boot string to start up the system in single user mode, and booted things up. "chkconfig openwsman off" to disable the service, and another reboot to get the machine back up and running to let me troubleshoot a little better. I took a look in /etc/init.d/openwsman to see what might be hanging, and nothing immediately looked suspicious. It was a pretty standard init script, with the extra feature of generating OpenSSL certificates if they didn't exist already:

if [ ! -f "/etc/openwsman/serverkey.pem" ]; then
        if [ -f "/etc/ssl/servercerts/servercert.pem" \
                -a -f "/etc/ssl/servercerts/serverkey.pem" ]; then
            echo "Using common server certificate /etc/ssl/servercerts/servercert.pem"
            ln -s /etc/ssl/servercerts/server{cert,key}.pem /etc/openwsman/
        else
            echo "Generating Openwsman server public certificate and private key"
            FQDN=`hostname --fqdn`
            if [ "x${FQDN}" = "x" ]; then
                FQDN=localhost.localdomain
            fi
cat << EOF | sh /etc/openwsman/owsmangencert.sh > /dev/null 2>&1
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
${FQDN}
root@${FQDN}
EOF
        fi
    fi

It's a little strange, but not unheard of practice to do this, and shouldn't cause any problems. (Puppet and Func, two other systems tools we use, generate their certs in the application which is a lot more common.)

I extracted the only possible culprit from the owsmangencert.sh script and tried running the openssl command manually:

openssl req -days 365 $@ -config /etc/openwsman/ssleay.cnf \
  -new -x509 -nodes -out cert.out \
  -keyout key.out

and it seemed that this was indeed the problem. It just sat there and didn't complete with the speediness I expect from OpenSSL. Time for strace!

cat << EOF | strace openssl req -days 365 -config ./ssleay.cnf.2    -new -x509 -nodes -out cert.out   -keyout key.out
> --
> SomeState
> SomeCity
> SomeOrganization
> SomeOrganizationalUnit
> test
> root@test
> EOF

This ended up doing a long read with output like:

open("/dev/random", O_RDONLY)           = 3
read(3, "\323K\372u_ya'\27\266\320\25\22\373\240\330~'\224\310\243\356\225\350.\245\362\3058\230Zb"..., 1024) = 128
read(3, "K\7:\273Zdr\274\25\227\263\366\260U\337Owp\6y\2333c\361\322\334\217\370.k\375]"..., 896) = 128
read(3, "dH\375V\327\230Bi\221\342\326\26R\301v^Qv5f\347\303g7\2747\345\360\207A!\227"..., 768) = 128
read(3, "X&\254r\331\353<:\36!\333\340\353", 640) = 13
read(3, "\357F\27\347\372atf", 627)     = 8
read(3, "\231\347\232\362\345\215n\227", 619) = 8
read(3, "\324\304\323\30\325\10G\332", 611) = 8

Looks like /dev/random wasn't returning random data nearly fast enough, which makes a whole lot of sense! /dev/random is "good" random data because it is based on environmental entropy and the entropy data is only used once, but on a modern multi-core systems doing lots of things, there usually isn't much entropy available. That means that while this command would eventually finish, it could take a very long time.

The fix: using /dev/urandom instead. It is "not quite as good" random data because the output may have less entropy than /dev/random, and it uses internal entropy bits multiple times to generate it's output, but it's "good enough" for generating cryptographic keys. And, it is non blocking which means that a caller will never have to wait inane amounts of time for enough "random" data. (See http://en.wikipedia.org/wiki//dev/random for a longer explanation.

I replaced the two occurrences of /dev/random, one in /etc/openwsman/ssleay.cnf and one in /etc/openwsman/owsmangencert.sh, and initial startup of openwsman (including key generation) became pretty instantaneous. "chkconfig --levels 2345 openwsmand on" to turn it back on, and a reboot (after removing the generated keys and certs) to confirm, and the machine booted up as expected. To make this work everywhere, I customized those two config files and added them to our Puppet system so that all Dell servers would get Openwsman set up properly when the update is run globally:

 file {
    "ssleay.cnf":
      path => "/etc/openwsman/ssleay.cnf",
      source => "puppet://$server/dell/ssleay.cnf",
  }
 
  file {
    "owsmangencert.sh":
      path => "/etc/openwsman/owsmangencert.sh",
      source => "puppet://$server/dell/owsmangencert.sh",
  }

Problem solved and all machines will automatically get the correct fix, so the next time a machine won't finish starting up, it will be a new and different problem to debug.

I've been burned too many times by hardware failures, so I keep all of my data on a pair of hard drives in my desktop mirrored with RAID-1. (Note that I'm not relying on this for backup, I have backups onsite and off, but it takes a long time to restore 100s of GB of data. This mirror is to prevent downtime when hard drives fail.) I started off with 2 300G drives several years ago, and set up a new mirror around 3 years ago with 2 750G drives, but the photos I take and videos I record add up pretty quickly. It was almost time to scale up some more when I stumbled across a great deal online and got 2 1.5T drives for about $160. Sunday, I finally got around to migrating things over and instead of building a new array and copying everything over, I tried something a little different that worked pretty well.

1. Shut down the machine and pull out one of the 750G drives and replace it with a 1.5T drive and turn things back on. Alternatively, if you have the space, you can put all of the drives in a machine to avoid a reboot or two.
2. Format the new drive using fdisk and use mdadm to add the new drive to the array. Replace "sdf1" with whatever the name of your new partition is and "md0" with whatever the name of your raid device is.
   

   # mdadm /dev/md0 --add /dev/sdf1

   If you have all the drives in the system, you'll need to fail out and remove an existing one first with something like

   # mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1

3. Run
   

   # watch cat /proc/mdstat

   so you can see when the array is finished rebuilding. (this took ~4 hours on my machine)

4. Repeat the same process, replacing the other 750G with the other 1.5T drive
5. Once both 1.5T drives are in the array and it's completely rebuilt (another 4 hours), it will still show up as a 750G RAID-1. Next, bump up the size of the array to the max size of the drives with:
   

   # mdadm --grow /dev/md0 --size=max

6. Again, run
   

   # watch cat /proc/mdstat

   until the array is finished growing. (another 4 hours)

7. Finally, it's time to resize the filesystem. This may be different if you're using an old version of Linux or a filesystem other than ext3, but on ext3 it's very straightforward and the filesystem can actually be resized while it is mounted and in use:
   

   # resize2fs -p /dev/md0

   That will grow the filesystem to fill the whole array and when this finishes in about 15 minutes you'll have a shiny new RAID-1 with a lot more space:

   # df -h /dev/md0
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/md0              1.4T  607G  699G  47% /mnt/storage

OpenVZ seems to be the hot open source container based virtualization tool these days. Instead of tools like VMWare and Xen which virtualize the hardware and allow each guest operating system to run their own kernel, OpenVZ uses operating system level virtualization. While less flexible and less "secure" in some instances, this allows for better performance of the guests due to lower overhead.

I've been tinkering with using OpenVZ for a project to provide rapidly deployable emergency copies of infrastructure for situations where the primary and secondary hardware go down (DNS servers, LDAP servers, etc). OpenVZ meets the need here because it has command line management tools, is low overhead, and these kinds of services don't depend on a specific kernel or hardware stack as much as some others might. The tricky part for me is that some of these services live on separate VLANs.

In this setup, each machine (including the OpenVZ host) has two Gigabit Ethernet interfaces bonded together to two ports on separate switches that are stacked together. This provides higher throughput and prevents interruption of service if a switch, cable, or interface fails. The hosts typically don't know about VLANs and the interfaces on the switches are in access mode which automatically tags all traffic to the proper vlan. However, the OpenVZ host will need access to multiple VLANs so that it's guest machines can get to the right places on the network, so some things need to change. It will need it's own VLAN as well as the VLAN for each guest machine.

Firstly, the switchports are configured to trunk the right VLANs to both of the ports that the OpenVZ host is plugged into. Note that if you do this, you'll loose access to the machine so make sure you're connected out-of-band to the console! On the switch in a config shell (Cisco IOS example):

# interface Gi1/0/1
# switchport trunk encapsulation dot1q
# switchport trunk allowed vlan 10,20-30
# switchport mode trunk
# interface Gi2/0/1
# switchport trunk encapsulation dot1q
# switchport trunk allowed vlan 10,20-30
# switchport mode trunk

Then the OpenVZ machine is configured to support VLANs by adding kernel modules and creating a new interface. Note that these instructions are for RHEL5/CentOS5:

1. Add "modprobe 8021q" and "modprobe vzethdev" to /etc/rc.modules
2. chmod +x /etc/rc.modules
3. Manually run /etc/rc.modules. It will be automatically run when the system boots
4. reconfigure the /etc/sysconfig/network-scripts/ifcfg-bond0 to have no IP or BOOTPROTO information, "ONBOOT=yes" and "MODE=trunk"
5. create /etc/sysconfig/network-scripts/ifcfg-bond0.10 like the following:
   

   DEVICE=bond0.10
     IPADDR=10.0.10.2
     NETMASK=255.255.255.0
     GATEWAY=10.0.10.1
     NETWORK=10.0.10.0
     BROADCAST=10.0.10.255
     ONBOOT=yes
     BOOTPROTO=none
     USERCTL=no
     VLAN=yes
     PHYSDEV=bond0
   

6. and load the interface with "ifcfg bond0.10 && ifup bond0.10"
7. make sure that proxy_arp and forwarding are enabled for bond0.10 in /proc/sys/net/ipv4/conf/bond0.10/. If not, you should reconfigure your system to set these by default. Consult your operating system documentation for instructions on this.

Once this is done, you should be able to use this host on the network (on VLAN 10) like nothing changed! If not, make sure routes are set up right, ifconfig looks right, etc. Assuming it works, you're halfway there! Up next is creating an interface for each vlan you want mapped. Here's an example for /etc/sysconfig/network-scripts/ifcfg-bond0.20 on VLAN 20:

DEVICE=bond0.20
  ONBOOT=yes
  BOOTPROTO=none
  USERCTL=no
  VLAN=yes
  PHYSDEV=bond0

Note that it doesn't have any IP information. We'll specify this inside of the OpenVZ instance. Next, we actually create a blank OpenVZ instance (you can use an existing one, but this is provided for completeness sake) and give it an eth0 interface. I'm using 20 as the ID here because this instance will be on VLAN 20, but this is not a requirement.

vzctl create 20 --ostemplate centos-5-x86_64-default-5.2-20081013 --config vps.basic
vzctl set 20 --onboot no --save
vzctl set 20 --hostname vlan20host.local --save
vzctl set 20 --numothersock 120 --save
vzctl set 20 --nameserver 10.0.10.1 --save
vzctl start 20
vzctl set 20 --netif_add eth0 --save

On each host, use "eth0" as the name of the interface. OpenVZ will automatically create the eth0 interface in the guest and an interface like "veth20.0" on the host where "20" in the name represents the guest ID and the .0 indicates that this is the default interface for the guest. You could add an eth0.21 interface to the guest with vzctl if you wanted VLAN 21 also piped into the guest, which would create a eth0.21 on the guest and veth20.21 on the host.

Now that it has the interface, enter the instance with "vzctl enter 20" and set up it's networking by creating /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
  IPADDR=10.0.20.1
  NETMASK=255.255.255.0
  GATEWAY=10.0.20.1
  NETWORK=10.0.20.0
  BROADCAST=10.0.20.255
  ONBOOT=yes
  BOOTPROTO=none
  USERCTL=no

Then "ifcfg eth0 && ifup eth0". You won't be able to send traffic yet, but configuration inside of the guest here is done. Head back to the OpenVZ host and set up a bridge to connect things. Here, we name the bridge so that it's recognizable as this VLAN and add the VLAN interface and OpenVZ host interface to it. Making one bridge per VLAN is probably the right thing to do, and if multiple guests are on the same VLAN, just add their host interfaces to the same bridge.

brctl addbr vzbr20
brctl addif vzbr20 bond0.20
brctl addif vzbr20 veth20.0
ifup vzbr20 0

Again, make sure that forwarding and arp_proxy are enabled for the bridge and the veth20.0 interface (created by adding eth0 to the guest). And that's it! You should be able to ping the guest's gateway from the guest. If you can't, run a ping from the guest and run tcpdump from the host to see where packets stop, looking at interfaces in this order:

veth20.0
vzbr20
bond0.20
bond0

Whichever one it stops on, make sure you have forwarding and proxy_arp enabled, and make sure that the bridge has the two things it should as members with "brctl show"

Each time the vm is rebooted, you'll need to add it's host interface to the bridge again and may need to re-enable forwarding and proxy_arp depending on how your OS is configured. If you are using a newer version of OpenVZ (>3.0.22) there is a easy workaround for this on the Veth wiki page. There is a more complex workaround for older OpenVZ versions there as well, but I should be using the newest version of OpenVZ when it's time to deploy this thing so I'm waiting!

Hopefully you find this useful, please let me know with a comment on the original article at ckdake.com or via an email if you have any suggestions, comments, or corrections!

Helpful References:
http://wiki.openvz.org/Veth
http://wiki.openvz.org/Venet
http://www.howtoforge.com/installing-and-using-openvz-on-centos5.2-p2

As I've noticed from watching hits on my site here, many of you have read my page on PHP security using mod_fastcgi and suexec. The logic on that page still holds, but Gentoo decided to make the switch from mod_fastcgi to mod_fcgid and it broke all sorts of things for me. I got things scratched back together without any security on my old server, and with the installation of my new server a few weeks ago, I set things up more securely again. I still think this way is the way to go for a server where many of the virtual hosts will seldomly see traffic, but if you're running lots of high traffic sites and have a little bit of RAM overhead, you might want to check out this article on mpm-peruser.

For this setup, I decided to stick to some standards. This means no more changing the suxec directory, using /data/, or anything like that. Other than that, the key differences from last time:

• All configuration is now done with with a setup script instead of using a mysql database. There was not really any point for the host names to be in a database, and it makes setup/teardown scripts easier to write as just a bash script.
• Some hosts have PHP, some don't, so no point in setting up all the overhead if a host isn't going to use PHP.
• Most hosts won't have any interest in having their own logs. Statistics can be done using client side things such as Google Analytics, and Apache is happier writing all the logs to 1 place instead of hundreds. I also have split-logs running when logs are rotated, so logs can easily be gathered per-site as needed, just not real time by one of my hosting customers. I've never known of one of my customers using live access to their logs.
• php.ini files are now stored with the wrapper script in the site's cgi-bin directory and file system extended attributes are used to protect it. This means no separate home for php.ini files, and it's easier for users to see what their PHP confguration is.

The script isn't quite ready for sharing yet, but here's what you can do to get a setup like this:

1. on Gentoo, make sure your USE contains: suexec, apache2, cgi, fastcgi, session.
2. on Gentoo, "emerge apache php mod_fcgid". On other platforms, consult your docs (or just download mod_fcgid and use apxs to install it. it should be pretty seamless)
3. Set up your global configuration. On Gentoo, this is done for you, but make sure this gets loaded into your global apache configuration:
   

   LoadModule fcgid_module modules/mod_fcgid.so
   SocketPath /var/run/fcgidsock
   SharememPath /var/run/fcgid_shm
   
   <Location /fcgid>
   SetHandler fcgid-script
   Options ExecCGI
   allow from all
   </Location>
   

4. Add a user and group for your first virtual host, test.example.com. call em "example" if you like
5. Set up the directory tree for the virtual host:
   

   /var/www/test.example.com/
   /var/www/test.example.com/tmp
   /var/www/test.example.com/htdocs
   /var/www/test.example.com/htdocs/cgi-bin
   

6. Make some files:
   

   
   hello HTML world!
   

   <? 
   /* /var/www/test.example.com/test.php  */
   print("hello PHP world!");
   ?>
   

   #!/bin/sh
   # /var/www/test.example.com/htdocs/fcgi
   PHPRC=/var/www/test.example.com/htdocs/cgi-bin/
   export PHPRC
   PHP_FCGI_CHILDREN=2
   export PHP_FCGI_CHILDREN
   PHP_FCGI_MAX_REQUESTS=25000
   export PHP_FCGI_MAX_REQUESTS
   exec /usr/bin/php-cgi
   

7. Copy your php.ini to /var/www/test.example.com/htdocs/fcgi and edit it so that directories are right. All you'll likely need to change is upload.tmp_dir and session.save_path, but you may want to change others.
8. Set fcgi to be executable, and make sure permissions are set on it so that it is owned by your test user/group and other users can't mess with it. If things don't work later, this is a frequent culprit
9. Set the immutable bit on php.ini and fcgi (you'll need to be using extended file system attributes on your filesystem to do this, check your OS documentation for details) by running 'chattr +i /var/www/test.example.com/htdocs/*'. You'll need to undo this with chattr -i if you want to change these files in the future.
10. Set up this host's configuration:
    

    <VirtualHost *:80>
            DocumentRoot /var/www/test.example.com/htdocs/
            ServerName test.example.com
            SuexecUserGroup example example
            <Directory /var/www/test.example.com/htdocs/>
                    Options +SymLinksIfOwnerMatch
                    AllowOverride All
                    Order allow,deny
                    Allow from all
                    DirectoryIndex index.html index.php
                    AddType application/x-httpd-fastphp .php
                    Action application/x-httpd-fastphp /cgi-bin/fphp
            </Directory>
    
            <Directory /var/www/test.example.com/htdocs/cgi-bin/>
                    SetHandler fcgid-script
                    FCGIWrapper /var/www/test.example.com/htdocs/cgi-bin/fphp .php
                    Options +ExecCGI -Includes
                    allow from all
            </Directory>
    
    </VirtualHost>
    
    <VirtualHost *:80>
            ServerName aerospace.com
            Redirect Permanent / http://test.example.com/
    </VirtualHost>
    

11. Give apache a restart and that should be it!

Check out the processes running on your server, and after you hit test.php you should see a php-cgi process running as the example user. If you have problems, error_log and suexec_log in /var/log/apache2/ (or /var/log/httpd/) tend to tell you everything you need to know.

An oh yeah, want to use APC to speed up your PHP applications significantly under this setup? Just install APC, then add the configuration for it to the bottom of the php.ini for any hosts that you want to enable this on. Given that APC isn't 100% perfect and crashes sometimes, the beauty of the fcgid setup is that it will take out the php-cgi process and the fcgid manager will just start a new one like nothing happened.

Something very weird happened this morning. I was checking email and all that sort of thing and turned on some music but there was no sound coming out of the sub. It was working fine yesterday and its a pretty nice setup so that was just odd. The sub has two cones in it and neither one was moving at all. I moved around some wires on my computer yesterday and plugged a paper shredder into the same outlet so i thought that might be related somehow so I plugged the line in into my ipod and put it on a different circut. Still no bass. Next step was opening the thing up and checking resistance across the speakers and driver circuits on the amp for it. (It's a BASH amp and it's pretty neat looking compared to a lot of other amps I've taken apart) Everything looked fine, but I notice that one of the wires to one of the cones was actually unplugged! I plugged it back in, plugged the power back in and it worked. How on earth did that one wire behind 9 screws unplug itself sometime when I was asleep? Crazy.Back to listening to the new Thrice CD: Vheissu. It's amazing and "Atlantic" on it may now be in the top 25 songs for me.Additionally I am no longer using Rhythmbox to play music. There's apparently a bug that causes gamin (A tool that informs userspace applications when a file on the filesystem changes using the inotify support in the kernel instead of constantly polling the directory for changes like the old "famd" did) to use up lots of cpu when Rhythmbox is running. The possible bugfix to completely disable music library monitoring is in CVS for them but hasn't made it into a release yet apparently, so hopefully 0.9.4 or 1.0 will fix it. We'll see. I'd like to keep using it as it has pretty good integration with Gnome and will mesh with my iPod nicely when support for that makes it into a release. Also, Songbird isn't out yet but if they get Linux support before Rhythmbox puts out another update, it may be time for a switch.