I carry a GPS with me on long bike rides and pull the resulting trace into Google Earth and Garmin's MapSource software. Google Earth is nice for looking at, but doesn't provide much useful information, and MapSource is pretty awful to look at (and will only run in Windows so I have to boot up VMware) but does provide elevation maps (as well as the ability to load maps). I recently started using a bike computer with cadence, and a heart rate monitor, and the last missing piece of information was total elevation gain over a ride. This information is nowhere in MapSource or Google Earth.

I can get GPX format (The standard interchangable format for GPS information) files out of MapSource and it's just XML, so after trying several tools online and several programs I downloaded that didn't work, I wrote a quick python script to get me the info I want. Hopefully this will help someone else:

from xml.dom import minidom

file = minidom.parse('./file.gpx')

min = 1000000
max = 0
gain = 0
loss = 0
last = 0

for node in file.getElementsByTagName("ele"):
        cur = float(node.childNodes[0].data)
        if (cur > max):
                max = cur
        if (cur < min):
                min = cur
        if (last != 0):
                if (cur > last):
                        gain = gain + (cur - last)
                elif (cur < last):
                        loss = loss + (last - cur)
        last = cur

print "max: %.2fft" % (float(max * 3.2808399))
print "min: %.2fft" %  (float(min * 3.2808399))
print "gain: %.2fft" % (float(gain * 3.2808399))
print "loss: %.2fft" % (float(loss * 3.2808399))

So for my 43 mile ride on sunday:
max: 1110.63ft
min: 773.16ft
gain: 3328.98ft
loss: 3232.78ft

Getting those numbers were a lot harder than it should have been! Good ride though..

I was going through some drawers and stumbled across my good old Griffin PowerMate that I got back before I started using Linux. It controlled iTunes in Mac OS 10.1 and was great because I could change volume and pause music without having to change programs or anything. These days I use Rhythmbox in Linux to listen to music and theres not a plugin for it. Yet!

Rhythmbox supports plugins written in python, a guy has some skeleton python code for talking to the powermate, and that means something could work out!

I got the powermate working by compiling and loading the powermate module for 2.6 linux kernels (In 2.6.23 it's in Device Drivers -> Input device support -> Miscellaneous devices -> Griffin PowerMate and Contour Jog support), adding a udev.d entry:

# /etc/udev/rules.d/45-powermate.rules
KERNEL=="event*", SYSFS{product}=="Griffin PowerMate", NAME="powermate", GROUP="users", MODE="0660"

I plugged it in, catted /dev/powermate, and with each twist or push it spit out garbage to the screen. Success!

A quick glance through everything shows that Rhythmbox doesn't support threads and the python code here uses polling so I'd need to delve into the Rhythmbox docs to figure out the best way to do that, but Rhythmbox also exposes itself through DBus and there are some examples of using this around the internet. In a few minutes, I hacked together something dirty to cover the basics and perhaps later on I'll make something that works as a Rhythmbox module. Right now pushing the button is play/pause, turning it adjusts the volume, and the LED shows volume when playing and pulses slowly when paused. Here ya go:

#!/usr/bin/python

import powermate
import dbus

EVENT_BUTTON_PRESS = 1
EVENT_RELATIVE_MOTION = 2

DBUS_START_REPLY_SUCCESS = 1
DBUS_START_REPLY_ALREADY_RUNNING = 2

bus = dbus.SessionBus()
(success, status) = bus.start_service_by_name('org.gnome.Rhythmbox')

proxy_obj = bus.get_object('org.gnome.Rhythmbox', '/org/gnome/Rhythmbox/Player')
         
player = dbus.Interface(proxy_obj, 'org.gnome.Rhythmbox.Player')

pm = powermate.PowerMate("/dev/powermate")
while 1:
	event = pm.WaitForEvent(-1)
	if (event[2] == EVENT_BUTTON_PRESS and event[4] == 0):
		player.playPause(1)
		if player.getPlaying():
			pm.SetLEDState((int)(player.getVolume() * 255), 0, 0, 0, 0)
		else:
			pm.SetLEDState(255, 252, 1, 1, 1);
	elif (event[2] == EVENT_RELATIVE_MOTION and player.getPlaying()):
		player.setVolumeRelative(event[4] * 0.02)
		pm.SetLEDState((int)(player.getVolume() * 255), 0, 0, 0, 0)

Download powermate.py and the code above, save the code above as whatever.py, run it, and you'll be able to control rhythmbox with your PowerMate in Linux!

First of all, at Google it's another day, another t-shirt. I added a partial album of Google T-Shirts from this summer so far. I haven't even managed to take pictures of all of em yet..

But on to this SWIGged C thing. I was fighting with another very strange problem in some code I was working on. I'm writing all of my code in Python, but a lot of the things that I need to use are in C++. Conveniently, there is a nifty tool called SWIG that lets someone expose C++ and C objects and methods to "a variety of higher level programming languages" of which Python is one of. (Cool!) So there was some C++ code I needed to use and a quick Python loop later, I thought I had something written that would create all of the C++ backed objects that I needed. Each C++ object had 3 other C++ objects assigned to it in Python. All sounds believable and looked like perfectly good code to me, but something wasn't right. When running it I would get strange core dumps that didn't make any sense, and the 3 people I go to at work to ask Python questions were all baffled. In addition to just crashing, very strange things would happen like when trying to reference an object, I would get back the contents of one of the sub objects of the other objects some of the time and nothing other times. I fought this most of the day, trying wrapper classes, using a create method instead of doing the work in a loop, etc, but nothing seemed to work. I thought it might be an issue with variable scoping, but Python isn't stupid and a variable private to a method is private to that method, etc. After a suggestion from one of the guys I work with, I created a global array and used a counter in the loop separate from the SWIGged objects and indexed the objects in the array with that. My manager agreed with me that this was quite a goofy solution and sent me down another path of figuring things out.

Turns out, the problem was my lack of understanding how swig ties in with C++ and Python memory management. Either C++ or Python "owns" an object and if Python thinks it owns the object (It did because I created it in Python) and thinks it no longer has any references to the object(I wasn't using the C++ objects outside of that loop as far as Python knew because of threading crazyness), garbage collection will come by and trash the object, and eventually something else will use that chunk of memory (hence my crazy crashes and things showing up in odd places). There is a mention of object ownership and destruction for SWIGged objects in Python in the SWIG documentation, but I was new to all of this (I didn't even know what SWIG was or that I was using it at the start of the day!) so I ended up needing some pointers to the right place to look from someone that had seen this before. The solution: After creating the object in Python, just calling myObject.__disown()__. If things continue at this rate, I'll know more about Python by the end of this summer than I do about Perl (after spending 4 years teaching myself) and PHP (after spending 7+ years using it myself and 3+ years managing Gallery where we do things like beta test versions of PHP before they are released, use unit testing, and have paid security audits that I get to read.).

Apparently people are reading this because I got enough comments about me not having updated all week and me having "stopped blogging" that I'm here writing something about something.

So a while ago I wrote a little tool to parse my log files from Gaim and make a pretty graph: pretty talking graph. It shows how many lines I've talked to people on AIM in all of my log files, was a bit of a pain to write, and now that it's been around a few years, I look in the code and wonder what some parts are doing. Plus, I wanted newer, cooler, graphs and something that would play nicely with other programs (like Adium on the Mac), and I've been doing a lot of Python at work, so it was time for a rewrite.

So I wrote a Python tool that crawls all of my logs and puts them in a database, and a graphing tool that does things on the database and spits out html for all kinds of graphs. It runs automatically every night and makes a page with all the details: pretty talking graph page. Everything there is done with HTML/CSS (Thanks to the trends graphs on Google Reader for inspiration) and I think it's pretty cool. More graphs in the future (like rates of change, and predictions for when people will pass other people, etc)! I'd also like to figure out some way to graph some e-mail things... We'll see what I can come up with.

I was doing some tinkering around at work in Python to familiarize myself with the way Python works on threads. Why threads? I'm working on something that needs to do lots of little independent things all at the same time and I figured Python would be a better language to use this summer than C++ because I've done so much Perl programming recently (and I had a heck of a lot of trouble just getting a simple "hello world" program to compile and run in C++ with the Google build system!). I hacked out some toy Python code to see how well the thing performed and all I could manage was 100% of one core on one cpu. Uh... With other languages on other boxes I'd been able to peg all the cores on all the CPUs, so it was time for some research. Apparently, Python has a massive lock in the interpreter called the GIL (Global Interpreter Lock). This lock is because not all of python is thread-safe and there are bad things that can happen when multiple threads try to access something non-thread-safe at the same time. The effect of this lock is that, even when using threads, Python is only really doing one thing at a time and can thus only use (the equivalent of) 1 CPU.

At first I was pretty annoyed by this because it sounds ridiculous for a modern programming language to have such a limitation, but after some reading around online I've come to a different conclusion. An e-mail on a mailing list by Guido (The creator of Python and a fellow Googler) got me thinking that threads might not actually be the best way to do things. Each thread has overhead of data structures and with each thread you run into more context switches that are required for your program to run for some amount of time. With thousands of threads I'd be wasting all kinds of CPU cycles! As multi-CPU and multi-core machines are becoming more and more dominant, programmers (like me) need to think about more effective ways to make use of the resources available to them. In my particular situation, running several separate processes that communicate with IPCs and/or shared memory makes a lot more sense. Each process can handle some portion of the independent actions, but can do them serially per "cycle" so things get done in the same amount of clock, but the CPU isn't trying to do thousands of things at the exact same time (and multiple CPUs can be used so more clock time can be used in less wall time). For very simple operations, this saves all kinds of CPU, but it's still useful for more complicated operations.

Python has been discussed as being a lot like Perl but only having one right way to do things, and the consequences of the GIL is an example of that. The right way to use up all the CPU on a machine is to do something other than threads! This should be pretty neat and gives me reason to learn about some parts of Python that are completely new to me. (I never did any IPC/shared memory in Perl or C so this will be completely new!)

Apparently there was some work done on removing the GIL but because of the much finer granularity of locks, it slowed Python down up to 2x on single-CPU machines. Ouch! This was a problem with the first OSs to support SMP, and they got around it by shipping a thread-safe OS and a threadless OS separately. They could do this with Python but it would me a lot more maintenance overhead for the language and it gets messy real quick when people accidentally mix thread-safe and threadless code together. Google around for "GIL" if you're interested in this. Theres a lot more that I read and found interesting!

Working at Google, I've had to start using Python. It's their scripting language of choice and even though all of my scripting experience is in Bash, Perl, and PHP, too bad! Using a new language isn't too bad because languages are just languages and once you know a couple you know them all, and Python isn't really any exception. I've had to search around for the basics like string operators, objects, and so on, but it's not to bad. I've probably written about 500 lines so far and things are doing what they are supposed to do. However, today I ran into something that stumped me for a while.

My current project had me making a tree like data structure (below is an example of something similar). Think boxes containing some number of smaller boxes which contained some number of smaller boxes, etc, which finally contain some number of tiny items in the smallest box. I set up all the classes to do this as well as methods to build test ones of arbitrary size and run some algorithms on it, but something weird was going on.

class Box:
  items = []

a = Box()
for b_label in range(2):
  b = Box()
  a.items.append(b)
  for c_label in range(2):
    c = Box()
    b.items.append(c)
    for d_label in range(2):
      d = Box()
      c.items.append(d)

for each in a.items:
  print '%s' % each

I expected this to print out the addresses of the two boxes in box a, however, it prints a list of all 14 boxes. (further exploring the data structure created shows me that each of the boxes at each level of the tree has each other box in it, totaling 2744 boxes at the inner most level. Uhm... there should only be 14 boxes! I tried all kinds of things, using a dictionary instead of an array for items, using __new__ and __init__ constructor code in the object, including a random string in the name of each object, including a random number in the constructor of the object that it stored to guarantee uniqueness, etc. But nothing worked! After spending too much time on this, I asked my Host (Google's word for the person interns work for) and she said it looked fine but to ask someone that knew Python better. Voila! The solution was so simple it would have been almost impossible to find searching for it. My usage of items was as a class variable. Even though it was inside the class, it acted as a singleton variable across all instances of the box object and was thus shared in each box. Every time anything was added to any box, it was added to all boxes instead of just it's parent box. The solution: declare the items variable in the constructor instead of the class so it would be scoped to the instance of the object:

class Box:
  def __init__(self):
    self._items = []

Problem solved! So if you're new to python and playing with things, make sure you scope everything properly! According to the guy that helped me out, he's pretty solid at python and still makes this mistake every now and then and it's very hard to track down.