Highgroove Hack Night: KML Heatmaps

Every month, Highgroove hosts a hack night where we order food, stock up on beer, and invite anyone and everyone to come hang out in our office to work on cool projects. Last night was a pretty busy one with a handful of open-source gems getting updated, and between helping out other people and building Space Cthulhu I played with turning GPS traces of bike rides into a heatmap.

First up was getting coordinate data for the heatmap out of Google Earth. I put together a quick script using nokogiri and ruby to grab the coordinates from a kml file and output them to CSV:

require 'nokogiri' # gem install nokogiri
@doc = Nokogiri::XML(File.open("my.kml"))
@doc.css('coordinates').each do |coordinates|
coordinates.text.split(' ').each do |coordinate|
(lat,lon,elevation) = coordinate.split(',')
puts "#{lat},#{lon}\n"
end
end
view raw mykml2csv.rb hosted with ❤ by GitHub

This gets run like:

ruby mykml2csv.rb > coords.csv

Then, I used heatmap.py in python to generate both an image and a KML overlay:

import heatmap # http://jjguy.com/heatmap/
import random
import csv
if __name__ == "__main__":
pts = []
for point in csv.reader(open('coords.csv')):
pts.append((float(point[0]), float(point[1])))
print "Processing %d points..." % len(pts)
hm = heatmap.Heatmap()
hm.heatmap(pts, "heatmap.png", 25, 128, (8192,8192))
hm.saveKML("heatmap.kml")
view raw csv2map.py hosted with ❤ by GitHub

This gets run like:

python csv2map.py

For my 2007 kml this was only ~3000 points and worked pretty quickly, but for 2008 there are ~60k points and I had to shrink dot size and bump up output size pretty significantly to get it to finish. csv2map.py ran for ~12 hours and finally outputted a pretty neat image:

Looks like I rode to Stone Mountain a lot in 2008. This could probably be generated in a few seconds with a more performant heatmap library, so perhaps I’ll hack one of those together in the future. That said, this was a fun proof of concept and may be useful for people with smaller KML files than me! My entire my.kml has over 1 Million points, so it may never build a heatmap of all of them.