Another Chapter?

2008-09 – GeoEstimator is born/takes off.

05/2011 – Pictometry acquires GeoEstimator (press release)

01/2013 – Pictometry merges with EagleView (press release)

01/2014 – Verisk acquires Eagleview (press release)


Javascript Mathematics Libraries… GO!

I needed to look at the performance of multiple different Javascript Math libraries, specifically around Matrices. So, I’ve created numerous tests to compare them on different items. I’d originally started with 3 libraries:

Google Closure:

After learning to use the libraries, comparing the performance, and just generally using them I parred it down to 2: Sylvester & Numeric. Sylvester is on the far OOP side of design, fluent interface, pretty good looking and readable code. Numeric is on the completely opposite end of the world with all static calls, no types just arrays, very functional looking, with code that is nearly IMPOSSIBLE to read. Google Closure on the other hand is the worst of both worlds. It’s OOP but the different libraries don’t play nice with each other (e.g. Vec3, Matrix) while it’s not easy to read the code and rather tedious in general. So, most of my later tests leave out gc.

Identity Matrix:
4×4 * 4×1 Multiplication:
Single 3D Transformation:
500 3D Rotates (T,R,T) w/ Unrolled:
1000 3D Rotates (T,R,T) w/ Unrolled:
1000 3D Rotates (T,R,T) w/ Unrolled not building Matrix:
Row Access:
Col Access:
4×4 & 3×3 Inversion:×4-3×3-inversion

When I got to the Row access, I thought things were going to be ‘similar’. MAN, were they not. Numeric kicked Sylvesters butt, like 100 to 1 style. I decided to look into it a bit and found that quite a bit happens on Matrix/Vector creation in Sylvester. Inputs are checked, arrays are copied/sliced/created, and all around good stuff when dealing with outside input. By simply removing all of this ‘stuff’ and simply creating a Vector and settings it’s elements to a known ‘good’ array from another matrix I was able to improve Sylvesters time by ~10x. That still left it years behind Numeric on row access so I dug deeper. Turns out it’s simple prototypal object creation that’s slowing Sylvester down. The vector constructor does absolutely nothign code-wise, simply instantiates an object. Simply putting ‘new Vector();’ in the numeric tests slowed down them down put them right on par with Sylvester. Leads me to believe that the speed of complex object creation is a big part of what is perceived as ‘slowness’ in Sylvester.

Simplified Constructor:

At the end of the day I’d recommend Numericjs for ‘raw performance’ and Sylvester for about everything else. Sylvester is easier to understand, has more implementation around common 3D constructs, is readable, and all round ‘nicer’. Numericjs cuts ALL cruft, is horrible to read, not nice at all, but is fast. Did I mention lots of variables named x,y,z (and not x, y, z coordinates) and functions named foo, dim, and other such worthless names? Did I mention that a LOT of the code is actually creating javascript on the fly with string concatenation?

Defining ‘Technical Debt’, breaking down agile, and how it all fits together

Here are some links to the slidedeck and other items of interest in relation to my talk on May 16th, 2013 at the Indy Software Artisans meetup.

My Slidedeck:
Technical Debt & Craftsmanship

Slides for most of the speakers at the Agile Indy Conference:

Ward Cunninghams Correction on Technical Debt:

Ron & Chet giving the same talk @ COHAA:

Cynefin Framework overview:

Monitoring MySQL replication with Nagios.

Fork me on GitHub
This isn’t a post about setting it all up. That would require quite a bit more space/time than I have as Nagios is fairly complex. I just wanted to let people know about some changes I’ve made to an existing package.

I found the check-mysql-all scripts on google code that already contained a number of nice scripts for monitoring with Nagios. What it didn’t support was the mk-heartbeat command in it’s check_maatkit command. In my github repo I’ve added this support. We’ll see if it ever makes it into the google code repo. Not having easy push/pull requests sucks! (maybe google code does if I used it?)

Handling Master/Slave replication servers in a Zend MVC app

Fork me on GitHub

Increasingly, our mysql db has been the bottleneck on performance in our application. After researching, looking around, weighing the options and all that jazz we decided to look at adding read-only slaves to our existing HA (drbd style) mysql server. When looking for solutions in Zend that would help to take advantage of this scenario I found a few projects out there but all they appeared to be were simple data structures to create, store, and retrieve Zend_Db_Adapters. This would require a code change everyplace the database is used to get either a master (update,insert,delete) or a slave (read).

I wanted plug and play, so I created the new Zend_Db_Adapter_Replication. It simply takes 2 adapters, anything implementing Zend_Db_Adapter_Abstract should do. From there it uses the correct connection depending on what operation you are running. If the update/insert/delete functions are called it passes them through to the master. If select is called it goes to slave. If the more generic query is called it looks at the query to determine which connection to use. If you start a transaction, it assumes you want master, and it forces everything else that happens inside that transaction to happen on the master.

This was enough for me to simply replace our current db in the Zend_Registry with the Replication db and everything ‘just worked’. You can watch munin/nagios/mysqladmin and the distribution looks perfect.

At the time of writing there’s no way to get an individual connect simply because I’ve not needed it. There’s also no way to force it one way or the other besides using transactions, again because I’ve not needed it.

So need it now? Where is it? Get the code on github!

behat – BDD for PHP

I’ve looked at implementing acceptance testing style tests for our website with cucumber/ruby in the past but never pulled the trigger due to the fact that I live in a PHP world at work. Any ruby setup code that was written for the step definitions would most likely be duplicate code that we already have written and support in PHP.

Recently I became aware of a BDD framework, nearly identical to cucumber, for PHP named behat. It uses a PHP cloned gherkin parser so the features themselves are nearly identical to cucumber

Feature: User sessions
In order to access their account
As a user
I need to be able to log into and out of the website

Scenario: Login
Given I am on "/account/login"
And I should see "Login"
When I fill in "username" with "username"
And I fill in "password" with "passwd"
And I press "Login"
Then I should be on "/account/my"
And I should see "Hello Mark Garringer"

Behind the scenes the step definitions are similar. Due to language limitations the regular expressions for matching are in doc blocks on methods rather than method names but very workable. I’m not sure I’m a fan of the more inheritance based nature of their step definitions but it does the job and doesn’t get in the way.

Much like cucumber out of the box it doesn’t do web, but instead you use a driver that does web. In this case it is Mink (by the exact same people). Mink provides headless (javascript-less) browsers for quick tests as well as drivers for selenium and sahi for testing javascript enabled pages.

All in all, what testing I have done, has proved very promising. The documentation is pretty good although the ‘google factor’ seems fairly low. Finding examples and help outside of the website and project resources is fairly hard. I’ve implemented some rather in depth tests putting it through it’s paces and it has worked like a champ. It suffers from the same javascript timing issues that other such frameworks suffer from but it does what it says, and does it well.

Most importantly it can output in junit format so integrating it with our build server and getting feedback was a cinch!

Behat FTW!!

Changing Default Column Values

A note to self so I remember when next I need this:

First, I don’t want to wade into the argument about where default values should, in the database or in the code (see this and this for example).

I left a default value off of a number of fields when I first created them and now I wish that I hadn’t. I found absolutely nothing in the guides on changing defaults and googling turned up nothing as well. Finally digging through the documentation on ActiveRecord migrations I turned up paydirt.

  1. t.change_default(:qualification, 'new')
  2. t.change_default(:authorized, 1)

or a more complete example

  1. def self.up
  2.   change_table users do |o|
  3.     o.change_default :active, true
  4.   end
  5. end

Starting Ruby on Rails: what platform…

Notice: I understand this is nearing religious boundaries, these are just my feelings from my experience..

I’ve been playing around with Ruby on Rails (RoR) for sometime now on the side, looking to get to know it so any ‘new’ projects at work can utilize it’s goodness. The reason for this could be a whole other post, but for time sake we’ll just say that for the last couple years on a PHP/Zend project I’ve found myself striving to implement things that RoR rails provides out of the box. The following is just my experience in learning RoR.

    Windows 7

In my playing I started out on my Windows 7 based desktop and the Windows 7 install on my dual-boot laptop as this was what I was in most of the time for work. My dev environment was pretty simple:

  • Windows
  • Emacs 23.?
  • Multiple modes, flymake, ecb, other emacs goodness
  • git bash (as a terminal)
  • Ruby 1.9.2 w/Devkit (
  • Rails 3 w/ various gems

I will say that it was easy to setup on Windows, no problems there. Some issues that I accounted to Windows were actually issues with most tutorials still being Rails2 at the time and the inclusion of bundler and the need for bundle exec in Rails3.

The reason I would NOT recommend Windows for Rails development is SPEED. If I’d gotten into more depth as far as gems and such it sounds like there might have been compatibility issues as well but I stopped well short of that due to speed. Running 10 unit tests (rspec) would take as much as 30 seconds OR MORE on Windows. Those same tests on other platforms would take less 5 seconds or less depending on circumstances. Starting console or server was just as painfully slow. I’m running an i7 quad core with 6gb of RAM, so it’s not like it was the machine lacking; it’s the most powerful of all 3 machines.

The other negative was no RVM on Windows. I don’t jump ruby versions much but having separate gemsets has turned out to be awesome. I’ve since heard of a windows copy called pik, but I can’t vouch for it.


I also have a mac mini at my desk for previous projects I’ve done for iPhone apps. Setting up my ‘standard’ environment in OSX was actually somewhat of a problem. Turns out that a number of the tools I use are baked into OSX and either upgrading them or replacing them is near impossible. Luckily, as it seems OSX is the platform of choice for RoR development, most of these problems have been overcome. One such tool is RVM. It’s an awesome tool in itself but it’s almost required on OSX if you want to use anything beyond the OSX baked in ruby 1.8.7 from 2009.

Once up and running OSX performed VERY nicely, especially after coming off of the very slow Windows. Having a better command line and more/better tools there is a definite plus as well.

Another negative that I’ve since found deals with the MacBook Pro and it’s keyboard layout. I setup my dev environment on my wife’s MBP and I find it very trying. If you’re an emacs user Ctrl & Alt are more important than the home row to you and your coding. Well, on the MBP, they put the FN key out there where every other keyboard I’ve used puts the Ctrl key and it is MESSING with me. I’ve tried getting used to it but can’t. Fortunately I think you can remap the keyboard, if I could only talk my wife into it…


My laptop has 2 hdd’s and currently it’s setup to dual boot between Windows 7 on one drive and Ubuntu Linux on the other drive. I think this is going to change to be a full Ubuntu install with a VM of windows on the second drive very soon. Setup was much the same as on windows: Emacs, Rails 3, Rails 1.9, etc. The only addition was RVM as with the OSX.

I’d definitely recommend Ubuntu (or linux) for a development environment for Ruby. Easy to install and setup. Very speedy (compared to windows) for running tests, console, and server. Linux in general can have some problems from time to time. Being all open, sometimes things just don’t work as nicely together as you’d want. But it tends to give you may more flexibility.

  1. Ubuntu (Linux)
  2. OSX
  3. Windows

Windows would be the one ‘do not recommend’. It’s doable but painful! The other two have there problems but Ubuntu wins for two reasons: I like Linux, and it’s almost certainly what your app is going to be running on in production anyways! Also, OSX has a rather high barrier to entry (cost), while Ubuntu has become almost stupidly simple to install and the inclusion of an ‘App Store’ makes getting all the big stuff together a breeze. Plus you can try it out inside a VM (like VirtualBox) in like an hour or two for free.

Granted, my post is based off a number of assumptions like using Emacs for an IDE. If I was a TextMate user I’d almost be forced to use OSX. I’d guess there are F/OSS editors that have tried to mimic TextMate you could get on Linux but TextMate itself is OSX.

From Left to Right: Mac Mini (2 screens), Windows 7 desktop, Dual-Boot laptop

But you don’t have to take my word for it!


errors with mysql gem on win x64

Quick information post here. While searching on this issue I saw lots of posts with “I uninstalled the mysql 64-bit version and installed the 32-bit version to make this work”.

It doesn’t have to be like that. Ruby doesn’t care in the slightest what version of mysql it’s talking to, it does so over mediums that are universal.

According to your PATH environment variable part of installing the ruby mysql gem is to add ‘mysql/bin’ to your PATH or to copy the libmysql.dll to ‘ruby/bin’. If you don’t you get errors like ‘The specified module could not be found’. If you do this and you’re running mysql for x64 then you can potentially get errors about ‘%1 is not a valid Win32 application’.

Having had very similar issues with the native libraries and a .NET program, I know that things can’t get weird. I’m going to guess that, like me, you installed a 32-bit version of Ruby, so it gets run in some compatibility mode for 32-bit programs, and all things it directly interacts with (loads), need to be 32-bit. Since you copied a 64-bit version of libmysql.dll it’s not happy. Simply copy a libmysql.dll from a 32-bit installation, I happened to have a 32-bit laptop with mysql installed laying around, and all should work fine.

It can get even more confusing with .NET because if you build your app when choosing the ‘x86′ platform, it works like mentioned above with everything always needing to be 32-bit. If you compile it with ‘any cpu’ selected for the platform it depends on the machine; if it’s 64-bit you need 64-bit things and if it’s 32-bit you need 32-bit.

Fighting with XML Deserialization

Hopefully somebody comes back in the comments with a better solution, as this feels dirty to me, but for someone else looking for “a” method here is what I came up with.

The scenario:
There is this XML document that is already specified and utilized throughout an entire system that contains a Person object. In the persistence framework the coder initially took the lazy route and stored the large binary data for the picture right in row with the rest of the Person data. The class looked something like this:

  1. public class Person
  2. {
  3.      // fields...
  5.      public virtual byte[] Picture
  6.      {
  7.           get;
  8.           set;
  9.      }
  10.      // more properties and other gooey goodness...
  11. }

All was well and good until the system was under any amount of load and the huge image was getting loaded into memory whenever the row was loaded. The OR/M of choice didn’t support lazy property loading and the best long term solution, in any case, was to move the column to it’s own table. Moving into the filesystem, while another choice that was possibly better, wasn’t an option.

So a little coding later, the picture byte[] data was moved into it’s own subcomponent / child object, the OR/M of choice was tweaked to lazy load that particular object via proxy and that left something like this

  1. public class Person
  2. {
  3.      // fields...
  5.      public virtual BinaryData Picture
  6.      {
  7.           get;
  8.           set;
  9.      }
  10.      // more properties and other gooey goodness...
  11. }
  13. public class BinaryData
  14. {
  15.      public virtual ulong Id
  16.      {
  17.           get;
  18.           set;
  19.      }
  21.      public virtual byte[] Data
  22.      {
  23.           get;
  24.           set;
  25.      }
  26.      // other gooey goodness...
  27. }

Easy fix in the persistence side of things, simple refactoring, and you’re off and saving your blobs in a much more performance friendly manner. Except that your simple “XmlSerializer.Deserialize” call is now failing because your Person object now has a different structure that doesn’t match the XML.

After much googling, documentation reading, and hand pounding no solution was found. All kinds of tricks for renaming properties in XML, making properties attributes, and otherwise effecting the XML but nothing in affecting the mapping of one ‘level’ of the hierarchy to another. Only fix found, was to essentially catch an exception which is never good. On the XmlSerializer object there was a “UnknownElement” event that gets fired when unknown elements are found that don’t map to a property.

Final Solution:

  1. public class Person
  2. {
  3.      // fields...
  5.      [XmlIgnore]
  6.      public virtual BinaryData Picture
  7.      {
  8.           get;
  9.           set;
  10.      }
  11.      // more properties and other gooey goodness...
  12. }
  14. public class BinaryData
  15. {
  16.      public virtual ulong Id
  17.      {
  18.           get;
  19.           set;
  20.      }
  22.      public virtual byte[] Data
  23.      {
  24.           get;
  25.           set;
  26.      }
  27.      // other gooey goodness...
  28. }
  30. // elsewhere in the deserializer.UnknownElement event handler
  31. private void serializer_UnknownElement(object sender, XmlElementEventArgs e)
  32. {
  33.     if (e.Element.Name == "Picture")
  34.         (e.ObjectBeingDeserialized as Person).Picture.Data = (Convert.FromBase64String(e.Element.InnerText));
  35. }

Note: This does work if you are deserializing a list of Person objects as well. The e.ObjectBeingDeserialized property is set to the Person object at the time of the event, not the collection that is eventually returned.