BasicDB – An update

It’s been a few weeks since I’ve talked about BasicDB. I’ve been on the road, so I haven’t had much time to hack on it, but this evening I managed finish a pretty nice replacement for the previous SQL parsing and subsequent data filtering code. The old code would simply parse (and validate) the semi-SQL provided through the API and return the parsed query as a list of strings. At that point, I had to re-analyze those strings to make sense of them and apply filtering.

The new SQL parser matches the SimpleDB API much more closely in terms of what’s allowed and what isn’t, and turns the WHERE expression into essentially a tree of expressions that can be easily applied to filter items from a domain. Additionally, constructing nice Javascript code for use in the Riak database turned out to be almost as easy.

As an example, an expression like:

colour == 'blue' AND size > '5' OR shape = 'triangular'

becomes something like this:


I can simply call a .match(item) method on the top level object to check if a given item matches. If you’ve written parsers and such before, this may be very basic stuff, but I thought it was really neat :)

The Javascript code generator follows a similar pattern where I call a method on the top level object and it ends up spitting out a javascript expression that checks whether a given item matches the WHERE expression:

((vals['colour'] == 'blue') && ((vals['size'] > '5') || (vals['shape'] == 'triangular')))

Again, this is probably beginner’s stuff for someone who has written parsers and/or code generators before, but I was pretty happy with myself when all the tests all of a sudden passed :)

4 thoughts on “BasicDB – An update

  1. Tim Bell

    BTW, just a few words of encouragement… it would be great to get an AWS-like BasicDB inside the OpenStack ecosystem… :-)

  2. Christopher Armstrong

    I looked up how secondary indexes work in Riak yesterday. The interface seems pretty awkward since Riak refuses to interpret the values of rows — you have to duplicate the value of each index in HTTP headers of your request. Since BasicDB does enforce a format for values, it seems to make sense for BasicDB to have support for secondary indexes that does the work for you.

  3. Soren Post author

    I believe secondary indexes require using the leveldb backend. Last I checked, leveldb had some memory requirements I was really hoping to avoid, but I must admit the details escape me and may not be relevant at all anymore.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>