Posts from May 2010

  • Dangers of Abstraction

    One of the more impressive things about Pharo/Squeak is the level of depth in the core libraries, and how those libraries build upon each other to create larger, complex structures. One need only look at the Collection hierarchy for an example of this, where myriad collection types are supported in a deep hierarchy that allows for powerful language constructs like:

    aCollection select: aPredicateBlock thenCollect: aMappingBlock
    

    to work across essentially every type of collection available. Unfortunately, building these large software constructs can have negative consequences when one attempts to analyze performance or complexity, and in this post I’ll outline one particular case that bit me a few weeks back.

    My problems all started while I was still experimenting with Magma. Magma, as you may or may not recall (depending on if you’ve read anything else I’ve posted… which you probably haven’t) is a pure-Smalltalk object-oriented database whose end goal is to provide the Smalltalk world with a free, powerful, transparent object store.

    Now, among Magma’s features is a powerful set of collections, which implement the aforementioned collection protocols, while also providing a much-needed feature: querying. In order to make use of this facility, any column that you wish to generate queries over must have an index defined over it, which is really a glorified hash table on the column1. Whenever you create one of these indexes on a collection, the index itself is squirreled away in a file on disk alongside the database. And that’s where the problems come in.

    In my application, a Go game repository, I had a fairly large number of collections sitting around holding references to Game objects (one per individual user, plus one per Go player), and I needed to be able to query each of these collections across a number of features (not the least of which, the tags applied to each game). That meant potentially many thousands of indexes in the system, at least2. And that meant thousands of files on disk for each of those indexes.

    Well, when I first hit the site, I found something rather peculiar: initially accessing an individual collection took a very long time. On the order of a few seconds, at least. Naturally this dismayed me, and so I started profiling the code, in order to pin down the performance issues. And I was, frankly, a little shocked at the outcome.

    It turns out that, deep in the bowels of the Magma index code, Magma makes use of the FileDirectory class to find the index file name for the index itself. Makes sense so far, right? As part of that, it uses some features of the FileDirectory class to identify files with a specific naming convention. And that code reads the entire directory, in order to identify the desired files.

    On the face of it, this should be fine.

    However, internally, that code does a bunch of work to translate those file names from Unicode to internal Squeak character/strings. And it turns out that little bit of code isn’t exactly snappy. Multiply that by thousands of files, and voila, you get horrible performance.

    So believe it or not, the index performance issues had nothing to do with Magma. It was all due to inefficiencies deep in the bowels of Squeak. And hence the subject of this article. Deep abstraction and code reuse is a very good thing, don’t get me wrong. But any time you build up what I think of as a “cathedral” of code, it’s possible for rotting foundations to bite you later.

    1. That, by itself, is a rather onerous requirement, but that’s really a separate issue 

    2. Granted, in retrospect, there may have been a better way to design the DB, but for a very small scale application, this approach sufficed 

  • The Seaside Web Framework

    While I’m aware that I have, what, maybe two readers of this blog, I thought I might actually start regularly writing a few posts on some of my recent work in the realm of software development. Why? Well, I enjoy writing, and I enjoy… let’s call it “self-gratification”, so posting on my blog seems like a great way to satisfy both of those needs.

    So, with all that said, I bring you the kickoff post, covering Seaside.

    A Little Introduction

    Anyone who’s done any amount of serious web development understands what an absolutely horrible place we, as a development community, find ourselves in. We’re still manually authoring HTML, hacking Javascript, writing AJAX callback hooks by hand, and generally doing all the nasty, gritty, ugly work to make rich web applications possible. Of course, frameworks and abstraction layers have come along to make this a bit easier (Google’s GWT is a great example), but in the end, many of us are still stuck in the dark ages when it comes to web development.

    Enter Seaside.

    Okay, no, wait, let’s back up one step further.

    A Little Pre-Introduction

    You all know what Smalltalk is, right? For those not in the know, it’s a nice, high-level, consistent, clean object-oriented programming language that is really the grandfather for many of the programming languages we see today.

    Of course, if that were it, we’d probably all be using Smalltalk today. But, alas, the history of Smalltalk is a messy one, sharing many similarities with the Unix battles of old, plagued by myriad, incompatible, expensive implementations that drove away developers to other solutions.

    Furthermore, it’s a little strange in at least one respect: rather than code being stored in files, and compiled into binaries, the entire environment, including all your code, is composed into a single “image” from which you must do all you work, including editing, debugging, and so forth. This has great advantages, for example:

    1. The entire environment is available to you and can be inspected and modified as you desire.
    2. Deploying an application involves just copying over an image and firing up a VM.

    But there’s also major disadvantages:

    1. You must use the tools provided in the environment (ie, editor, debugger, etc).
    2. Integration with version control systems isn’t necessarily that great.
    3. It can be tough to figure out where your code ends and the system begins.

    So the picture is certainly mixed. But the sheer power of Smalltalk, the language, and the encompassing environment makes it, at the very least, incredibly intriguing.

    As for implementations, for hobbyists, the most commonly used environment is Squeak, or it’s more professional cousin Pharo. I’ve settled on the latter, as it seems to be taking a more professional tack, but it’s really a matter of preference.

    By the way, what I’ve said isn’t actually true of GNU Smalltalk, but having never used it, I can’t really speak to it’s viability as a platform. Of course, feel free to take a look at it and let me know what you think!

    Where Were We

    Oh yeah. Enter Seaside.

    So what’s Seaside? Well, it provides an advanced web development framework for Smalltalk that allows the developer to just, you know, get on with it already.

    Yeah yeah, I know, you’ve heard that before. So let me illustrate an example for you, and perhaps you’ll see why Seaside excites me so much.

    The Example

    The program we want to develop is incredibly simple:

    1. It presents a counter to the user.
    2. It presents a “decrease” link which lowers the counter.
    3. It presents an “increase” link which increases the counter.

    That’s it. Now imagine, in a traditional web framework, how you would do this. Well, obviously, you need some amount of state, here, in order to track the counter. You could squirrel the value away in a hidden field in a page form (seriously ugly). Or you could assign the user some kind of session ID, and then track the state on the server, using that session ID as a reference (somewhat complicated). Either way, you, the developer, have to focus on how, exactly, that state will be managed.

    Now let’s look at how this program would be expressed in Seaside. First, a class declaration:

    WAComponent subclass: #Counter
          instanceVariableNames: 'count'
          classVariableNames: ''
          poolDictionaries: ''
          category: 'Counter'
    

    This is a simple class declaration describing a subclass of WAComponent named Counter, and containing an instance variable called ‘count’. Okay, so now we need an initializer:

    Counter>>initialize
       
      super initialize.
    
      count := 0.
    

    Again, nothing too special here, we just want to initialize our superclass and our counter. But now comes the meat of the program, and the magic:

    Counter>>renderContentOn: html
    
      html heading: counter.
    
      html anchor
        callback: [ counter := counter + 1 ];
        with: 'increase'.
    
      html space.
    
      html anchor
        callback: [ counter := counter - 1 ];
        with: 'decrease'.
    

    Voila, that’s the entire application, including links and state management.

    No, really, that’s it. The whole thing.

    So, how does it work? Well, first…

    A Bit On Blocks

    Like other high-level languages such as Perl, C#, and others, Smalltalk supports the concept of a closure, which is called a block, encapsulating a chunk of code along with it’s lexical scope. That code can then later be invoked at your leisure. For example:

    | var block |
    
    var := 5.
    
    block := [ Transcript show: 'Hello world, my value is '; show: var; cr ].
    

    The variable ‘block’ now contains a reference to a closure which we can then invoke later with:

    block value.
    

    This block remembers everything in it’s lexical scope, so, for example, the variable ‘var’ will retain it’s value, 5, and be emitted on the transcript. This fact, that closures are stateful code objects, is key to the way Seaside works.

    Back To The Example

    So, in Seaside, you never hand-write HTML. There aren’t even any templating languages. You generate all your HTML with code.

    Yes, I know, this is weird, but bear with me.

    You see, this has a major advantage. Consider the following piece of code from the example:

    html anchor
      callback: [ counter := counter + 1 ];
      with: 'increase'.
    

    Of course, this spits out an anchor. Nothing fancy there. But notice how we didn’t specify a URL? That’s weird enough. But notice something else? There’s an argument called ‘callback’, and we’re providing it a block of code. Can you guess what’s happening here?

    That’s right. Under the covers, Seaside generates a URL for us. When the link is clicked, Seaside invokes the callback automatically. And because the block remembers the lexical scope, it can fiddle with the counter variable, incrementing it.

    So because we let Seaside generate the HTML, suddenly our program is incredibly simple. Under the covers, Seaside manages all our state for us, associating an instance of the Counter object with our browser session. When those links are clicked, the callbacks are invoked in the context of that Counter instance and can manipulate the state of the system. Suddenly we’re no longer hacking HTML, parsing CGI parameters, and all that hideous garbage. We simply write what we want (‘when the user clicks this link, increment the counter’), and Seaside does the rest.

    Conclusion

    So there you go. A really quick intro to Smalltalk and Seaside. As you can tell, this is incredibly exciting to me. Why? Well, developing web applications has always struck me as incredibly tedious. Rather than just being able to write my damn application, I’m stuck parsing query parameters, managing state, manually handling state transitions, and a whole bunch of other garbage that’s really only peripheral to the actual act of building an application. Seaside, on the other hand, gets rid of all that tedium and lets me focus on the important thing: building a powerful application.

    And note, I’ve only just scratched the surface here. Among Seaside’s other powerful features, it has cleanly integrated:

    1. JQuery
    2. Prototype
    3. Scriptaculous
    4. A general AJAX framework for doing partial page updates
    5. And probably a whole bunch of other stuff.

    Mighty cool if you ask me.

    So, all this said, again, the picture isn’t completely rosy. As with all things, there are many issues that Seaside developers must face:

    1. Myriad persistence solutions that are of mixed quality.
    2. Code management issues.
    3. Deployment issues.
    4. Scaling and performance challenges.

    And probably other stuff, too. Which will, of course, be fodder for further posts on this topic.