Ad-Hockery

This blog has moved

2011-11-29T00:11:00.001Z

Ad-Hockery is no longer updated at this location. Please redirect bookmarks to adhockery.github.com for more awesome content.

Fear & loathing in functional testing land

2011-11-24T00:59:00.001Z

As projects grow the two things I’ve repeatedly found to be particularly painful have been functional testing and data fixtures. I might write up some thoughts on data fixtures another time but what follows is a brain-dump of my troubled relationship with functional testing.

Disclaimers: I have more questions than answers and I’m completely open to the idea that I’m doing it all wrong. I’m not trying to diss any tool or technique here. I have spent a lot of time over the last few years writing functional test coverage so I think I at least have some perspective on the issues if no clue how to solve them.

When I say functional testing I mean in the Grails sense of an end-to-end test via the browser. Some people are quite resistant to writing such tests and whilst I agree that as much testing as possible should be done at as low a level as possible there are certain things that really only can be tested in that way. I’m also a big fan of the GOOS approach of working outside in; starting with a (failing) functional test that defines the desired behaviour in a user-centric way and building in to the low-level implementation with its unit tests then back out to watch the original end-to-end test (hopefully) pass.

Why is functional testing difficult?

Test development cadence

The main issue I find when constructing functional tests is what I’ll call the test development cadence; that is the time it takes to go round the loop (and sub-loops) of

write a bit of test
watch it fail
write some code to make it work
watch it still fail
figure out if your expectation is wrong or your code doesn’t work
fix it
repeat last 3 steps n times
watch it pass
celebrate

With a unit test that time is typically fast, a keystroke in an IDE and the results are available in at most a couple of seconds. Functional tests are considerably slower. Even assuming you can optimise so that the application is running and you can run the test from an IDE then Selenium has to start up a browser instance, queries execute, views need to render, etc. In the worst case you’re switching to the terminal and using grails test-app functional: Blarg or equivalent then waiting for the webapp to start up before the test can even start to run and shut down again before the report is generated.

A slow test development cadence leads to distraction (checking Twitter, making coffee, getting drawn into a discussion of the finer points of mixing an old fashioned, etc.) and distraction leads to context-switching which slows things still further.

Test diagnostics

GOOS makes a great point about the importance of test diagnostics suggesting that the TDD mantra of “red, green, refactor” should be replaced with “red, decent diagnostics, green, refactor”. When a test breaks, especially one someone else wrote (or that I wrote more than a week ago and have consequently lost all recollection of), I want to be able to see what’s broken without re-running the test with added logging or resorting to a debugger. Testing further from the browser hurts diagnostics as you can’t see what’s not working and so have to rely solely on the quality of your assertion output. That’s not an easy thing to get right. Geb’s output when a waitFor condition fails is just Condition did not pass in x seconds. Even with direct assertions and nice power assert output its not always clear whether the functionality didn’t work or the expectation is incorrect. Selenese is by no means great in this regard (a humble Condition timed out isn’t much help) but at least you can step back with the Selenium IDE and watch the damn thing not working much more easily.

Bad test diagnostics coupled with a slow test development cadence make for a horrible experience.

The quest for the functional testing holy grail

The most productive I’ve ever been when writing functional tests has been when using Selenium IDE. That’s quite an admission for someone who’s spent a considerable amount of time & energy over the last few years trying to find or build something better!

The test development cadence is fast. Really fast. When you’re writing tests with Selenium IDE (and I do mean write them, I’ve almost never used the recording functionality) the app is running, the browser is running and you can execute the test, a portion of the test or an individual command very quickly. You can step through the test, set breakpoints, etc. When using a framework like Grails that lets you make changes to the app without restarting you can rock along pretty rapidly.

That said, the downsides are not inconsiderable:

Abstraction is typically poor; you’re dealing with fine details of page structure (DOM element ids, CSS selectors) and copy ‘n pasting sequences of commands that would in a sane world be defined as functions or macros. You can write custom JavaScript commands but with considerable limitations such as the fact that any wait for x step must be the last thing the command does. Lack of abstraction means lack of maintainability. As the project goes on any change in page rendering probably means picking apart a bunch of tests that fail not because the functionality they are testing has stopped working but because the implementation of that functionality has changed.
Atomicity is difficult. Because each test probably requires a few lines of setup it’s tempting for developers to add new assertions to an existing test. This violates the principle of having a single (logical) assertion per test. Part of the problem I think is that with Java, Groovy, Ruby, etc. each file can contain multiple tests whereas with Selenese each file is a single test script. The right thing to do is to have lots of small Selenese test files but it’s tempting to fall into the trap of munging tests together into the testing equivalent of a run-on sentence. One of the worst side-effects of this is that as a test suite grows it becomes really hard to identify where certain features are tested and to find redundancy or obsolete tests.

Despite these significant failings writing tests in Selenium IDE is very effective. Maintaining a suite of such tests is another matter. Working on a long-running project the failings of Selenese tests start to increase logarithmically. The reason I created the Grails Selenium RC plugin was to try to build something I could use in future projects that would combat the failings of Selenese. I wanted to use a ‘real’ language with selection and iteration and to be able to build a robust abstraction so that tests are not dealing with the fine details of page markup. Geb is another step along this road. It provides a nice way of defining page objects and modules and handles things like tracking which page class is the ‘current’ one and how and when that should change.

What do I want from a functional testing tool/language?

I’m convinced that the goal of writing tests in the same language as the application is a pretty vapid one. Working inside one’s comfort zone is all very well but too many times I’ve seen things tested using Selenium or Geb that would be better tested with a unit-level JavaScript test. I’m guilty of this myself. I’m a better Groovy coder than I am a JavaScript coder so it’s easy initially to break out a new Geb spec than a new Jasmine spec. Functional testing tools are really bad at testing fine-grained JavaScript behaviour, though. These sort of tests are really flaky, false negatives are a fact of life. They’re wastefully slow as well. JavaScript unit tests are fast, faster than Groovy unit tests. As a Grails developer I’ve looked enviously at how fast tests run in Rails apps but that’s nothing compared to watching a couple of hundred Jasmine tests run in under a second. To get back to the point, I have no problem with writing my functional tests in something other than Groovy if I can hit my goals of productivity and maintainability.

I was at one time convinced that the ability to use loops and conditional statements in Groovy made it a more suitable testing language than Selenese but honestly, how often are such constructs really required for tests? The The single most essential thing for a maintainable suite of functional tests is the ability to create a decent abstraction. Without that you’ll be building brittle tests that fail when the implementation changes 100 times more often than they fail because the functionality they’re testing is actually broken.

Abstraction is key

The abstraction layer needs to be powerful but simple. I’ve seen test suites crippled by badly written page object models and I’m starting to feel that the whole idea is too formalized. Building Geb content definitions with deeply nested layers of Module types is time consuming & difficult. With Selenium RC there’s not even the page transition logic Geb provides so you end up having to write that as well (probably getting it wrong or implementing it in several different ways in different places).

I can’t help thinking the page object model approach is coming at the problem from the wrong angle. Instead of abstracting the UI shouldn’t we be abstracting the behaviour? After all the goal is to have tests that describe how users interact with the application rather than how the various components that make up the application relate to one another. I’d rather have a rusty toolbox of lightweight macros and UI module definitions than a glittering palace of a page component model that I find awkward to use, extend or change. The abstraction has to be there - when I change the implementation I don’t want to spend half a day finding and fixing 100 subtly different CSS selectors scattered throughout the tests - but I don’t think it has to be particularly deep.

Where do I go from here?

A better Selenese?

An interesting possibility for creating better Selenese tests is the UI-Element extension library that allows a UI abstraction layer to be built on top of Selenese. It also introduces the concept of rollup rules (paramaterized macros) that are a more powerful way of abstracting command sequences than custom Selenese commands. From what I’ve seen the tool support in Selenium IDE looks impressive too. I need an opportunity to use UI-Element seriously but it certainly appears promising.

The most impressive Selenium extension I’ve seen is Steve Cresswell’s Natural Language Extensions that layers something like JBehave’s feature definition language on top of Selenese. Energized Work used this on a couple of projects (unfortunately not ones I was involved with) and I’ve heard great stories of how it enabled really rich cooperation between developers, QA and project stakeholders. I was pleasantly surprised with how simple the underlying code appeared to be given the radical difference in the test language.

Other options?

The tools I really need to look into are:

Cucumber which syntactically looks like the answer to my prayers. I want to see how fast the test development cadence is. Since there’s now a pure JVM implementation I really have no excuse for not getting up to speed with it pronto.
FuncUnit is much lower level and I’m not sure how easy it would be to build an effective abstraction layer that kept the tests readable and maintainable but it’s fast and runs right in the browser which are potentially compelling advantages.

Grails Gotcha: Beware HEAD requests when rendering binary output in controllers

2011-08-26T17:19:00.001+01:00

Although most Grails controllers render HTML, JSON or XML output it is possible to use them to render binary data as well. We use a controller to render images uploaded by editors into our content management interface. The theory is simple enough, instead of using the render dynamic method or returning a model the controller action just writes bytes directly to the HTTP response stream. Our action looked something like this:

def show = {
    def image = Image.read(params.id)
    if (image) {
        response.contentType = image.contentType
        response.outputStream.withStream { stream ->
            stream << image.bytes
        }
    } else {
        response.sendError SC_NOT_FOUND
    }
}

This seemed to work well enough. However when writing a test I noticed an odd thing. I was using RESTClient to scrape resource URLs out of and make a HEAD request against them to ensure the URLs were valid. Javascript and CSS files were working fine but all the non-static images in the page were getting 404s. Initially I suspected a data setup problem and spent some time ensuring my test was setting data up properly. It was only once I put some debug logging in the controller action that I saw that the controller was actually loading images. The 404 was not coming from the else block in the action as I initially assumed. I tried changing the RESTClient call from head to get and suddenly the image URLs started working!

Once I did that I realised what the problem was. An HTTP HEAD request does not expect a response, in fact a server receiving a HEAD request must not return a response. The response stream that our controller action is writing to is, when the request method is HEAD, actually a no-op stream. When the action completes Grails checks to see if anything has been committed to the response stream and since it has not assumes that we want to render a view by convention. You can probably see where this is going now. The convention is that the request gets forwarded to grails-app/views/<controller>/<action>.gsp which of course does not exist. The forwarded request sets a response code of 404 because there is no GSP!

We caught this bug in our app completely by accident but it could actually have been quite serious. Caching proxies and CDNs may well use a HEAD request to revalidate content and on getting a 404 assume that the URL is no longer valid. If the 404 response itself then gets cached we could get broken images on our site because the CDN tells the client browser there's nothing there.

The solution is simple enough. I changed the controller action to simply set a 200 response code when it gets a HEAD request for a valid image:

def show = {
    def image = Image.read(params.id)
    if (image) {
        if (request.method == "HEAD") {
            render SC_OK
        } else {
            response.contentType = image.contentType
            response.outputStream.withStream { stream ->
                stream << image.bytes
            }
        }
    } else {
        response.sendError SC_NOT_FOUND
    }
}

A neater solution might be to use Grails’ support for mapping actions to request methods so that GET and HEAD requests dispatch to different actions.

Data-driven variation with Spock

2011-08-18T21:42:00.000+01:00

Spock’s where: block is commonly used with a data table but can also be driven by any Iterable data. It’s worth bearing in mind that the data driving the where: block doesn’t have to be hardcoded, it can be dynamic. For example, today we implemented a spec to ensure that every table in our database schema has a primary key (because it’s required by HA-JDBC and not automatically added by GORM on join tables). In this spec the where: block is driven by the list of table names read from the database metadata.

import grails.plugin.spock.IntegrationSpec import java.sql.Connection import javax.sql.DataSource import spock.lang.* class DatabaseSchemaSpec extends IntegrationSpec { @Shared def dataSource @Shared List tableNames def setupSpec() { DataSource.mixin(DataSourceCategory) tableNames = [] dataSource.withConnection {connection -> def rs = connection.metaData.getTables(null, null, '%', ['TABLE'] as String[]) while (rs.next()) { tableNames << rs.getString(3) } } } @Unroll("the #table table has a primary key") void "all tables have a primary key"() { expect: dataSource.withConnection { Connection connection -> assert connection.metaData.getPrimaryKeys(null, null, table).next() } where: table << tableNames } } @Category(DataSource) class DataSourceCategory { static void withConnection(dataSource, Closure closure) { Connection connection = dataSource.connection try { closure(connection) } finally { connection?.close() } } }

Something like this could be done with JUnit, of course. A test could iterate over the table names and assert that each has a primary key. However, such a test would fail fast whereas with the power of Spock’s @Unroll annotation the spec creates a separate test result for each database table and will run each individually regardless of whether any others pass or fail. The command line output from this spec will be enough to tell you which tables do not have primary keys as @Unroll puts the table name right in the test name.

The other great thing about this spec is that it doesn’t require maintenance; as we add more domain classes to our app the spec will automatically check the associated tables.

Avoiding accidental i18n in Grails

2011-08-02T13:01:00.002+01:00

We’re developing an app that’s exclusively for a UK audience so i18n really isn’t an issue for us. However recently we got bitten by some i18n creeping in where we didn’t want it. Specifically, when using Grails’ g:dateFormat tag the default behaviour is to format the date according to the Locale specified in the user’s Accept-Language header. Even though we are explicitly specifying a format pattern for the date Java is aware of localized day names for some languages so the output can vary. The result is that on a page full of English text there suddenly appears a Spanish or Swedish day name. What makes things worse is that as we use server-side content caching and a CDN if a user with a non-English Accept-Language header is the first to see a particular page or bit of dynamically retrieved content then the cache is primed and until it expires everyone will see the non-English day name text.
The solution in a Grails app is as simple as replacing Spring’s standard localeResolver bean with an instance of FixedLocaleResolver. Just add the following to grails-app/conf/spring/resources.groovy:

localResolver(org.springframework.web.servlet.i18n.FixedLocaleResolver, Locale.UK)

This changes the way Spring works out the request locale and any locale-aware tags should just fall into place.

Resources plugin and modular web components for Grails apps

2011-05-21T21:59:00.000+01:00

I recently gave a talk on ‘Building progressive UIs with Grails’ at gr8conf in Copenhagen and was really pleased with the feedback & comments I received afterwards. There was a question asked at the end that I felt in retrospect I could have answered better, though. I had mentioned that the Grails AJAX tags were something that should be avoided as they write script blocks directly into the page and event handlers directly onto HTML elements. I was pitching an approach based on a clean separation of clean semantic markup and script enhancements and inline script violates that separation.

I was asked if there might be any kind of tags that could be developed that provided a more appropriate replacement and answered that since modern JavaScript frameworks such as jQuery make decorating elements with AJAX functionality so easy that I didn’t think there was much point.

I wouldn’t want anyone to understand me to mean that creating taglibs and GSP templates for modular components of your pages is a bad thing. I’d just advocate keeping the script out of them. Custom tags that write out markup or delegate to templates can be really useful for building complex reusable modules. Pairing those with external JavaScript files that enhance the generated markup would be very effective.

If you’re using the resources plugin (and really… you should be) then there’s a really neat way to tie the taglib or template to the JavaScript file as well. I did briefly mention this in my talk but it’s worth expanding on here as it’s a technique with a lot of potential.

The resources plugin’s r:use tag doesn’t write out anything directly to the page but rather adds a resource module to a list that will get written out at the appropriate place in the page when the r:layoutResources tag is used. This means modular components throughout the page can declare a dependency on JavaScript and CSS resources by simply calling r:use. In a complex app this can be a real boon as it might not be obvious from the top level GSP exactly which modules are going to get rendered. Also if a module is later added to or removed from a page you don’t need to worry about fiddling around with resource declarations in the top level GSP. Taglibs or templates become real drop in components whilst maintaining a nice clean separation of markup and script. Even better, if you use the same resource dependency multiple times the plugin ensures the resources are actually only linked once.

Since r:layoutResources is typically used in a SiteMesh layout the resources your module depends on can even be ones that need to appear in the head of the document. The SiteMesh is rendered after the GSP it decorates so any r:use calls will already have been made.

CSS box model hacking

2011-03-26T16:25:00.008Z

Want to make an HTML element fill 100% of its parent’s width but also give it a border and/or some padding? Since the width of an element is exclusive of its border and padding this can be a pain. However there’s a fairly simple CSS solution that works cross-browser.

Here’s an example. Both input boxes are set to width: 100% and have padding: 5px. The first input shows the problem. Because the padding and border are added to the width of the element it overflows its container. The box model of the second input has been modified so that the padding and border are inside the declared width.

Field 1 Field 2

The trick to modifying the box model is to set box-sizing: border-box. Unfortunately that’s not a cross-browser property, only Opera supports it at the moment. To get the same effect in other browsers you will also need to set browser-specific versions as well:

   -moz-box-sizing: border-box;
-webkit-box-sizing: border-box;
    -ms-box-sizing: border-box;
        box-sizing: border-box;

Note that Internet Explorer only supports the -ms-box-sizing property from version 8 upwards so you should probably be judicious with this technique or use an alternative method to get a similar effect in IE7 and below.

Styling with HTML5 form validation

2011-03-25T00:10:00.001Z

HTML5 specifies a number of enhancements to form inputs that are gradually being implemented by newer browsers. Among the enhancements is support for some level of automatic form validation. When inputs have invalid values the browser may refuse to submit the form. Currently Opera, Safari, Chrome and Firefox 4 have some support for this. IE8 does not. I haven’t experimented with IE9 yet.

Let’s look at an example. A simple registration form has inputs for the name, email and website of a prospective user. Of these name and email are required. The input elements are as follows:

<input type="text" name="name" required>
<input type="email" name="email" required>
<input type="url" name="website">

The required attribute is new in HTML5 as are the input types email and url. All of these have implications for validation. Additionally the new input types, whilst they appear just like regular text inputs in desktop browsers are great for mobile users as they are given optimised virtual keyboard layouts.

CSS3 has :invalid, :required and :optional pseudo-classes that are supported by current versions of Firefox, Chrome, Safari and Opera. Some browsers also refuse to submit the form if a required field is not filled in or invalid values are entered in url or email type fields. Obviously without appropriate feedback this could be a pretty disastrous user experience and thankfully even the Webkit browsers which until recently were blocking form submission without feedback are now handling things better.

Firefox 4, Opera 11.01 and Chrome 10.0.648.204 all block form submission if there is an empty required field or invalid value and display a message next to the first such field.
Safari 5.0.4 will apply :invalid CSS rules but allows the form to be submitted.

Applying styles to invalid fields is pretty easy. For example to highlight invalid fields with a red border and background:

input {
    border: 1px solid #ccc;
}

input:invalid {
    background: #fff3f3;
    border-color: #f66;
}

You can also combine the :invalid pseudo-class in combination with others. For example to highlight focused fields with a fancy CSS3 box-shadow:

input:focus {
    border-color: #99f;
    outline: none;
     -moz-box-shadow: 0 0 0.25em #99f; 
  -webkit-box-shadow: 0 0 0.25em #99f; 
          box-shadow: 0 0 0.25em #99f; 
}

input:focus:invalid {
     -moz-box-shadow: 0 0 0.25em #f66; 
  -webkit-box-shadow: 0 0 0.25em #f66; 
          box-shadow: 0 0 0.25em #f66;
}

It’s worth noting that Firefox 4 automatically adds a red box-shadow to invalid fields (and not just focused ones). To turn it off you can specify:

input:invalid {
    -moz-box-shadow: none;
}

Of course, the browsers that display messages when they refuse to submit a form do so with different text and different visual effects so creating a consistent cross-browser look and feel with both client and server side validation messages is going to be an interesting challenge.

Date and time inputs vs. usability

2011-03-06T20:08:00.001Z

Form input controls for date/time values have always been problematic. The more I experiment with different options, the more I think there is no one-size-fits-all solution.

Calendar widgets

Calendar widgets such as the jQuery UI date picker or the Opera HTML5 date input are very popular and can be a slick looking addition to a form. I don’t think they’re entirely positive though and certainly don’t work in all situations. They’re great when you don’t know exactly the date you want (I want to book a flight around Easter time) or you need to have an overview of the calendar (I need to set a reminder for GGUG on the third Tuesday of every month). However, when trying to enter your date of birth the widget is just in the way. Even if the widget is keyboard accessible and doesn’t force you to page month-by-month to the correct year you can almost certainly type the date quicker than you can find and select it in a calendar widget.

Multi-field inputs

A multi-field input works well for entering familiar dates such as your date of birth or for transcribing data. They’re not so good when you need the context a calendar widget affords (please don’t force your users to mentally calculate what date the 3rd Tuesday of next month is).

With a multi-field input it’s vital to remember that users in different places will expect to enter dates in a different order. For example, here in the UK we use the completely logical dd mm yyyy order whilst an American will, bizarrely, want to put the month before the day. It’s not particularly difficult to build a tag that renders the fields in a different order according to the request locale. Using the HTML5 placeholder attribute (or a suitable fallback for older browsers) is invaluable to ensure that users know which field is which.

A possible optimisation is to auto-tab between fields after the user has typed the requisite number of digits. In that case you should allow for the user to correct mistakes, so backspacing or using the cursor keys should also automatically jump between fields otherwise the auto-tabbing behaviour becomes more a hinderance than a help for fat-fingered typists like me.

Selects in a multi-field input

It’s very common to use a select input for some or all portions of the date. Personally, I pretty much detest this practice as it combines all the inconvenience of a calendar widget with none of the contextual information. Grails’ <g:datePicker> tag does exactly this. Not only are select elements like this bulking out the page with huge option lists (Grails’ datePicker dumps 8.6Kb of markup in your page without time fields!) but they are cumbersome to input values into with the keyboard because so many values start with the same character. An arguable exception is a select for the month field as presenting textual values rather than numbers is a little friendlier and the option list is short enough and the values distinct enough to allow for reasonably quick keyboard selection.

I suspect the rationale for using select elements is often that it all but ensures the user can’t enter an invalid value (I’ve even seen significant hoop-jumping done in Javascript to ensure values such as 30th February can’t be entered). I think it’s preferable to give the user a clear input and trust them to do the right thing with it rather than forcing them to use an awkward input so that they can’t get it wrong.

Simple textual inputs

You shouldn’t rule out using just a basic text input. In fact for monotonous data entry purposes it is probably optimal. The keystrokes to enter a date become muscle memory and there’s no fiddling about tabbing between day, month and year fields or grabbing the mouse to pick options from a select element. Obviously the down-side is that a plain text input is error-prone and very liable to confuse users unless the expected format is made clear.

HTML5

HTML5 specifies various new date and time input types which seems like good news but has some significant drawbacks at the moment. I can only imagine the new input types are designed to be rendered as native browser widgets (currently only Opera actually does so) as the required format is just about the most inconvenient imaginable for text entry. Sure, as a developer it’s appealing to require that dates have to be entered as yyyy-mm-dd as you no longer have to support varied input formats but no user in the world would naturally type a date that way. Therefore, to make such inputs usable you really need to use Javascript to proxy them with a widget or a set of multi-field inputs. Fine, but what about users who disable Javascript or use mobile or assistive devices that might not cope particularly well with the widgets? The fallback - a text input into which the user must type a machine-readable RFC3339 date - is pretty unfriendly.

What’s worse is that Webkit based browsers such as Chrome and Safari will render the new input types as regular text inputs but refuse to submit a form containing incorrectly formatted values and unbelievably provide no user feedback whatsoever when they do so.

I’m inclined to think that the useful support for the new input types is so minimal and the actively user-hostile handling of them so much more widespread that they’re best avoided altogether.

So, to sum up:

Think about who your users are and what they are trying to accomplish with this particular input field and give them the right input for the job at hand.
If they need to type some or all of the value make sure it’s clear what format is expected or which field is which.
Allow for people to enter values in the order they are comfortable with and above all don’t force them to type it in a machine-readable format.
Don’t strand Chrome and Safari users with broken HTML5 input types.

JavaScript's history object, pushState and the back button

2011-02-22T22:31:00.000Z

I’m not sure if it’s the immaturity of the browser support or my general uselessness but I’ve been having some trouble with the JavaScript history API.

I won’t try to explain the history API here, it’s pretty well covered at Mozilla Developer Network and W3. The basics are simple enough:

The API provides two methods; pushState which allows you to add a new entry to the browser history and replaceState which modifies the current history entry.
New entries added to history using pushState can be navigated via the browser’s back and forward buttons and a popstate event is fired on the window object when this happens.
Both methods allow you to attach arbitrary data to the history entry that you can use to reconstruct the appropriate page state when the user uses the back or forward buttons.

I imagine a pretty typical use-case is what I’ve been trying to do with the pagination and sorting on the list pages of Grails scaffolding. Instead of pagination links and column headers causing a full page reload when clicked I intercept the click event and send an AJAX request getting back a page fragment I can use to update the list in the page. Easy enough, however without the history API it will break the back and forward buttons and make deep linking impossible. This isn’t an acceptable option so in true progressive enhancement style I’ve used Modernizr and only apply the AJAX behaviour if the browser supports the history API.

The essence of the script involved is this:

var links =     //... pagination and sorting links
var container = //... the region of the page that will be updated with AJAX

// override clicks on the links
links.live('click', function() {
    // grab the link's URL
    var url = $(this).attr('href');
    // add a new history entry
    history.pushState({ path: url }, '', url);
    // load the page fragment into the container with AJAX
    container.load(url);
    // prevent the link click bubbling
    return false;
});

// handle the back and forward buttons
$(window).bind('popstate', function(event) {
    // if the event has our history data on it, load the page fragment with AJAX
    var state = event.originalEvent.state;
    if (state) {
        container.load(state.path);
    }
});

// when the page first loads update the history entry with the URL
// needed to recreate the 'first' page with AJAX
history.replaceState({ path: window.location.href }, '');

At first glance this works pretty nicely. In browsers that support history (right now that’s just Chrome, Safari and its mobile variants) paginating and sorting the list does not refresh the entire page but the browser’s location bar is updated so copy-pasting or bookmarking the URL will give a valid link to the current page. What’s more, the back and forward buttons can be used to step back through the list pages just as if we reloaded the whole page. In non-history-compliant browsers the list page behaves just like it always did; the links reload the entire page.

Unfortunately there’s a problem that was reported to me on GitHub shortly after I uploaded a demo of my scaffolding work. Where everything falls down is when you paginate the list, follow a link off the page (or just use a bookmark or type in a URL), then use the back button to return to it. In Chrome and iPad/iPhone variants of Safari the browser displays just the page fragment from the last AJAX call, losing all the surrounding page along with styling, scripts, etc.

Where things get very odd is that adding a Cache-Control: no-cache header to the AJAX responses makes the problem disappear, presumably because the browser then doesn’t cache the AJAX response and has to use the historical URL to reload the entire page. Remember, in good progressive enhancement style the URL for the full page or the fragment is the same. The server uses the X-Requested-With header to decide whether to return a fragment or the whole page. Obviously, forgoing caching is hardly an acceptable compromise but it’s interesting in that it reveals what the browser is doing. It can’t, surely, be right for the browser to treat a page fragment the same as a full document!

Curiously in desktop Safari this doesn’t happen and the full page is loaded as you would hope. Looking at the User-Agent header it looks like Safari is using an older version of WebKit (533.19.4 rather than 534.13).

You can see the behaviour in my scaffolding sample app. I also ran into the exact same issue today at work whilst trying to add history navigation to this page (which is a good example of why you’d want to use history in the first place). I don’t think it’s just me, either. The same problem can be seen with the demo linked from this post about the history API.

If there are any JavaScript experts out there who can point out my obvious incompetence here, that would be great. Otherwise I guess I’ll have to wait for updates to WebKit to hopefully fix the issue before I can start applying this technique with confidence.

Running Geb tests from your IDE

2011-02-06T15:02:00.002Z

A frequent complaint about functional testing in Grails is that the start up / shut down cycle time of the app makes prototyping a functional test prohibitive. There has been some recent progress in this are with Luke Daley's Functional test development plugin but it would be really nice to be able to just run functional tests from inside your IDE in the same way as a similar unit test.

With Geb and IntelliJ Idea at least this is actually pretty straightforward (I'm sure the same thing is possible with Eclipse/STS I'm just not familiar enough with it).

Geb's dependence on the Grails lifecycle is pretty minimal so the tests can run without needing any magic to happen in any _Events scripts (unlike the Selenium RC plugin where this would be significantly more difficult). One extra step is required; the Grails Geb plugin normally picks up its base URL from Grails configuration and the test will not be running in the same JVM as the Grails app if it runs from the IDE so configuration will not be available. Just add the following method to your test that extends GebTests or GebSpec (or add it to a base class):

@Override String getBaseUrl() {
    super.baseUrl ?: "http://localhost:8080"
}

Now Geb will use the Grails configured base URL if it is in the same JVM as the app or default to http://localhost:8080 otherwise (obviously you need to change that default if that's not the host or port where your app runs).

You can then just run up the app from the command line, open the test in IntelliJ and hit Ctrl + Shift + F10 to run it. When the test completes the app is still running so you can make changes & run again with minimal turnaround time. Since the app is running in development mode you can make application changes without a server restart as well.

IntelliJ Idea's test integration is pretty good and it even groks Spock specifications. You can also debug and step through the test. The only downside is that you can't directly access domain classes or other application components in the test as they are only available from the app's JVM.

Encapsulating page state and actions in Geb

2010-11-30T13:47:00.001Z

Geb Page and Module classes are just regular Groovy classes. In addition to the content DSL you can declare methods just as you would in any other class. Those methods can use content properties directly. This is really useful for encapsulating state and actions in a reusable way.

In this post I want to run through a couple of simple examples of re-usable Module classes that are enhanced with methods. Although the examples deal with Module classes everything here is equally applicable to Page classes.

Page object methods tend to fall into two categories:

Observers

report on some aspect of page state (e.g. whether a user is logged in or not).

have non-void return types (I prefer explicit return types on methods but def would work too).

can be used for making assertions.

Actions

perform an action such as clicking a link or button.

will typically be void (note that it's not necessary to return a Page instance or class from a navigation method in Geb).

throw IllegalStateException if the action is invalid in the current page state.

First let's consider an authentication module that can be used across every page in a site. The markup of the module can be completely different depending on whether a user is logged in or not. When not logged in a username/password form is shown:

<aside id="auth"> 
    <fieldset> 
        <legend>Log In</legend> 
        <form action="/auth/login" method="post" > 
            <label>Username: <input name="username"/></label> 
            <label>Password: <input type="password" name="password"></label> 
            <button name="login" type="submit">Log In</button> 
        </form> 
    </fieldset> 
</aside>

When logged in, a welcome message is shown:

<aside id="auth"> 
    Welcome back <span class="username">blackbeard</span> 
    <a href="/auth/logout" name="logout">Log Out</a> 
</aside>

To model this I'll create a Geb Module like this:

class AuthModule extends Module {
    static base = { $("aside#auth") }

    static content = {
        username(required: false) { $(".username").text() }
        logoutButton(required: false, to: HomePage) { $("a[name=logout]") }
        form(required: false) { $("form") }
        loginButton(required: false, to: HomePage) { $("button[name=login]") }
    }
}

The required: false declaration is used to declare content properties that may or may not be on the page. The username and logoutButton properties are only present in the logged-in state and the form and loginButton properties are only present in the logged-out state.

I can use the module in some tests like this:

def "user can log in"() {
    when:
    authModule.form.username = "blackbeard"
    authModule.form.password = "yohoho!"
    authModule.loginButton.click()

    then:
    at HomePage
    authModule.username == "blackbeard"
}

def "user can log out"() {
    when:
    authModule.logoutButton.click()

    then:
    at HomePage
    authModule.username == null
}

This is a good start but there are several assumptions and bits of page detail creeping into the test. The test is using the presence or absence of the username to determine if someone is logged in or not. That doesn't lead to a very meaningful assertion in the test and is assuming things about the page detail. Likewise the login step is quite long-winded and likely to be repeated in a lot of tests. Not only that but it won't work if a user is already logged in as the form fields won't be present on the page.

To encapsulate the module's state and actions better I'll add the following methods:

boolean isLoggedIn() { username }

void login(String username, String password = "password") {
    if (loggedIn) throw new IllegalStateException("already logged in")
    form.username = username
    form.password = password
    loginButton.click()
}

void logout() {
    if (!loggedIn) throw new IllegalStateException("already logged out")
    logoutButton.click()
}

The methods declared by the module abstract some detail away very effectively. The isLoggedIn method means I can change the login detection mechanism later and just change the module's method rather than a bunch of tests. The login and logout methods abstract away the how of logging in and out so the test can just deal with the what. I've used IllegalStateException for cases where a method is called when the module is not in the correct state. The tests now look much clearer:

def "user can log in"() {
    when:
    authModule.login "blackbeard"

    then:
    at HomePage
    authModule.username == "blackbeard"
}

def "user can log out"() {
    when:
    authModule.logout()

    then:
    at HomePage
    !authModule.loggedIn
}

Another good example of encapsulating state and behaviour like this is a typical pagination module that would appear on a list or search results page. The markup would look something like this (I've omitted the link href attributes for clarity):

<nav class="pagination">
    <a class="prevLink">Previous</a>
    <a class="step">1</a>
    <span class="currentStep">2</span>
    <a class="step">3</a>
    <a class="nextLink">Next</a>
</nav>

The previous and next links will only appear when there is a previous or next page and no links will be present at all if there is only a single page. The following Module class models the state and actions of this component:

class Pagination extends Module {
    static content = {
        links(required: false) { $("a") }
        currentPage(required: false) { $(".currentStep")?.text()?.toInteger() ?: 1 }
        nextLink(required: false) { links.filter(".nextLink") }
        previousLink(required: false) { links.filter(".prevLink") }
    }

    boolean isFirstPage() {
        previousLink.empty
    }

    boolean isLastPage() {
        nextLink.empty
    }

    void toPage(int pageNumber) {
        def link = links.filter(text: "$pageNumber")
        if (!link) throw new IllegalArgumentException("Page number $pageNumber not present in pagination")
        link.click()
    }

    void nextPage() {
        if (lastPage) throw new IllegalStateException("Already on the last page")
        nextLink.click()
    }

    void previousPage() {
        if (firstPage) throw new IllegalStateException("Already on the first page")
        previousLink.click()
    }
}

Breaking the Module down in detail:

The currentPage property returns the current page number as an int and defaults to 1 if there is no pagination present in the page.
The isFirstPage and isLastPage observer methods use the absence of the previous and next links respectively to determine if the current page is the first or last one.
The toPage method finds a numbered link and clicks it, throwing IllegalArgumentException if no such link is present.
The nextPage and previousPage action methods throw IllegalStateException if the relevant link is not on the page.

The Pagination class now neatly encapsulates the detail of the pagination elements and presents a higher-level façade to the tests.

Fuller versions of the examples in this post can be found on GitHub.

Modelling repeating structures with Geb page objects

2010-11-26T00:17:00.001Z

Geb is the hot new thing in Grails functional testing. One of its most powerful features is the concise DSL for defining page objects. The reasons for using page objects are well enumerated elsewhere but the basic point is to allow your tests to interact with pages in a manner agnostic of the detail of their structure. This is both practical (you can change markup structure without having to fix numerous tests that only fail because they were tightly coupled to that structure) and aesthetic (your tests read more like a user’s interaction with the page - the what rather than the how).

I want to put together a few short blog posts dealing with patterns that I find useful for defining page objects and modules in Geb. As I go I’ll keep adding to a very simple Grails project showing working examples which is available on GitHub.

In this post I want to talk about repeating data structures such as lists and tables and how to model them effectively.

A content property in a Geb Page or Module can be any type; whatever is returned from the defining closure. This will frequently be a Geb Navigator instance or a String but can be whatever is useful for the tests you’re writing. A good rule of thumb is that the test should be dealing with as simplified a view of the data as possible. All the complexity of traversing HTML elements and manipulating them into a useful form should be hidden away in the page objects and modules. When handling repeating data structures such as ol or table elements you probably want to be able to treat the content as a List so that tests can use Groovy features such as iterator methods, indexing and slicing to make very expressive assertions.

Simple repeating structures

For example, imagine we want to verify an ol element like this:

<ol id="recent-books">
    <li>Zero History</li>
    <li>Surface Detail</li>
    <li>The Machine of Death</li>
</ol>

The most useful type would probably be a List<String> which we can get easily enough by defining our content like this:

static content = {
    recentBooks { $("ol#recent-books li")*.text() }
}

This is then very easy to use in a test:

expect:
recentBooks == ["Zero History", "Surface Detail", "The Machine of Death"]

Complex repeating structures using Modules

A more complex example of a repeating structure is a table, where each row contains several fields. Here we can use a Geb Module to represent each row, with content properties to get data from each cell. Let’s say we want to verify the contents of the following table of search results:

<table id="book-results">
    <thead>
        <tr>
            <th>Title</th>
            <th>Author</th>
            <th>Format</th>
            <th>Price</th>
            <th>Release Date</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Zero History</td>
            <td>William Gibson</td>
            <td>Hardback</td>
            <td>£12.29</td>
            <td>2 Sep 2010</td>
        </tr>
        <tr>
            <td>Zero History</td>
            <td>William Gibson</td>
            <td>Kindle</td>
            <td>£11.99</td>
            <td>2 Sep 2010</td>
        </tr>
        <tr>
            <td>Spook Country</td>
            <td>William Gibson</td>
            <td>Paperback</td>
            <td>£5.00</td>
            <td>31 Jul 2008</td>
        </tr>
        <tr>
            <td>Pattern Recognition</td>
            <td>William Gibson</td>
            <td>Paperback</td>
            <td>£4.99</td>
            <td>24 Jun 2004</td>
        </tr>
    </tbody>
</table>

We can define our module like this:

static content = {
    bookResults { i -> module BookRow, $("table#book-results tbody tr", i) }
}

class BookRow extends Module {
    static content = {
        cell { i -> $("td", i) }
        title { cell(0).text() }
        author { cell(1).text() }
        format { cell(2).text() }
        price { cell(3).text()[1..-1].toDouble() }
        releaseDate { cell(4).text() }
    }
}

The bookResults content closure takes a row number parameter and uses it to select the corresponding tr from the body of the table and use that to construct a module. The module itself defines content properties with meaningful names that map to the text in each cell.

This isn’t bad as far as it goes. We can use the module pretty effectively in tests like this:

expect:
bookResults(0).title == "Zero History"
bookResults(3).price == 4.99

However, bookResults isn’t a List. We can’t easily get all the book titles at once or make an assertion that all the authors are the same or find the lowest price. Even querying how many rows there are would require an additional content property or method to retrieve $("tbody tr").size(). The table is a repeating data structure and it would be nice to treat it as one!

This ought to be possible bearing in mind 3 things:

The type of a content property is simply whatever you return from the defining closure, there’s no reason we can’t return a List<BookRow>.
There’s nothing special about the expression that constructs the module itself: module BookRow, $("tbody tr", i) is just a call to a method called module passing a Class<? extends Module> which is the module type we want and a Navigator pointing to the module’s root element.
The Geb Navigator class returned by the $ expression implements Iterable<Navigator> and can be treated like a List of all the selected HTML elements.

In fact we can get a List<BookRow> easily enough if we redefine the bookResults property like this:

static content = {
    bookResults {
        $("tbody tr").collect {
            module BookRow, it
        }
    }
}

The key here is that we iterate over the tr elements inside the content definition collecting a new BookRow instance for each one. Now the page object doesn’t require the test to pass in the index of the row it’s interested in. This enables our test to do some much more powerful and interesting things:

expect:
bookResults.size() == 4
bookResults[0].title == "Zero History"
bookResults.title.unique() == ["Zero History", "Spook Country", "Pattern Recognition"]
bookResults.every { it.author == "William Gibson" }
bookResults[2..3].every { it.format == "Paperback" }
bookResults.price.sum() == 34.27

I’ve tried to show a couple of reasonably simple examples here. Others are easy to imagine; a Map representing the dt and dd elements inside an HTML definition list, a list of modules representing a group of labelled radio buttons or news items with images and links, a tree-like multi-level navigation structure, etc.

Stubbing access to the g:message tag in unit tests

2010-09-24T13:48:00.001+01:00

Grails controllers and tag libs can access any tag as though it were a method. The most common use for this is probably accessing i18n messages via the g:message tag. However, because tag access is magic wired up by Grails it's not available in unit tests without some effort.

This is a perennial nuisance. Not exactly difficult to solve and yet something I find myself solving over and over in different tests and different projects.

I've come up with what I think is a pretty good and re-usable solution. It allows you to specify messages if you want to, or just use the message code if it's not something you care about in a particular test. As an aside, at a unit test level, I think testing that the correct message code is being used is probably the right thing to do; I'd leave testing actual message text to end-to-end tests.

Here's an example. Imagine we're testing the following controller that displays a simple greeting:

Here's a spec that shows the behaviour both when a message is specified and when it isn't:

import grails.plugin.spock.*
import org.springframework.context.*
import org.springframework.context.support.*
import spock.lang.*

class GreetingControllerSpec extends ControllerSpec {

@Shared def messageSource = new StaticMessageSource()
@Shared def pirateEnglish = new Locale("en", "BV")

def setupSpec() {
messageSource.useCodeAsDefaultMessage = true
messageSource.addMessage "greeting.message", pirateEnglish, "Ahoy there {0}!"
}

def setup() {
mockMessageTag(controller, messageSource)
}

@Unroll
def "greeting is rendered by index action"() {
given:
if (name) controller.params.name = name
if (locale) controller.request.addPreferredLocale(locale)

expect:
controller.index() == [message: message]

where:
name | locale | message
null | null | "greeting.message"
"Rob" | null | "greeting.message"
"Rob" | pirateEnglish | "Ahoy there Rob!"
}

// in reality this would be static imported from a helper class
static void mockMessageTag(artefact, MessageSource messageSource) {
artefact.metaClass.message = { attrs ->
messageSource.getMessage(attrs.code, attrs.args as Object[], delegate.request.locale)
}
}
}

A few things to note:

The stubbed message tag returns the code if there is no message defined
message arguments are ignored unless there is a message defined

Although in the example I've used Spock, this technique would work equally well with JUnit tests extending ControllerUnitTestCase. It will also work just as well for tag libs tests extending TagLibUnitTestCase or TagLibSpec.

Auto-generate Spock specs for Grails artifacts

2010-08-12T22:46:00.005+01:00

When creating artifacts such as domain classes, controllers and tag libs Grails generates a JUnit test case. If, like me, you're digging writing specifications with Spock you'd probably rather have Grails generate one of those. The last thing I want is to manually transform every generated test case into a specification for every artifact I create.

It's very simple to create a CreateUnitSpec or CreateIntegrationSpec script with a template specification. Hooking in to the other types of artifact creation turned out to be fiddlier. Each create-* command calls a Closure called createUnitTest. Reassigning that Closure should be the solution. The trick is in figuring out where that can be done.

Any time one of its Gant targets is invoked the Grails build system fires an event. You can respond to those events by declaring a closure called event<target name>Start in scripts/_Events.groovy. The only Gant target directly invoked when an artifact is created is called 'default'. It is possible to intercept that although that means the event handler will be invoked any time any Gant target called 'default' runs. For this purpose that's no problem since we're just overriding a closure in the build binding.

The other factor is that the superclass for the unit test is specified by the individual create-* scripts (or defaulted to GrailsUnitTestCase). Rather than having to override those scripts as well, I've just mapped the non-standard unit test superclasses to the Spock equivalents.

Here's the code for your _Events.groovy script:

eventDefaultStart = {
createUnitTest = { Map args = [:] ->
def superClass
// map unit test superclass to Spock equivalent
switch(args["superClass"]) {
case "ControllerUnitTestCase":
superClass = "ControllerSpec"
break
case "TagLibUnitTestCase":
superClass = "TagLibSpec"
break
default:
superClass = "UnitSpec"
}
createArtifact name: args["name"], suffix: "${args['suffix']}Spec", type: "Spec", path: "test/unit", superClass: superClass
}
}

The template specification should be placed in src/templates/artifacts/Spec.groovy and is simply:

@artifact.package@import spock.lang.*
import grails.plugin.spock.*

class @artifact.name@ extends @artifact.superclass@ {
def "feature method"() {

}
}

It goes without saying that this is a slightly hairy and it would be great if Grails provided a proper hook for overriding the test generation. I can live with some fun-size evil in _Events.groovy for the sake of the convenience of getting template specs for all my artifacts, though.

Rendering Grails Joda-Time date inputs cross browser with HTML5, jQuery and Modernizr

2010-07-10T14:30:00.006+01:00

Yesterday I released a new version of the Grails Joda-Time plugin that includes support for the various new date and time input types in the HTML5 standard. Right now only Opera supports these types with proper date-picker type controls, the other browsers just render them as text inputs. However this doesn’t mean you can’t or shouldn’t start using them right away. There’s a very handy JavaScript library called Modernizr that you can use to detect the features supported by the client’s browser and render alternatives using script. In this post I’m going to walk through how to combine the Joda-Time plugin, Modernizr and the jQuery datepicker to render an HTML5 date input field that will bind to a LocalDate property on a domain object.

Install the Joda-Time plugin

If you don’t currently have version 1.1+ of the Joda-Time plugin installed, run grails install-plugin joda-time.

Once the plugin is installed you will need to enable HTML5 support so that data binding will use the HTML5 date/time formats. Simply add the following to your Config.groovy

jodatime.format.html5 = true

Create a domain class with a LocalDate property

For this example I’ll create a simple domain class representing a person:

import org.joda.time.*
import org.joda.time.contrib.hibernate.*

class Person {
    String name
    LocalDate birthdate
    static mapping = {
        birthdate type: PersistentLocalDate
    }
}

If you have default mappings set up (which the Joda-Time plugin will attempt to do for you when it installs) then you can omit the mapping block but I’ve included it for completeness.

Execute grails generate-all Person to create a controller and scaffolded views for the Person class. Finally change the create.gsp and edit.gsp files to use an HTML5 input instead of the standard Grails date picker tag. Replace the <g:datePicker name="birthdate"… with this:

<joda:dateField name="birthdate" value="${personInstance.birthdate}"/>

In the unlikely event that all your site’s visitors are using Opera then your work is done at this point. However right now anyone using another browser will have to type in the birthdate value in the correct format in order to get the forms to submit properly which isn’t the friendliest user experience.

Install jQuery and jQuery-UI

You can include jQuery and jQuery-UI by simply downloading the JavaScript files into your app, or linking to them on Google. Alternatively you can use the Grails plugins. If you want to go the plugin route this is what you will need to do. First install the plugins:

grails install-plugin jquery-ui

The jQuery plugin is installed automatically by the jQuery-UI plugin.

Then you will need to add the libraries to your SiteMesh template (or you could include it just on the relevant pages, it doesn’t matter). Add the following in the head section:

<g:javascript library="jquery" plugin="jquery"/>
<jqui:resources components="datepicker" mode="normal" />

Install Modernizr

Download Modernizr and put it in web-app/js/modernizr. You can link to the script direct from there or define a library by adding this to BootStrap.groovy:

import org.codehaus.groovy.grails.plugins.web.taglib.JavascriptTagLib

class BootStrap {
    def init = { servletContext ->
         JavascriptTagLib.LIBRARY_MAPPINGS.modernizr = ["modernizr/modernizr-1.5.min"]
    }

Then include Modernizr in your SiteMesh template by adding the following in the head section:

<g:javascript library="modernizr"/>

To get Modernizr to work its magic you should add a class="no-js" to the html element at the top of your SiteMesh template. Modernizr will replace this when the document loads with a whole bunch of classes that it uses to detect the various features supported by the browser.

Bind the jQuery-UI datepicker to the field

The last step is to ensure that any time a user hits the page with a browser that doesn’t support a native widget for the date input type they get a jQuery datepicker instead. To do this create a JavaScript file and link to it from your SiteMesh template or the pages using date inputs. The script simply needs to contain the following code:

$(document).ready(function() {
    if (!Modernizr.inputtypes.date) {
        $("input[type=date]").datepicker({dateFormat: $.datepicker.W3C});
    }
});

What this does is create a function that runs on page load that uses Modernizr to determine if the date input type is supported and if not initialises a jQuery-UI datepicker for every <input type="date"… found in the page. The dateFormat argument ensures the jQuery widget will update the input with the correct date format when a value is selected.

It’s that simple. Now the create and edit person pages will use the native date input widget when the visitor’s browser supports one and the jQuery widget when it doesn’t.

So how does it look? Here's a screenshot of the page in Firefox when a visitor has focused the date input:

Here's the same thing rendered in Opera using the native datepicker widget (yes, you could be forgiven for thinking the jQuery version is prettier):

With a little adaptation the same thing can be done for the other input types; month, week, time, datetime and datetime-local. Of those month should work perfectly by adding this to your script:

if (!Modernizr.inputtypes.month) {
    $("input[type=month]").datepicker({dateFormat: 'yy-mm');
}

Other types would require different widgets to enable users to input the time portion of the value. There are several jQuery based options available. Of course, instead of jQuery you could use YUI, script.aculo.us or any other date/time widgets. I’ve gone with jQuery because I’m familiar with it and the initialisation is beautifully simple even if you’ve got a number of different inputs on a page.

Acyclic relationship validation in Grails

2010-06-29T16:27:00.001+01:00

A common domain class use-case is for a self-referential relationship that must not be circular (a directed acyclic graph). For example, a Person class may have a parent property that is a many-to-one relationship with another Person instance. However a given Person cannot be his own parent or ancestor.

We're using just such a relationship for trees of pages that inherit certain characteristics from their ancestors. In order to validate that our users aren't setting up circular references I implemented the following constraint:

import org.codehaus.groovy.grails.validation.AbstractConstraint
import org.springframework.validation.Errors

class AcyclicConstraint extends AbstractConstraint {

static final String DEFAULT_MESSAGE_CODE = "default.acyclic.violation.message"
static final String NAME = "acyclic"

private boolean validateConstraint

protected void processValidate(Object target, Object propertyValue, Errors errors) {
if (validateConstraint && propertyValue) {
if (isCyclic(target, propertyValue)) {
def args = [constraintPropertyName, constraintOwningClass, propertyValue] as Object[]
rejectValue(target, errors, DEFAULT_MESSAGE_CODE, "${NAME}.violation", args)
}
}
}

void setParameter(Object constraintParameter) {
if (!(constraintParameter instanceof Boolean)) {
throw new IllegalArgumentException("Parameter for constraint [$NAME] of property [$constraintPropertyName] of class [$constraintOwningClass] must be a boolean value")
}
validateConstraint = constraintParameter
super.setParameter(constraintParameter)
}

boolean supports(Class type) { true }

String getName() { NAME }

private boolean isCyclic(original, node) {
boolean cyclic = false
while (node != null) {
if (node.id == original.id) {
cyclic = true
break
}
node = node."$propertyName"
}
return cyclic
}
}

Then I just needed to register my new constraint by adding ConstrainedProperty.registerNewConstraint(AcyclicConstraint.NAME, AcyclicConstraint) to grails-app/conf/spring/resources.groovy. Using the constraint is as simple as this:

class Person {

    String name
    Person parent

    static constraints = {
        parent acyclic: true
    }
}

The constraint can be mixed with others such as nullable: true. The really nice thing is that the constraint implementation doesn't reference any of my domain classes directly, meaning it can be re-used in any other domain class.

When you're using similar validation logic in multiple classes defining a constraint like this is a much better option than using a validator closure and like many things in Grails it turns out to be pretty easy to implement.

Defaulting Joda Time Hibernate Types with Grails

2010-06-24T06:57:00.001+01:00

Grails 1.2 added the ability to define default Hibernate mappings that apply to all GORM classes. It turns out this is incredibly useful if you're using Joda-Time types instead of java.util.Date in your domain objects. Previously you had to specify the Hibernate user type for every single field, like this:

import org.joda.time.*
import org.joda.time.contrib.hibernate.*

DateTime dateCreated
DateTime lastUpdated
LocalDate birthday

static mapping = {
    dateCreated type: PersistentDateTime
    lastUpdated type: PersistentDateTime
    birthday type: PersistentLocalDate
}

Now you can just add this to your Config.groovy once and do away with the type mappings in the individual classes:

grails.gorm.default.mapping = {
    "user-type" type: PersistentDateTime, class: DateTime
    "user-type" type: PersistentLocalDate, class: LocalDate
}

Correction: I had previously posted that the method call in the default mapping DSL could be anything, but on further investigation it turns out it has to be user-type. Apologies for the error.

I'll look into the possibility of the Joda-Time plugin doing this itself to make things even easier.

Export Selenium IDE scripts for Grails

2010-06-15T14:18:00.005+01:00

Thanks to the documentation provided on Adam Goucher's blog I've created a simple Firefox plugin that extends the Selenium IDE and adds formatters that allow you to export a script as a Grails Selenium RC test case. You have the option of JUnit 3 (extends GroovyTestCase, uses assertEquals) or JUnit 4 (doesn't extend anything, uses annotations and assertThat) formats.

I think everything is working. There are some things I'd still like to do:

Custom commands are currently exported as commented out which they don't need to be given Grails Selenium RC test cases can call Javascript extension functions directly.

I haven't yet figured out how to output a command such as

selenium.waitForAlertPresent()

rather than

selenium.waitFor {
    selenium.isAlertPresent()
}

The Firefox plugin is available from Selenium HQ. You will need to have Selenium IDE installed for it to work, obviously. Once installed you should find options

Export Test Case As -> Grails Selenium RC (JUnit 3)
Export Test Case As -> Grails Selenium RC (JUnit 4)

have been added to Selenium IDE's File menu.

Whilst I would certainly not leave the exported test case as is it's a quick way to get started. From the exported script you can easily refactor out some page objects or helper methods.

Grails Spring Events Plugin

2010-05-28T13:27:00.000+01:00

Following on from my last post I've developed a Grails plugin that packages the asynchronous events behaviour up and adds some extra useful functionality.

In addition to the asynchronous event processing the plugin gives you:

A publishEvent method attached to all domain classes, controllers and services.
A Hibernate session bound to the listener thread for the duration of the notification so that listeners can access lazy-loaded properties, etc.
The ability to have a "retry policy" for certain types of failed notifications on individual listeners. This is particularly useful for listeners that do things like invoking external web-services that may be periodically unavailable.

To install the plugin just use:

grails install-plugin spring-events

The code and some more detailed documentation is on GitHub. I'll be migrating the docs to the plugin's page on grails.org soon.

Asynchronous application events in Grails

2010-05-03T21:59:00.003+01:00

On the project for my current client we've been using JMS in a rather naïve way for some time now. We've also experienced a certain amount of pain getting JMS and ActiveMQ configured correctly. However, all we're really using JMS for is asynchronous event broadcasting. Essentially we have a handful actions such as flushing hardware caches and notifying external systems that take place when a document changes. We don't want these things blocking the request thread when users save data.

After wrestling with JMS one too many times we decided to take a look at Spring's event framework instead. It turns out it's extremely easy to use for these kinds of asynchronous notifications in a Grails application.

Essentially any artefact can publish an event to the Spring application context. A simple publishing service can be implemented like this:

import org.springframework.context.*

class EventService implements ApplicationContextAware {

    boolean transactional = false

    ApplicationContext applicationContext

    void publish(ApplicationEvent event) {
        println "Raising event '$event' in thread ${Thread.currentThread().id}"
        applicationContext.publishEvent(event)
    }
}

So a Grails domain class can then do something like this:

def eventService

void afterInsert() {
    eventService.publish(new DocumentEvent(this, "created"))
}

void afterUpdate() {
    eventService.publish(new DocumentEvent(this, "updated"))
}

void afterDelete() {
    eventService.publish(new DocumentEvent(this, "deleted"))
}

Grails services make ideal ApplicationListener implementations. As services are singleton Spring beans they are automatically discovered by Spring's event system without any configuration required. For example:

import org.springframework.context.*

class EventLoggingService implements ApplicationListener<DocumentEvent> {

    boolean transactional = false

    void onApplicationEvent(DocumentEvent event) {
        println "Recieved event '$event' in thread ${Thread.currentThread().id}"
    }
}

Of course, multiple listeners can respond to the same events.

If you run the code you will notice that by default Spring's event system processes events synchronously. The EventService and ApplicationListener will print out the same Thread id. This is not ideal if any of the listener implementations might take any time. Luckily it's easy to override the ApplicationEventMulticaster bean in resources.groovy so that it uses a thread pool:

import java.util.concurrent.*
import org.springframework.context.event.*

beans = {
    applicationEventMulticaster(SimpleApplicationEventMulticaster) {
        taskExecutor = Executors.newCachedThreadPool()
    }
}

Running the code again will show the event being published in one thread and consumed in another. If you have multiple listeners each one will be executed in its own thread.

Oddly, I would have thought it was possible to override the taskExecutor property of the default ApplicationEventMulticaster in Config.groovy using Grails' property override configuration, but I found the following didn't work:

beans {
    applicationEventMulticaster {
        taskExecutor = Executors.newCachedThreadPool()
    }
}

Selenium RC plugin v1.0

2010-05-03T07:59:00.002+01:00

The Selenium RC plugin for Grails has hit version 1.0 with:

Firefox 3.6 support
grabbing of screenshots on test failures
the ability to call JavaScript user extensions on the selenium object as though they were regular Groovy methods

among other features. Check out the documentation at http://robfletcher.github.com/grails-selenium-rc and install with grails install-plugin selenium-rc.

Building your app on Hudson with Multiple Grails versions

2010-05-03T06:35:00.001+01:00

For my Grails plugins I generally write a test project (or more than one). I thought it would be useful to be able to run these against multiple versions of Grails on my build server so I can spot any incompatibilities with the versions the plugin is supposed to support.

Using Hudson this turned out to be pretty straightforward.

You will need all the Grails versions you want to test against installed on your Hudson box. I put them in /opt so replace that with wherever you have them in the steps below. Also, if you want the builds to run in parallel and will be running any functional tests you'll need the Hudson Port Allocator plugin.

Start by creating a new job in Hudson and choosing "Build multi-configuration project".
The configuration dialog has an extra section "Configuration Matrix" where you can set up the different config combinations that will run. We're just interested in varying the Grails version so check the "Axes" and create a new axis named "GRAILS_VERSION" with the different versions you want in the "values" box, e.g. "1.2.0 1.2.1 1.2.2 1.3.0.RC2"
Set up your source code repository and build triggers as you would for any other project.
If you want to run the builds in parallel and are running any functional tests you will need to make sure that Grails starts up on a unique port. In the "Build environment" section add a new "Plain TCP port" named "GRAILS_PORT". If you're running anything else that needs a port such as a Selenium server you'll need one for that as well.

Add an "Execute shell" build step. Unfortunately the Hudson Grails plugin does not allow you to use a variable to specify the Grails version so you'll have to go old-school:

export GRAILS_HOME=/opt/grails-$GRAILS_VERSION
export PATH=$GRAILS_HOME/bin:$PATH
# assign Grails a unique port (you can skip this if you're only running unit/integration tests)
export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$GRAILS_PORT"
grails clean
grails upgrade --non-interactive
grails test-app --non-interactive

Check "Publish JUnit test result report" in the "Post-build Actions" section and specify target/test-reports/*.xml as the "Test report XMLs"

When you run the Hudson job you should see it kick off one sub-job for each Grails version. Each will check out the project, upgrade it to the relevant Grails version and run the tests.

I have my plugin test projects triggered by the plugin build and have them grab the plugin zip itself from the archived artefacts in the plugin's Hudson job. So I've added this before the grails test-app step:

grails install-plugin http://my.hudson.server/hudson/job/my-plugin/lastSuccessfulBuild/artifact/my-plugin-1.0.zip --non-interactive

There's a bug in Grails 1.3.0.RC2 that means you will need to split this into a wget followed by installing the zip from a file path rather than a URL. This is fixed in trunk so won't be a problem in Grails 1.3 final.

If you're using SVN I think you may need to revert any changes the grails upgrade has made to application.properties, etc. otherwise you'll get merge conflicts on the next run. This does not seem to be a problem with Git.

A Grails JUnit 4 test template

2010-04-21T14:26:00.002+01:00

Grails 1.3 upgrades the bundles JUnit to the newer JUnit 4 API. However, the test template used to generate classes by the Grails create-* scripts is still very JUnit 3-ish. Here's a replacement that will generate a skeleton test using JUnit 4 conventions:

To use this just run grails install-templates then copy the contents over the file src/templates/artifacts/Tests.groovy. You can delete anything else in the src/templates directory that you don't need. Then every time you use a Grails command such as grails create-service foo the generated test will use the JUnit 4 template.

Customising collection binding in Grails

2010-03-17T06:53:00.003Z

Following up on my earlier post about using custom PropertyEditor implementations to bind association properties I started looking into the options for custom binding one-to-many associations.

For example, tags, as seen on many sites (including Blogger) would typically be modelled as a Set of either String or some kind of Tag domain object. With tag functionality we really don't want to be selecting tags using a multi-select combo box. It would be ridiculously huge and hard to find anything in it. An auto-completer that recognises commas as delimiters would be the way to go. Script.aculo.us has an AJAX Autocompleter that can handle this kind of tokenised input. Similarly, the Grails RichUI plugin offers an autocomplete tag that uses a Yahoo! UI component. There are comparable controls available if you're using jQuery or some other Javascript library. What you will end up with is the browser submitting a single parameter containing a comma-separated String of tag names. The trick is to turn that into a collection of String or Tag domain instances.

For the former case binding is pretty easy. All you need is a property editor that converts a comma-separated String into a Set of String and vice-versa:

class TagListEditor extends PropertyEditorSupport {
void setAsText(String text) {
value = text.split(/,\s*/) as Set
}

String getAsText() {
value?.join(", ")
}
}

Then just register the editor in your PropertyEditorRegistrar implementation:

registry.registerCustomEditor List, "tags", new TagListEditor()

If you've gone the latter route and used a domain object to represent a tag things have been trickier. Grails didn't allow you to register custom property editors against one-to-many association properties as it considered it knew how to handle such binding.

I raised and fixed a Grails issue to allow for registering custom editors on one-to-many properties so from 1.2.2 and 1.3-M2 you will be able to use a property editor something like this:

class TagListEditor extends PropertyEditorSupport {
void setAsText(String text) {
value = text.split(/,\s*/).collect {
Tag.findByName(it) ?: new Tag(name: it)
} as Set
}

String getAsText() {
value?.name?.join(", ")
}
}

Grails plugins on Hudson

2010-03-11T17:50:00.006Z

Yesterday I spent some time setting up Hudson continuous integration build for my various Grails plugins. It was pretty straightforward but I've been asked to document the steps involved.

Each plugin has one or more test applications under test/projects each of which has grails.plugin.location.whatever = "../../.." in BuildConfig.groovy. So each plugin CI build needs to run not only the plugin's own unit tests but any integration and/or functional tests included in the test apps as well.

Hudson plugins

I installed the essential plugins I would need:

The Grails plugin makes building Grails projects much easier and allows you to build against multiple versions of Grails.
The Git plugin enables Hudson to pull from a Git repository such as GitHub.
The GitHub plugin adds links to the GitHub project in the Hudson build.
The Chuck Norris plugin encourages you to keep your tests from failing.

Configuring Grails

Under Manage Hudson -> Configure System there is a Grails Builder section where I entered the name and path of the various versions of Grails installed on the build box.

Connecting to GitHub

In the project configuration:

I pasted the GitHub project URL in the Github project field (shown if you have the GitHub plugin installed).
Under Source Code Management I selected Git then pasted the read-only repository URL into the URL of repository field.
I entered master in the Branch Specifier field as I've seen Hudson do odd things picking up changes from other branches before. I don't really need this behaviour and haven't had time to investigate it further yet.
I ticked Poll SCM under Build Triggers then set a cron expression of */5 * * * * so Hudson will poll GitHub every 5 minutes.

Grails build steps

Select Build with Grails from the Add build step drop down, then:
- Select the Grails Installation you want to use.
- Enter the Targets to run. These are the Grails commands that Hudson will execute. For example, I entered "test-app unit: -clean --non-interactive" package-plugin for the to run unit tests then package the plugin. The quotes allow arguments to be passed to individual targets so I'm telling the test-app phase to build clean and run non-interactively (i.e. Grails won't ask about things like installing or uninstalling plugins).
- The other fields can be left blank for now.
Add further Grails build steps for test applications. The only difference in configuring these is that the targets will likely be different and Project Base Directory needs to be set to the relative path where the test apps lives, e.g. test/projects/content-caching

Isolating each build

One thing I found is that it's probably a good idea to specify separate working directory for each build. I found that simultaneous builds would stomp over each other's compiled classes otherwise. This is particularly an issue for me where I have a build for the Selenium RC plugin whilst one of the test apps for the Springcache plugin uses the latest Selenium RC release to run tests.

Hudson provides various environment variables so I just set grails.work.dir to /tmp/.grails/${JOB_NAME} in every case.

The other consideration is the port that Grails applications will use. If you are running any kind of functional tests the Grails application will start and it each Hudson job needs to use a separate server port or simultaneous jobs will experience port contention. The server port is also configured in the Grails build step. I've just assigned a different one to each job but I couldn't think of a particularly clever way to automate the port assignment.

Test reports

Under Post-build Actions I ticked Publish JUnit test result report then entered a comma-separated set of paths to the XML reports generated by each build step. For example, the Springcache plugin has the following reports configured:

target/test-reports/*.xml,test/projects/springcache-test/target/test-reports/*.xml,test/projects/content-caching/target/test-reports/*.xml

That's picking up the root plugin's unit test reports and the reports generated by the springcache-test and content-caching test applications. Hudson does a good job of merging these together into a unified test report.

Running tests that need a browser

Some of my tests use Selenium RC which drives a real browser. Given that Hudson is running on a headless box I had to get the display exported to a box with an X Windows environment.

Still to do

Right now all the plugins are building against Grails 1.2.1 only so the next task is to set up parallel builds using Grails 1.3-RC1. Ideally I would like to use Hudson's multi-configuration project capability to do that but I haven't had a chance to look into it yet. I'd also like to use Gradle to build the various project with one build step but I'd lose the convenience in switching Grails versions that the Hudson Grails plugin gives me.

Page fragment caching with Grails Springcache plugin

2010-03-05T06:42:00.003Z

Yesterday I released the new version of the Springcache plugin for Grails. The new feature this brings to the table is page fragment caching driven by annotations on controller actions. The feature is based on EhCache Web and is even simpler and more powerful than the full page caching I blogged about recently. With the page fragment caching feature you can:

Define @Cacheable and @CacheFlush annotations on controller actions.
Have SiteMesh decorate cached and uncached output alike.
Use SiteMesh to mix dynamic page sections with cached sections.
Use SiteMesh and <g:include> to have multiple areas of the page that are cached and flushed independently of one another.
Cache the output of controller actions that use content negotiation separately according to the requested format.

There are some examples in the plugin documentation and the plugin source code has a test project with a variety of page fragment caching mechanisms used and tested.

Using a custom data binder with Grails domain objects

2010-02-25T13:04:00.002Z

Yesterday I read a post by Stefan Armbruster on how to register a custom data binder to lookup Grails domain objects by any arbitrary property. I wanted to go a little further so that I could not only bind existing domain instances but create new ones as well.

For example, let's say I have Artist and Album domain classes where Artist hasMany Albums and Album belongsTo Artist. Artist has a name property that is unique. On my create album page I want to be able to type in the artist name and have the save action in the controller automatically lookup an existing Artist instance or create a new one if it doesn't exist. Doing this I don't want to have to add or change anything in the save action itself - I could theoretically use dynamic scaffolding.

Adapting Stefan's PropertyEditor implementation I created this:

import java.beans.PropertyEditorSupport
import org.apache.commons.lang.StringUtils

class DomainClassLookupPropertyEditor extends PropertyEditorSupport {

Class domainClass
String property

String getAsText() {
value."$property"
}

void setAsText(String text) {
value = domainClass."findBy${StringUtils.capitalize(property)}"(text)
if (!value) {
value = domainClass.newInstance((property): text)
}
}
}

The crucial change is the if (!value) block which creates the new instance and populates the relevant property.

To make everything work I just need to:

Add the PropertyEditorRegistrar and place it in resources.groovy as per Stefan's post.
Have a text input or autocompleter with the name "artist" in my create album form.
Add artist cascade: "save-update" to the mapping block in Album so that when the Album is saved the new Album will get saved as well.

Full page caching in Grails with Ehcache-Web

2010-02-23T12:47:00.000Z

I'm on the verge of releasing a new version of the Springcache plugin that will feature a pretty cool new annotation-driven page fragment caching feature based on Ehcache-web. However one of the things that came up during discussions on the mailing list was full page caching. I mentioned that it was pretty straightforward and promised to blog about how to do it, so…

Install Ehcache-web

Add the following to your dependencies in BuildConfig.groovy:

runtime("net.sf.ehcache:ehcache-web:2.0.0") {
 excludes "ehcache-core"
}

Install Grails templates

Run: grails install-templates

Set up a page caching filter

Edit src/templates/war/web.xml and add this to the filters section:

<filter>
 <filter-name>MyPageCachingFilter</filter-name>
 <filter-class>net.sf.ehcache.constructs.web.filter.SimplePageCachingFilter</filter-class>
 <init-param>
  <param-name>cacheName</param-name>
  <param-value>myPageCache</param-value>
 </init-param>
</filter>

And this as the first entry in the filter-mappings section:

<filter-mapping>
 <filter-name>MyPageCachingFilter</filter-name>
 <url-pattern>/controller/action/*</url-pattern>
 <dispatcher>REQUEST</dispatcher>
</filter-mapping>

Configure the cache

Unfortunately you can't configure the cache in grails-app/conf/spring/resources.groovy using EhCacheFactoryBean instances as the servlet filter initialises before the Spring context. The cache definitions need to be added to a ehcache.xml which would normally be placed in grails-app/conf. Here is an example that works for the filter definitions above:

<ehcache>

    <diskStore path="java.io.tmpdir"/>

    <defaultCache
            maxElementsInMemory="10"
            eternal="false"
            timeToIdleSeconds="5"
            timeToLiveSeconds="10"
            overflowToDisk="true"
            />

    <cache name="myPageCache"
           maxElementsInMemory="10"
           eternal="false"
           timeToIdleSeconds="10000"
           timeToLiveSeconds="10000"
           overflowToDisk="true">
    </cache>

</ehcache>

You will need to add another filter and filter-mapping and potentially another cache for each URL pattern you want to cache separately.

Full page caching is fine for simple pages but isn't as flexible as fragment caching. For example, any authenticated state (such as a "Welcome back xxx" label) on the page would get cached across all users. Because Grails uses Sitemesh and has the <g:include> tag for including output from other controllers a fragment caching solution is a good fit. The 1.2 release of the Springcache plugin will add fragment caching functionality. However, I can imagine using full-page caching like this for things like RSS feeds.

A Gradle build for Grails plugins with test apps

2010-01-17T20:35:00.002Z

There's a fairly popular technique in Grails plugin development where the plugin has 'test apps' stored in test/projects/*, each of which references the plugin by including grails.plugin.location."my-plugin" = "../../.." in its BuildConfig.groovy. Doing this allows you as a plugin developer to:

Automate tests for sample projects using the plugin with significantly different configurations.
Test using domain classes, controllers, etc. that shouldn't be packaged with the plugin.
Test drive, changing code in both the test app and the plugin without the need to use grails package-plugin or grails install-plugin to pick up changes.

The downside is that to run all the plugin project's tests it's necessary to run the plugin's own tests, then change directory into each of the test apps and run their tests. Continuous integration config is also fiddlier for the same reason.

I wanted to find a way to automate this so I could run all my plugins' tests with a single command. I could write a bash script or an Ant build I guess, or even use maven <shudder>. However keeping things in Groovy-land I decided to try using Gradle which I've been meaning to look into for a while now. I saw Hans Dockter's presentation on Gradle at the London Groovy & Grails Exchange back in December and was impressed with how terse and expressive the syntax is, especially compared to XML based formats. Unfortunately one thing Gradle doesn't grok yet is Grails.

The solution I've come up with is based on a post by Helmut Denk on the gradle-user list. Gradle just uses Ant's exec task to shell out and execute the appropriate Grails command. Combining this with Gradle's multi-project build support I now have plugin builds that can be run with gradle test that will automatically descend into the test apps under test/projects/*.

The build.gradle file at the root of the project defines clean and test tasks: The test task will run the plugin's unit tests. The clean task is defined just once for all projects as there's no difference in how it is done - a nice examply of Gradle's DRY-ness.

Then in settings.gradle I tell Gradle where to find the various sub-projects:
Finally each test app has it's own build.gradle that defines its test task: This is only necessary if the test apps run tests differently to the plugin - here my app is running integration rather than unit tests - otherwise I could have defined test in the same DRY manner as clean.

The process is not as fast as it could be if Grails were wired in to Gradle properly. gradle clean test for the Springcache plugin and its two test apps takes just over 2 minutes on my MBP. Also, my Gradle-fu leaves a lot to be desired right now so I'm sure there are improvements that could be made with the way I'm handling the sub-projects. But, for the purposes of a pre-commit test run or Hudson build this works pretty well.

Upgrading Grails 1.1 -> 1.2

2010-01-04T21:22:00.004Z

We've just successfully upgraded our app from Grails 1.1 to 1.2 and pushed it to our QA environment. I thought I'd write up some of the issues we encountered in case anyone is bashing their heads against them.

Custom constraints that execute queries

Grails 1.2 now executes domain object validation on any unsaved objects when the Hibernate session flushes. One upshot of this is that if you have a custom constraint that executes a query you can end up with a StackOverflowError; the query causes the session to flush, causing the object to be validated, causing the query to run, causing the session to flush, etc.

The solution is to use one of the new domain class dynamic methods withNewSession to execute the query in a separate Hibernate session.

For example, we have a domain class where only a single instance can exist in a 'live' state. A simple unique constraint won't work as any number of instances can exist in other states. The final constraint looks something like this:

static constraints = {
    state validator: { value, self ->
        boolean valid = true
        if (value == State.LIVE) {
            Homepage.withNewSession {
                valid = Homepage.countByStateAndIdNotEqual(State.LIVE, self.id) == 0
            }
        }
        return valid
    }
}

Errors set outside constraints

The new validation on flush behaviour also caused us a more subtle and difficult to trace problem. One of our controllers checks that a date/time property on a domain class instance is in the future when the instance is saved. Doing this in a constraint isn't desirable as it makes it awkward to set up test data or re-save instances in other contexts - the rule really only applies to the domain class when the user creates or updates it on the one particular form.

The validation code in the controller looks something like this:

domainInstance.validate()
if (domainInstance.goLiveTime < new Date()) {
    domainInstance.rejectValue "goLiveTime", "min.notMet"
}
if (!domainInstance.hasErrors() && domainInstance.save()) {
    // redirect
} else {
    // re-render form with errors
}

When upgrading one of our Selenium tests started failing as, although the save was failing, the error message did not get rendered on the form. What was happening is that at the end of the controller action the entity was not saved as it had errors but Hibernate attempted to flush and triggered re-validation which wiped out the error the controller had set.

If save had been called and validation failed then the object would have been made read-only so Hibernate would not attempt to flush it. My solution was to do this explicitly in the else block before rendering the form using GrailsHibernateUtil.setObjectToReadOnly(domainInstance, sessionFactory). It's certainly not ideal that the controller should be aware of this kind of low-level detail so some refactoring is in order here, but it solved the problem.

Specifying fetch modes on associations in criteria queries

We have one particularly nasty query that retrieves a domain object instance and loads an entire hierarchy of associations using several fetchMode "property", FetchMode.JOIN in the criteria query. For example if Author has many books and each Book has many chapters the query previously looked like:

Author.withCriteria {
    // some criteria
    fetchMode "books", FetchMode.JOIN
    fetchMode "chapters", FetchMode.JOIN
}

This always seemed syntactically odd as it appears that chapters is a property of Author when it is in fact a property of each member of Author.books. In Grails 1.2 the syntax makes much more sense as you nest the fetchMode declarations just as you would other criteria. So the example above would become:

Author.withCriteria {
    // some criteria
    fetchMode "books", FetchMode.JOIN
    books {
        fetchMode "chapters", FetchMode.JOIN
    }
}

Date binding format

Date binding now uses a fixed format of yyyy-MM-dd HH:mm:ss.S rather than the short format of the request locale. We have a couple of forms where dates are entered as text rather than using a date picker or rich control and the users (and our tests) expect to be able to enter dates as dd/MM/yy. In order to keep this functionality working I had to create a class that implements PropertyEditorRegistrar and deploy it in grails-app/conf/spring/resources.groovy. The class registers a custom date editor using the required format.

Because the custom property editor registration happens on a per-request basis it would be simple to use the same technique to allow users to enter dates in the locale format they are used to (Americans with their crazy month-day-year format and everyone else with something sensible, for example).

Setting String values in g:set tags

One of the main features of Grails 1.2 is the enhanced GSP rendering performance. One way this was achieved as I understand it is that mutable streams are used rather than immutable String instances where possible. One minor change I had to make to support this was changing g:set tags in the format <g:set var="name">${value}</g:set> to <g:set var="name" value="${value}"/>Variable created in the first manner are now instances of StreamCharBuffer rather than java.lang.String which sometimes caused problems when they were used later.

Flash scope and null values

The flash scope object available to controllers is now an instance of ConcurrentHashMap which means none of its keys can map to null. There were a few places where we were setting flash.message to the result of a method call that might return null. I simply ensured the empty string was used instead.

Ultimately the upgrade took a lot less time and effort than when we went from Grails 1.0.3 to 1.1 and we thankfully didn't encounter any blocking issues. The impression I have is that the platform's really maturing nicely. Will be cool to start using some of the new features of 1.2 in the next few weeks.

Selenium RC tests with a running Grails app

2010-01-02T12:23:00.000Z

I posted recently about the new remote mode feature of the Selenium RC plugin. One thing I forgot to mention is that this feature can also be used to run your Selenium tests interactively against a running app instance. You can start your app as normal using grails run-app then either open a second terminal or background the process (Ctrl-Z in a bash shell) then use grails -Dselenium.remote=true test-app other:selenium <test name> to run individual tests without stopping or re-starting the app. With the app running development mode you can effectively test-drive using Selenium tests and Grails' artefact reloading capabilities.

One thing I should clarify is that direct domain class access will not work in remote mode. I'm thinking about ways to add fixtures support to the Selenium RC plugin so there's a good alternative approach that will work in remote mode.

Using GMock to complement Grails mockDomain

2010-01-01T14:33:00.002Z

Since Grails 1.1 we've had pretty good unit testing support via GrailsUnitTestCase and its sub-classes. The mockDomain method is particularly useful for simulating the various enhancements Grails adds to domain classes. However, there are some domain class capabilities, such as criteria queries and the new named queries, that can't really be simulated by mockDomain.

So assuming we're trying to unit test a controller that uses criteria methods or named queries on a domain class how can we enhance the capabilities of mockDomain? One of my favourite Groovy libraries is GMock which I use in preference to Groovy's built in mock capabilities. One of its really powerful features is the ability to use 'partial mocks', i.e. to mock particular methods on a class whilst allowing the rest of the class to continue functioning as normal. This means we can layer a mocked createCriteria, withCriteria or named query call on to a domain class that is already enhanced by mockDomain.

First off you need to add the GMock dependency to your BuildConfig.groovy. Since GMock supports Hamcrest matchers for matching method arguments you'll probably want those as well:

    dependencies {
        test "org.gmock:gmock:0.8.0"
        test "org.hamcrest:hamcrest-all:1.0"
    }

If you're using an earlier version of Grails you'll need to just grab the jar files and put them in your app's lib directory.

Then in your test case you need to import GMock and Hamcrest classes and add an annotation to allow GMock to work:

    import grails.test.*
    import org.gmock.*
    import static org.hamcrest.Matchers.*

    @WithGMock
    class MyControllerTests extends ControllerUnitTestCase {

Adding criteria and named query methods is now fairly simple:

Mocking a withCriteria method

    def results = // whatever you want your criteria query to return
    mock(MyDomain).static.withCriteria(instanceOf(Closure)).returns(results)
    play {
        controller.myAction()
    }

Breaking this example down a little

mock(MyDomain) establishes a partial mock of the domain class.
instanceOf(Closure) uses a Hamcrest instanceOf matcher to assert that the withCriteria method is called with a single Closure argument (the bit with all the criteria in).
returns(results) tells the mock to return the specified results which here would be a list of domain object instances.

In this example we're expecting the withCriteria method to be called just once but GMock supports more complex time matching expressions if the method may be called again.

Mocking a createCriteria method

The withCriteria method returns results directly but createCriteria is a little more complicated in that it returns a criteria object that has methods such as list, count and get. To simulate this we'll need to have the mocked createCriteria method return a mocked criteria object.

    def results = // whatever you want your criteria query to return
    def mockCriteria = mock() {
        list(instanceOf(Closure)).returns(results)
    }
    mock(MyDomain).static.createCriteria().returns(mockCriteria)
    play {
        controller.myAction()
    }

This is only a little more complex than the previous example in that it has the mocked list method on another mock object that is returned by the domain class' createCriteria method. mock() provides an un-typed mock object as we really don't care about the type here.

Mocking a named query

For our purposes named queries are actually pretty similar to createCriteria.

    def results = // whatever you want your criteria query to return
    def mockCriteria = mock() {
        list(instanceOf(Closure)).returns(results)
    }
    mock(MyDomain).static.myNamedQuery().returns(mockCriteria)
    play {
        controller.myAction()
    }

Some other examples:

Mocking a named query with an argument

    mock(MyDomain).static.myNamedQuery("blah").returns(mockCriteria)

For simple parameters you don't need to use a Hamcrest matcher - a literal is just fine.

Mocking a withCriteria call using options

You can pass an argument map to withCriteria, e.g. withCriteria(uniqueResult: true) { /* criteria */ } will return a single instance rather than a List. To mock this you will need to expect the Map as well as the Closure:

    def result = // a single domain object instance
    mock(MyDomain).static.withCriteria(uniqueResult: true, instanceOf(Closure)).returns(result)

Mocking criteria that are re-used

It's fairly common in pagination scenarios to call a list and count method on a criteria object. We can just set multiple expectations on the mock criteria object, e.g.

    def mockCriteria = mock() {
        list(max: 10).returns(results)
        count().returns(999)
    }
    mock(MyDomain).static.myNamedQuery().returns(mockCriteria)

The nice thing about this technique is that it doesn't interfere with any of the enhancements mockDomain makes to the domain class, so the save, validate, etc. methods will still work as will dynamic finders.

Be aware however, that what we're doing here is mocking out the criteria queries, not testing them! All the interesting stuff inside the criteria closure is being ignored by the mocks and could, of course, be garbage. Named queries are pretty easy to test by having integration test cases for your domain class. Criteria queries beyond a trivial level of complexity should really be encapsulated in service methods or named queries and integration tested there. Of course, GMock then makes an excellent solution for mocking that service method in your controller unit test.

Selenium RC tests with a remote app

2009-12-29T12:18:00.001Z

I've just released a new snapshot of the Selenium RC plugin. The main new feature is the ability to run your Selenium tests against a running instance of your application instead of having the application start and stop as part of the test phase. A number of people have requested this. It's really useful for CI environments where you may want to start the app up on a different server from where the tests are being run.

All you need to do is set selenium.remote = true in your SeleniumConfig.groovy and set selenium.url to the root URL of the server where the app is running. Once this is done you can run tests as normal (note: when remote mode is used Selenium tests run in the other phase rather than the functional phase).

The really useful thing is that SeleniumConfig.groovy can contain environment blocks just as other Grails config files can. This means you can enable remote mode in your CI environment but continue to test as normal locally. For example:

selenium {
    remote = false
    // etc.
}

environments {
    hudson {
        selenium.remote = true
        selenium.url = "http://my.test.server/"
    }
}

The plugin release is a snapshot so you need to specify the version explictly: grails install-plugin selenium-rc 0.2-SNAPSHOT and you need to be using Grails 1.2.

New Springcache Plugin Version

2009-12-12T16:49:00.001Z

After saying it might take me a while I got stuck in and rewrote the Springcache plugin from scratch. The new version is up now. The plugin now requires Grails 1.2.0-M3 or greater as it's reliant on Spring 3.0.

The plugin allows you to declare caching and flushing behaviour on service methods (well, any Spring bean methods, but services are typical) using @Cacheable and @CacheFlush annotations. This is really useful for service methods that perform long-running or expensive tasks such as retrieving data from web services, network resources, etc. The plugin is less appropriate for service methods that retrieve data using GORM/Hibernate as you could just use the 2nd level cache, although there's nothing stopping you doing so if it makes sense in your domain.

The documentation and source code are in the usual places. Here's a quick summary of the changes and new features:

No longer depends on the (discontinued and non-Spring 3 compatible) spring-modules-cache library.
No more mapcache, the plugin now uses ehcache by default.
Only ehcache is supported directly but it's really easy to implement an adapter for other caching libraries if you need to.
Slightly simplified configuration. Some minor tweaks will be needed if you're upgrading from an earlier version of the plugin.
Bean names are now prefixed with "springcache" so they're much less likely to clash with other things in your Spring context.

There has been some interest in some new features such as a taglib for caching chunks of views which I may start looking at shortly.

Springcache Plugin Status

2009-11-26T22:56:00.000Z

A couple of people have asked me about the Springcache plugin and Grails 1.2. Currently the plugin is not compatible with 1.2-M3 and onwards. The reason for this is that the spring-modules-cache library that the plugin depends on is not compatible with Spring 3.0 which is used by the newest versions of Grails. Not only that but the spring-modules project has been discontinued altogether. There is a fork of it on Github but that doesn't look terribly active either. This means that at some point I'm going to have to sit down and write some code to bring the annotation driven caching inside the plugin itself and drop the spring-modules dependency. However, just to compound things the Spring documentation on the relevant area hasn't been updated yet and still references the classes that have been removed from the API which is what's causing the incompatibility in the first place!

So, it's on my to-do list but don't hold your breath!

Why do Strings behave like a Collection?

2009-11-21T14:17:00.001Z

Mostly I love Groovy but every now and then some behaviour gets on my nerves. For example, why do the iterator methods on String behave as though it was a collections of characters rather than an Object? If I define a method with a dynamic-typed parameter like:

void printNames(names) {
    names.each {
        println it
    }
}

Passing a collection of Strings results in each String being printed. Passing a single String results in each character of the String being printed separately. I would have thought that the Object behaviour was a more appropriate default here (iterators on Object are called once passing the object itself). After all the .chars property is available if you really want to represent a String as a collection of characters.

The solution is to override the method:

void printNames(String name) {
    printNames([name])
}

Unlike in Java dispatch to such overridden methods in Groovy is based on the runtime rather than declared type of the parameter.

You might think this is a trivial point, but the scenario that keeps biting me is HTTP parameters passed into a Grails controller. If you have multiple inputs with the same name, or a multi-select then sometimes params.x is a String and sometimes it's a List of Strings. Any code dealing with that parameter has to handle the two cases separately.

Slides from GGUG

2009-11-21T06:45:00.002Z

I gave a talk last night at the London Groovy & Grails User Group about the Selenium RC plugin and how to use it. I pitched the talk assuming that people were somewhat familiar with Selenium and had probably used the Selenium IDE. I'm not sure if that was the right way to go but the talk went pretty well and people seemed interested in what I had to say. I ended up talking a lot about the page object pattern and how awesome it is for building manageable suites of Selenium tests.

I really wanted to show a few quick examples but I had neglected to consider that a mini display port VGA adapter might be a useful thing to bring and unfortunately with Skills Matter being in the middle of an office move there wasn't one to hand. Luckily I was able to borrow a laptop so at least I could get my slides up on the projector.

Testing Grails Applications With Selenium Rc

View more presentations from Robert Fletcher.

Grails Selenium RC Plugin Released

2009-11-05T20:58:00.001Z

I've finally got the first release of the Selenium RC plugin out of the door. Just run grails install-plugin selenium-rc to install.

For anyone unfamiliar with Selenium RC basically the plugin allows you to write and run Selenium tests in Groovy to functionally test Grails apps using a real browser. The tests run in the functional phase using grails test-app or grails test-app -functional.

In terms of compatibility there is a very minor issue with Safari and some slightly more annoying ones with IE - all of which the plugin's default configuration will work around for you. Unfortunately Firefox 3.5 on OSX Snow Leopard doesn't want to play at all due to an open Selenium bug. Firefox on other platforms, Google Chrome and Opera all appear to be 100% compatible.

I've tried to make writing the tests themselves as similar to regular Grails unit and integration testing as possible. Not only that, if you have the Spock plugin installed you can write your Selenium tests as Spock Specifications.

Documentation is here, source code is on GitHub and issues and feature requests can be raised on Codehaus' JIRA. I've got plenty of things to work on for future releases such as Selenium Grid integration and automatic screen-grabbing when assertions fail.

Talk on Selenium RC Testing In Grails

2009-11-02T17:24:00.000Z

I'll be delivering a short talk about testing Grails apps with the Selenium RC plugin I'm working on at the London Groovy and Grails User Group on 20th November. Details are here.

The plugin is basically release ready, I'll try to make the final push some time this week so that it's out in the wild in advance of the talk (and hopefully any particularly idiotic bugs are reported and fixed before I make a total fool of myself).

Git and Hudson

2009-11-02T10:48:00.000Z

I just encountered a rather annoying problem when running Hudson as a service on Ubuntu. I was getting the following exception whenever the build checked code out of GitHub:

hudson.plugins.git.GitException: Could not apply tag hudson-Selenium_RC_Plugin-23
 at hudson.plugins.git.GitAPI.tag(GitAPI.java:265)

It turns out Git needs a username to be set and the hudson user that the Debian package creates when Hudson is installed doesn't have one. Easily fixed by using sudo nano /etc/passwd to add Hudson,,, into the hudson user's entry (if you look at your own entry you should see where it needs to go).

Testing Grails Apps With Selenium-RC

2009-09-14T16:00:00.000+01:00

I mentioned in my last post that I'd started to think we were going about Selenium testing all wrong. For years I've been writing Selenium tests using the Selenium IDE, so the tests are in HTML files. When doing pre-commit testing and on CI the tests are run with the HtmlTestRunner. This technique is fine and well - developing tests with the IDE is fast and they can be easily worked on an re-tested in the IDE until they are correct. However, some things have always bothered me and Adam Goucher's thoughts on the Selenium Value Chain crystallised some of those suspicions.

So what are the big advantages of using Selenium RC over just the IDE and HtmlTestRunner?

Test design

Selenium tests written in HTML tend to either become an amorphous blob, not testing one simple thing or proliferate to the point of having hundreds of small test files. With Selenium RC you can write Selenium test classes with multiple test case methods just as you would unit tests. Each test case can be neat, self-contained and share set up with other test cases in the same class. Also by abstracting out common command sequences into utility methods you can start to develop a higher level DSL for testing your application. In HTML tests this can be done by writing extension functions in Javascript but it's not as straightforward.

Data fixtures

Data set up is the biggest headache when writing Selenium tests in HTML. You can't drop out of selenese and set up a couple of domain objects so you end up either trying to run all your tests against a monolithic data fixture (a Sisyphean task if any of your tests make modifications to the data) or write some sort of URL endpoints that can set up and tear down data. We've done the latter and it's resulted in the single most bloated, horrible, misused bit of code in our app. Fixtures get re-used by other tests because they set up approximately appropriate data - until some change requires a change in the fixture and suddenly a bunch of unrelated tests fail. Selenium RC tests don't have this problem; you're in Groovy-land so you can access domain classes directly and use the invaluable build-test-data plugin to set data up.

Data verification

I mentioned previously the anti-pattern of false moniker testing and HTML based Selenium tests are terrible for this. If you have a test that creates some data, how do you verify it has done it correctly? The only way is to scrape the data on another screen - e.g. a CRUD list or show view. Ugh! With Selenium RC you can read the data from the database with standard Grails domain classes and verify your expectations directly.

I started experimenting over the weekend with getting Selenium RC tests written in Groovy to run using grails test-app -functional. It turns out to be really fairly straightforward and I've put some alpha code up on GitHub.

Thoughts on Testing

2009-09-11T02:56:00.004+01:00

These are some of the practices I try to live by when writing test coverage. Some of the detail may be Grails-centric since that's been my development platform of choice for the last couple of years, but the gist of the points is generally applicable.

I've been thinking about posting something like this for quite a while and was pushed over the edge by reading some interesting things...

Simon Baker recommended that I read what J.B. Rainsberger has to say about integration tests which chimed with my disgruntlement with them and my tendency to refactor old integration tests into unit tests when I touch related code.
Tim Ottinger posted about stopping the 'bad test death spiral'.
Jerome Pimmel pointed me at this article that adds to my growing suspicion that we're going about Selenium testing all wrong!

Make just one (logical) assertion per test.

The prime attribute of a good test case is legibility. When my change breaks your test I don't what to be scratching my head trying to figure out what it is your the test is trying to prove. The best way to achieve legibility is to have each test case test just one thing. Sure, it may take more than one actual assert statement to test it properly. For example to test a query returns correct values you may want to assert that the return value is non-null, that it is a collection with the correct size and that each element of the collection complies with the query parameters. Those are separate assertion statements but in the service of a single logical assertion.

One of the worst habits you can get into when adding new functionality is to just bung a few more asserts into an existing test case. Imagine we're testing a list action on a Grails controller class. There's an existing test case to confirm the list of instances in the model are correct. You implement a new requirement to load some sidebar content for the page by adding some new stuff to the model (leaving aside the fact that there are better ways to achieve this). This change should be tested with a separate (set of) test case(s) to those testing the list model is correct. That may mean some duplication with several test cases invoking the same controller action with the same parameters but the trade-off is that each test case is self-contained, legible and will not break due to further changes that aren't related to the functionality the test is supposed to be verifying.

Name test cases according to what they are testing

This sounds obvious, but how often do you come across test cases named something like testSaveArticle that doesn't make it obvious what it is testing about saving an article? A good test case name is verbose but only as verbose as it needs to be; names like testListReturnsEmptyListWhenNoResultsFound, testListOnlyReturnsLiveAssets, testUserIsRedirectedToLoginFormIfNotLoggedIn tell me what I have broken when my changes cause those tests to fail. On the other hand if a test case name is too long or contains a lot of 'and's & 'or's it's probably a sign that the test is doing too much and not making a single logical assertion.

Avoid obliquely named 'helper' methods

Few things bug me more when trying to decipher other peoples' tests than helper methods whose names either don't describe properly what it is they do or that have a raft of side effects. These things make tests harder to read, not easier! Names like initMocks (to do what?), createArticles (how many? anything special about them? what behaviour are they trying to drive out?) or checkModel (what about it? how reusable is this?) require the person maintaining your test cases to find the implementation and figure out what it's doing.

Worse, such methods have a tendency to accumulate complex parameters over time, (often with defaults designed to minimise impact on existing code at the expense of making any sense) as they get re-used in new test cases that require them to do slightly different things.

Avoid the test-per-method anti-pattern

The 'test-per-method' anti-pattern is characterised by test cases named testMethodNameX that attempt to test all the paths through a given API method in a single test case (or worse, test only the happy path). A lot of people start out writing unit tests this way, probably because a lot of unit testing tutorials are written this way.

The problem is this pattern bundles a bunch of assertions into a single test and if the earlier ones fail the later ones won't even run meaning you can waste time uncovering layers of broken functionality as you work your way through the failures and re-run the test.

Legibility suffers as well as it may not be obvious at a glance what edge cases, boundary conditions and side-effects are covered and what the expectations for them are. If any kind of state has to be reset between different calls to a method then you will end up writing code that you simply wouldn't if you separated things out into multiple test cases.

Test edge cases and variant behaviour

Tests that only cover the happy path give you false confidence. Coverage should include things like...

What happens when null is passed to this method?
What is returned when a query finds no results?
What happens when you can't connect to that web service or the connection times out?
What happens when a user submits a form with incorrect data?
What happens when someone figures out your URL scheme includes domain instance ids and starts crafting their own URLs to retrieve records that don't exist?
What happens when someone figures out your URL includes a maxResults parameter and crafts a URL to request 10¹⁰⁰ results?

TDD is your friend here; if you don't write the code until you've got a test for it, it's a lot harder to miss coverage of edge cases. Similarly the first thing you do when a bug is uncovered is to write a test that fails then fix it.

Test the thing you care about not a signifier

Max introduced me to the anti-pattern of false moniker testing and it's really changed the way I write assertions. It's an easy trap to fall into. Imagine you're testing that a query only returns 'live' records. Do you set up some live and some non-live records with particular names and then test that the names you expect show up in the results, or do you set up the data then assert that all the results actually are 'live' records? The former is terribly error prone (you assume you've set the data up correctly) and much less legible (which records are we expecting? Why that one in particular?)

Groovy's every iterator method is a godsend here enabling you to write code like:

def results = MyDomain.findAllLiveRecords()
assertFalse results.isEmpty()
assertTrue results.every { it.state == 'live' }

Avoid the monolithic setUp method

Much as it's bad to overload individual test cases with too many assertions it quickly becomes a problem when your setUp method is trying to set up data for too many test cases. Different test cases will probably require slightly different data to drive them which results in setUp trying to be all things to all cases or doing things that only a subset of the test cases actually need. As new functionality is developed the setUp becomes a black hole, accumulating more and more mass and becoming less legible and harder to refactor. The cohesion of the individual tests is harmed as well as it's hard to tell at a glance what data exists when an individual test case is run.

Ideally setUp should only be used for bootstrapping things that are truly common between all test cases and the test cases themselves should be responsible for setting up data to drive the test case's assertions.

Really bad examples of this problem will have setUp create a bunch of data and then some of the test cases actually delete it again before doing anything!
The worst example I've ever seen was from a project I worked on where the entire integration suite used a single DBUnit fixture! It was next to impossible to figure out what data would be returned in particular scenarios, impossible to change anything in the fixture without breaking a raft of other tests and the setUp and tearDown phases were so slow that the integration suite took 3 1/2 hours to run!

Keep data fixtures simple

On a related note it's worth striving to keep test data fixtures as simple as possible. For example, if a test has to wire up a bunch of properties and sub-graphs on domain object instances to satisfy constraints it can be really hard to see the wood for the trees when trying to figure out what data the test actually cares about.

When testing Grails apps the build-test-data plugin is a fantastic tool here. It allows sensible defaults to be configured so that tests only need to specify the properties they need to drive their assertions. With well thought out defaults in TestDataConfig.groovy the resulting test code can be very simple.

In unit tests the situation is easier. Since domain objects don't need to pass constraints to simulate persistent instances via mockDomain you don't need to jump through hoops setting properties and wiring up sub-graphs, test cases can set up simple skeleton objects that just contain sufficient state to drive the functionality and assertions the test case is making.

Test at the appropriate level

Slow-running integration tests that are not testing anything that couldn't be adequately tested with unit tests are just wasting time. In the Grails pre 1.1 this was a real issue as the enhancements that Grails makes to artefacts (domain classes, controllers, etc.) were not available in unit tests and therefore integration tests were the only practical way to Grails artefacts. Now with the unit testing support found in the grails.test package this is much less of an issue.

Integration tests definitely have their place but, I believe, should be used sparingly for things that a unit test cannot test and not just written as a matter of course. In Grails apps, for example, criteria queries and webflows cannot be tested with unit tests. I would also argue that there is value in integration testing things like complex queries or transactional behaviour where the amount of mocking required to get a unit test to work would result in you simply testing your own assumptions or expectations.

Do...

Write integration tests that add coverage that unit tests cannot.
Learn about the Grails unit testing support and understand the differences between unit and integration tests.

Don't...

Write an integration test when you should be writing a unit test or functional test.
Write integration tests that re-cover what is already adequately covered by unit tests.

I find that integration tests classes named for a single class (e.g. if I have a SecurityController and an integration test called SecurityControllerTests) are a bad code smell worth looking out for. They often tend to be an integration test masquerading as a unit test.

Refactoring Grails integration tests into unit tests doesn't (usually) take long, speeds up your build and I find often results in up to 50% less test code with the same amount of coverage.

Write functional test coverage

Okay, so your unit test verifies that your controller's list action will accept a sort parameter and return query results appropriately but it can't test that the associated view renders a column-heading link that sends that param or that clicking on it again reverses the order. When you start developing rich functionality on a web front-end a back-end unit test isn't going to do you a lot of good either.

Grails' functional testing plugin is a pretty good tool although for rich interaction testing I'd rather use Selenium.

Discussion of practices for these kind of tests really justifies a whole other post.

Session Bound Temporary Files

2009-09-07T06:04:00.005+01:00

I've just released Session Temp Files, a very simple plugin for managing a temporary file storage space bound to the HTTP session. I'm working on something where a user uploads a file at the start of a webflow conversation, the flow then validates the file and may require the user to enter extra information before finally the file is saved and an associated record is added to the database. The uploaded files needed to be stored somewhere between the initial upload phase and the successful outcome of the flow where they will be copied in to a permanent storage space. The file is really too big to keep in memory in the flow scope. It's easy enough to create a temp file and delete it at the end of the flow (even if the user cancels out of the flow) however users can also do things like close the browser, suddenly decide to do something different and type in a new URL, etc. The webflow then gets abandoned in an intermediate state. The flow itself will be destroyed when the HTTP session ends but the temp files I created will hang around until the OS decides to sweep the temp directory.

The plugin simply allows you to create a directory within the normal temp directory - System.properties."java.io.tmpdir" - that will get deleted when the HTTP session expires. It binds two new methods on to HTTPSession: getTempDir() returns the session's temp directory, creating it if it doesn't exist and createTempFile(prefix,suffix) works like File.createTempFile except that the file is created inside the directory returned by getTempDir.

The code is up on GitHub and the plugin is available from the standard Grails plugin repository via grails install-plugin session-temp-files.

JSON Rendering Your Classes

2009-08-12T06:06:00.006+01:00

It turns out adding custom JSON renderers for your own types to Grails is really easy. Since I habitually use Joda Time instead of the horrible java.util.Date and even more horrible java.util.Calendar I need to be able to render classes such as Joda's DateTime as JSON so that domain objects with fields of those types will convert properly.

Implementing a renderer for a type is as easy as this: DateTimeMarshaller.java. After that all that's required is to register the new renderer in BootStrap.groovy or some other appropriate spot with:

grails.converters.JSON.registerObjectMarshaller new DateTimeMarshaller()

I'll be adding this to the next release of the Joda Time Plugin so that it's completely transparent.

Update: Actually it's even easier than I thought. This being Groovy you can just use closures like so:

JSON.registerObjectMarshaller(DateTime) {
    return it?.toString("yyyy-MM-dd'T'HH:mm:ss'Z'")
}

Update: This is now built in to version 0.5 of the Joda Time Plugin.

Auditing data with the Acegi plugin and Grails upgrade pain

2009-07-10T09:02:00.001+01:00

I'm in the middle of trying to upgrade our app (again). We're running on Grails 1.1 now and I'm attempting to get us to 1.1.1 and from there to 1.2-M1. As Marc Palmer points out the more people using it the more likely it is that 1.2 final will be rock solid.

The problem that's biting us right now is GRAILS-4453 (or, more accurately, HHH-2763). We're using Grails' Hibernate events support to track the user that created and last updated assets in our system. This isn't just us being anal, the site editors frequently search the data using those criteria so it's an essential feature.

In Grails 1.1 this is simplicity itself. The following code goes in the domain class and that's all there is to it:

 User createdBy
 User updatedBy

 def authenticateService

 def beforeInsert = {
  createdBy = authenticateService.userDomain()
 }

 def beforeUpdate = {
  updatedBy = authenticateService.userDomain()
 }

Yes, it is quite 'exciting' that you're able to inject a service into a domain class instance. GORM only maps explicitly typed properties to the database so anything declared using def is effectively transient.

Unfortunately Grails 1.1.1 includes a newer version of Hibernate that introduces a particularly horrible problem. When saving any update to our domain object now we're faced with the error: collection [User.authorities] was not processed by flush(). The problem appears to be that the User instance attached to createdBy cannot be flushed when the beforeUpdate closure executes because it has a lazy-loaded collection of authorities. Even declaring the authorities collection as lazy: false doesn't help as the relationship is a bi-directional many-to-many - each Authority also has a collection of all the Users who have been granted that role. Given that for the purposes of displaying data to the audience of our site this audit data doesn't matter a damn I really don't want to be eager fetching it. Also, given the nature of the User-Authority relationship, casual eager fetching could result in rather a lot of data being loaded in to memory (the User, his roles, all the other users with that role, all their roles...)

Our options seem to be:

Explicitly eager fetch the User data in places where the owning object will get updated. Since the domain class in question is the root of a heirarchy of 13 sub-classes (things this project has taught me #63: never do this) and varieties get updated by a service or two, at least one controller and one Quartz and several hundred Selenium test fixtures, it's going to be a massive PITA and just as bad to remove if/when HHH-2763 ever gets fixed.
Break referential integrity and store username or id rather than an actual domain object relationship. This feels horribly wrong and is likely to cause problems down the line.
Store the created/updated information as a domain object of its own. This would make the query to find data by who created or updated it more complex (although not impossibly so) and might actually be prone to the same original bug

I don't know if anyone might have come across this problem and has some kind of workaround (preferably one that isn't an evil hack). I'd really appreciate any pointers.

Update: This is fixed in Grails 1.2-M2

Scripting Plugin and Dependency Installation

2009-06-23T18:17:00.002+01:00

Another quick script that I find quite useful, particularly on continuous integration servers is my Prepare.groovy script. It solves a couple of problems

Some plugins are not as well behaved as others on installation, for example the Functional Testing plugin automatically installs when test-app or run-app is installed but will then immediately crash with a ClassNotFoundException because the libraries it depends on are not on the classpath yet. Re-running the command will then work but if the commands are part of a CI build it's probably already bombed out and reported a broken build.
The Ivy plugin is great but test-app and run-app won't invoke its get-dependencies target to pull down libraries the way they will automatically install plugins.
Sometimes the grails command you want the build to execute is supplied by a plugin so there's no convenient way to run it directly from a brand new workspace because the plugin isn't installed yet.

The script simply ensures all plugins are installed and then invokes the Ivy plugin's get-dependencies target (only if the Ivy plugin is installed - it won't blow up on you if you don't use Ivy). At that point you should have a fully workable workspace and be able to run any grails command you like without your app complaining that some library or other isn't present.

Joda Time Auto-Timestamping Issue on Grails 1.1.1

2009-06-18T05:54:00.004+01:00

I've not upgraded most of my code to Grails 1.1.1 yet so I hadn't noticed this problem until it was brought to my attention by Manuel Vio on the grails-user mailing list.

I raised a bug a few days ago around the fact that GORM identifies Joda Time types (and presumably any non-default property type) as a one-to-one association. Up to now the only problem this caused me was that it means the Joda Time plugin has to do some slightly hacky tricks in the scaffolding templates to prevent crashes when Grails tries to render the properties as though they were associated domain instances only to find they have no id property. However, on Grails 1.1.1 the bug is causing a nasty failure on auto-timestamping.

Under Grails 1.1 you can use a Joda DateTime for an auto-timestamped fields:

DateTime dateCreated
DateTime lastUpdated

static mapping = {
    dateCreated type: PersistentDateTime
    lastUpdated type: PersistentDateTime
}

These behave exactly like the regular Date fields with those names, i.e. they're set automatically on save and update. In Grails 1.1.1 the same class will fail on save with a GroovyRuntimeException Could not find matching constructor for: java.lang.Object(java.lang.Long).

There is a (somewhat strange) workaround - declare the timestamped fields as non-lazy!

DateTime dateCreated
DateTime lastUpdated

static mapping = {
    dateCreated type: PersistentDateTime, lazy: false
    lastUpdated type: PersistentDateTime, lazy: false
}

Digging into the Grails code a little I found that association properties have their getter wrapped in a lazy-loading proxy. Since GORM thinks Joda Time properties are associations, this happens to them. The auto-timestamping code initialises the properties using property.getType().newInstance(System.currentTimeMillis()). Unfortunately, because property is actually the lazy-load proxy rather than the real property property.getType() returns Object.

Checking for plugin updates

2009-06-15T11:24:00.003+01:00

If you're anything like me you probably like to keep all your Grails plugins updated to the latest versions. It's quite easy to miss a release announcement and not realise there's a new version available. Although grails list-plugins will list out the available versions and what version you currently have installed it doesn't really highlight the plugins you could upgrade. I threw this script together this morning to do exactly that. It will simply check the plugin versions you have installed against all configured repositories and notify you of any that could be updated.

Querying by Association Redux

2009-06-13T06:25:00.007+01:00

A couple of months back I posted about the problems with criteria queries where you want results with a collection property that contains some value.

Last night I was reading through my new copy of Glen Smith and Peter Ledbrook's Grails In Action and my eye was caught by a particular example where they use aliases in a criteria query. Sure enough a quick test this morning confirms that this is the solution to the problem.

To recap. If I have these domain classes:

class Pirate {
    String name
    static belongsTo = Ship
    static hasMany = [ships: Ship]
}

class Ship {
    String name
    static hasMany = [crew: Pirate]
}

and I want to query for all the Ships containing a particular Pirate I would probably try to do this:

def ships = Ship.withCriteria {
    crew {
        eq("name", "Blackbeard")
    }
}

Unfortunately, while I end up with the correct Ship instances, each of their crew collections only contains the item(s) that matched the criteria regardless of any others that may actually exist.

Using an alias we can do this type of query without resorting to HQL:

def ships = Ship.withCriteria {
    createAlias("crew", "c")
    eq("c.name", "Blackbeard")
}

A complete test case that proves the point using the domain classes above is:

class PirateTests extends GroovyTestCase {

    void setUp() {
        Ship.withSession {session ->
            def blackbeard = new Pirate(name: "Blackbeard")
            def jack = new Pirate(name: "Calico Jack")
            def bart = new Pirate(name: "Black Bart")
            [blackbeard, jack, bart]*.save()

            def ship1 = new Ship(name: "Queen Anne's Revenge")
            ship1.addToCrew blackbeard
            ship1.addToCrew jack

            def ship2 = new Ship(name: "Royal Fortune")
            ship2.addToCrew blackbeard
            ship2.addToCrew bart

            def ship3 = new Ship(name: "The Treasure")
            ship3.addToCrew jack
            ship3.addToCrew bart

            [ship1, ship2, ship3]*.save()

            session.flush()
            session.clear()
        }
    }

    void testQueryByAssoc() {
        def ships = Ship.withCriteria {
            crew {
                eq("name", "Blackbeard")
            }
            order("name", "asc")
        }

        assertEquals 2, ships.size()
        assertEquals "Queen Anne's Revenge, Royal Fortune", ships.name.join(", ")
        assertEquals "Expected 2 crew but found ${ships[0].crew.name}", 2, ships[0].crew.size()
        assertEquals "Expected 2 crew but found ${ships[1].crew.name}", 2, ships[1].crew.size()
    }

    void testQueryByAssocUsingAlias() {
        def ships = Ship.withCriteria {
            createAlias("crew", "c")
            eq("c.name", "Blackbeard")
            order("name", "asc")
        }

        assertEquals 2, ships.size()
        assertEquals "Queen Anne's Revenge, Royal Fortune", ships.name.join(", ")
        assertEquals "Expected 2 crew but found ${ships[0].crew.name}", 2, ships[0].crew.size()
        assertEquals "Expected 2 crew but found ${ships[1].crew.name}", 2, ships[1].crew.size()
    }
}

The first test fails as each Ship returned by the query only has Blackbeard in the crew despite the fact that they were created and saved with 2 crew members each. The second test retrieves the correct results.

Logging out the SQL generated by the Hibernate queries shows the difference in what it's doing under the hood. The first (incorrect) query is:

select * from ship
left outer join ship_crew on ship.id = ship_crew.ship_id
left outer join pirate on ship_crew.pirate_id = pirate.id
where pirate.name = ?
order by ship.name asc

The second (correct) query is:

select * from ship
inner join ship_crew on ship.id = ship_crew.ship_id
inner join pirate on ship_crew.pirate_id = pirate.id
where pirate.name = ?
order by ship.name asc

So the first query is using left outer joins and the second is using inner joins. Running those queries directly in SQuirreL returns basically the same result set so the discrepancy must be in how Hibernate treats the results. It seems Hibernate has populated the Ship instance and its crew collection after the first query and therefore considers the collection to be initialized, although actually the data in the result set was not the complete collection. In contrast, after the second query you can see the individual SQL select statements as the lazy-loading fires when the assertions are done, so Hibernate obviously only populated the root Ship instance from the query results and is treating the collection property as uninitialized.

Hacking on Grails scaffolding templates

2009-06-13T05:51:00.005+01:00

Grails scaffolding is a fantastic mechanism for getting you up and running quickly. I love the fact that you can provide your own implementation of some controller actions or views and scaffolding will just fill in the gaps. That way you can start hand-crafting things when complexity increases past a certain point but still have Grails do a bunch of work for you.

I'd done some customising of scaffolding templates before for the Joda Time plugin but recently started playing with them again with a view to creating some I could re-use for future projects. Because scaffolding templates are not only used for dynamic views but also provide the basis of generated views that you go on to customise it's worth having a solid baseline. A couple of things I wanted to do were:

Mark up create and edit forms without using tables
Provide automatic indication of mandatory fields

I'm pretty happy with the result. This is how a standard create form now appears. The grab is from Safari 4 but the rendering is all but consistent across Firefox 3 on my Ubuntu box, Safari 4 on my Mac and even IE6, IE7, Google Chrome and Opera 9 on the Windows XP VirtualBox instances I use for cross-browser testing.

Forms Without Tables

Each form element is simply:

<div class="prop">
    <label for="${p.name}">Property Name</label>
    ${renderEditor(p)}
</div>

Which is reasonable semantic markup. Styling it cross-browser took me a while (I know just enough CSS to make me dangerous) but with a few helpful pointers from our resident front-end guru Jeff I was able to achieve that with a lot less browser-specific tweaking than I would have thought would be necessary.

The necessary CSS is just:

.prop {
    margin: 1em 0;
}
label {
    display: inline-block;
    margin-right: 1.5em;
    text-align: right;
    width: 8em;
}
input, select {
    vertical-align: middle;
}
textarea {
    vertical-align: top;
}

Making the label inline-block meant I can apply a width to it so all the fields will line up nicely but also keeps the label inline so I can then use vertical-align on the input to centre it on the text line of the label. The results are consistent cross-browser. Most solutions I've seen around the internet are horribly inconsistent between browsers (and operating systems with different font rendering) and often rely on nasty pixel-perfect margin and padding tweaks to try to align things nicely. That approach rapidly becomes a game of whack-a-mole as a tweak that fixes some alignment on IE will screw it up on Opera, fixing that will make Firefox do something odd, etc.

Automatically Indicating Mandatory Fields

A polite form should really indicate to the user what fields are mandatory and it turns out this isn't hard to achieve automatically in Grails scaffolded views. The scaffolding templates already use the constrained properties of the domain class to determine whether to display a form field for each property. It's a very small step to use the isNullable() and in the case of String properties isBlank() methods to decide whether to render a mandatory indicator.

I simply output a span with an asterisk inside the label then styled it with:

label {
    /* other label properties as above */
    position: relative;
}
label .mandatory {
    color: #c00;
    position: absolute;
    right: -1.25em;
}

The absolute positioning takes the asterisk out of the flow of the page so the labels and inputs line up neatly regardless of whether there's an asterisk or not and the right: -1.25em shoves it over into the space of the label's right margin. Positioning the asterisk perfectly in between the label and the input is tricky and not reliable cross browser. On IE the asterisk is too far to the right.

I tried other techniques such as disabling the label's right margin when the asterisk is present and setting the asterisk's span to the exact same size the margin would have been. Unfortunately it seems 1.5em as an element width is not quite the same thing as 1.5em as a right margin so the alignment of the labels and inputs was thrown off. Absolute positioning is necessary to maintain that alignment which is far more important to the eye than pixel-perfect placement of the asterisk itself.

Source Code

Here's the source code. Any suggestions for improvements would be very welcome. I based the templates on Marcel Overdijk's excellent i18n-templates, the only differences are in the rendering of the form fields and the surrounding fieldset.

web-app/css/forms.css (needs to be added to grails-app/views/layouts/main.gsp)
src/templates/scaffolding/create.gsp
src/templates/scaffolding/edit.gsp

Something this exercise has driven home for me is that when it comes to cross-browser styling less is definitely more. Taking away everything, setting the font size consistently cross-browser, then gradually building up with the simplest markup and styling possible yields the best results.

Faster, Pussycat! Kill! Kill!

2009-05-15T14:58:00.004+01:00

I mentioned previously that we'd seen some speed improvements in our page load time after upgrading to Grails 1.1. It turns out that for the week 3rd May to 10th May Sky Showbiz ranked 8th in the UK for page response time in the Gomez Benchmark Report with an average of 1.84 seconds. We came in ahead of some heavyweights like Amazon, HSBC, British Airways, etc. Sure, we weren't even close to Google (0.35 seconds!) and we rely on heavy caching for most of our responsiveness, but it's not bad for a site that's basically 8 virtual nodes (6 web and 2 DB) running a Grails app on Jetty and PostgreSQL.

Grails Upgrade

2009-04-28T14:55:00.023+01:00

We've just finished upgrading our sites to Grails 1.1 and I thought it would be worth sharing some of the fun we had with the process.

First and foremost was a call site caching bug in Groovy 1.6.0 that caused chaos with our unit and integration tests. Basically any time we mocked a class then demanded behaviour on a static method or any time we used the metaClass to override a static method (among other things that's pretty much any unit test that goes anywhere near a domain class) the behaviour wouldn't get torn down again. This caused symptoms such as stubbed dynamic finders leaking from unit tests and breaking integration tests that would pass when re-run in isolation. We got as far as refactoring a whole bunch of test code that mocked static methods before Burt Beckwith pointed out to me that simply replacing $GRAILS_HOME/lib/groovy-all-1.6.0.jar with the newer version from the Groovy 1.6.1 or 1.6.2 release would fix the problem.

Unfortunately the same bug has another effect that was quite catastrophic for us. Any Hibernate proxy of an instance of a domain class involved in an inheritance hierarchy will throw ClassCastException when you try to access a subclass property or do an instanceof check. I blogged already about the instanceof issue but it's actually wider ranging than I realised at the time. It can cause very unpredictable behaviour because complex relationships between objects and the way they are loaded by a controller before a page is rendered can cause this error to pop up under obscure conditions that are very hard to nail down with test coverage. To solve the problem we've had to use eager fetching in places where associations are typed as the base class of an inheritance hierarchy and to use explicit checks for proxies such as:

if (o instanceof HibernateProxy) {
    o = GrailsHibernateUtil.unwrapProxy(o)
}

Hopefully this bug will be fixed in Grails 1.1.1 and we will be able to use the default lazy loading behaviour in all relationships.

When is a Pirate not a Pirate? When it's a HibernateProxy

2009-04-15T02:49:00.004+01:00

At work we're in the middle of upgrading the Sky Entertainment sites from Grails 1.0.3 to Grails 1.1 - a long blog post will follow with a tale of our woes! One of the changes in Grails 1.1 is that one-to-one domain associations are now lazy by default. This raised an interesting problem for us as it meant we were now sometimes dealing with HibernateProxy instances where before we weren't.

For example, let's imagine we have a Pet domain class that is has an owner that is a Person and there exists another domain class, Pirate that is a specialisation of Person:

Pet.groovy

class Pet {
    String type
    String name
    Person owner
}

Person.groovy

class Person {

    String name

    boolean equals(Object o) {
        if (this.is(o)) return true
        if (o == null) return false
        if (!(o instanceof Person)) return false
        return name == o.name
    }


    // hashCode and toString ommitted
}

Pirate.groovy

class Pirate extends Person {

    String nickname

    boolean equals(Object o) {
        if (!super.equals(o)) return false
        if (!(o instanceof Pirate)) return false
        return nickname == o.nickname
    }

    // hashCode and toString ommitted
}

We want to test that either a regular Person or a Pirate can own a Pet. To do so we write an integration test like this:

void testAssignPersonAsPetOwner() {
    Person terry = new Person(name: 'Terry Elbow')
    Pet rex = new Pet(type: 'Dog', name: 'Rex', owner: terry)
    Pet.withSession {session ->
        assert terry.save(flush: true)
        assert rex.save(flush: true)
        session.clear()
    }
    assertEquals(terry, Pet.findByName('Rex').owner)
}

void testAssignPirateAsPetOwner() {
    Person longJohn = new Pirate(name: 'John Silver', nickname: 'Long John')
    Pet capnFlint = new Pet(type: 'Parrot', name: "Cap'n Flint", owner: longJohn)
    Pet.withSession {session ->
        assert longJohn.save(flush: true)
        assert capnFlint.save(flush: true)
        session.clear()
    }
    assertEquals(longJohn, Pet.findByName("Cap'n Flint").owner)
}

Simple enough. In both cases we just create a Pet and its owner, ensure they save properly, then assert that the owner is correct when we read the Pet back from the database. However the second test, where we assign a Pirate as the owner, fails. The final assertion fails with the message:

expected:<Pirate[Long John]> but was:<Pirate[Long John]>

Great... what?

When we read the Pet instance back from the database the owner association is set to lazy load so Pet.findByName("Cap'n Flint").owner is a HibernateProxy. The failure is caused by the fact that as owner is declared as being of type Person the proxy generated extends Person and therefore the instanceof check in Pirate's equals method fails...

if (!(o instanceof Pirate)) return false

In our case o is an instance of Person and an instance of HibernateProxy which if loaded would yield an instance of Pirate but is not itself an instance of Pirate.

Our options at this point seem to be...

Set owner to eager fetch.
Remove the instanceof check from the equals implementation in Pirate
Initialise the HibernateProxy so equals is working with the real object

The first option is just admitting defeat! Loading objects from the database when it's not necessary is wasteful and we don't want to go creating that kind of tech debt just to get our test working. The second option is not acceptable as it would mean the equals implementation on Pirate would violate the general contract of equals as aPerson == aPirate would return false but aPirate == aPerson would throw MissingPropertyException when it tried to execute the line:

return nickname == o.nickname

This leaves us with figuring out a way to initialise the proxy so we're always dealing with real objects. You may think it's an undesirable side-effect of the equals implementation to potentially force a database read but consider that it will do so anyway to perform the comparison of the nickname property.

There is a convenience initialize method in org.hibernate.Hibernate that can force a proxy to load, but it is void so our equals method would still have a reference to the HibernateProxy that is not actually an instance of Pirate. What we need to do is:

if (o instanceof org.hibernate.proxy.HibernateProxy) {
    o = o.hibernateLazyInitializer.implementation
}
if (!(o instanceof Pirate)) return false

As it turns out we can't do this directly in the equals implementation. It appears that getAt(String) is overridden in HibernateProxy's meta class so we get the error:

No such property: hibernateLazyInitializer for class: Person_$$_javassist_0

As luck would have it Grails has a utility method for doing exactly what we want and as it's written in Java no amount of meta class trickery will hide the hibernateLazyInitializer property from it. Our final, working equals implementation for Pirate looks like:

boolean equals(Object o) {
    if (!super.equals(o)) return false
    if (o instanceof HibernateProxy) {
        o = GrailsHibernateUtil.unwrapProxy(o)
    }
    if (!(o instanceof Pirate)) return false
    return nickname == o.nickname
}

Note: this is a simple example so we'll gloss over the fact that Pirate's equals isn't symmetric as

new Person(name: 'X') == new Pirate(name: 'X', nickname: 'Y')

returns true while flipping the operands causes it to return false. The problem and the solution apply any time inheritance and lazy-loading run up against class checking whether via instanceof, Class.isAssignableFrom, switch statements using a Class as a case, etc.

Grails 1.1 Plugins and IntelliJ Idea

2009-04-02T13:09:00.004+01:00

Good news. The recent release of the IntelliJ Idea EAP has added proper handling of the new plugin installation system in Grails 1.1 - i.e. it automatically picks up your plugins installed under ~/.grails/1.1/projects/<name>/plugins.

Querying By Association With Grails Criteria

2009-04-01T12:46:00.007+01:00

A common requirement is to select all instances of a domain class where one of its many-to-many associations contains a particular instance of another domain class. Consider these domain classes where a Ship has a crew composed of Pirates and any particular Pirate can be a part of the crew of several Ships:

class Ship {
   static hasMany = [crew: CrewMember]
   String name
}

class CrewMember {
   static belongsTo = [ship: Ship]
   Pirate pirate
}

class Pirate {
   String name
}

How can we use a criteria query to find all the Ship instances where a particular Pirate is a member of the crew? You might think it's pretty simple, surely:

Ship.withCriteria {
   crew {
       eq('pirate', blackbeard)
   }
}

However, this has a curious side-effect. For example:

Ship.withSession {session ->
   blackbeard = new Pirate(name: 'Blackbeard')
   jack = new Pirate(name: 'Calico Jack')
   bart = new Pirate(name: 'Black Bart')
   [blackbeard, jack, bart]*.save()

   def ship1 = new Ship(name: "Queen Anne's Revenge")
   ship1.addToCrew new CrewMember(pirate: blackbeard)
   ship1.addToCrew new CrewMember(pirate: jack)

   def ship2 = new Ship(name: "Royal Fortune")
   ship2.addToCrew new CrewMember(pirate: blackbeard)
   ship2.addToCrew new CrewMember(pirate: bart)

   def ship3 = new Ship(name: "The Treasure")
   ship3.addToCrew new CrewMember(pirate: jack)
   ship3.addToCrew new CrewMember(pirate: bart)

   [ship1, ship2, ship3]*.save()

   session.flush()
   session.clear()
}

def blackbeardsShips = Ship.withCriteria {
   crew {
       eq('pirate', blackbeard)
   }
}
blackbeardsShips.each {
   println "$it.name: ${it.crew.pirate.name.join(', ')}"
}

You might expect the output to be:

Queen Anne's Revenge: Blackbeard, Calico Jack
Royal Fortune: Blackbeard, Black Bart

but it is actually:

Queen Anne's Revenge: Blackbeard
Royal Fortune: Blackbeard

The criteria query has restricted the content of the associations to only the ones matching the eq('pirate', blackbeard) criterion. This is quite a problem as it may well not be immediately obvious that it's happened and even using ships*.refresh() (which would be a horrible hack, especially if we were likely to get more than a couple of results) doesn't seem to restore the missing entries. I don't think there is any criterion that does a 'contains' type match for a persistent collection. My guess is this problem isn't a Grails thing but a Hibernate thing - Grails' HibernateCriteriaBuilder is a very thin layer over Hibernate itself.

In this example there is an alternative as the association is bi-directional. We can query from the other side of the association:

def blackbeardsShips = CrewMember.withCriteria {
   projections {
       property('ship')
   }
   eq('pirate', blackbeard)
}

If the association wasn't bidirectional this wouldn't be possible and we'd probably have to resort to using HQL to make the query exhibit the correct behaviour.

Joda-Time and Grails Auto-Timestamping

2009-03-31T14:55:00.002+01:00

The Joda-Time Plugin docs state that Grails' auto-timestamping works with Joda-Time properties which is the case. However, when testing it can be useful to take advantage of Joda Time's DateTimeUtils class to mock the current time. This enables you to have new DateTime objects, for example, use a predictable timestamp. Unfortunately it doesn't play nicely with Grails' auto-timestamping which under the covers uses System.currentTimeMillis(). There are a couple of solutions to this. You can disable the auto-timestamping and define your own beforeInsert event which enables you to use DateTimeUtils.setCurrentMillisFixed. For example:

static mapping = {
    autoTimestamp false
}

def beforeInsert = {
    dateCreated = new DateTime()
}

The other option would be to mock out the value returned by System.currentTimeMillis(). This has the advantage of meaning you don't have to add code to your domain class to enable tests to work, but on the other hand it's all to easy to have such meta class modifications leak from test to test by not being torn down properly.

On a related note the next release of the Joda-Time plugin will bind additional methods to DateTimeUtils to scope mocking of the current timestamp, for example:

DateTimeUtils.withCurrentMillisFixed(aLong) {
    // do some stuff that requires a mocked current time
}

No more forgetting to call DateTimeUtils.setCurrentMillisSystem() in your tearDown method!

Internationalizing Domain Classes and Enums

2009-03-10T09:30:00.008Z

It's common to provide a toString implementation on a domain object that may well end up being used in the view. However, this doesn't allow for internationalization in the view. A good solution that I've used a number of times is to have classes implement Spring's MessageSourceResolvable interface. Consider this domain class that represents an image file:

grails-app/domain/Image.groovy

class Image {
    String name
    String path
    User uploadedBy
    org.joda.time.DateTime dateCreated

    static transients = ['file']
    File getFile() {
        new File(ConfigurationHolder.config.image.base.dir, path)
    }

    String toString() {
        "$name uploaded by $uploadedBy.username on $dateCreated - ${file.size()} bytes"
    }
}

The toString implementation is all well and good if we're targeting an English-speaking audience but with some simple changes we can make it i18n compliant:

grails-app/domain/Image.groovy

class Image implements org.springframework.context.MessageSourceResolvable {

    // properties as above

    static transients = ["codes", "arguments", "defaultMessage"]

    Object[] getArguments() {
        [name, uploadedBy.username, dateCreated.toDate(), file.size()] as Object[]
    }

    String[] getCodes() {
        ['image.info'] as String[]
    }

    String getDefaultMessage() {
        "$name uploaded by $uploadedBy.username on $dateCreated - ${file.size()} bytes"
    }
}

In our message properties file we can add:

grails-app/i18n/messages.properties

image.info={0} uploaded by {1} on {2,date} - ${3,number,integer} bytes

In the view we can display our object like this:

<g:message error="${imageInstance}"/>

Yes, that is the error attribute we're passing to the message tag! Grails intends the attribute to be used for outputting validation errors but the underlying mechanism is the same - Spring's ObjectError implements MessageSourceResolvable and that's how Grails' message tag resolves the displayed error message. Rather than passing separate code, args and default attributes to the tag we can pass the single MessageSourceResolvable instance and its implementation will take care of supplying those values.

Note: I added a message attribute to the message tag to avoid the confusion caused by using error. This is in Grails from version 1.2-M1 onwards.

We can now add translations of our object description. For example:

grails-app/i18n/messages_af.properties

image.info={0} opgelaai deur {1} op {2,date} - ${3,number,integer} grepe

It's worth noting that the format of the dateCreated property will be automatically determined by the request locale so the value will be formatted correctly for the user.

I've found this technique can also be very useful on enum classes. For example:

src/groovy/com/mycompany/Season.groovy

package com.mycompany

enum Season implements org.springframework.context.MessageSourceResolvable {
    SPRING, SUMMER, AUTUMN, WINTER

    Object[] getArguments() { [] as Object[] }

    String[] getCodes() {
        ["${getClass().name}.${name()}"] as String[]
    }

    String getDefaultMessage() { name() }
}

grails-app/i18n/messages.properties

com.mycompany.Season.SPRING=Spring
com.mycompany.Season.SUMMER=Summer
com.mycompany.Season.AUTUMN=Autumn
com.mycompany.Season.WINTER=Winter

grails-app/i18n/messages_en_US.properties

com.mycompany.Season.AUTUMN=Fall

Persisting DateTime with zone information

2009-03-03T13:31:00.005Z

I was stuck a while ago trying to figure out how to map PersistentDateTimeTZ in a GORM class. It's an implementation of Hibernate's UserType interface that persists a Joda DateTime value using 2 columns - one for the actual timestamp and one for the time zone. The DateTime class contains time zone information but a SQL timestamp column is time zone agnostic so you can lose data when the value is saved (the same exact problem exists when persisting java.util.Date values).

I could never figure out how to map the user type to my domain class property correctly. Just doing:

static mapping = {
    dateTimeProperty type: PersistentDateTimeTZ
}

Failed with:

org.hibernate.MappingException: property mapping has wrong number of columns.

I seem to remember someone on the Grails mailing list suggesting I tried treating the value as an embedded type. That also didn't work as GORM embedded types have to be Groovy classes in grails-app/domain and PersistentDateTimeTZ is written in Java and lives in a library jar.

I finally found the solution in the 2nd Edition of The Definitive Guide to Grails (which I can't recommend enough, by the way). The trick is to pass a closure specifying the two columns to the property definition in the mapping builder. The working code looks like this:

static mapping = {
    dateTimeProperty type: PersistentDateTimeTZ, {
        column name: "date_time_property_timestamp"
        column name: "date_time_property_zone"
    }
}

The order of the columns corresponds to the order of the values returned by the sqlTypes() method of whatever UserType implementation you're using.

Why would I ever want to disable the L2 cache?

2009-03-02T17:28:00.000Z

This question came up when pairing last week. We were going through our code-base adding the cache directive to a bunch of our domain classes. Grails is all about sensible defaults and it seems slightly odd that the level 2 cache is configured by default in DataSource.groovy but not actually used unless cache(true) is added to the mapping closure in each domain class. I wonder if anyone has any ideas why it might ever be a bad idea to use the L2 cache?

The only scenario I can come up with is situations where updates are made direct to the DB, bypassing Hibernate. This is, I would think, pretty rare (we do it for some rather naïve hit tracking and for voting on polls). Sure, in this circumstance the L2 cache will likely give you stale results. However, it's very much the exception rather than the rule.

Most other objections I've heard have been some variety of worry about caches eating up vast amounts of heap space and crashing the JVM (which is why cache implementations like ehcache use time-based and LRU eviction).

Oh, and yeah, we did have some issues with our changes but most seem to be to do with cruft and tech debt in our code (mostly now-redundant workarounds to GORM issues from older releases of Grails).

Hibernate POGOs, Immutability and Encapsulation

2009-03-01T11:21:00.000Z

I had a confusing experience this morning. I'm working on a simple web based game as a learning exercise.

I use a custom tag to determine if the currently logged in user is the active player (i.e. the player whose turn it is).

  <g:isNextPlayer>
    <!-- display something only active player should see -->
  </g:isNextPlayer>

The tag is used at various places in the page - displaying an "it's your turn" message, switching on bits of script, rendering buttons, etc. Suddenly I found the last couple of calls to the tag in a page weren't rendering the tag body.

The tag uses a transient property on the Game domain class that looks like this:

  Player getNextPlayer() {
    return players.empty ? null : players[turn % players.size()]
  }

The variable turn is simply an integer that gets incremented when a turn completes.

After poking around with debug logging I figured out that the implementation of getNextPlayer was fine and the value of turn wasn't being accidentally changed but the collection property players was getting re-ordered. What I'd done was introduce some code mid-way in the page that displays the players' names and scores - ordered by score.

  <g:each var="player" in="${gameInstance.players.sort { -it.score }}">
    <tr>
      <td>${player.user.username}</td>
      <td>${player.score}</td>
    </tr>
  </g:each>

I had assumed when I did this that Collection.sort(Closure) would return a copy (which is what the docs state - not that I'd checked). However, it seems that is only the case if the collection it's called on is not a List. On List instances the method acts as a mutator and does an in-place sort.

  def c = [1,2,3] as Set
  def s = c.sort { it }
  assert !(c.is(s))

Remove the 'as Set' and the assertion fails.

Although I'd argue that Groovy's sort implementation doesn't exactly stick to the principal of least surprise that's not really what concerns me. In my domain it doesn't make sense for me to be able to mutate the players property of Game. Once a game is created the players are fixed, they can't leave, can't take turns out of order. The state of an individual Player can change - his score can change, for example, but not the collection property of Game. In a regular non-persistent class this sort of invariant would be enforced by overriding the getter and setter:

  private List players
  List getPlayers() { players.asImmutable() }
  void setPlayers(List players) {
    this.players = []
    this.players.addAll players
  }

If my Game class was implemented that way I'd have got an UnsupportedOperationException from the sort call. If nothing else it would have saved me 5 minutes of "WTF?" and 10 minutes of trying to figure out if it was something to do with the second-level cache. Unfortunately, this technique doesn't quite work satisfactorily in a GORM domain class though. For a start declaring the property as private will result in a compilation failure - 'The field 'players' is declared multiple times.' Declaring the property without private gets you past the compilation phase but Grails 'addTo*' dynamic method isn't bound.

One of the things that's always nagged at the back of my mind when using GORM/Hibernate is the fact that all to often the domain classes end up resembling the anæmic domain model anti-pattern even to the extent of a design something like the cautionary tale of POTS. I like to put a certain amount of business logic in my domain classes and give them a reasonably rich API for managing their state without violating encapsulation but there's always an uncomfortable feeling that properties that really should be part of an object's private state are hanging out in the wind so that the persistence framework can access them.

Watchmen Widget

2009-02-12T06:05:00.001Z