Grails Upgrade

We've just finished upgrading our sites to Grails 1.1 and I thought it would be worth sharing some of the fun we had with the process.

First and foremost was a call site caching bug in Groovy 1.6.0 that caused chaos with our unit and integration tests. Basically any time we mocked a class then demanded behaviour on a static method or any time we used the metaClass to override a static method (among other things that's pretty much any unit test that goes anywhere near a domain class) the behaviour wouldn't get torn down again. This caused symptoms such as stubbed dynamic finders leaking from unit tests and breaking integration tests that would pass when re-run in isolation. We got as far as refactoring a whole bunch of test code that mocked static methods before Burt Beckwith pointed out to me that simply replacing $GRAILS_HOME/lib/groovy-all-1.6.0.jar with the newer version from the Groovy 1.6.1 or 1.6.2 release would fix the problem.

Unfortunately the same bug has another effect that was quite catastrophic for us. Any Hibernate proxy of an instance of a domain class involved in an inheritance hierarchy will throw ClassCastException when you try to access a subclass property or do an instanceof check. I blogged already about the instanceof issue but it's actually wider ranging than I realised at the time. It can cause very unpredictable behaviour because complex relationships between objects and the way they are loaded by a controller before a page is rendered can cause this error to pop up under obscure conditions that are very hard to nail down with test coverage. To solve the problem we've had to use eager fetching in places where associations are typed as the base class of an inheritance hierarchy and to use explicit checks for proxies such as:

if (o instanceof HibernateProxy) {
    o = GrailsHibernateUtil.unwrapProxy(o)
}

Hopefully this bug will be fixed in Grails 1.1.1 and we will be able to use the default lazy loading behaviour in all relationships.

Other problems we encountered included:

• The onLoad Hibernate event handler on domain classes now seems to run at a slightly different time in the object life cycle: before any eager fetched collections are loaded. We had one domain class that was doing some initialisation (calculating the percentage of votes each answer to a poll had received and storing it in a transient property) that no longer worked as the persistent collection it was trying to iterate over was always empty. This was simple to refactor out and in fairness was pretty horrible anyway.


• Config.groovy and related external config files are no longer loaded when running unit tests so any unit test that excercises code doing ConfigurationHolder.config.some.value.or.other blew up. We solved this by simply adding the following to _Events.groovy to load up config prior to the unit tests running:

  eventTestPhaseStart = { phase ->
      if (phase == 'unit') {
          ConfigSlurper slurper = new ConfigSlurper(GrailsUtil.environment)
          def configClass = getClass().classLoader.loadClass("Config")
          def config = slurper.parse(configClass)
          ConfigurationHelper.initConfig config // this step loads the external environment config file(s)
          ConfigurationHolder.config = config
      }
  }



• Logger configuration has completely changed (for the better) in Grails 1.1


• Some of our tests were explicitly setting the level of particular loggers to 'OFF' because the test is deliberately causing an error condition and we didn't want the console polluted with a huge stack trace. This no longer works as the logging abstraction has changed to SLF4J which does not allow you to directly set the level on its Logger class. Having better things to do than unwrapping logging abstractions I just turned logging off altogether in the test Grails environment.

  log4j = {
      root {
          off()
      }
  }



• It looks like binding errors don't get reported on nullable properties when using DomainClass.properties = params in a controller. I've raised a bug.


• There's been a minor change to the <g:select> tag. Previously the disabled attribute followed the HTML convention (disabled="disabled") but now the attribute value needs to be a boolean. I guess this actually makes more sense than the goofy HTML convention but I really expect most of the attributes on those sorts of tags to be pure pass-through.


• The mockDomain method used by unit tests seems to be a little inconsistent when dealing with inheritance heirarchies. If you do mockDomain(BaseClass) then try to use subClassInstance.subClassProperty it fails with a MissingPropertyException. I've raised a bug.


• We had customised Package.groovy to deploy our app at the root context. It seems this is now directly supported as an option in Grails 1.1 by setting grails.app.context = '/' in Config.groovy. I think this option was already available in Grails 1.0.3 but our patch pre-dated us upgrading to 1.0.3. One of the things I was trying to achieve while upgrading was to ensure we were patching Grails and any plugins as little as possible.


• I'd upgraded our selenium plugin (not the one in the Grails plugin repository) a while back to cope with Grails 1.1 and upgraded the selenium server it contained at the same time. It turns out the new selenium server will time out attempting to click any anchor that uses the javascript: protocol in its href. Luckily this was only being done in 2 places in our app and was easily fixed by using href="#" instead.

One other thing we decided to do was to replace our existing suite of webtests with functional tests. Webtest has never been popular with the developers on our team and we had made a number of modifications to version 0.4 of the plugin in order to support issues like config loading and running the app at the context root of the server. These modifications caused some headaches when trying to upgrade the plugin so we decided to drop it and port the tests over to the functional test plugin. Generally I'd still rather be writing Selenium tests than using either webtest or functional tests but the functional test syntax is a little nicer and less "pseudo-XML" than webtest's.

When is a Pirate not a Pirate? When it's a HibernateProxy

At work we're in the middle of upgrading the Sky Entertainment sites from Grails 1.0.3 to Grails 1.1 - a long blog post will follow with a tale of our woes! One of the changes in Grails 1.1 is that one-to-one domain associations are now lazy by default. This raised an interesting problem for us as it meant we were now sometimes dealing with HibernateProxy instances where before we weren't.

For example, let's imagine we have a Pet domain class that is has an owner that is a Person and there exists another domain class, Pirate that is a specialisation of Person:

Pet.groovy

class Pet {
    String type
    String name
    Person owner
}

Person.groovy

class Person {

    String name

    boolean equals(Object o) {
        if (this.is(o)) return true
        if (o == null) return false
        if (!(o instanceof Person)) return false
        return name == o.name
    }


    // hashCode and toString ommitted
}

Pirate.groovy

class Pirate extends Person {

    String nickname

    boolean equals(Object o) {
        if (!super.equals(o)) return false
        if (!(o instanceof Pirate)) return false
        return nickname == o.nickname
    }

    // hashCode and toString ommitted
}

We want to test that either a regular Person or a Pirate can own a Pet. To do so we write an integration test like this:

void testAssignPersonAsPetOwner() {
    Person terry = new Person(name: 'Terry Elbow')
    Pet rex = new Pet(type: 'Dog', name: 'Rex', owner: terry)
    Pet.withSession {session ->
        assert terry.save(flush: true)
        assert rex.save(flush: true)
        session.clear()
    }
    assertEquals(terry, Pet.findByName('Rex').owner)
}

void testAssignPirateAsPetOwner() {
    Person longJohn = new Pirate(name: 'John Silver', nickname: 'Long John')
    Pet capnFlint = new Pet(type: 'Parrot', name: "Cap'n Flint", owner: longJohn)
    Pet.withSession {session ->
        assert longJohn.save(flush: true)
        assert capnFlint.save(flush: true)
        session.clear()
    }
    assertEquals(longJohn, Pet.findByName("Cap'n Flint").owner)
}

Simple enough. In both cases we just create a Pet and its owner, ensure they save properly, then assert that the owner is correct when we read the Pet back from the database. However the second test, where we assign a Pirate as the owner, fails. The final assertion fails with the message:

expected:<Pirate[Long John]> but was:<Pirate[Long John]>

Great... what?

When we read the Pet instance back from the database the owner association is set to lazy load so Pet.findByName("Cap'n Flint").owner is a HibernateProxy. The failure is caused by the fact that as owner is declared as being of type Person the proxy generated extends Person and therefore the instanceof check in Pirate's equals method fails...

if (!(o instanceof Pirate)) return false

In our case o is an instance of Person and an instance of HibernateProxy which if loaded would yield an instance of Pirate but is not itself an instance of Pirate.

Our options at this point seem to be...

1. Set owner to eager fetch.
2. Remove the instanceof check from the equals implementation in Pirate
3. Initialise the HibernateProxy so equals is working with the real object

The first option is just admitting defeat! Loading objects from the database when it's not necessary is wasteful and we don't want to go creating that kind of tech debt just to get our test working. The second option is not acceptable as it would mean the equals implementation on Pirate would violate the general contract of equals as aPerson == aPirate would return false but aPirate == aPerson would throw MissingPropertyException when it tried to execute the line:

return nickname == o.nickname

This leaves us with figuring out a way to initialise the proxy so we're always dealing with real objects. You may think it's an undesirable side-effect of the equals implementation to potentially force a database read but consider that it will do so anyway to perform the comparison of the nickname property.

There is a convenience initialize method in org.hibernate.Hibernate that can force a proxy to load, but it is void so our equals method would still have a reference to the HibernateProxy that is not actually an instance of Pirate. What we need to do is:

if (o instanceof org.hibernate.proxy.HibernateProxy) {
    o = o.hibernateLazyInitializer.implementation
}
if (!(o instanceof Pirate)) return false

As it turns out we can't do this directly in the equals implementation. It appears that getAt(String) is overridden in HibernateProxy's meta class so we get the error:

No such property: hibernateLazyInitializer for class: Person_$$_javassist_0

As luck would have it Grails has a utility method for doing exactly what we want and as it's written in Java no amount of meta class trickery will hide the hibernateLazyInitializer property from it. Our final, working equals implementation for Pirate looks like:

boolean equals(Object o) {
    if (!super.equals(o)) return false
    if (o instanceof HibernateProxy) {
        o = GrailsHibernateUtil.unwrapProxy(o)
    }
    if (!(o instanceof Pirate)) return false
    return nickname == o.nickname
}

Note: this is a simple example so we'll gloss over the fact that Pirate's equals isn't symmetric as

new Person(name: 'X') == new Pirate(name: 'X', nickname: 'Y')

returns true while flipping the operands causes it to return false. The problem and the solution apply any time inheritance and lazy-loading run up against class checking whether via instanceof, Class.isAssignableFrom, switch statements using a Class as a case, etc.

Grails 1.1 Plugins and IntelliJ Idea

Good news. The recent release of the IntelliJ Idea EAP has added proper handling of the new plugin installation system in Grails 1.1 - i.e. it automatically picks up your plugins installed under ~/.grails/1.1/projects/<name>/plugins.

Querying By Association With Grails Criteria

A common requirement is to select all instances of a domain class where one of its many-to-many associations contains a particular instance of another domain class. Consider these domain classes where a Ship has a crew composed of Pirates and any particular Pirate can be a part of the crew of several Ships:

class Ship {
   static hasMany = [crew: CrewMember]
   String name
}

class CrewMember {
   static belongsTo = [ship: Ship]
   Pirate pirate
}

class Pirate {
   String name
}

How can we use a criteria query to find all the Ship instances where a particular Pirate is a member of the crew? You might think it's pretty simple, surely:

Ship.withCriteria {
   crew {
       eq('pirate', blackbeard)
   }
}

However, this has a curious side-effect. For example:

Ship.withSession {session ->
   blackbeard = new Pirate(name: 'Blackbeard')
   jack = new Pirate(name: 'Calico Jack')
   bart = new Pirate(name: 'Black Bart')
   [blackbeard, jack, bart]*.save()

   def ship1 = new Ship(name: "Queen Anne's Revenge")
   ship1.addToCrew new CrewMember(pirate: blackbeard)
   ship1.addToCrew new CrewMember(pirate: jack)

   def ship2 = new Ship(name: "Royal Fortune")
   ship2.addToCrew new CrewMember(pirate: blackbeard)
   ship2.addToCrew new CrewMember(pirate: bart)

   def ship3 = new Ship(name: "The Treasure")
   ship3.addToCrew new CrewMember(pirate: jack)
   ship3.addToCrew new CrewMember(pirate: bart)

   [ship1, ship2, ship3]*.save()

   session.flush()
   session.clear()
}

def blackbeardsShips = Ship.withCriteria {
   crew {
       eq('pirate', blackbeard)
   }
}
blackbeardsShips.each {
   println "$it.name: ${it.crew.pirate.name.join(', ')}"
}

You might expect the output to be:

Queen Anne's Revenge: Blackbeard, Calico Jack
Royal Fortune: Blackbeard, Black Bart

but it is actually:

Queen Anne's Revenge: Blackbeard
Royal Fortune: Blackbeard

The criteria query has restricted the content of the associations to only the ones matching the eq('pirate', blackbeard) criterion. This is quite a problem as it may well not be immediately obvious that it's happened and even using ships*.refresh() (which would be a horrible hack, especially if we were likely to get more than a couple of results) doesn't seem to restore the missing entries. I don't think there is any criterion that does a 'contains' type match for a persistent collection. My guess is this problem isn't a Grails thing but a Hibernate thing - Grails' HibernateCriteriaBuilder is a very thin layer over Hibernate itself.

In this example there is an alternative as the association is bi-directional. We can query from the other side of the association:

def blackbeardsShips = CrewMember.withCriteria {
   projections {
       property('ship')
   }
   eq('pirate', blackbeard)
}

If the association wasn't bidirectional this wouldn't be possible and we'd probably have to resort to using HQL to make the query exhibit the correct behaviour.