Export Selenium IDE scripts for Grails

Thanks to the documentation provided on Adam Goucher's blog I've created a simple Firefox plugin that extends the Selenium IDE and adds formatters that allow you to export a script as a Grails Selenium RC test case. You have the option of JUnit 3 (extends GroovyTestCase, uses assertEquals) or JUnit 4 (doesn't extend anything, uses annotations and assertThat) formats.

I think everything is working. There are some things I'd still like to do:

• Custom commands are currently exported as commented out which they don't need to be given Grails Selenium RC test cases can call Javascript extension functions directly.
• I haven't yet figured out how to output a command such as

  selenium.waitForAlertPresent()

  rather than

  selenium.waitFor {
      selenium.isAlertPresent()
  }

The Firefox plugin is available from Selenium HQ. You will need to have Selenium IDE installed for it to work, obviously. Once installed you should find options

• Export Test Case As -> Grails Selenium RC (JUnit 3)
• Export Test Case As -> Grails Selenium RC (JUnit 4)

have been added to Selenium IDE's File menu.

Whilst I would certainly not leave the exported test case as is it's a quick way to get started. From the exported script you can easily refactor out some page objects or helper methods.

Grails Spring Events Plugin

Following on from my last post I've developed a Grails plugin that packages the asynchronous events behaviour up and adds some extra useful functionality.

In addition to the asynchronous event processing the plugin gives you:

• A publishEvent method attached to all domain classes, controllers and services.
• A Hibernate session bound to the listener thread for the duration of the notification so that listeners can access lazy-loaded properties, etc.
• The ability to have a "retry policy" for certain types of failed notifications on individual listeners. This is particularly useful for listeners that do things like invoking external web-services that may be periodically unavailable.

To install the plugin just use:

grails install-plugin spring-events

The code and some more detailed documentation is on GitHub. I'll be migrating the docs to the plugin's page on grails.org soon.

Selenium RC plugin v1.0

The Selenium RC plugin for Grails has hit version 1.0 with:

• Firefox 3.6 support
• grabbing of screenshots on test failures
• the ability to call JavaScript user extensions on the selenium object as though they were regular Groovy methods

among other features. Check out the documentation at http://robfletcher.github.com/grails-selenium-rc and install with grails install-plugin selenium-rc.

Grails plugins on Hudson

Yesterday I spent some time setting up Hudson continuous integration build for my various Grails plugins. It was pretty straightforward but I've been asked to document the steps involved.

Each plugin has one or more test applications under test/projects each of which has grails.plugin.location.whatever = "../../.." in BuildConfig.groovy. So each plugin CI build needs to run not only the plugin's own unit tests but any integration and/or functional tests included in the test apps as well.

Hudson plugins

I installed the essential plugins I would need:

• The Grails plugin makes building Grails projects much easier and allows you to build against multiple versions of Grails.
• The Git plugin enables Hudson to pull from a Git repository such as GitHub.
• The GitHub plugin adds links to the GitHub project in the Hudson build.
• The Chuck Norris plugin encourages you to keep your tests from failing.

Configuring Grails

Under Manage Hudson -> Configure System there is a Grails Builder section where I entered the name and path of the various versions of Grails installed on the build box.

Connecting to GitHub

In the project configuration:

• I pasted the GitHub project URL in the Github project field (shown if you have the GitHub plugin installed).
• Under Source Code Management I selected Git then pasted the read-only repository URL into the URL of repository field.
• I entered master in the Branch Specifier field as I've seen Hudson do odd things picking up changes from other branches before. I don't really need this behaviour and haven't had time to investigate it further yet.
• I ticked Poll SCM under Build Triggers then set a cron expression of */5 * * * * so Hudson will poll GitHub every 5 minutes.

Grails build steps

• Select Build with Grails from the Add build step drop down, then:
  
  • Select the Grails Installation you want to use.
  • Enter the Targets to run. These are the Grails commands that Hudson will execute. For example, I entered "test-app unit: -clean --non-interactive" package-plugin for the to run unit tests then package the plugin. The quotes allow arguments to be passed to individual targets so I'm telling the test-app phase to build clean and run non-interactively (i.e. Grails won't ask about things like installing or uninstalling plugins).
  • The other fields can be left blank for now.
• Add further Grails build steps for test applications. The only difference in configuring these is that the targets will likely be different and Project Base Directory needs to be set to the relative path where the test apps lives, e.g. test/projects/content-caching

Isolating each build

One thing I found is that it's probably a good idea to specify separate working directory for each build. I found that simultaneous builds would stomp over each other's compiled classes otherwise. This is particularly an issue for me where I have a build for the Selenium RC plugin whilst one of the test apps for the Springcache plugin uses the latest Selenium RC release to run tests.

Hudson provides various environment variables so I just set grails.work.dir to /tmp/.grails/${JOB_NAME} in every case.

The other consideration is the port that Grails applications will use. If you are running any kind of functional tests the Grails application will start and it each Hudson job needs to use a separate server port or simultaneous jobs will experience port contention. The server port is also configured in the Grails build step. I've just assigned a different one to each job but I couldn't think of a particularly clever way to automate the port assignment.

Test reports

Under Post-build Actions I ticked Publish JUnit test result report then entered a comma-separated set of paths to the XML reports generated by each build step. For example, the Springcache plugin has the following reports configured:

target/test-reports/*.xml,test/projects/springcache-test/target/test-reports/*.xml,test/projects/content-caching/target/test-reports/*.xml

That's picking up the root plugin's unit test reports and the reports generated by the springcache-test and content-caching test applications. Hudson does a good job of merging these together into a unified test report.

Running tests that need a browser

Some of my tests use Selenium RC which drives a real browser. Given that Hudson is running on a headless box I had to get the display exported to a box with an X Windows environment.

Still to do

Right now all the plugins are building against Grails 1.2.1 only so the next task is to set up parallel builds using Grails 1.3-RC1. Ideally I would like to use Hudson's multi-configuration project capability to do that but I haven't had a chance to look into it yet. I'd also like to use Gradle to build the various project with one build step but I'd lose the convenience in switching Grails versions that the Hudson Grails plugin gives me.

Page fragment caching with Grails Springcache plugin

Yesterday I released the new version of the Springcache plugin for Grails. The new feature this brings to the table is page fragment caching driven by annotations on controller actions. The feature is based on EhCache Web and is even simpler and more powerful than the full page caching I blogged about recently. With the page fragment caching feature you can:

• Define @Cacheable and @CacheFlush annotations on controller actions.
• Have SiteMesh decorate cached and uncached output alike.
• Use SiteMesh to mix dynamic page sections with cached sections.
• Use SiteMesh and <g:include> to have multiple areas of the page that are cached and flushed independently of one another.
• Cache the output of controller actions that use content negotiation separately according to the requested format.

There are some examples in the plugin documentation and the plugin source code has a test project with a variety of page fragment caching mechanisms used and tested.

A Gradle build for Grails plugins with test apps

There's a fairly popular technique in Grails plugin development where the plugin has 'test apps' stored in test/projects/*, each of which references the plugin by including grails.plugin.location."my-plugin" = "../../.." in its BuildConfig.groovy. Doing this allows you as a plugin developer to:

• Automate tests for sample projects using the plugin with significantly different configurations.
• Test using domain classes, controllers, etc. that shouldn't be packaged with the plugin.
• Test drive, changing code in both the test app and the plugin without the need to use grails package-plugin or grails install-plugin to pick up changes.

The downside is that to run all the plugin project's tests it's necessary to run the plugin's own tests, then change directory into each of the test apps and run their tests. Continuous integration config is also fiddlier for the same reason.

I wanted to find a way to automate this so I could run all my plugins' tests with a single command. I could write a bash script or an Ant build I guess, or even use maven <shudder>. However keeping things in Groovy-land I decided to try using Gradle which I've been meaning to look into for a while now. I saw Hans Dockter's presentation on Gradle at the London Groovy & Grails Exchange back in December and was impressed with how terse and expressive the syntax is, especially compared to XML based formats. Unfortunately one thing Gradle doesn't grok yet is Grails.

The solution I've come up with is based on a post by Helmut Denk on the gradle-user list. Gradle just uses Ant's exec task to shell out and execute the appropriate Grails command. Combining this with Gradle's multi-project build support I now have plugin builds that can be run with gradle test that will automatically descend into the test apps under test/projects/*.

The build.gradle file at the root of the project defines clean and test tasks: The test task will run the plugin's unit tests. The clean task is defined just once for all projects as there's no difference in how it is done - a nice examply of Gradle's DRY-ness.

Then in settings.gradle I tell Gradle where to find the various sub-projects:
Finally each test app has it's own build.gradle that defines its test task: This is only necessary if the test apps run tests differently to the plugin - here my app is running integration rather than unit tests - otherwise I could have defined test in the same DRY manner as clean.

The process is not as fast as it could be if Grails were wired in to Gradle properly. gradle clean test for the Springcache plugin and its two test apps takes just over 2 minutes on my MBP. Also, my Gradle-fu leaves a lot to be desired right now so I'm sure there are improvements that could be made with the way I'm handling the sub-projects. But, for the purposes of a pre-commit test run or Hudson build this works pretty well.

Selenium RC tests with a running Grails app

I posted recently about the new remote mode feature of the Selenium RC plugin. One thing I forgot to mention is that this feature can also be used to run your Selenium tests interactively against a running app instance. You can start your app as normal using grails run-app then either open a second terminal or background the process (Ctrl-Z in a bash shell) then use grails -Dselenium.remote=true test-app other:selenium <test name> to run individual tests without stopping or re-starting the app. With the app running development mode you can effectively test-drive using Selenium tests and Grails' artefact reloading capabilities.

One thing I should clarify is that direct domain class access will not work in remote mode. I'm thinking about ways to add fixtures support to the Selenium RC plugin so there's a good alternative approach that will work in remote mode.

Selenium RC tests with a remote app

I've just released a new snapshot of the Selenium RC plugin. The main new feature is the ability to run your Selenium tests against a running instance of your application instead of having the application start and stop as part of the test phase. A number of people have requested this. It's really useful for CI environments where you may want to start the app up on a different server from where the tests are being run.

All you need to do is set selenium.remote = true in your SeleniumConfig.groovy and set selenium.url to the root URL of the server where the app is running. Once this is done you can run tests as normal (note: when remote mode is used Selenium tests run in the other phase rather than the functional phase).

The really useful thing is that SeleniumConfig.groovy can contain environment blocks just as other Grails config files can. This means you can enable remote mode in your CI environment but continue to test as normal locally. For example:

selenium {
    remote = false
    // etc.
}

environments {
    hudson {
        selenium.remote = true
        selenium.url = "http://my.test.server/"
    }
}

The plugin release is a snapshot so you need to specify the version explictly: grails install-plugin selenium-rc 0.2-SNAPSHOT and you need to be using Grails 1.2.

New Springcache Plugin Version

After saying it might take me a while I got stuck in and rewrote the Springcache plugin from scratch. The new version is up now. The plugin now requires Grails 1.2.0-M3 or greater as it's reliant on Spring 3.0.

The plugin allows you to declare caching and flushing behaviour on service methods (well, any Spring bean methods, but services are typical) using @Cacheable and @CacheFlush annotations. This is really useful for service methods that perform long-running or expensive tasks such as retrieving data from web services, network resources, etc. The plugin is less appropriate for service methods that retrieve data using GORM/Hibernate as you could just use the 2nd level cache, although there's nothing stopping you doing so if it makes sense in your domain.

The documentation and source code are in the usual places. Here's a quick summary of the changes and new features:

• No longer depends on the (discontinued and non-Spring 3 compatible) spring-modules-cache library.
• No more mapcache, the plugin now uses ehcache by default.
• Only ehcache is supported directly but it's really easy to implement an adapter for other caching libraries if you need to.
• Slightly simplified configuration. Some minor tweaks will be needed if you're upgrading from an earlier version of the plugin.
• Bean names are now prefixed with "springcache" so they're much less likely to clash with other things in your Spring context.

There has been some interest in some new features such as a taglib for caching chunks of views which I may start looking at shortly.

Springcache Plugin Status

A couple of people have asked me about the Springcache plugin and Grails 1.2. Currently the plugin is not compatible with 1.2-M3 and onwards. The reason for this is that the spring-modules-cache library that the plugin depends on is not compatible with Spring 3.0 which is used by the newest versions of Grails. Not only that but the spring-modules project has been discontinued altogether. There is a fork of it on Github but that doesn't look terribly active either. This means that at some point I'm going to have to sit down and write some code to bring the annotation driven caching inside the plugin itself and drop the spring-modules dependency. However, just to compound things the Spring documentation on the relevant area hasn't been updated yet and still references the classes that have been removed from the API which is what's causing the incompatibility in the first place!

So, it's on my to-do list but don't hold your breath!

Slides from GGUG

I gave a talk last night at the London Groovy & Grails User Group about the Selenium RC plugin and how to use it. I pitched the talk assuming that people were somewhat familiar with Selenium and had probably used the Selenium IDE. I'm not sure if that was the right way to go but the talk went pretty well and people seemed interested in what I had to say. I ended up talking a lot about the page object pattern and how awesome it is for building manageable suites of Selenium tests.

I really wanted to show a few quick examples but I had neglected to consider that a mini display port VGA adapter might be a useful thing to bring and unfortunately with Skills Matter being in the middle of an office move there wasn't one to hand. Luckily I was able to borrow a laptop so at least I could get my slides up on the projector.

Testing Grails Applications With Selenium Rc

View more presentations from Robert Fletcher.

Grails Selenium RC Plugin Released

I've finally got the first release of the Selenium RC plugin out of the door. Just run grails install-plugin selenium-rc to install.

For anyone unfamiliar with Selenium RC basically the plugin allows you to write and run Selenium tests in Groovy to functionally test Grails apps using a real browser. The tests run in the functional phase using grails test-app or grails test-app -functional.

In terms of compatibility there is a very minor issue with Safari and some slightly more annoying ones with IE - all of which the plugin's default configuration will work around for you. Unfortunately Firefox 3.5 on OSX Snow Leopard doesn't want to play at all due to an open Selenium bug. Firefox on other platforms, Google Chrome and Opera all appear to be 100% compatible.

I've tried to make writing the tests themselves as similar to regular Grails unit and integration testing as possible. Not only that, if you have the Spock plugin installed you can write your Selenium tests as Spock Specifications.

Documentation is here, source code is on GitHub and issues and feature requests can be raised on Codehaus' JIRA. I've got plenty of things to work on for future releases such as Selenium Grid integration and automatic screen-grabbing when assertions fail.

Talk on Selenium RC Testing In Grails

I'll be delivering a short talk about testing Grails apps with the Selenium RC plugin I'm working on at the London Groovy and Grails User Group on 20th November. Details are here.

The plugin is basically release ready, I'll try to make the final push some time this week so that it's out in the wild in advance of the talk (and hopefully any particularly idiotic bugs are reported and fixed before I make a total fool of myself).

Testing Grails Apps With Selenium-RC

I mentioned in my last post that I'd started to think we were going about Selenium testing all wrong. For years I've been writing Selenium tests using the Selenium IDE, so the tests are in HTML files. When doing pre-commit testing and on CI the tests are run with the HtmlTestRunner. This technique is fine and well - developing tests with the IDE is fast and they can be easily worked on an re-tested in the IDE until they are correct. However, some things have always bothered me and Adam Goucher's thoughts on the Selenium Value Chain crystallised some of those suspicions.

So what are the big advantages of using Selenium RC over just the IDE and HtmlTestRunner?

Test design

Selenium tests written in HTML tend to either become an amorphous blob, not testing one simple thing or proliferate to the point of having hundreds of small test files. With Selenium RC you can write Selenium test classes with multiple test case methods just as you would unit tests. Each test case can be neat, self-contained and share set up with other test cases in the same class. Also by abstracting out common command sequences into utility methods you can start to develop a higher level DSL for testing your application. In HTML tests this can be done by writing extension functions in Javascript but it's not as straightforward.

Data fixtures

Data set up is the biggest headache when writing Selenium tests in HTML. You can't drop out of selenese and set up a couple of domain objects so you end up either trying to run all your tests against a monolithic data fixture (a Sisyphean task if any of your tests make modifications to the data) or write some sort of URL endpoints that can set up and tear down data. We've done the latter and it's resulted in the single most bloated, horrible, misused bit of code in our app. Fixtures get re-used by other tests because they set up approximately appropriate data - until some change requires a change in the fixture and suddenly a bunch of unrelated tests fail. Selenium RC tests don't have this problem; you're in Groovy-land so you can access domain classes directly and use the invaluable build-test-data plugin to set data up.

Data verification

I mentioned previously the anti-pattern of false moniker testing and HTML based Selenium tests are terrible for this. If you have a test that creates some data, how do you verify it has done it correctly? The only way is to scrape the data on another screen - e.g. a CRUD list or show view. Ugh! With Selenium RC you can read the data from the database with standard Grails domain classes and verify your expectations directly.

I started experimenting over the weekend with getting Selenium RC tests written in Groovy to run using grails test-app -functional. It turns out to be really fairly straightforward and I've put some alpha code up on GitHub.

Session Bound Temporary Files

I've just released Session Temp Files, a very simple plugin for managing a temporary file storage space bound to the HTTP session. I'm working on something where a user uploads a file at the start of a webflow conversation, the flow then validates the file and may require the user to enter extra information before finally the file is saved and an associated record is added to the database. The uploaded files needed to be stored somewhere between the initial upload phase and the successful outcome of the flow where they will be copied in to a permanent storage space. The file is really too big to keep in memory in the flow scope. It's easy enough to create a temp file and delete it at the end of the flow (even if the user cancels out of the flow) however users can also do things like close the browser, suddenly decide to do something different and type in a new URL, etc. The webflow then gets abandoned in an intermediate state. The flow itself will be destroyed when the HTTP session ends but the temp files I created will hang around until the OS decides to sweep the temp directory.

The plugin simply allows you to create a directory within the normal temp directory - System.properties."java.io.tmpdir" - that will get deleted when the HTTP session expires. It binds two new methods on to HTTPSession: getTempDir() returns the session's temp directory, creating it if it doesn't exist and createTempFile(prefix,suffix) works like File.createTempFile except that the file is created inside the directory returned by getTempDir.

The code is up on GitHub and the plugin is available from the standard Grails plugin repository via grails install-plugin session-temp-files.

Scripting Plugin and Dependency Installation

Another quick script that I find quite useful, particularly on continuous integration servers is my Prepare.groovy script. It solves a couple of problems

• Some plugins are not as well behaved as others on installation, for example the Functional Testing plugin automatically installs when test-app or run-app is installed but will then immediately crash with a ClassNotFoundException because the libraries it depends on are not on the classpath yet. Re-running the command will then work but if the commands are part of a CI build it's probably already bombed out and reported a broken build.
• The Ivy plugin is great but test-app and run-app won't invoke its get-dependencies target to pull down libraries the way they will automatically install plugins.
• Sometimes the grails command you want the build to execute is supplied by a plugin so there's no convenient way to run it directly from a brand new workspace because the plugin isn't installed yet.

The script simply ensures all plugins are installed and then invokes the Ivy plugin's get-dependencies target (only if the Ivy plugin is installed - it won't blow up on you if you don't use Ivy). At that point you should have a fully workable workspace and be able to run any grails command you like without your app complaining that some library or other isn't present.

Checking for plugin updates

If you're anything like me you probably like to keep all your Grails plugins updated to the latest versions. It's quite easy to miss a release announcement and not realise there's a new version available. Although grails list-plugins will list out the available versions and what version you currently have installed it doesn't really highlight the plugins you could upgrade. I threw this script together this morning to do exactly that. It will simply check the plugin versions you have installed against all configured repositories and notify you of any that could be updated.

Grails Upgrade

We've just finished upgrading our sites to Grails 1.1 and I thought it would be worth sharing some of the fun we had with the process.

First and foremost was a call site caching bug in Groovy 1.6.0 that caused chaos with our unit and integration tests. Basically any time we mocked a class then demanded behaviour on a static method or any time we used the metaClass to override a static method (among other things that's pretty much any unit test that goes anywhere near a domain class) the behaviour wouldn't get torn down again. This caused symptoms such as stubbed dynamic finders leaking from unit tests and breaking integration tests that would pass when re-run in isolation. We got as far as refactoring a whole bunch of test code that mocked static methods before Burt Beckwith pointed out to me that simply replacing $GRAILS_HOME/lib/groovy-all-1.6.0.jar with the newer version from the Groovy 1.6.1 or 1.6.2 release would fix the problem.

Unfortunately the same bug has another effect that was quite catastrophic for us. Any Hibernate proxy of an instance of a domain class involved in an inheritance hierarchy will throw ClassCastException when you try to access a subclass property or do an instanceof check. I blogged already about the instanceof issue but it's actually wider ranging than I realised at the time. It can cause very unpredictable behaviour because complex relationships between objects and the way they are loaded by a controller before a page is rendered can cause this error to pop up under obscure conditions that are very hard to nail down with test coverage. To solve the problem we've had to use eager fetching in places where associations are typed as the base class of an inheritance hierarchy and to use explicit checks for proxies such as:

if (o instanceof HibernateProxy) {
    o = GrailsHibernateUtil.unwrapProxy(o)
}

Hopefully this bug will be fixed in Grails 1.1.1 and we will be able to use the default lazy loading behaviour in all relationships.

Other problems we encountered included:

• The onLoad Hibernate event handler on domain classes now seems to run at a slightly different time in the object life cycle: before any eager fetched collections are loaded. We had one domain class that was doing some initialisation (calculating the percentage of votes each answer to a poll had received and storing it in a transient property) that no longer worked as the persistent collection it was trying to iterate over was always empty. This was simple to refactor out and in fairness was pretty horrible anyway.


• Config.groovy and related external config files are no longer loaded when running unit tests so any unit test that excercises code doing ConfigurationHolder.config.some.value.or.other blew up. We solved this by simply adding the following to _Events.groovy to load up config prior to the unit tests running:

  eventTestPhaseStart = { phase ->
      if (phase == 'unit') {
          ConfigSlurper slurper = new ConfigSlurper(GrailsUtil.environment)
          def configClass = getClass().classLoader.loadClass("Config")
          def config = slurper.parse(configClass)
          ConfigurationHelper.initConfig config // this step loads the external environment config file(s)
          ConfigurationHolder.config = config
      }
  }



• Logger configuration has completely changed (for the better) in Grails 1.1


• Some of our tests were explicitly setting the level of particular loggers to 'OFF' because the test is deliberately causing an error condition and we didn't want the console polluted with a huge stack trace. This no longer works as the logging abstraction has changed to SLF4J which does not allow you to directly set the level on its Logger class. Having better things to do than unwrapping logging abstractions I just turned logging off altogether in the test Grails environment.

  log4j = {
      root {
          off()
      }
  }



• It looks like binding errors don't get reported on nullable properties when using DomainClass.properties = params in a controller. I've raised a bug.


• There's been a minor change to the <g:select> tag. Previously the disabled attribute followed the HTML convention (disabled="disabled") but now the attribute value needs to be a boolean. I guess this actually makes more sense than the goofy HTML convention but I really expect most of the attributes on those sorts of tags to be pure pass-through.


• The mockDomain method used by unit tests seems to be a little inconsistent when dealing with inheritance heirarchies. If you do mockDomain(BaseClass) then try to use subClassInstance.subClassProperty it fails with a MissingPropertyException. I've raised a bug.


• We had customised Package.groovy to deploy our app at the root context. It seems this is now directly supported as an option in Grails 1.1 by setting grails.app.context = '/' in Config.groovy. I think this option was already available in Grails 1.0.3 but our patch pre-dated us upgrading to 1.0.3. One of the things I was trying to achieve while upgrading was to ensure we were patching Grails and any plugins as little as possible.


• I'd upgraded our selenium plugin (not the one in the Grails plugin repository) a while back to cope with Grails 1.1 and upgraded the selenium server it contained at the same time. It turns out the new selenium server will time out attempting to click any anchor that uses the javascript: protocol in its href. Luckily this was only being done in 2 places in our app and was easily fixed by using href="#" instead.

One other thing we decided to do was to replace our existing suite of webtests with functional tests. Webtest has never been popular with the developers on our team and we had made a number of modifications to version 0.4 of the plugin in order to support issues like config loading and running the app at the context root of the server. These modifications caused some headaches when trying to upgrade the plugin so we decided to drop it and port the tests over to the functional test plugin. Generally I'd still rather be writing Selenium tests than using either webtest or functional tests but the functional test syntax is a little nicer and less "pseudo-XML" than webtest's.

Grails 1.1 Plugins and IntelliJ Idea

Good news. The recent release of the IntelliJ Idea EAP has added proper handling of the new plugin installation system in Grails 1.1 - i.e. it automatically picks up your plugins installed under ~/.grails/1.1/projects/<name>/plugins.