Fear & loathing in functional testing land

As projects grow the two things I’ve repeatedly found to be particularly painful have been functional testing and data fixtures. I might write up some thoughts on data fixtures another time but what follows is a brain-dump of my troubled relationship with functional testing.

Disclaimers: I have more questions than answers and I’m completely open to the idea that I’m doing it all wrong. I’m not trying to diss any tool or technique here. I have spent a lot of time over the last few years writing functional test coverage so I think I at least have some perspective on the issues if no clue how to solve them.

When I say functional testing I mean in the Grails sense of an end-to-end test via the browser. Some people are quite resistant to writing such tests and whilst I agree that as much testing as possible should be done at as low a level as possible there are certain things that really only can be tested in that way. I’m also a big fan of the GOOS approach of working outside in; starting with a (failing) functional test that defines the desired behaviour in a user-centric way and building in to the low-level implementation with its unit tests then back out to watch the original end-to-end test (hopefully) pass.

Why is functional testing difficult?

Test development cadence

The main issue I find when constructing functional tests is what I’ll call the test development cadence; that is the time it takes to go round the loop (and sub-loops) of

1. write a bit of test
2. watch it fail
3. write some code to make it work
4. watch it still fail
5. figure out if your expectation is wrong or your code doesn’t work
6. fix it
7. repeat last 3 steps n times
8. watch it pass
9. celebrate

With a unit test that time is typically fast, a keystroke in an IDE and the results are available in at most a couple of seconds. Functional tests are considerably slower. Even assuming you can optimise so that the application is running and you can run the test from an IDE then Selenium has to start up a browser instance, queries execute, views need to render, etc. In the worst case you’re switching to the terminal and using grails test-app functional: Blarg or equivalent then waiting for the webapp to start up before the test can even start to run and shut down again before the report is generated.

A slow test development cadence leads to distraction (checking Twitter, making coffee, getting drawn into a discussion of the finer points of mixing an old fashioned, etc.) and distraction leads to context-switching which slows things still further.

Test diagnostics

GOOS makes a great point about the importance of test diagnostics suggesting that the TDD mantra of “red, green, refactor” should be replaced with “red, decent diagnostics, green, refactor”. When a test breaks, especially one someone else wrote (or that I wrote more than a week ago and have consequently lost all recollection of), I want to be able to see what’s broken without re-running the test with added logging or resorting to a debugger. Testing further from the browser hurts diagnostics as you can’t see what’s not working and so have to rely solely on the quality of your assertion output. That’s not an easy thing to get right. Geb’s output when a waitFor condition fails is just Condition did not pass in x seconds. Even with direct assertions and nice power assert output its not always clear whether the functionality didn’t work or the expectation is incorrect. Selenese is by no means great in this regard (a humble Condition timed out isn’t much help) but at least you can step back with the Selenium IDE and watch the damn thing not working much more easily.

Bad test diagnostics coupled with a slow test development cadence make for a horrible experience.

The quest for the functional testing holy grail

The most productive I’ve ever been when writing functional tests has been when using Selenium IDE. That’s quite an admission for someone who’s spent a considerable amount of time & energy over the last few years trying to find or build something better!

The test development cadence is fast. Really fast. When you’re writing tests with Selenium IDE (and I do mean write them, I’ve almost never used the recording functionality) the app is running, the browser is running and you can execute the test, a portion of the test or an individual command very quickly. You can step through the test, set breakpoints, etc. When using a framework like Grails that lets you make changes to the app without restarting you can rock along pretty rapidly.

That said, the downsides are not inconsiderable:

• Abstraction is typically poor; you’re dealing with fine details of page structure (DOM element ids, CSS selectors) and copy ‘n pasting sequences of commands that would in a sane world be defined as functions or macros. You can write custom JavaScript commands but with considerable limitations such as the fact that any wait for x step must be the last thing the command does. Lack of abstraction means lack of maintainability. As the project goes on any change in page rendering probably means picking apart a bunch of tests that fail not because the functionality they are testing has stopped working but because the implementation of that functionality has changed.
• Atomicity is difficult. Because each test probably requires a few lines of setup it’s tempting for developers to add new assertions to an existing test. This violates the principle of having a single (logical) assertion per test. Part of the problem I think is that with Java, Groovy, Ruby, etc. each file can contain multiple tests whereas with Selenese each file is a single test script. The right thing to do is to have lots of small Selenese test files but it’s tempting to fall into the trap of munging tests together into the testing equivalent of a run-on sentence. One of the worst side-effects of this is that as a test suite grows it becomes really hard to identify where certain features are tested and to find redundancy or obsolete tests.

Despite these significant failings writing tests in Selenium IDE is very effective. Maintaining a suite of such tests is another matter. Working on a long-running project the failings of Selenese tests start to increase logarithmically. The reason I created the Grails Selenium RC plugin was to try to build something I could use in future projects that would combat the failings of Selenese. I wanted to use a ‘real’ language with selection and iteration and to be able to build a robust abstraction so that tests are not dealing with the fine details of page markup. Geb is another step along this road. It provides a nice way of defining page objects and modules and handles things like tracking which page class is the ‘current’ one and how and when that should change.

What do I want from a functional testing tool/language?

I’m convinced that the goal of writing tests in the same language as the application is a pretty vapid one. Working inside one’s comfort zone is all very well but too many times I’ve seen things tested using Selenium or Geb that would be better tested with a unit-level JavaScript test. I’m guilty of this myself. I’m a better Groovy coder than I am a JavaScript coder so it’s easy initially to break out a new Geb spec than a new Jasmine spec. Functional testing tools are really bad at testing fine-grained JavaScript behaviour, though. These sort of tests are really flaky, false negatives are a fact of life. They’re wastefully slow as well. JavaScript unit tests are fast, faster than Groovy unit tests. As a Grails developer I’ve looked enviously at how fast tests run in Rails apps but that’s nothing compared to watching a couple of hundred Jasmine tests run in under a second. To get back to the point, I have no problem with writing my functional tests in something other than Groovy if I can hit my goals of productivity and maintainability.

I was at one time convinced that the ability to use loops and conditional statements in Groovy made it a more suitable testing language than Selenese but honestly, how often are such constructs really required for tests? The The single most essential thing for a maintainable suite of functional tests is the ability to create a decent abstraction. Without that you’ll be building brittle tests that fail when the implementation changes 100 times more often than they fail because the functionality they’re testing is actually broken.

Abstraction is key

The abstraction layer needs to be powerful but simple. I’ve seen test suites crippled by badly written page object models and I’m starting to feel that the whole idea is too formalized. Building Geb content definitions with deeply nested layers of Module types is time consuming & difficult. With Selenium RC there’s not even the page transition logic Geb provides so you end up having to write that as well (probably getting it wrong or implementing it in several different ways in different places).

I can’t help thinking the page object model approach is coming at the problem from the wrong angle. Instead of abstracting the UI shouldn’t we be abstracting the behaviour? After all the goal is to have tests that describe how users interact with the application rather than how the various components that make up the application relate to one another. I’d rather have a rusty toolbox of lightweight macros and UI module definitions than a glittering palace of a page component model that I find awkward to use, extend or change. The abstraction has to be there - when I change the implementation I don’t want to spend half a day finding and fixing 100 subtly different CSS selectors scattered throughout the tests - but I don’t think it has to be particularly deep.

Where do I go from here?

A better Selenese?

An interesting possibility for creating better Selenese tests is the UI-Element extension library that allows a UI abstraction layer to be built on top of Selenese. It also introduces the concept of rollup rules (paramaterized macros) that are a more powerful way of abstracting command sequences than custom Selenese commands. From what I’ve seen the tool support in Selenium IDE looks impressive too. I need an opportunity to use UI-Element seriously but it certainly appears promising.

The most impressive Selenium extension I’ve seen is Steve Cresswell’s Natural Language Extensions that layers something like JBehave’s feature definition language on top of Selenese. Energized Work used this on a couple of projects (unfortunately not ones I was involved with) and I’ve heard great stories of how it enabled really rich cooperation between developers, QA and project stakeholders. I was pleasantly surprised with how simple the underlying code appeared to be given the radical difference in the test language.

Other options?

The tools I really need to look into are:

• Cucumber which syntactically looks like the answer to my prayers. I want to see how fast the test development cadence is. Since there’s now a pure JVM implementation I really have no excuse for not getting up to speed with it pronto.
• FuncUnit is much lower level and I’m not sure how easy it would be to build an effective abstraction layer that kept the tests readable and maintainable but it’s fast and runs right in the browser which are potentially compelling advantages.

Export Selenium IDE scripts for Grails

Thanks to the documentation provided on Adam Goucher's blog I've created a simple Firefox plugin that extends the Selenium IDE and adds formatters that allow you to export a script as a Grails Selenium RC test case. You have the option of JUnit 3 (extends GroovyTestCase, uses assertEquals) or JUnit 4 (doesn't extend anything, uses annotations and assertThat) formats.

I think everything is working. There are some things I'd still like to do:

• Custom commands are currently exported as commented out which they don't need to be given Grails Selenium RC test cases can call Javascript extension functions directly.
• I haven't yet figured out how to output a command such as

  selenium.waitForAlertPresent()

  rather than

  selenium.waitFor {
      selenium.isAlertPresent()
  }

The Firefox plugin is available from Selenium HQ. You will need to have Selenium IDE installed for it to work, obviously. Once installed you should find options

• Export Test Case As -> Grails Selenium RC (JUnit 3)
• Export Test Case As -> Grails Selenium RC (JUnit 4)

have been added to Selenium IDE's File menu.

Whilst I would certainly not leave the exported test case as is it's a quick way to get started. From the exported script you can easily refactor out some page objects or helper methods.

Selenium RC plugin v1.0

The Selenium RC plugin for Grails has hit version 1.0 with:

• Firefox 3.6 support
• grabbing of screenshots on test failures
• the ability to call JavaScript user extensions on the selenium object as though they were regular Groovy methods

among other features. Check out the documentation at http://robfletcher.github.com/grails-selenium-rc and install with grails install-plugin selenium-rc.

Selenium RC tests with a running Grails app

I posted recently about the new remote mode feature of the Selenium RC plugin. One thing I forgot to mention is that this feature can also be used to run your Selenium tests interactively against a running app instance. You can start your app as normal using grails run-app then either open a second terminal or background the process (Ctrl-Z in a bash shell) then use grails -Dselenium.remote=true test-app other:selenium <test name> to run individual tests without stopping or re-starting the app. With the app running development mode you can effectively test-drive using Selenium tests and Grails' artefact reloading capabilities.

One thing I should clarify is that direct domain class access will not work in remote mode. I'm thinking about ways to add fixtures support to the Selenium RC plugin so there's a good alternative approach that will work in remote mode.

Selenium RC tests with a remote app

I've just released a new snapshot of the Selenium RC plugin. The main new feature is the ability to run your Selenium tests against a running instance of your application instead of having the application start and stop as part of the test phase. A number of people have requested this. It's really useful for CI environments where you may want to start the app up on a different server from where the tests are being run.

All you need to do is set selenium.remote = true in your SeleniumConfig.groovy and set selenium.url to the root URL of the server where the app is running. Once this is done you can run tests as normal (note: when remote mode is used Selenium tests run in the other phase rather than the functional phase).

The really useful thing is that SeleniumConfig.groovy can contain environment blocks just as other Grails config files can. This means you can enable remote mode in your CI environment but continue to test as normal locally. For example:

selenium {
    remote = false
    // etc.
}

environments {
    hudson {
        selenium.remote = true
        selenium.url = "http://my.test.server/"
    }
}

The plugin release is a snapshot so you need to specify the version explictly: grails install-plugin selenium-rc 0.2-SNAPSHOT and you need to be using Grails 1.2.

Slides from GGUG

I gave a talk last night at the London Groovy & Grails User Group about the Selenium RC plugin and how to use it. I pitched the talk assuming that people were somewhat familiar with Selenium and had probably used the Selenium IDE. I'm not sure if that was the right way to go but the talk went pretty well and people seemed interested in what I had to say. I ended up talking a lot about the page object pattern and how awesome it is for building manageable suites of Selenium tests.

I really wanted to show a few quick examples but I had neglected to consider that a mini display port VGA adapter might be a useful thing to bring and unfortunately with Skills Matter being in the middle of an office move there wasn't one to hand. Luckily I was able to borrow a laptop so at least I could get my slides up on the projector.

Testing Grails Applications With Selenium Rc

View more presentations from Robert Fletcher.

Grails Selenium RC Plugin Released

I've finally got the first release of the Selenium RC plugin out of the door. Just run grails install-plugin selenium-rc to install.

For anyone unfamiliar with Selenium RC basically the plugin allows you to write and run Selenium tests in Groovy to functionally test Grails apps using a real browser. The tests run in the functional phase using grails test-app or grails test-app -functional.

In terms of compatibility there is a very minor issue with Safari and some slightly more annoying ones with IE - all of which the plugin's default configuration will work around for you. Unfortunately Firefox 3.5 on OSX Snow Leopard doesn't want to play at all due to an open Selenium bug. Firefox on other platforms, Google Chrome and Opera all appear to be 100% compatible.

I've tried to make writing the tests themselves as similar to regular Grails unit and integration testing as possible. Not only that, if you have the Spock plugin installed you can write your Selenium tests as Spock Specifications.

Documentation is here, source code is on GitHub and issues and feature requests can be raised on Codehaus' JIRA. I've got plenty of things to work on for future releases such as Selenium Grid integration and automatic screen-grabbing when assertions fail.

Talk on Selenium RC Testing In Grails

I'll be delivering a short talk about testing Grails apps with the Selenium RC plugin I'm working on at the London Groovy and Grails User Group on 20th November. Details are here.

The plugin is basically release ready, I'll try to make the final push some time this week so that it's out in the wild in advance of the talk (and hopefully any particularly idiotic bugs are reported and fixed before I make a total fool of myself).

Testing Grails Apps With Selenium-RC

I mentioned in my last post that I'd started to think we were going about Selenium testing all wrong. For years I've been writing Selenium tests using the Selenium IDE, so the tests are in HTML files. When doing pre-commit testing and on CI the tests are run with the HtmlTestRunner. This technique is fine and well - developing tests with the IDE is fast and they can be easily worked on an re-tested in the IDE until they are correct. However, some things have always bothered me and Adam Goucher's thoughts on the Selenium Value Chain crystallised some of those suspicions.

So what are the big advantages of using Selenium RC over just the IDE and HtmlTestRunner?

Test design

Selenium tests written in HTML tend to either become an amorphous blob, not testing one simple thing or proliferate to the point of having hundreds of small test files. With Selenium RC you can write Selenium test classes with multiple test case methods just as you would unit tests. Each test case can be neat, self-contained and share set up with other test cases in the same class. Also by abstracting out common command sequences into utility methods you can start to develop a higher level DSL for testing your application. In HTML tests this can be done by writing extension functions in Javascript but it's not as straightforward.

Data fixtures

Data set up is the biggest headache when writing Selenium tests in HTML. You can't drop out of selenese and set up a couple of domain objects so you end up either trying to run all your tests against a monolithic data fixture (a Sisyphean task if any of your tests make modifications to the data) or write some sort of URL endpoints that can set up and tear down data. We've done the latter and it's resulted in the single most bloated, horrible, misused bit of code in our app. Fixtures get re-used by other tests because they set up approximately appropriate data - until some change requires a change in the fixture and suddenly a bunch of unrelated tests fail. Selenium RC tests don't have this problem; you're in Groovy-land so you can access domain classes directly and use the invaluable build-test-data plugin to set data up.

Data verification

I mentioned previously the anti-pattern of false moniker testing and HTML based Selenium tests are terrible for this. If you have a test that creates some data, how do you verify it has done it correctly? The only way is to scrape the data on another screen - e.g. a CRUD list or show view. Ugh! With Selenium RC you can read the data from the database with standard Grails domain classes and verify your expectations directly.

I started experimenting over the weekend with getting Selenium RC tests written in Groovy to run using grails test-app -functional. It turns out to be really fairly straightforward and I've put some alpha code up on GitHub.