Fear & loathing in functional testing land

As projects grow the two things I’ve repeatedly found to be particularly painful have been functional testing and data fixtures. I might write up some thoughts on data fixtures another time but what follows is a brain-dump of my troubled relationship with functional testing.

Disclaimers: I have more questions than answers and I’m completely open to the idea that I’m doing it all wrong. I’m not trying to diss any tool or technique here. I have spent a lot of time over the last few years writing functional test coverage so I think I at least have some perspective on the issues if no clue how to solve them.

When I say functional testing I mean in the Grails sense of an end-to-end test via the browser. Some people are quite resistant to writing such tests and whilst I agree that as much testing as possible should be done at as low a level as possible there are certain things that really only can be tested in that way. I’m also a big fan of the GOOS approach of working outside in; starting with a (failing) functional test that defines the desired behaviour in a user-centric way and building in to the low-level implementation with its unit tests then back out to watch the original end-to-end test (hopefully) pass.

Why is functional testing difficult?

Test development cadence

The main issue I find when constructing functional tests is what I’ll call the test development cadence; that is the time it takes to go round the loop (and sub-loops) of

1. write a bit of test
2. watch it fail
3. write some code to make it work
4. watch it still fail
5. figure out if your expectation is wrong or your code doesn’t work
6. fix it
7. repeat last 3 steps n times
8. watch it pass
9. celebrate

With a unit test that time is typically fast, a keystroke in an IDE and the results are available in at most a couple of seconds. Functional tests are considerably slower. Even assuming you can optimise so that the application is running and you can run the test from an IDE then Selenium has to start up a browser instance, queries execute, views need to render, etc. In the worst case you’re switching to the terminal and using grails test-app functional: Blarg or equivalent then waiting for the webapp to start up before the test can even start to run and shut down again before the report is generated.

A slow test development cadence leads to distraction (checking Twitter, making coffee, getting drawn into a discussion of the finer points of mixing an old fashioned, etc.) and distraction leads to context-switching which slows things still further.

Test diagnostics

GOOS makes a great point about the importance of test diagnostics suggesting that the TDD mantra of “red, green, refactor” should be replaced with “red, decent diagnostics, green, refactor”. When a test breaks, especially one someone else wrote (or that I wrote more than a week ago and have consequently lost all recollection of), I want to be able to see what’s broken without re-running the test with added logging or resorting to a debugger. Testing further from the browser hurts diagnostics as you can’t see what’s not working and so have to rely solely on the quality of your assertion output. That’s not an easy thing to get right. Geb’s output when a waitFor condition fails is just Condition did not pass in x seconds. Even with direct assertions and nice power assert output its not always clear whether the functionality didn’t work or the expectation is incorrect. Selenese is by no means great in this regard (a humble Condition timed out isn’t much help) but at least you can step back with the Selenium IDE and watch the damn thing not working much more easily.

Bad test diagnostics coupled with a slow test development cadence make for a horrible experience.

The quest for the functional testing holy grail

The most productive I’ve ever been when writing functional tests has been when using Selenium IDE. That’s quite an admission for someone who’s spent a considerable amount of time & energy over the last few years trying to find or build something better!

The test development cadence is fast. Really fast. When you’re writing tests with Selenium IDE (and I do mean write them, I’ve almost never used the recording functionality) the app is running, the browser is running and you can execute the test, a portion of the test or an individual command very quickly. You can step through the test, set breakpoints, etc. When using a framework like Grails that lets you make changes to the app without restarting you can rock along pretty rapidly.

That said, the downsides are not inconsiderable:

• Abstraction is typically poor; you’re dealing with fine details of page structure (DOM element ids, CSS selectors) and copy ‘n pasting sequences of commands that would in a sane world be defined as functions or macros. You can write custom JavaScript commands but with considerable limitations such as the fact that any wait for x step must be the last thing the command does. Lack of abstraction means lack of maintainability. As the project goes on any change in page rendering probably means picking apart a bunch of tests that fail not because the functionality they are testing has stopped working but because the implementation of that functionality has changed.
• Atomicity is difficult. Because each test probably requires a few lines of setup it’s tempting for developers to add new assertions to an existing test. This violates the principle of having a single (logical) assertion per test. Part of the problem I think is that with Java, Groovy, Ruby, etc. each file can contain multiple tests whereas with Selenese each file is a single test script. The right thing to do is to have lots of small Selenese test files but it’s tempting to fall into the trap of munging tests together into the testing equivalent of a run-on sentence. One of the worst side-effects of this is that as a test suite grows it becomes really hard to identify where certain features are tested and to find redundancy or obsolete tests.

Despite these significant failings writing tests in Selenium IDE is very effective. Maintaining a suite of such tests is another matter. Working on a long-running project the failings of Selenese tests start to increase logarithmically. The reason I created the Grails Selenium RC plugin was to try to build something I could use in future projects that would combat the failings of Selenese. I wanted to use a ‘real’ language with selection and iteration and to be able to build a robust abstraction so that tests are not dealing with the fine details of page markup. Geb is another step along this road. It provides a nice way of defining page objects and modules and handles things like tracking which page class is the ‘current’ one and how and when that should change.

What do I want from a functional testing tool/language?

I’m convinced that the goal of writing tests in the same language as the application is a pretty vapid one. Working inside one’s comfort zone is all very well but too many times I’ve seen things tested using Selenium or Geb that would be better tested with a unit-level JavaScript test. I’m guilty of this myself. I’m a better Groovy coder than I am a JavaScript coder so it’s easy initially to break out a new Geb spec than a new Jasmine spec. Functional testing tools are really bad at testing fine-grained JavaScript behaviour, though. These sort of tests are really flaky, false negatives are a fact of life. They’re wastefully slow as well. JavaScript unit tests are fast, faster than Groovy unit tests. As a Grails developer I’ve looked enviously at how fast tests run in Rails apps but that’s nothing compared to watching a couple of hundred Jasmine tests run in under a second. To get back to the point, I have no problem with writing my functional tests in something other than Groovy if I can hit my goals of productivity and maintainability.

I was at one time convinced that the ability to use loops and conditional statements in Groovy made it a more suitable testing language than Selenese but honestly, how often are such constructs really required for tests? The The single most essential thing for a maintainable suite of functional tests is the ability to create a decent abstraction. Without that you’ll be building brittle tests that fail when the implementation changes 100 times more often than they fail because the functionality they’re testing is actually broken.

Abstraction is key

The abstraction layer needs to be powerful but simple. I’ve seen test suites crippled by badly written page object models and I’m starting to feel that the whole idea is too formalized. Building Geb content definitions with deeply nested layers of Module types is time consuming & difficult. With Selenium RC there’s not even the page transition logic Geb provides so you end up having to write that as well (probably getting it wrong or implementing it in several different ways in different places).

I can’t help thinking the page object model approach is coming at the problem from the wrong angle. Instead of abstracting the UI shouldn’t we be abstracting the behaviour? After all the goal is to have tests that describe how users interact with the application rather than how the various components that make up the application relate to one another. I’d rather have a rusty toolbox of lightweight macros and UI module definitions than a glittering palace of a page component model that I find awkward to use, extend or change. The abstraction has to be there - when I change the implementation I don’t want to spend half a day finding and fixing 100 subtly different CSS selectors scattered throughout the tests - but I don’t think it has to be particularly deep.

Where do I go from here?

A better Selenese?

An interesting possibility for creating better Selenese tests is the UI-Element extension library that allows a UI abstraction layer to be built on top of Selenese. It also introduces the concept of rollup rules (paramaterized macros) that are a more powerful way of abstracting command sequences than custom Selenese commands. From what I’ve seen the tool support in Selenium IDE looks impressive too. I need an opportunity to use UI-Element seriously but it certainly appears promising.

The most impressive Selenium extension I’ve seen is Steve Cresswell’s Natural Language Extensions that layers something like JBehave’s feature definition language on top of Selenese. Energized Work used this on a couple of projects (unfortunately not ones I was involved with) and I’ve heard great stories of how it enabled really rich cooperation between developers, QA and project stakeholders. I was pleasantly surprised with how simple the underlying code appeared to be given the radical difference in the test language.

Other options?

The tools I really need to look into are:

• Cucumber which syntactically looks like the answer to my prayers. I want to see how fast the test development cadence is. Since there’s now a pure JVM implementation I really have no excuse for not getting up to speed with it pronto.
• FuncUnit is much lower level and I’m not sure how easy it would be to build an effective abstraction layer that kept the tests readable and maintainable but it’s fast and runs right in the browser which are potentially compelling advantages.

Running Geb tests from your IDE

A frequent complaint about functional testing in Grails is that the start up / shut down cycle time of the app makes prototyping a functional test prohibitive. There has been some recent progress in this are with Luke Daley's Functional test development plugin but it would be really nice to be able to just run functional tests from inside your IDE in the same way as a similar unit test.

With Geb and IntelliJ Idea at least this is actually pretty straightforward (I'm sure the same thing is possible with Eclipse/STS I'm just not familiar enough with it).

Geb's dependence on the Grails lifecycle is pretty minimal so the tests can run without needing any magic to happen in any _Events scripts (unlike the Selenium RC plugin where this would be significantly more difficult). One extra step is required; the Grails Geb plugin normally picks up its base URL from Grails configuration and the test will not be running in the same JVM as the Grails app if it runs from the IDE so configuration will not be available. Just add the following method to your test that extends GebTests or GebSpec (or add it to a base class):

@Override String getBaseUrl() {
    super.baseUrl ?: "http://localhost:8080"
}

Now Geb will use the Grails configured base URL if it is in the same JVM as the app or default to http://localhost:8080 otherwise (obviously you need to change that default if that's not the host or port where your app runs).

You can then just run up the app from the command line, open the test in IntelliJ and hit Ctrl + Shift + F10 to run it. When the test completes the app is still running so you can make changes & run again with minimal turnaround time. Since the app is running in development mode you can make application changes without a server restart as well.

IntelliJ Idea's test integration is pretty good and it even groks Spock specifications. You can also debug and step through the test. The only downside is that you can't directly access domain classes or other application components in the test as they are only available from the app's JVM.

Encapsulating page state and actions in Geb

Geb Page and Module classes are just regular Groovy classes. In addition to the content DSL you can declare methods just as you would in any other class. Those methods can use content properties directly. This is really useful for encapsulating state and actions in a reusable way.

In this post I want to run through a couple of simple examples of re-usable Module classes that are enhanced with methods. Although the examples deal with Module classes everything here is equally applicable to Page classes.

Page object methods tend to fall into two categories:

Observers

report on some aspect of page state (e.g. whether a user is logged in or not).

have non-void return types (I prefer explicit return types on methods but def would work too).

can be used for making assertions.

Actions

perform an action such as clicking a link or button.

will typically be void (note that it's not necessary to return a Page instance or class from a navigation method in Geb).

throw IllegalStateException if the action is invalid in the current page state.

First let's consider an authentication module that can be used across every page in a site. The markup of the module can be completely different depending on whether a user is logged in or not. When not logged in a username/password form is shown:

<aside id="auth"> 
    <fieldset> 
        <legend>Log In</legend> 
        <form action="/auth/login" method="post" > 
            <label>Username: <input name="username"/></label> 
            <label>Password: <input type="password" name="password"></label> 
            <button name="login" type="submit">Log In</button> 
        </form> 
    </fieldset> 
</aside> 

When logged in, a welcome message is shown:

<aside id="auth"> 
    Welcome back <span class="username">blackbeard</span> 
    <a href="/auth/logout" name="logout">Log Out</a> 
</aside>

To model this I'll create a Geb Module like this:

class AuthModule extends Module {
    static base = { $("aside#auth") }

    static content = {
        username(required: false) { $(".username").text() }
        logoutButton(required: false, to: HomePage) { $("a[name=logout]") }
        form(required: false) { $("form") }
        loginButton(required: false, to: HomePage) { $("button[name=login]") }
    }
}

The required: false declaration is used to declare content properties that may or may not be on the page. The username and logoutButton properties are only present in the logged-in state and the form and loginButton properties are only present in the logged-out state.

I can use the module in some tests like this:

def "user can log in"() {
    when:
    authModule.form.username = "blackbeard"
    authModule.form.password = "yohoho!"
    authModule.loginButton.click()

    then:
    at HomePage
    authModule.username == "blackbeard"
}

def "user can log out"() {
    when:
    authModule.logoutButton.click()

    then:
    at HomePage
    authModule.username == null
}

This is a good start but there are several assumptions and bits of page detail creeping into the test. The test is using the presence or absence of the username to determine if someone is logged in or not. That doesn't lead to a very meaningful assertion in the test and is assuming things about the page detail. Likewise the login step is quite long-winded and likely to be repeated in a lot of tests. Not only that but it won't work if a user is already logged in as the form fields won't be present on the page.

To encapsulate the module's state and actions better I'll add the following methods:

boolean isLoggedIn() { username }

void login(String username, String password = "password") {
    if (loggedIn) throw new IllegalStateException("already logged in")
    form.username = username
    form.password = password
    loginButton.click()
}

void logout() {
    if (!loggedIn) throw new IllegalStateException("already logged out")
    logoutButton.click()
}

The methods declared by the module abstract some detail away very effectively. The isLoggedIn method means I can change the login detection mechanism later and just change the module's method rather than a bunch of tests. The login and logout methods abstract away the how of logging in and out so the test can just deal with the what. I've used IllegalStateException for cases where a method is called when the module is not in the correct state. The tests now look much clearer:

def "user can log in"() {
    when:
    authModule.login "blackbeard"

    then:
    at HomePage
    authModule.username == "blackbeard"
}

def "user can log out"() {
    when:
    authModule.logout()

    then:
    at HomePage
    !authModule.loggedIn
}

Another good example of encapsulating state and behaviour like this is a typical pagination module that would appear on a list or search results page. The markup would look something like this (I've omitted the link href attributes for clarity):

<nav class="pagination">
    <a class="prevLink">Previous</a>
    <a class="step">1</a>
    <span class="currentStep">2</span>
    <a class="step">3</a>
    <a class="nextLink">Next</a>
</nav>

The previous and next links will only appear when there is a previous or next page and no links will be present at all if there is only a single page. The following Module class models the state and actions of this component:

class Pagination extends Module {
    static content = {
        links(required: false) { $("a") }
        currentPage(required: false) { $(".currentStep")?.text()?.toInteger() ?: 1 }
        nextLink(required: false) { links.filter(".nextLink") }
        previousLink(required: false) { links.filter(".prevLink") }
    }

    boolean isFirstPage() {
        previousLink.empty
    }

    boolean isLastPage() {
        nextLink.empty
    }

    void toPage(int pageNumber) {
        def link = links.filter(text: "$pageNumber")
        if (!link) throw new IllegalArgumentException("Page number $pageNumber not present in pagination")
        link.click()
    }

    void nextPage() {
        if (lastPage) throw new IllegalStateException("Already on the last page")
        nextLink.click()
    }

    void previousPage() {
        if (firstPage) throw new IllegalStateException("Already on the first page")
        previousLink.click()
    }
}

Breaking the Module down in detail:

• The currentPage property returns the current page number as an int and defaults to 1 if there is no pagination present in the page.
• The isFirstPage and isLastPage observer methods use the absence of the previous and next links respectively to determine if the current page is the first or last one.
• The toPage method finds a numbered link and clicks it, throwing IllegalArgumentException if no such link is present.
• The nextPage and previousPage action methods throw IllegalStateException if the relevant link is not on the page.

The Pagination class now neatly encapsulates the detail of the pagination elements and presents a higher-level façade to the tests.

Fuller versions of the examples in this post can be found on GitHub.

Modelling repeating structures with Geb page objects

Geb is the hot new thing in Grails functional testing. One of its most powerful features is the concise DSL for defining page objects. The reasons for using page objects are well enumerated elsewhere but the basic point is to allow your tests to interact with pages in a manner agnostic of the detail of their structure. This is both practical (you can change markup structure without having to fix numerous tests that only fail because they were tightly coupled to that structure) and aesthetic (your tests read more like a user’s interaction with the page - the what rather than the how).

I want to put together a few short blog posts dealing with patterns that I find useful for defining page objects and modules in Geb. As I go I’ll keep adding to a very simple Grails project showing working examples which is available on GitHub.

In this post I want to talk about repeating data structures such as lists and tables and how to model them effectively.

A content property in a Geb Page or Module can be any type; whatever is returned from the defining closure. This will frequently be a Geb Navigator instance or a String but can be whatever is useful for the tests you’re writing. A good rule of thumb is that the test should be dealing with as simplified a view of the data as possible. All the complexity of traversing HTML elements and manipulating them into a useful form should be hidden away in the page objects and modules. When handling repeating data structures such as ol or table elements you probably want to be able to treat the content as a List so that tests can use Groovy features such as iterator methods, indexing and slicing to make very expressive assertions.

Simple repeating structures

For example, imagine we want to verify an ol element like this:

<ol id="recent-books">
    <li>Zero History</li>
    <li>Surface Detail</li>
    <li>The Machine of Death</li>
</ol>

The most useful type would probably be a List<String> which we can get easily enough by defining our content like this:

static content = {
    recentBooks { $("ol#recent-books li")*.text() }
}

This is then very easy to use in a test:

expect:
recentBooks == ["Zero History", "Surface Detail", "The Machine of Death"]

Complex repeating structures using Modules

A more complex example of a repeating structure is a table, where each row contains several fields. Here we can use a Geb Module to represent each row, with content properties to get data from each cell. Let’s say we want to verify the contents of the following table of search results:

<table id="book-results">
    <thead>
        <tr>
            <th>Title</th>
            <th>Author</th>
            <th>Format</th>
            <th>Price</th>
            <th>Release Date</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Zero History</td>
            <td>William Gibson</td>
            <td>Hardback</td>
            <td>£12.29</td>
            <td>2 Sep 2010</td>
        </tr>
        <tr>
            <td>Zero History</td>
            <td>William Gibson</td>
            <td>Kindle</td>
            <td>£11.99</td>
            <td>2 Sep 2010</td>
        </tr>
        <tr>
            <td>Spook Country</td>
            <td>William Gibson</td>
            <td>Paperback</td>
            <td>£5.00</td>
            <td>31 Jul 2008</td>
        </tr>
        <tr>
            <td>Pattern Recognition</td>
            <td>William Gibson</td>
            <td>Paperback</td>
            <td>£4.99</td>
            <td>24 Jun 2004</td>
        </tr>
    </tbody>
</table>

We can define our module like this:

static content = {
    bookResults { i -> module BookRow, $("table#book-results tbody tr", i) }
}

class BookRow extends Module {
    static content = {
        cell { i -> $("td", i) }
        title { cell(0).text() }
        author { cell(1).text() }
        format { cell(2).text() }
        price { cell(3).text()[1..-1].toDouble() }
        releaseDate { cell(4).text() }
    }
}

The bookResults content closure takes a row number parameter and uses it to select the corresponding tr from the body of the table and use that to construct a module. The module itself defines content properties with meaningful names that map to the text in each cell.

This isn’t bad as far as it goes. We can use the module pretty effectively in tests like this:

expect:
bookResults(0).title == "Zero History"
bookResults(3).price == 4.99

However, bookResults isn’t a List. We can’t easily get all the book titles at once or make an assertion that all the authors are the same or find the lowest price. Even querying how many rows there are would require an additional content property or method to retrieve $("tbody tr").size(). The table is a repeating data structure and it would be nice to treat it as one!

This ought to be possible bearing in mind 3 things:

1. The type of a content property is simply whatever you return from the defining closure, there’s no reason we can’t return a List<BookRow>.
2. There’s nothing special about the expression that constructs the module itself: module BookRow, $("tbody tr", i) is just a call to a method called module passing a Class<? extends Module> which is the module type we want and a Navigator pointing to the module’s root element.
3. The Geb Navigator class returned by the $ expression implements Iterable<Navigator> and can be treated like a List of all the selected HTML elements.

In fact we can get a List<BookRow> easily enough if we redefine the bookResults property like this:

static content = {
    bookResults {
        $("tbody tr").collect {
            module BookRow, it
        }
    }
}

The key here is that we iterate over the tr elements inside the content definition collecting a new BookRow instance for each one. Now the page object doesn’t require the test to pass in the index of the row it’s interested in. This enables our test to do some much more powerful and interesting things:

expect:
bookResults.size() == 4
bookResults[0].title == "Zero History"
bookResults.title.unique() == ["Zero History", "Spook Country", "Pattern Recognition"]
bookResults.every { it.author == "William Gibson" }
bookResults[2..3].every { it.format == "Paperback" }
bookResults.price.sum() == 34.27

I’ve tried to show a couple of reasonably simple examples here. Others are easy to imagine; a Map representing the dt and dd elements inside an HTML definition list, a list of modules representing a group of labelled radio buttons or news items with images and links, a tree-like multi-level navigation structure, etc.