BDD, Regression Testing and some feature requests

We write scenarios to help us know when we’re done… not

I’ve been quite happily espousing the idea that we write scenarios in order to help us (devs) know when we’re done, which helps drive our design, and everything else is a lucky by-product.

Dan referred in our OOPSLA tutorial to something he calls ‘Beer Driven Development'; how soon can I get down the pub? As a business analyst, it helps to assume that all devs are horribly lazy and want beer, so if you haven’t specified the scope of the problem, you won’t get that bit of the solution you’re after. Conversely, if you do include some behaviour in the scope of a scenario, you’ll most likely get the simplest thing that could possibly work. This is good, because it cuts down the number of bugs.

There are a couple of other reasons for writing scenarios: regression, and documentation. It turns out (from feedback at OOPSLA) that a lot of people care about automated regression tests. It’s a nice way for devs to know that we’re still done, and as long as the QAs have confidence in the automated tests, they don’t have to manually cover that ground. (In practice I’ve found that they usually do, just to make sure that the tests themselves aren’t faulty, but less often than they would.)

BDD, it turns out, lends itself quite nicely to regression testing.

A scenario is a set of steps

Given <some context>, when <some event occurs> then <some outcomes should result>.

There may be more than one context, event and outcome. It may also be useful to exercise the behaviour a few times in the same scenario, so perhaps there will be more than one event-outcome set.

Scenarios are regression tests once the code that makes them run has been written

Sometimes, though, we like to change scenarios or acceptance tests for the specific purpose of regression testing. This is because the frameworks that we have to test the UI are horribly slow.

I’ve worked on one project where, while running the suite of 500 Fitnesse scenarios, the CPU buzzed at 100% and the whole suite took 5 minutes to run. A lot of technical jiggery-pokery was needed to get this to happen, but if you can do it, you don’t need to differentiate between scenarios and regression tests (the one becomes the other as soon as the code that makes it run is in place). There are some frameworks which are faster than others – Simon Stewart’s WebDriver, and the tiny Swing harness packaged with JBehave, which I’m still incredibly proud of – but generally these fast frameworks aren’t mature, or full-featured yet. And, even if they are, once you start getting above 500 scenarios you may still want to think about cutting down the length of time it takes to get feedback from your suite.

Acceptance tests are hard to read, which makes it hard to merge them

I’ve seen (and written) a number of acceptance tests which look something like this (only they weren’t using pretty fluent interfaces, so they were even worse):

I can refund a customer’s purchase

TillScreen.addItem("Panatachi Television 3X").pay("2000").with(new CreditCard("2345123478902468")).done();
int originalNoOfTelevisionsInStock = StockScreen.count("Panatachi Television 3X");
Money originalBalance = FinanceScreen.getBalance();

TillScreen.findMostRecentBill("2345123478902468").refundItem("Panatachi Television 3X").with(new CreditCard("2345123478902468")).done();

assertThat(StockScreen.count("Panatachi Television 3X"), eq(originalNoOfTelevisionsInStock));
assertThat(FinanceScreen.getBalance(), eq(originalBalance - 2000));

I can replace a customer’s purchase

TillScreen.addItem("Panatachi Television 3X").pay("2000").with(new CreditCard("2345123478902468")).done();
int originalNoOfTelevisionsInStock = StockScreen.count("Panatachi Television 3X");
Money originalBalance = FinanceScreen.getBalance();

TillScreen.findMostRecentBill("2345123478902468").replace("Panatachi Television 3X").done();

assertThat(StockScreen.count("Panatachi Television 3X"), eq(originalNoOfTelevisionsInStock - 1));
assertThat(FinanceScreen.getBalance(), eq(originalBalance));

Did you actually read those, or did you just skip to this line? Chances are that you just glossed over them. I would. BDD helps clear this up when the scenarios are written, so instead of this we’d have:

I can refund a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and a Panatachi Television 3X costs $2000and he paid with credit card number 2345123478902468
When I search for the most recent bill with credit card number 2345123478902468
and I refund the Panatachi Television 3X
Then the stock of Panatachi Television 3X should be unchanged
and we should have $2000 less.

I can replace a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and a Panatachi Television 3X costs $2000
and he paid with credit card number 2345123478902468
When I search for the most recent bill with credit card number 2345123478902468
and I replace the Panatachi Television 3X
Then the stock of Panatachi Television 3X should be one less
and we should have the same balance.

(If you think it’s impossible to write code this way, take a look at the latest RSpec features, brought to you by David Chelimsky. I’m completely blown away by this, and can’t wait to convert all my breakable toy scenarios to RSpec running on JRuby. Anyway…)

It’s easier to merge scenarios

With BDD, it’s fairly easy to see which contexts, events and outcomes are common between the two scenarios, and we can imagine a situation in which we could exercise both features associated with these stories.

You can see that the context of the scenarios both start with a customer owning a previously-purchased Panatachi television. In most frameworks, this is accomplished by running the scenario in which a customer purchases a television; we’re using a scenario as the context for another scenario. Running scenarios takes time! Wouldn’t it be great if we could just run this once?

To make these into one scenario, we need the outcome from one to resemble the context of the other, then we don’t need to set up that context again. In the same way that we’d refactor code to look similar, we can refactor the context of the first to be identical to the outcome of the second.

Interestingly, as I come to do this, I realise we’re missing a couple of contexts! They’re implicit in both scenarios – what’s the actual balance? and the actual stock levels? I’ve also swapped the scenarios around, because that’s the order they’ll happen in now:

I can replace a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and our balance is $23150
and we have 10 Panatachi Television 3Xs in stock
and he paid with credit card number 2345123478902468
When I search for the most recent bill with credit card number 2345123478902468
and I replace the Panatachi Television 3X
Then we should have 9 Panatachi Television 3Xs in stock
and our balance should be $23150.

I can refund a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and our balance is $23150
and we have 9 Panatachi Television 3Xs in stock
and a Panatachi Television 3X costs $2000
and he paid with credit card number 2345123478902468
When I search for the most recent bill with credit card number 2345123478902468
and I refund the Panatachi Television 3X
Then we should have 9 Panatachi Television 3Xs in stock
and our balance should be $21150

So, now we can merge the two scenarios:

I can replace or refund a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and our balance is $23150
and we have 10 Panatachi Television 3Xs in stock
and he paid with credit card number 2345123478902468
When I search for the most recent bill with credit card number 2345123478902468
and I replace the Panatachi Television 3X
Then we should have 9 Panatachi Television 3Xs in stock
and our balance should be $23150.
When I search for the most recent bill with credit card number 2345123478902468
and I refund the Panatachi Television 3X
Then we should have 9 Panatachi Television 3Xs in stock
and our balance should be $21150

Trying to do this with code, instead of English, sucks. It’s hard to see the difference between the setup of a context and the occurrence of an event. It’s a lot easier if the code is written in English, to the extend that most people will skip that second step and move straight to the third.

Even so, we’ve lost some of the emphasis. We don’t want to know that the balance is $23150. We want to know that the balance is the same. The implicitness of the two missing contexts was actually quite beautiful. Wouldn’t it be lovely if we could bring those back?

I want some new features!

The scenario I really want looks like this:

I can replace or refund a customer’s purchase

Given that a customer purchased a Panatachi Television 3X
and he paid with credit card number 2345123478902468
When I search for the most recent bill with that credit card number
and I replace the Panatachi Television 3X
Then we should have 1 less Panatachi Television 3X in stock
and our balance should be the same
When I search for the most recent bill with that credit card number
and I refund the Panatachi Television 3X
Then we should have 1 less Panatachi Television in stock
and our balance should be $2000 less.

So, as a scenario writer (aka BA for happy paths, or QA for full regression, or some combination with a dev to help turn it into code):

  • I want outcomes to be able to record the state of the world before the events that will result in those outcomes, so that I can verify comparative outcomes and retain meaningful emphasis (This is possible in JBehave, but horribly ugly and ties the Givens to the Outcomes – I don’t want to do that!)
  • I want all scenario steps to be aware of the world in which they are running, so that they can share aspects of the world instead of duplicating them (‘that credit card number’ instead of ‘2345123478902468’. JBehave already does this, but I’m not sure if RSpec does. Will ask…)
  • I want to be able to associate a scenario with more than one story, so that I can change the scenarios for several stories into a single regression test (RSpec lets you tag things; JBehave doesn’t. Yet.)

As a regression tester (aka dev) being asked to save QA some work:

  • I want to be able to run scenarios as well as stories (JBehave doesn’t do this yet).
  • I want to be able to associate a scenario with several features, benefits, etc., so that I can test a feature or check that a benefit is still being provided (see tagging, above.)
  • I want to have the option to check for the outcome of a scenario being used as the context of another scenario, so that I know whether I need to run them or not. (Since a scenario might be used as the context for a scenario that’s being used as a context, etc., at some point we are likely to find ourselves in a world we can use. Dan says this is a ‘backwards-chaining rules engine’ and is quite excited by the idea. He also says this is incredibly dangerous and will only work for Given/When/Then, not Given/When/Then/When/Then, so it should be used with some thought.)

Fortunately, as well as being a scenario writer (at least for my breakable toys) and a regression tester, I’m a JBehave dev, so I have some control over when I get these features. Looks like I have some work to do.

This entry was posted in bdd. Bookmark the permalink.

2 Responses to BDD, Regression Testing and some feature requests

  1. ext_67968 says:

    It was great meeting you at ooPSLA, even if I didn’t get to go to the BDD tutorial. I think Beer-Driven Development is definitely the way to go. ;)

    (I’m that Student Volunteer from the “Art of Telling your Design Story” tutorial, and I’m sorry if I missed you and Dan on Thursday night, I was around the conference centre until about 7pm before going out to get beer with the other SV’s.)

    -Eitan

  2. sirenian says:

    Was good to meet you too! All the SVs were fantastic and their beer much deserved; please pass my thanks on if you see any of them in the future.

    Hope to run into you at another event?

Comments are closed.