Liz Keogh's blog

A discussion on the Yahoo XP group regarding refactoring long methods into smaller ones coincided with me writing the code below. This is from my most recent test class, which is 180+ lines and growing:

    public void testThatRefundingTwoToysBoughtOnDifferentDaysDoesNotCauseError() throws Exception {

        goBackTwoDaysFromRealDate();
        String transactionRefA = buyToy();

        goBackOneDayFromRealDate();
        String transactionRefB = buyToy();

        goBackToRealDate();
        String refundTransactionRef = refundItems(transactionRefA , transactionRefB);

        verifyNoErrors(refundTransactionRef);
    }

I’m glad I did this, because now when I go back to it after finishing this quick post I’ll know exactly what I was doing. I don’t have to wade through the 180+ lines to find out which bits have the TODO comments in. Plus, when the next person comes to look at the weird configuration file I’m about to change, they’ll be able to tell from a glance at the code why I removed the lines I have. They can work with the change instead of around it.

This exercise made me realise that there seem to be a lot of very long tests around. In case it’s a prevalent thing and not just a statistical glitch, please let me beg on behalf of the campaign for legible tests – split ‘em up! Make them readable. Your tests define the behaviour of your application. If you’re racing along in an Agile environment then they may well be the only documentation you have.

(Did you know that in JUnit you can write an abstract test class, and any tests in it will be called when the subclass test is run? You did? Did you know that you can also do it with behaviour classes in <plug>JBehave</plug>?)

Of course, one has to be careful when refactoring tests. It’s easy enough to tell if your test fails as a result of refactoring. It’s harder to see if it no longer tests the same thing as before, and passes regardless. For instance, when writing abstract JUnit test classes, be careful not to put tset or should instead of test at the beginning of the method, unless you have a particular fondness for the word “Doh!”.

Update: Time travel is now parameterized too:

    public void testThatRefundingTwoToysBoughtOnDifferentDaysDoesNotCauseError() throws Exception {

        goBackInTime(2);
        String transactionRefA = buyToy();

        goBackInTime(1);
        String transactionRefB = buyToy();

        goBackInTime(0);
        String refundTransactionRef = refundItems(transactionRefA , transactionRefB);

        verifyNoErrors(refundTransactionRef);
    }

Thanks to the XP group for hints and tips on how to make vaguely similar bits of code turn into almost identical bits of code (Summary: put all the similar stuff together and move the dissimilar stuff somewhere else.)

I spoke to some of my colleagues back in London and was reliably informed (again) that the ratio of idealists to pragmatists amongst them is in fact lower than I made out in my last post. We did agree, though, that it’s the idealists making most of the noise.

New things from last night’s conversation, to be investigated later:

• Domain-specific languages. This (briefly and inaccurately) is what happens if you notice that something is changing a lot in code, and move those changes to a configuration file. The file itself is then a form of language. (A better definition can be found here).
• The database problem: part 1. If you’ve got several client versions in production, the server database has to be backwards compatible. This makes it hard to change, which can make the domain model hard to change too. What solutions are there for getting round this problem?
• The database problem: part 2. A grid which relies on a central database can’t scale as well as we’d like. The database forms a bottleneck. What solutions are there for getting round this problem? (And what kind of things require a central database?)
• Zen and the Art of Motorcycle Maintenance. I only got halfway through this last time I tried, about three years ago. Now I know a little more about philosophy and the twisted nature of the human mind. Time to try again?

arrogant
adj.

1. Having or displaying a sense of overbearing self-worth or self-importance.

2. Marked by or arising from a feeling or assumption of one’s superiority toward others: an arrogant contempt for the weak.

Thank you, dictionary.com.

“Do you know,” said Roy, “that eighty percent of people who’ve come into contact with Thoughtworkers think that we’re arrogant?”

“Really?” I asked. “How many Thoughtworkers think Thoughtworkers are arrogant?”

“About ninety percent,” he conceded.

We both agreed that this was odd, since TWers are amongst the least arrogant people I know. We are, in general, always interested in learning new things; ready to be told that there’s a better way than our own; never fooled into thinking that we know everything; open to critisism. So why do Thoughtworkers so often come across as arrogant?

As someone who gets mistakenly labelled as arrogant all the time, I’m obviously the best person to answer this question. So I have.

A sense of self-worth

There’s a difference between self-worth and self-importance. Self-importance is delusional; it suggests that the team, the project, the company or the world will fall apart without you. Perhaps that happens in an environment where knowledge is closely guarded, but most TWers love new things and hate to be stuck on the same project for months. I think we’re more likely to give away our knowledge; to ensure that someone else can take over our role, and to make ourselves as redundant as possible. I don’t think that we regard ourselves as important, or even want to be.

We do, however, have a sense of self-worth which is greater than that of your average IT bod. We know that our opinions matter, that we can make a difference, and that we all have responsibility for everything (thank you, Collective Code Ownership, management transparency and a flat hierarchical structure). We also know that this is true of everyone else in the company. That kind of mutual respect makes for a great place to work. The problem is that the enthusiasm generated by our environment often translates into “Look! Our world is so great! You should do this too!” (This is also the traditional TW greeting given to newcomers to the company, along with a garland of books and articles detailing the exact architecture required to build the better world.)

One of the problems with suggesting a better way is that it casts a reflection of something worse; something wrong. In most of the world, doing something wrong equates to a certain level of punishment, be it in lower salaries, stressful emails and memos from on high, rehabilitative courses or plain sacking. So until the world gets rid of the blame culture, there will always be people who aren’t prepared to be told that they’re wrong – especially if no one is interested in the stuff they’re doing right.

A feeling of superiority

Is our way really better? Which companies, exactly, are we superior to? Which projects?

There’s still a debate out there as to whether Agile is useful on, say, large-scale projects involving > 20 people or mission critical applications like aeroplane flight controls, and I have my own gut feelings about its practical application to legacy code.

How do you know, before you’ve listened, that a company might not be doing some things right already?

I’m pairing up to run the Agile induction course tomorrow and I’m very nervous about it. One of my colleagues here told me, “Don’t worry. Everyone started Agile somewhere; they worked out that there was a better way and played it by ear until they got it right.” And there are people playing around out there. Agile-like practices which I’ve seen outside of Agile projects include:

• test-first design (less disciplined than test-driven, but it still works)

• self-documenting code
• collaborative code ownership (a natural practice for any team disinclined to blame ?)
• feature-driven design (similar to story-driven design)
• a zero-overtime policy (at 37.5 hours, even better than the 40 hour week)
• pairing
• stand-up meetings
• frequent code integration

I don’t think any of the companies I’ve seen doing these things are familiar with the concepts of Agile or Extreme Programming. They were just doing these things because they worked; because there were a few bright sparks running things who worked out how to prevent the problems they’d encountered on previous projects. The founders of the Agile movement went a little further, but they all started somewhere too.

I’d swear that one company I worked with has a secret Agile practitioner converting the way they work. “Wouldn’t it be better, instead of writing a method this way, to break it up into reusable chunks? Wouldn’t it be great if this variable was renamed so we knew what it meant? How about we inject this via an interface so we can put that functionality into a new class? What if we have low-level tests for our code, so we know if it fails before we send it off to our poor beleaguered test team? What happens if we integrate more regularly so that the merges become easier and don’t break everything quite so often?”

Thinking back on his approach, I am once again filled with real respect for this cunning rebel. He knew he was right – of course he was! – but he asked questions to make other people do the hard work of realising it; by which time, of course, they were right too. I never once heard him say, “Listen! Look!”, but I bet he sits back at his desk every day with the smuggest smile on his face.

Guess what? He doesn’t work for Thoughtworks. I really, really want to meet the people like this who do. Stop being the quiet voice, whoever you are.

Contempt for the weak

Ah. See, this is the problem; I don’t think it’s weakness we’re contemptuous of. I suspect it’s pragmatism.

Pragmatists are prepared to compromise their ideals in order to get a job done. Most Thoughtworkers are idealists, and worse; some of us are evangelists who are convinced that we can change the world. Here are three idealist thoughts on why people might practice pragmatism, particularly with respect to Agile evangelism:

• Not wanting to upset people by telling them that there’s a better way than theirs

• Pandering to a team’s inability to absorb a number of changes at once
• Denial of responsibility for the problems caused by poor procedure.

Or, as the pragmatist might phrase them:

• Tact

• Consideration, and respect for the learning curve
• Recognition that not everyone shares the same ideals, and that there’s more than one way to skin a cat.

From an extreme idealist’s point of view, failure to follow ideals is a weakness. If there’s some reason why an ideal can’t be met, then that needs changing too. What – too many changes at once? Courage! Courage! It’ll all work out in the end. Why can’t you trust us? Why can’t you see how right we are? Why won’t you bet your time, your career and your company’s money on us being so very, very right?

I’m a self-confessed idealist, so I find it hard to see things from a pragmatist’s point of view; but I’ve tried, and will probably write again about it in the future. One day I may even practice a little pragmatism myself. The point I’m trying to make is that pragmatism isn’t a weakness; it’s just a different way of approaching the problem. Idealists have a destination. We plot a crows-flight path and try to follow it as closely as possible (and the evangelist will try to drive everyone else that way too). Pragmatists just want to get there quickly. If there’s a traffic jam on the motorway then they’ll take the back roads; otherwise, the most proven route with the least effort wins out.

Idealists build new roads, and will one day invent the Personal Flying Vehicle to eliminate the necessity for new roads at all. Pragmatists widen the existing roads, and will one day comprise the majority of Personal Flying Vehicle customers.

Clay Shirky’s article on whether Wikipedia is a good thing (or the part which Carlos summarised nicely) gave me another of those beautiful moments of epiphany. There’s nothing wrong with radial or cartesian people; with pragmatists or idealists. The world needs both. Maybe it’s time for us idealists to start listening to the pragmatist’s point of view.

Humility

humbleness
n

1. the state of being humble and unimportant

2. a humble feeling; “he was filled with humility at the sight of the Pope”
3. a disposition to be humble; a lack of false pride

A common Thoughtworker expression, albeit paraphrased, is that each of us looks back on our work six months ago with astonishment for how poorly it was done – and correspondingly, though fewer mention it, a sense of accomplishment when we think of how far we’ve come.

Perhaps that’s the problem with my humility and that of other Thoughtworkers too. It always comes with a sense of glory. Every realisation that we’ve done something wrong comes as a result of learning a better way, and we love that feeling; not the humility, but the epiphany that accompanies it; the ‘Papal visit’ of Agile evangelism. Humbleness is there, I promise. If you ever want to see humility on a Thoughtworker’s face, just ask him, “What’s the most stupid thing you’ve ever done?” and watch his expression change as he mentally runs down the list.

We have no egos; at least, not ones that care about their appearance. Our sense of self-worth is collective, not individual. Our idealism comes out of a desire to do right, not to be right. Sometimes it gets mistaken for arrogance, but it isn’t. Really. Why anyone would think that, I have no idea.

Disclaimer

I’ve never named Thoughtworks in my blog before, mostly because I’ve only been here for six months, and partly because I wasn’t sure they’d want to be associated with my ramblings. I’m not an expert on Thoughtworkers, but I am one. I’ve spoken to a lot of them. I am not a psychologist, a psychiatrist or in any way associated with the dissection of character, except as a writer. I have my fair share of both arrogance and humility. One is louder than the other. This piece is not intended as an apology or an excuse for anything I may have written previously.

I was asked to write on arrogance at Thoughtworks, and have tried to do so thoughtfully. It’s not an instruction manual. It’s not perfect. I have generalised horribly about various categories of people, especially my colleagues, ex-colleagues and client staff. Some of you are less arrogant than others. I love working with you all.

This post represents my personal views, etc., and hasn’t even been read by the TW management at the time of posting, let alone authorised.

Now you’ve read the disclaimer, if you’re still offended by this post, please let me know. I’m keeping score.

Liz.

I’ve not touched Ruby yet, but I’m insatiably curious, so I talked to Darren about it.

As far as I can work out, dynamic languages in general are fantastic if you’re running Test Driven Design; because it’s not the compiler that tells you you’re wrong, it’s the tests. (I’ve never tried this with a dynamic language, but am looking forward to the experience.)

I have a bit of a problem with this idea, though. As much as I like to go on about “the right way”, in practice I’ve not yet seen a project where all the developers write tests for absolutely everything, however much they might like to. The temptation to “just hack that bit of code” seems to be too much. I think this is particularly true of project tools; those little scripts which lend themselves nicely to Perl or Ruby and which aren’t really part of the code base.

Does Ruby tend to encourage people to write tests for everything? Darren thinks not. I’ve certainly hacked a few Perl scripts together in the past; things which had ten line comments for one strange regular expression, or little scripts which made perfect sense to me then, but I couldn’t tell you what they did or how they worked today. They always grow; have you noticed that? I like to think I’m a bit more disciplined now than I used to be, but the idea of thousands of developers out there with another language that lends itself to hacking makes me uncomfortable. It’s like seeing a good friend try on a dress, asking, “Does this make me look fat?” and trying to work out how to tell her that really, it’s not the fault of the dress. Imagine the expression on my face. It’s the same one I’m wearing now (the expression, not the dress).

It’s not Ruby’s fault; I’m sure it’s a great language. I believe, though, that there’s another way to encourage people into the golden path of TDD, even full-on Agile. Ruby’s a relatively new language*, and from this outsider’s point of view, it looks as though it’s gaining a culture and a following all of its own.

Is TDD part of that culture? Do the manuals, forums, etc. for Ruby say, “If you use this language, you should consider reading more about Test Driven Design”?

Just curious… and lazy…

I have vague, unformed visions of a monster language which is absolutely bound to definitions and tests for the behaviour of each class. I have even more vague ideas of how to implement that. I have a better idea of how horribly unwieldy it might be. Back to JBehave…

Update: Thanks to commenters and to Steven, who wrote this illuminating post in response. I think I’m starting to grasp the “Rails” concept of Ruby on Rails – it’s a web development framework with integrated testing (thanks, Steven). Will have to get round to practicing this at some point. If nothing else, I’d love to get a feel for the speed of the TDD / dynamic language cycle.

It’s good to see a bunch of people being so enthusiastic about their work. Hope for the future.

* I’m told that my perception of Ruby as a new language is wrong; it’s been around for as long as Java, and unit testing is built into the standard library. Sweet.

Last week, Alan pointed me towards Dale Emery’s post on Posted by at 10:05 am

All problems, from wars to software bugs, are caused by a combination of the following:

1. lack of communication
2. lack of courage
3. lack of imagination
4. lack of time.

It’s arguable that the first three are products of the last.

Matt Ryall responded to my post on documentation with some suggestions for making it easier. I finally persuaded my brain to give up some of the thoughts that it didn’t like (because they’d result in doing documentation). This isn’t particularly structured, just a few random ideas and concerns.

Stable code

What happens when part of the system is working perfectly, the customer is very happy with it and there are no changes likely in the future? No one is working on the code any more. No new developers are being introduced to the code base through pairing. Knowledge is slowly being lost from the system. Is there any way to identify code that’s reached this state? Are there any easy ways to produce a quick, high-level overview of the code?

If there are big sections of the code base which are stable, should they perhaps be separate sub-projects? Can a customer sign off sections of the project before the whole thing is finished? eg: an administration console, a look-and-feel or gui component palette, a logger, or a related standalone application?

Splitting a project up this way might help to reduce build times and complexity. The code would, however, have to be documented. For a library that may mean javadocs and clear code naming. For an application, that may mean a user manual.

Wikis

Could a Wiki be scripted in some way so that out-of-date documentation is flagged? Do we need some kind of agile methodology for stopping Wikis from getting too big, too complex, and duplicating information? I can’t find anything I’m looking for on ours, because the bits and pieces which people put there three years ago are still there, and hardly any of it is relevant any more.

Good things to put on Wikis include:
• details of commonly used test data
• TCP IP numbers of servers, etc.
• Links to useful tech sites, eg: J2SE 1.4.2, JMock, Hibernate, etc.
• VM parameters (we have about 20 of these and I don’t think they’re documented anywhere!)
• HowTos
• Reasons for choice, eg: “Why we switched to Hibernate”
• A map showing the local pubs, bus stops etc.

Bad things to put on Wikis include:
• CRs, bug reports, etc. – there are better ways of tracking important bugs
• Anything which people have to read regularly (use email updates; Wikis are passive)
• Notes for new joiners (this should be emailed too, if only because people don’t like to email inaccuracies and out-of-date information, so it might actually stay relevant)
• Anything which requires you to go through 3 other unrelated Wiki pages to find it
• Lists of people on the project, half of whom left six months ago
• Invitations to a special event in April 2003.

The flashing red light

My experience tells me that if there isn’t some big flashing light that goes off when something’s wrong, it won’t get fixed. For code, we have the red bar. What do we have for documentation? Matt mentioned using references to code names, etc., so that they are automatically updated.

Make documentation part of the build process, eg: if you’re using Javadocs and linking from one piece of code to another, run a script which checks for broken links as part of the build.

Here’s a quick test for the fence from my last post. My customer wants a fence which allows sheep to graze in a field, and prevents sheep from wandering outside of the field.

public class FenceTest extends TestCase {

    public void testShouldAllowSheepToGrazeInField() {

        Field field = new Field(3, 4);
        Sheep sheep = new Sheep();

        Fence fence = new Fence(field);

        boolean sheepGrazes = fence.sheepGrazes(sheep, 3, 3);
        assertTrue(sheepGrazes);
    }

    public void testShouldPreventSheepFromWanderingOutsideField() {

        Field field = new Field(3, 4);
        Sheep sheep = new Sheep();

        Fence fence = new Fence(field);

        boolean sheepGrazes = fence.sheepGrazes(sheep, 3, 5);
        assertFalse(sheepGrazes);
    }
}

It’s easy to write code which implements this, so imagine that I’ve done it. Hurrah! Green bar. That was easy.

Now supposing that my customer wants to define the strength of my fence, so that it gives way if a really strong sheep comes along. (This is how things like Sims get started!) Here goes:

    public void testShouldAllowStrongSheepToBreakFenceAndWanderOutsideOfField() {

        Field field = new Field(3, 4)
        Sheep strongSheep = new Ram();

        Fence fence = new Fence(field);

        boolean strongSheepGrazes = fence.sheepGrazes(strongSheep, 3, 5);
        assertTrue(strongSheepGrazes);
    }

Another easily implemented test. Bang! Green bar. The customer is very happy with the way in which the Ram can escape the clutches of the fence and graze beyond the boundary of the field.

But wait! My pair has spotted something I missed. He speaks briefly to the customer about the definition of ‘break the fence’, then writes this excellent failing test:

    public void testShouldAllowAnySheepToWanderOnceFenceBroken() {

        Field field = new Field(3, 4)
        Sheep weakSheep = new Lamb();
        Sheep strongSheep = new Ram();

        Fence fence = new Fence(field);

        fence.sheepGrazes(strongSheep, 3, 5); // result covered by other test

        boolean weakSheepGrazes = fence.sheepGrazes(weakSheep, 6, 6);
        assertTrue(weakSheepGrazes);
    }

So now we implement a “broken” state in the fence, to allow sheep to graze. We know that the fence is still whole when sheep are originally put into it, because the second test still passes. We should probably rename the second test to testShouldPreventSheepFromWanderingOutsideFieldWhenUnbroken. Now no one will misinterpret the “documentation” provided by the test names, and the code will ensure that anyone who, say, puts Horses which can leap over the fence into the same field, doesn’t accidentally cause the Rams to start leaping over the fences too.

(In the real world I would mock the sheep out so that the fence’s behaviour was independent of the sheep. But you get the picture.)

Jim and his colleagues (hope you don’t mind me using you as an example) write documentation as the reference point to code. Then, one group of people write the tests, and one group of people implement the functionality, both working from the same set of documents. They then run the functionality against the tests to see if they’ve done it correctly (or if perhaps the documentation is wrong).

(If the documentation is wrong, then documentation, test and code must all need rewriting. Do you have to wait for the documentation to be produced before the two teams of testers and coders can get on with the next bit?)

With Test Driven Design you write tests first, thinking all the time of the behaviour you want to achieve, then you write the functionality to pass the tests. So it doesn’t matter who writes the tests and who writes the functionality (and pairing helps you to avoid any bad design decisions that might otherwise get made).

The tests form the reference point. Because the full test suite is run every time anyone changes the build (about once an hour) you know very quickly if something is broken. Then, the person who breaks the build can look at the ‘broken’ code and tests and decide whether they’ve actually caused a bug, or whether the code should behave in a different way to the way it’s defined in the tests (usually this happens when a customer decides to change the requirements, or a bug is found because no one thought that a user would ever do something that obviously silly, even though they always do). The story cards provided by the customer also form a reference point, but if you don’t have a story card in your hand which directly contradicts the code behaviour, you know that the stories from previous iterations are still valid and still working.

When a requirement changes, or a bug is found, we write tests which fail, to prove that the software no longer does what we want it to. Then we change the code so that the tests pass again. There’s an excellent form of pair TDD called “Ping-pong programming” in which coder A writes a failing test which coder B must make pass, then coder B writes a failing test for coder A to implement, etc. The two coders sit together to do this. It’s a lot of fun and results in code that does as little as it possibly can to get the job done (ie: minimalist, clean, and very legible if me* and people like me** have beaten them over the head often enough).

Documentation makes it harder to change code, and is one of the reasons for the exponential cost-of-change curve seen on top-down projects. If your code is defined by its tests, then you can do whatever you like to them – refactor, rename, remove – and know that you haven’t broken any existing functionality. Five years down the line, the tests will tell you what the code ought to do, as well as giving you confidence that it still does it.

Oh, yeah – I think I’ve written a single one-line comment this week, and no documentation. You can tell what my code does because my tests say things like “testShouldAllowSheepToGrazeInTheField” and “testShouldPreventSheepFromWanderingOutsideField”. There are parsers around which will go through tests like this and produce documentation for you, eg:

Class Fence:
- should allow sheep to graze in the field
- should prevent sheep from wandering outside field

Class ElectricFence*** extends Fence:
- should dissuade sheep from breaking this fence

* Should be “I and people like me”. Agile and English have this in common: it takes real effort to get it right, even the strictest proponents aren’t perfect, and when you do get it right, it can feel very strange.

** By people like me I mean people who have been beaten over the head regarding code legibility by people like me, repeatedly, until they got it. Like me.

**My ElectricFenceTest class extends the FenceTest class, so ElectricFence still maintains the behaviour of a Fence. Nothing to teach about TDD there; I just love geeky things like that.

My friend Jim says, “Have to say, if I were a customer I would stipulate that you left me documentation. I wouldn’t buy services from a company that didn’t ”

One of the responsibilities of an Agile Coach (or perhaps this the role of a ‘Business Coach’ or ‘Client Principal’ – but a role isn’t the same as a person) is to find out what it is that the customer really wants. In the example I gave, it’s maintainability, not documentation. If documentation is the only way to provide maintainability – for instance, if the customer hasn’t been able to provide any in-house staff for pairing on the project – I might suggest writing it in the weeks before handover, so that it’s accurate and up-to-date. There’s no point doing this as you go along in Agile, because you know that the customer requirements, the application and therefore any processes / documentation are subject to rapid and frequent change. A pragmatist might document the more stable aspects of the project, such as architecture or persistence mechanisms, but even they’ve changed during the lifecycle of the project I’m on.

Another reason why customers like to see documentation is because it gives them some idea of how the project is going. Agile development provides other ways of measuring this. Most particularly, the customer can see progress in the number of stories (requirements) implemented, and by sitting down and playing with their new toy as it evolves.

Some projects may not be suitable for Agile development (mission critical?). Even so, I bet my lunch that there are still Agile practices which can improve the quality and speed of code production.

When someone wants something that doesn’t make sense to me, I usually ask, “Why?” Sometimes it helps that person clarify what they really mean, and sometimes I learn something new from the answer. It’s always a fun question to ask, and often the most important one.