LSSC 2012 was very, very different – challenging a ton of sacred cows, trashing some long-held ideas and giving me a bunch of new thoughts and understanding. The format was perfect: seemingly flawless organisation; amazing speakers, including some that we wouldn’t have heard if we weren’t in Boston; a Lean Camp unconference where I actually felt panic at not being able to attend all the sessions; long breaks giving us plenty of time to talk, network, reflect and synthesize ideas; innovative sponsors whose presence added value; lightning and ignite talks with hours of brilliance compacted into small spaces of time (thanks for letting me rant instead!); great venue, food and service and of course the fantastic mix of attendees.

Here are some of the joyous nuggets that I learnt and the people from whom I learnt them, paraphrased for your quick enjoyment.

Hillel Glazer: Demand pulls most of us to start projects, but it only really works when it pulls us to finish them and produce something. For that, we need to work on capacity. David Joyce echoed Hillel’s sentiments in his talk when he told us to stop disbanding teams right when they’re at their most productive. That’s part of the problem.

Steven Spear: If you can’t make predictions, you lose your ability to be surprised, which means you can’t learn. This spawned a Twitter conversation with Karl Scotland and others about complexity and whether expectations apply in the complex space, which helped me with my ignite talk. I really love Karl’s take – that we should at least predict impact if not outcome. Steven was talking about nuclear reactors, which I hope are a little more predictable than the software we produce.

David Anderson: It turns out that the community is doing more than just software these days, so the Lean Software Systems Consortium has rebranded as the Lean Systems Society. It’s modelled on the Royal Society, and I’m honoured to be named as a fellow. Particularly interesting was David’s comment that they’d observed the behaviours and values of the community, and drawn up their Credo or value statement accordingly – not the other way round! Talk about making process policies explicit…

Michael Kennedy: From TPS – “Product development is not about developing cars, it is about developing knowledge about cars.” Michael showed how sharing knowledge across projects can help us to build quality in. He showed how Toyota explore new ideas, resolving complexity while developing expertise before using them in production, which falls right into place alongside some of the Cynefin conversations that have been happening in the UK, as well as Dan North’s call for Deliberate (as opposed to accidental) Discovery. One excellent quote: “They’re only standards because they’re the best we can do right now. Engineers are expected to break the standards. Managers decide which standards they want to be broken.”

Gregory Howell: It turns out that the construction metaphor we’ve been using in IT doesn’t even work out in construction! One study found that 85% of construction managers had underestimated the complexity of the project. Pushing people to go faster isn’t as useful as working on the interfaces between people. The stories he told mostly centred on slowing things down and increasing quality and fluidity of hand-offs in order to go faster.

Robert Charette: All exchanges of goods and services are exchanges of risks and opportunity. Robert used the example of a plumber who’s taken on risk by investing in his own skills. He has an interesting model for risk and particularly market share, and challenged us to think about the kind of things which might end up bringing a company as successful as Apple down.

Jim Benson: If you have a repetitive task, record the metrics for how it went each time on your kanban board. Jim was talking in terms of Personal Kanban, but I can see how that could be useful for eg: retrospectives. Already added the “exercise” column to mine! Jim also taught one table at the Brickell Key awards that if you play buzzword drinking games with the Lean & Kanban crowd, you’re going to get what you measure… dangerous stuff. Congratulations to both Brickell Key winners, Jim and Arne Roock.

Mike Burrows: Limit WIP in portfolio too. Mike also suggests limiting the amount of money carried in the portfolio, allowing you to have a few big projects or several small ones. Richard Hensley apparently thinks a similar way. Karl Scotland shared another of Mike’s suggestions: limit dots in progress. Put dots on the cards every day they’re in progress and you’ll be able to see the flow very clearly.

Karl Scotland and Larry Maccherone: Our data is really dirty. If you start using 2-sigma thresholds to make SLAs, you’ll probably get stung. It doesn’t follow a normal distribution, in part because developers simply don’t update the statuses on time – they’re too busy doing the work! Outliers are also interesting. Rally has some neat tools to show you where your percentiles really are, but they’re in competition with Lean Kit’s lounge with its hot dogs and ice-cream. Lean Kit definitely win the prize for brand awareness this trip.

Yochai Benkler: This was the most mind-blowing talk for me. It turns out that our desire to collaborate is partly genetic, and only prevalent in 60-70% of the population. The industrialised world is slowly moving towards models in which we *do* collaborate – it does actually form part of our evolution, and making collaboration the norm can help – but our innate desire to collaborate can easily be disrupted by extrinsic rewards and punishment for failure to collaborate, amongst other things. Motivating people to collaborate is way more complex than we thought.

This is only part of the learning I take away from LSSC 2012. Already signed up for LSSC 2013 – hope to see you there!

First off, two tutorials:

I’m running my first Deliberate Discovery, Cynefin and Real Options tutorial at Skills Matter on 29th May. These three ways of thinking and modelling software development and the world in general have really helped me, and I’d like to pass the techniques on. Highly workshop-driven and not at all technical.

At the end of September I’ll be running my 1-day BDD tutorial as part of Agile Cambridge. This is the only BDD tutorial I’ll be running this year, so if you’re interested, get in there now! Dan North (my mentor in all things BDD and Agile) and David Snowden (Complexity Thinking and Cynefin guru) are keynote speakers, so it promises to be an excellent conference!

I’ll also be speaking and running workshops through the year:

The Next Generation Testing Conference organised by Unicom is on the 23rd. I’ll be on the panel talking about Agile, and particularly ranting about our obsession with granularity and our need for certainty even where it doesn’t really exist.

I’m going to be talking about Real Options at Dev Tank in London on the 29th, after the Progressive .NET Tutorials. It shouldn’t be a long talk, so lots of opportunity to catch up with your fellow devs over a beer. More details to follow – watch the space!

On June 7th, I’ll be giving an overview of BDD, how to do it well and why it works at Agile East Anglia. If you’ve been out of the loop on BDD, this is a great chance to get into it, and I’ll be answering any questions you have too. I think there may be just one ticket left…

In August I’ll be at Agile 2012 in Dallas, running “Turning and turning in the Widening Gyre”, a workshop on complexity and deliberate discovery, and “BDD: Look, no Frameworks!” on how to do BDD using custom DSLs instead of BDD tools, while keeping steps maintainable and readable.

On 21st – 22nd September I’m honoured to be one of the keynote speakers at Lean Agile Scotland. Topic still to be decided, but it’s going to be related to people and the inside of our heads; one of my favourite minefields.

I’m also speaking at GOTO this year (October 1st to 3rd, Aarhus, Denmark). In my talk, “To be honest…” I’ll be looking at why honesty is so important and yet so hard to do.

In the meantime, I have some days free for in-house training or coaching. Please get in touch now, before the rest of the year gets this busy too!

I’m half-tempted to get “Options have value. Options expire,” added to my tattoo. The principles which Chris Matts and Olav Maassen champion have become such guiding forces in my life that many of the decisions I make are only done in order to increase the number of options available to me. Once the whole innate phobia of uncertainty is out of the way, it’s a fantastic way to live life in freedom and with fun.

Real Options are also a phenomenal model for managing risk on software project – and not the only brilliant idea that Chris and Olav have!

So imagine my absolute joy when I saw this in the works – a graphical novel, fun and easy to read and understand, dealing with all kinds of ideas around risk management. The skiing story at the end of the first chapter (downloadable from the link) is the same one that Chris shared with me to help me understand his ideas.

Chris and Olav describe it as “a business graphic novel on managing risk. The book is a culmination of their collaboration on the real options model over the last six years. It provides examples of how to manage a project using the real options model and outlines a simple technique for making better (informed) decisions. It also covers more advanced topics such as information arrival process, game theory, feature injection, paradox of choice and how to deal with uncertainty. While geared towards project managers, the book benefits anyone making decisions in their job or daily life. Chris and Olav focus on making risk management easily understandable as everybody does this every day without being aware of it. ”

I am very confident that this book is going to be ground-breaking. I am so confident that I’ve already laid out £500 of hard-earned cash for a chance to star in the book (sorry, that option has expired). I don’t regard this as charity. This is me doing something small to pay Chris and Olav back for their guidance and help over the last few years, and it’s money they’ve helped me earn anyway. Estimated publishing date is Winter 2012… assuming they manage to raise the funds to kick it off.

If you would like to help these two fantastic people get this novel out of the door and benefit from the same help they’ve given me, please go to the Sponsume site before April 14th and pick an option. You never know – you could be the proud owner of a limited edition of a book that turns out to be as ground-breaking as “The Goal”*.

I’d like to shortcut some of that with Cynefin, because I think it’s kind of cool, I’ve found it useful, and it’s not actually that hard to get your head around once you make the small mindshift. I’m going to share a bit about what I know of it, then talk about how it might actually be a useful concept for a dev to have in their head.

Some of us went to the Calm Alpha meet up, created a list afterwards, and have been using it to discuss which kinds of software can be categorised by which domains. I think our need to categorise everything is part of the problem, but I haven’t worked out how to solve it yet. In this post, I’ll be categorising things according the Cynefin definition of the terms, so resist the urge to say, “No! What I’m doing is simple!” or “But it’s complicated!” because Dave Snowden, who created the Cynefin model, uses those terms in a slightly different way to our common usage.

So here’s Dave’s Cynefin model.

Dave Snowden, released under CC BY 3.0 – thank you!

The Cynefin model consists of four domains – simple, complicated, complex and chaotic – with disorder in the middle. The edges of the domains aren’t strict – they’re domains, not quadrants. There’s also a little fold beneath the “simple” to “chaotic” boundary, to show how easily simplicity can tip into chaos. If you’re going to use this for a lot of items, it might be worth using the domains as attractors, rather than categories. Put the items down on a table, then work out where the lines go afterwards. If you can get your head round that, you’re already on your way to understanding complexity thinking.

Here’s how I tell what kind of programming I’m doing.

Simple

Everyone can work out how to do this. Like, everyone. The example the CE guys used was a bicycle chain falling off. It’s easy to work out how to get it back on again. If you’re programming, I imagine stuff like the turtle at the science museum, or Lego Mindstorms with its graphical drag-and-drop interface, might fall into this space. Children and non-programmers can do it.

In a simple environment, you sense, categorize and respond. You say, “Oh, it’s one of those problems.” No analysis is required.

Complicated

Complicated stuff is predictable, but requires expertise to understand. A watch is complicated. If you’re programming, complicated stuff will be well-understood, done before, not going to change as you develop it. Writing yet another CRUD form probably falls into this space.

In a complicated environment, you sense, analyze and respond. You say, “Let me have a look at this problem and I’ll tell you how to solve it, because I’m an expert at this.”

(I have a theory that most devs get really bored by doing the same predictable but complicated thing over and over again. We tend to turn it into open-source or automate it, reducing it to the far smaller but more complex problem of how to do the automation or use the open-source. Devs are drawn to complexity like moths to a bonfire. And we make it when we don’t have it…)

I reckon that if everything in software development was merely simple or complicated, Waterfall would work very well. You’d be able to set out to achieve some goal, work out how to do it, achieve the goal and say, “Job done.” Software doesn’t consist of only simple and complicated stuff, though. So let’s have a look at the other two domains.

Complex

My favourite way to understand complexity is that acting in the space causes the space to change, and cause and effect can only be understood in retrospect.

When you start writing tests, or having discussions, and the requirements begin changing underneath you because of what you discover as a result, that’s complex. You can look back at what you end up with and understand that it’s much better, but you can’t come up with it to start with, nor can you define what “better” will look like and try to reach it. It emerges as you work.

In a complex environment, you probe, sense and respond. You do something that can fail, safely, and it tells you things about the environment which you respond to, changing the environment . This is the land of high-feedback, risk and innovation: generally stuff you’ve never done before, anything that the business are unsure about, new technologies, etc. This is the domain in which Agile techniques really flourish. If you have a look at Cynefin’s pyramids, this is the one with very loose leadership and the people at the bottom all connected together, collaborating to solve a problem.

This is the most interesting domain for me. It’s the reason why we do things like BDD – using examples to discover more about what we’re doing – but it’s also the reason why, if we focus on trying to pin every small requirement down, we fail.

Chaotic

Chaos is your house catching fire. Chaos is accident and emergency. Chaos is that bug you released to production that brought your site down on the day of release, and you need to drop everything and fix it now.

In chaos, you act. You get out of the house. You stem the bleeding. You do something to get the situation under better control. When Egor Homakov hacked Github this week, Github responded by suspending his account immediately. They acted on the threat. After that they analyzed his actions, considered what he had done and reinstated his account. Act, sense, respond.

So why should I care as a dev?

I’ve found this model really useful for understanding why certain ways of approaching software work best in certain situations, and fail at other times.

The biggest failures I’ve seen have come from treating complex problems as if they’re complicated. For instance, a common Agile practice is to divide a problem into small chunks that we call “stories”, then start working on them. That’s actually a good way to work when most of the problem is complicated, but if you’re doing something new then you may want to go down the “probe, sense, respond” route instead. Hack something out, and get feedback on it. There’s no point guessing how you’re going to reach the goal, because as soon as you get feedback, there’s a good chance the goal itself will change.

Also, sensemaking is itself a complex thing. By trying to get feedback, you might find out how easy or hard it is just to get feedback. That could change the way in which you engage with the business. It might affect how much time you put into preparing for a planning meeting, whether you make a feature work fully or just fire off a screenshot, etc. As you seek to get that feedback the business will change the way in which they respond, too, so it might become easier or harder, and you need to be watching for those signs so you can help to change the process to match. As a dev, you’ll be the one feeling the frustration or ease from the process. It’s no good just relying on your coach or Scrum Master to help you, because he or she needs your insight too.

Knowing this model helps me know when to do TDD or automate BDD – defining a well-understood outcome, and working creatively with the software to reach it – and when to just use examples as ways of discovering more about what we’re trying to do. If I can clearly articulate the outcome and everyone agrees on it, then probably it’s good for TDD and BDD automation. Otherwise, having conversations is more important than automation, whether it’s with a pair-programmer at a class level or a business spokesperson or tester at a system level. Being able to tell the difference can help ensure that the conversations are the most interesting and effective conversations we can have. If I spot the conversations becoming boring, with people yawning or drifting off, then I know we’re trying to apply a complex method to a complicated or simple space and I can say, “Okay, I think we understand this well enough,” and work on something different instead.

(Also, knowing that the act of sensemaking is complex itself makes me look out for situations in which we’re misapplying methods.)

When chaos erupts, I know that letting my PM order us about for a bit is OK. Dropping everything we’re doing is also OK – forget the whole “developers need to be left alone to do their work” idea that the Scrum timebox normally provides. We fix the bug, hack the workaround, take the feature down, do whatever we need to do to get things under control. Chaos almost matches the “expedite” class of service in Kanban, except that I think you probably don’t even bother putting a card on the board until the need to expedite is over. After the emergency is over, we can look at whether it was just a one-off or whether there’s something we need to do to avoid happening again. For instance, being able to roll back a deployment cleanly and quickly gives us the ability to probe a release instead of crossing our fingers, moving something that might become chaotic into a complex space instead.

These are the kind of decisions that we make intuitively anyway, as developers, but human intuition is often flawed. We’re often uncomfortable with uncertainty, so we usually try to define outcomes regardless of whether they can be clearly defined or not. We’re definitely uncomfortable with chaos, and often make the mistake of treating a one-off chaotic incident as systemic, stamping controls over everything when what we really need is a way of probing, or trying things out safely. Having this model in my head has really helped me to become more comfortable with those situations, and to find approaches that fit them much better. Of course, this is massively useful when I’m coaching and acting as a change agent or helping a PM work out how to lead a team – but even as a dev, having this model makes a big difference to me and my day-to-day work.

I hope it will help you too.

Or, “A feature is what you’re building. A story is how you’re going to build it.”

Or, “When you’re doing TDD, don’t worry about the internals of the class. The API is what it does. The internals are how it does it.”

Here’s the thing. When we write code, that’s how we’re creating a story or a feature. It’s also how we’re implementing the architecture. It’s how we’re managing security and providing an audit trail and doing a bunch of other stuff.

And that’s how we’re selling more goods. And how we’re keeping things maintainable for the future. And how we’re preventing data theft. And how we’re correcting our mistakes.

And that’s how we’re staying in business. And how we’ll be able to react in the future. And how we’re able to sleep at night. And how we learn.

• The code is how we deliver a story.
• The story is how we deliver a feature.
• The feature is how we give the users the capability to do something.
• The users’ capabilities are how they deliver a business goal.
• The business goals are how stakeholders implement a vision.
• The vision is an idea of how to make money, save money or protect revenue.
• And we could keep going if we wanted to…

Every goal, no matter how big or small, is the how for someone’s what. (The how and the what may come from the same person, but the interesting stuff happens when they’re different people. Or behave like they are.)

If only life worked by being able to divide a bigger what up into smaller pieces and manage them appropriately, then it would all be fine! Unfortunately the code has to work with other code to deliver its value. The features have to work with other features. The system has to work with other systems, users have to work with other users and the goals of one stakeholder have to align with the goals of another.

This is why we have root cause analysis and the “5 Whys” – because when we can see the higher-level goals and deeper-rooted problems, we can understand better how our own actions fit into them, and what they deliver – for better or worse. This is also true for other domains than software.

It doesn’t really matter much whether we call them features, stories or tasks, as long we appreciate that they’re how we’re delivering to those higher-level whats, and we have a pretty good understanding of the whats and how to test that we’ve achieved them, even (especially!) if the what is a learning or exploratory goal.

Of course, we’ll get the how wrong occasionally and fail to deliver the what. But that’s what feedback, in its many forms, is for.

There are notes underneath each slide which are a cut-down version of the kind of things I talk about. I’ve even left the exercises in, with a description of what I do in each.

I am releasing these under my usual favourite licence – Creative Commons share-alike. That means that as long as you attribute me, and are similarly generous with any derived work you make, you can use the slides commercially and base your own work on them.

Yes, that means that if you want to, and you feel qualified to do so, you can use the slides, run the course and get paid for it. If you don’t feel qualified, feel free to experiment. Just please give a nod my way when you do!

Dan explained to me, “If you start with the word ‘will’, you’re already making the assumption that you understand what you’re doing. Using ’should’ allows you to question whether you understand or not.”

For me, that was the moment that I suddenly got it. BDD and TDD aren’t actually about making sure that something works well. They’re about uncovering the parts you don’t understand; the parts that are hard, and the gaps. Dan’s post introducing “Deliberate Discovery” takes this idea even further, but it started here: replacing the word “test” with the word “should”.

This is why I find it more useful to look for the gaps in the examples; scenarios we haven’t thought of, or don’t know what to do. Chris Matts calls this, “Breaking the model”. Often whatever we do here is wrong, so we need to write down our suggestions and then get feedback on them.

If you do anything which starts suggesting to people that these examples are correct and complete – like, for instance, automating them or referring to them as “specifications” – then you may find yourself running into trouble. Automation is a commitment, specification can be seen as one, and we know by now that we never commit early unless we know why.

You might also want to keep a small record of the looser requirements or behaviour that spawned the examples – for instance, as blurb in a scenario or as a method name – just so that you can give yourself a chance of spotting those gaps later. You won’t always get it right first time, and that should be OK.

The outcome of the unconference wasn’t really stated. When you understand that a complex domain is one in which the cause of an outcome can’t be perceived except in retrospect, this might make more sense. The only thing we were trying to do was see if there was a way of using complexity theory to help inform our practices, and if there were some practices from Agile and Lean that complexity theorists might find interesting – a mash-up!

There’s one problem with this.

Currently, the best-known leadership of Complexity Theory revolves around the company Cognitive Edge. These guys have some amazing methods for making sense of domains, spotting complex problems and providing data which calls out “weak signals” that might otherwise be lost. I paid good money and took time off work for the course last year, and it was worth every penny. For the non-initiated and tl;dr, imagine five new types of retrospective, a method for reducing planning meetings to five minutes, and six different ways of making the output from them heard, and you’ll get a vague idea of the impact and scope. Oh, and they’ve got software for running the retro across countries.

Except…

I can’t currently use the methods they taught me, not as a professional coach. The methods are open-sourced, but released under a non-commercial, non-derivative Creative Commons license.

Cognitive Edge, your Wiki says (emphasis mine):

The Cognitive Edge wiki exists to provide a collaborative space for accredited members of the Cognitive Edge Network. All accredited practitioners should feel welcome to contribute to the ideas and concepts in these pages.

The licence prevents me from using your methods as a professional coach:

You may not use this work for commercial purposes.

The licence also prevents non-accredited people, which is most of our communities and a lot of CALMalpha attendees, from creating their own ideas:

You may not alter, transform, or build upon this work.

While I might be able to build on your work, I’m unwilling to do so as long as my efforts fall under this licence. I also can’t pass on anything to the people I work with for their contribution.

Can you see how this doesn’t mesh with the idea of a “mash-up”, and goes completely against your ideas around multiplying perspectives?

So here’s my request.

Cognitive Edge, please, please open your licence up for commercial and derivative work.

The stuff you do is amazing. If you were working solely for the money, you wouldn’t have come up with these ideas. I can only assume that you, like us, are trying to make the world a better place. We will continue to attribute the methods to you and talk about how amazing they are. Those of us who’ve seen it will continue to point people towards your SenseMaker software (which is ground-breaking, world-changing, worth paying for the 1-day demo, and deserves the more rigorous patent applied to it – I look forward to the day when it’s a bit cheaper!)

As it stands, we can’t do anything useful with your methods. Worse, because you’re working in a space full of narratives and I’m working in a space full of very similar examples, I have to be very careful that my work – released on my blog under CC, non-commercial, non-derivative – is actually based on other sources (mostly Dan North and Chris Matts) and not on yours.

Please. Be generous. Reach out to your contributors, ask them, and release what you can.

Sometimes it’s worth doing something you can’t go back from.

Our industry is in an awful state. A really awful state. It took me a day and a half to recognize one of the problems. There’s a prevailing sentiment that I keep hearing that, “If only we provide the right bucket for them to deliver in, teams will deliver.”

You know what? This isn’t true.

Of the many developers I’ve worked with over the years, I’ve been lucky enough to work with the best. These are the developers who can code well, work in teams, learn quickly and respect and learn from differing opinions.

I shouldn’t have to be lucky to work in teams like this. Engineers don’t get away with it. Architects, pilots, restaurant staff and dustbin men don’t get away with it*. We have such a low barrier to entry in this industry, it hurts. I shouldn’t have gotten away with it for as long as I did!

While I was at the conference, a gentleman linked to one of my blog posts about Scrum and Kanban, calling it “pontificating rubbish”. I wrote it mostly because I was already fed up of the fighting between the two communities: there’s more similarity than difference. This gentleman said, “Can we just ship some software already?” Iain, I need to thank you. Your comments must have stuck with me, because by the second day I found myself thinking, “He’s right.”

One of the things we strive for in Agile and Lean is a high-trust environment; one in which the business give us the space and time that we need to deliver the things that they really want. As an industry, though, we don’t have that trust, because we haven’t deserved it. Even Agile and Lean haven’t solved the problem. Even Agile and Lean don’t ship on time, and even Scrum projects occasionally wait until the last moment to tell people it’s going to be late (and yes, I know this isn’t how you ought to run a Scrum project). So here’s the first problem:

Many of the developers I’ve interviewed over the last decade can’t communicate. They can’t code, even when they’ve been told that it’s part of the interview. I had one dev provide a code sample that they were completely unable to work with – I’m guessing a friend wrote it. I’ve had samples submitted which were ripped off of the internet. I’ve worked “with” devs who were sidelined onto useless projects, deleted afterwards, because they were no good at coding and too nervous to help. No wonder, as an industry, we can’t ship. No wonder those of us who spend our time at conferences and self-improvement workshops are befuddled as to why. The faculty at CALMalpha talked about how crews work, and the training they undergo. We don’t do that.

One of the things that David Snowden taught us is to aim for fail-safe probes – a thing that we try, in order to make a change, which might fail and which is safe to do so. He also taught us not to attack the problem directly. I’m not sure that this is even a complex problem, but the fact that it hasn’t been solved yet suggests it might be. So rather than saying, “Just ship it already!”, here’s what I suggest (and please, if you have an understanding of complexity theory and can see some stuff that might not be safe to fail, let us know quickly – I’m new at this).

Get devs to code in the interviews.

Do it, and spread your success stories. I’ve not yet found a company who’s done this and stopped. Code with them – pair on some silly, simple problem – and you’ll also find out what they’re like to work with, and whether they’re able to learn and talk to you.

The idea is not necessarily to filter the developers. If this happens industry-wide, we’ll quickly run out of devs who can code, and something else will happen. Perhaps a lot of them will be able to learn quickly, and you may find after some time that you’re frustrated looking for the ones who can code and you’ll settle for the ones who can learn. Or maybe the universities will up their game. Perhaps it will become accepted practice that developers spend time learning, and companies will give them more space.

If this fails, and we find that we can’t get people to code in the interview, we’ll be no worse off than we were and we might find out something more about the problem.

*Dear Dustbin Men

Without you life would very quickly grind to a messy and disease-ridden halt. You are my heroes. You do the job that I do least want to be doing, for a lot less pay than I’d do it for. This is why I’ve picked on you – not because I think you’re low on the list of people who need to work as a team, but because your job is so stressful in my estimation that I believe you have the right to work in any way you want. That I always see you working together, usually smiling, carting away all our rubbish and doing extra work to make up for Christmas etc., is amazing. Thank you.

(I have never yet seen a Dustbin Woman).

Devs, our job is ranked as really not all that stressful. We do not have the excuse.

I was in a planning meeting with my project manager and several of the devs. “What happened?” the project manager said. “Why did this one story take so long?”

“There was some functionality we needed and didn’t know about,” I replied. “We managed to get it in before the deadline, though.” The business had been quite happy with that, and they were notoriously hard to please.

“If they’re going to change their mind like this,” the PM replied, “we’re going to have to introduce some kind of change control.”

“Please don’t,” I begged. “If you do that, the business will spend more time investing in getting things right to start with.”

“Exactly.”

“But they’ll still get it wrong. No amount of planning would have spotted that missing piece before we showed it to them! When they get it wrong now, though, they’ll encounter the change control and they’ll want to spend even more time getting it right first time. And they’ll still get it wrong, but now we’ll have made it more expensive for them to be wrong. We’ll have a formal process which means it takes even longer for us to find out what’s missing, by which time us devs will have to work to remember the code and the change will take longer. It will slow us down. So they’ll see that, and spend even more time trying to get it right, and before you know it they’ll be planning whole projects up front.”

We have a word for that. It’s called Waterfall.

That desire to control change creates a reinforcing feedback loop, in which the cost of change makes us want to invest up-front, which makes the change expensive later on, which makes us want to invest up-front, and so on.

In this case, it would have been pointless, except as a way of shifting blame and risk back to the analysts (and this was an internal project). The cost of change was quite low; we had clean code with a good suite of tests. It was only the cost of discovery, and the implementation that followed, that was really expensive.

Don’t confuse the cost of discovery with the cost of change.

Discovering something later on only costs more than planning for it if you’ve made a commitment to something else.

In fact, if you don’t plan for it, it can cost less. The newly-discovered knowledge will be fresh in everyone’s minds. Because the ideas haven’t been known for long, nobody’s mentally committed to them either, which makes them easier to question and clarify.

This is even more important when there’s a chance that the analysis might be wrong (and there’s always that chance; we learnt this from Waterfall). If you plan for something that you later have to revert, you’ve introduced a cost of change right there. Perhaps the cost is just changing some documents. Perhaps the developers have designed for the plan, and built on top of the thing which needs changing.

But we need to do some planning, right? Otherwise the chances of us being wrong and building on top of the wrong thing are even bigger, and there’s even more chance that we’ll have made commitments the wrong way. So how should we plan?

I’ve got few more guidelines I’d like to share. Here’s the first:

Keep the cost of change low.

This is more important even than planning, because there’s always a chance that we’ll get something wrong.

This is what we’re aiming for: the ideal, low cost of change on an Agile project.

The first thing to notice about this is that the cost of change is not zero. That’s going to become important in the next section, and it’s what drives the team’s desire for change control and starts kicking off that Waterfall loop.

The second thing to notice is that this bears little resemblance to actual, real Agile projects.

On a real Agile project, it’s likely that we have fluctuating levels of stress, concentration, experience and desire for feedback. All of these – or lack of them – will lead us to occasionally write code that is, shall we say, less than ideal.

It takes discipline to write code that’s easy to change. On a real Agile project, we tend not to do it all the time. Oh, we might say we do, but we don’t. Not always. And if we do, our team-mates don’t. The real skill isn’t in writing clean code – it’s in cleaning up the horrible mess we made the week before. And it takes even more discipline to do it afterwards, especially if you’re under pressure to just hack the next working thing in too.

If we don’t clean up and keep that cost of change low, we’re making a commitment to the wrong thing. The longer that commitment stays in place, the higher the cost of change will become. We’ll find ourselves on that Waterfall cost of change curve, and the longer we’re on it, the more expensive it is.

I’ve found the skill to clean up afterwards is even more important in high-learning projects, where a lot of the technology or domain is new, at least to the team. There’s no point in writing tests up front for a class when you don’t even know what’s possible, or what other frameworks and libraries will do for you, or what the business really want. In those environments, rather than TDD or BDD, teams tend to Spike and Stabilize. The spikes aren’t really prototypes – they’re small pieces of features, often with hard-coded data behind them, designed to get some feedback about techology or requirements. Dan North, who gave me the term, might write more about this later if we ask nicely, but for this post, we can simply bear in mind that the skill to stabilize later – ensuring that the cost of change is lowered – is often more important than the skill to keep the cost of change low up-front.

Because we get it wrong too.

Assume you got it wrong.

Human beings are hard-wired to try and get things right, and to pretend that they did even when they didn’t. I love this list of cognitive biases on Wikipedia. These are just some of the ways in which we get it wrong and don’t even notice.

If we assume that we got it wrong, then we start to look for feedback, and quickly. This is difficult for most human beings. We much prefer to get validation than feedback; to be told that we did it right, rather than finding out what we did wrong. Our brain gives us these little kicks of nice chemicals when we learn that we did something right, and it feels much better than the other kind.

If we can remember, though, that we probably got it wrong, our focus will change. Instead of trying to invest in good requirements and nice code, we’ll try to find out what we got wrong in the things we’ve already done. Of course, we need to invest in stabilizing what we’ve done, or the cost of change goes up, and that will make it more expensive later if we find out we were wrong, which is the assumption we were trying to make in the first place… ah, the paradox!

There’s a fine balance to be struck between getting quick feedback – often itself an expensive proposition, given the business of most domain experts – and getting it right up front. So where does the balance lie?

Don’t sweat the petty stuff.

If it’s easy to change, don’t worry about it. Analysts, learn what’s easy to change. Typos, colours, fonts, labels, sizes, placement on the screen, tab order, an extra field… these are all usually easy to change, and do not normally need to be specified up-front. Even if you have a particular style you want to see on a page or form, this can usually be abstracted out and changed later – just let the devs know that you want that consistency at some point.

It’s more important for us to know the rough shape of what you want to see, and the flow and process of that information. We don’t want to know every field on an order item. We just want to know that it needs to be sent to the warehouse and stored locally because you’re going to check the money in the till and count the stock. The fine detail of that is pretty easy to change, so we can get feedback on it later. Getting the fine detail right would definitely be an investment, and we might have got the big picture wrong.

Deliberately discover things you’ve never done before.

Dan North wrote an excellent post on Deliberate Discovery, and I’ve been using it to manage risk on my projects for a while now. It’s one of the most important tools in my toolbox, along with Real Options to which it’s strongly related, so I want to cover how I use it here.

I really like using Cynefin to help me work out what to target for discovery. We treat a lot of software as if it’s complex, and we talk about self-organising teams and high learning environments, but in reality there are huge chunks in most applications which follow well-established rules, have been done a thousand times before and probably have libraries or third-party apps to do the job for you. They’re not complex. They’re complicated. They require the application of a bit of expertise, and are likely to be done right and never changed again. User registration and logging in are great examples of this. You don’t need a big, thick document to describe them. The fine details might change, of course, but we already know not to sweat the petty stuff.

It is OK to plan some aspects of a system as if it’s Waterfall – for instance, deciding up-front whether you want to use your own login or let Google authenticate. Even better than requirements documents, and quicker, is to say, “It’s user registration. Make it work like Twitter’s, but we also need the user’s loyalty card number. We should offer to send them a card if they don’t have one.” Dan North calls this pattern “Ginger Cake” – it’s like a chocolate cake, but with ginger. He even cuts and pastes code. And it’s OK! Honestly, it is! This code is also absolutely prime for TDDing – if you actually have to write it yourself, that is, since it’s been done before so someone’s probably written something to do it for you already. You can also give this code to junior devs, for whom it’s new, and guide them in TDDing, making it perfect pair-programming territory. Everything you have ever been told about Agile software development applies particularly in this place.

Fortunately, most applications have a minimum set of requirements that they share with other, similar applications. David Anderson calls these commodities – table stakes that you have to have just to play the game – so *most* code in an application will end up going this way.

The places in which we’re most likely to get it wrong, and need fast feedback, are places where we’re doing something new. They might be technological, particular to a domain, or just things that the team themselves have never looked at. My favourite book for understanding risk is “Waltzing with Bears”, which starts the first chapter with, “If a project has no risks, don’t do it.” It’s these new, risky aspects of the project that differentiate it from others and make it valuable in the first place!

For new and risky aspects of a project, the best thing to do is assume you got it wrong, and work out how quickly you can get feedback on how wrong you are.

Any new or unknown aspect of a project will need to be changed.

I was chatting to one of our analysts. “I can see this feature is in analysis at the moment,” I said. “Does that mean it’s the next thing we want the developers to do?”

“Oh, no,” the analyst said. “It’s only there because the analysis is quite complex. It’s all new stuff, so we’re having to be careful with it and it’s taking a bit of time. Once we get the analysis done, the development should be very easy, so we’ll do it later.”

“Oh, the development will be easy, I’m sure… but wouldn’t you like to find out what you did wrong now, rather than later, while it’s still fresh in your mind?”

The analyst smiled. The company was very much more used to Waterfall, and the idea that it was OK to get it wrong was something very new.

It’s OK to get it wrong, as long as you get feedback quickly, while the cost of change is still low. By working out which parts of a project are unknown or new, and targeting those first, we make small investments while the cost of change is still low.

Keep your options open – do the risky stuff first and keep tech debt low.

Anyone who’s run into me at conferences will know how much I love Real Options, and it’s really at the heart of the cost of change.

The only reason change costs more is because of the commitment that we already made. Chris Matts describes technical debt as an “unhedged call option”. Edit: while Chris came up with the metaphor, it was Steve Freeman who described it. He says, “You give someone the right to buy Chocolate Santas from you at 30 cents each. That’s fine, as long as the price of chocolate stays low. As soon as it goes up, you still have to pay to make the Santas, and now you’re in trouble and your company is going bust, because you didn’t give yourself the option to get the chocolate somewhere else.”

Similarly, technical debt is absolutely fine until we’re called on to act, and act fast. At that point, we’re in trouble. This is the biggest reason for keeping the cost of change low – because it gives us the option for change, later. It’s a frequently-cited reason for replacing legacy projects – and, bizarrely, often forgotten when the pressure mounts and the business want their replacement app.

This isn’t helped by common practices of estimation and the associated promises, which often lead to that pressure building up in the first place. Rather than making these promises up-front, why not try the risky bits first? I often hard-code data so that I can get feedback on a new UI early, or I spike something out using a new library or framework, or connect to that third-party application just to see what the API is really like to use, or have a chat with the team writing that other system we’re going to need in June, so I can find out how communicative and receptive to feedback they are. Doing this means that we give ourselves the most time to recover from any unexpected discoveries, and we can worry about the more predictable aspects of a system later.

Once we’ve got spikes out of the way, adding tests to act as documentation and examples for any legacy code we’ve created, cleaning it up so it’s self-commenting, ensuring that architectural and modular components are properly decoupled, etc., all help us to stabilize the code. At the same time, the effort involved in creating stable code is itself an investment. If there’s a good chance that the code might be wrong, it could be worth getting feedback on that – knocking up integration tests, showing it to the business, testing it, getting it live – before it’s made stable.

That way, the commitment made is small, and the cost of change is low.

Just remember to clean up and keep it that way!

Update: Dan is currently writing about Spike and Stabilize and Ginger Cake as part of his “Patterns of Effective Delivery”. If you’re interested in finding out more, you might like to watch his Roots talk on the topics.