Wednesday, October 14, 2015

TDD and Getting Lost in the Trees (Part 2)

"There is no such thing as complete when it comes to stories. Stories are infinite. They are as infinite as worlds." – Kelly Barnhill

This is the sixth of a series of posts about Test-Driven Development (TDD).  Here's a recap of what I've posted so far:


In the last post, I started describing the situation when, in doing TDD, it starts to feel a lot like you're getting lost in the woods. You get that sinking feeling of helplessness when you've lost your bearings and become confused. There's probably even some dismay, desperation, and fear mixed in that bag of doom and gloom. Invariably, people in this situation ask themselves "What in the world have I gotten myself into and how the heck do I get out of it?"

There are two aspects to addressing this problem: prevention and remediation.

Let's talk about prevention first.

Extending another olive branch


Before we get back to eavesdropping on the imaginary conversation between Phil, aka "Tandy", and Carol in their simple calculator TDD session, I'd like to extend another olive branch to the practical-minded folks who have stuck it out through the five philosophically-laden posts that came before. That's a long time to hold out hope that you'll see some sample code written through TDD.

I'm going to take a slight detour from the calculator program and demonstrate a few things from a very simple exercise in refactoring. It will illustrate three simple tools that I believe can help you vastly improve 80% of the code you write.

I was looking for an analogy for these and started out with the venerable Swiss Army Knife but after thinking about it, these techniques are really like your basic oral hygiene kit: a toothbrush, floss, and mouthwash. If you use these properly and consistently, not only can you go about your daily business with freshness and confidence, you can also avoid a host of bigger problems and pain in the long term.

The Code Hygiene Starter Kit


The code hygiene starter kit includes the following techniques:

  1. Rename
  2. Extract
  3. Compose Method

That's it.

You might be saying to yourself right now that it can't be that simple. Yes, in my experience it really is that simple. However, my observation is probably consistent with what dental hygienists see with their patients: there are a lot of people who do not use these tools diligently or consistently. Only about one out of ten programmers I talk to have even heard of the Compose Method refactoring or use it on a regular basis when they are refactoring code.

In my experience, diligently applying these techniques in every TDD cycle keeps you moving forward with clean, readable, and well-factored code 80% of the time. Oftentimes, these are all that's required for 100% of your refactoring needs.

Joshua Kerievsky's Compose Method example has become my go-to example for this trio of refactoring techniques. I like to use Joshua's example not only because there's a stark improvement in the code after just a few simple changes, but it also embodies the strategy that I use to avoid getting into the "lost in the woods" predicament we're talking about.

I won't rehash what Joshua discussed in his article so go ahead and read that before you continue here.

Notice that Joshua uses the three refactoring techniques in our basic code hygiene kit. Mixed in there are some bonus goodies: the Arrow Code antipattern and the Guard Clause, and the Single Level of Abstraction Principle or SLAP, which I mentioned in passing in the first post of this series. It also has the elements of the strategy I mentioned. More on that later.

It'd be great to be a fly on the wall


In the previous post, we heard Carol tell Phil that the way we get there is just as important as what we get in the end when we do TDD.  The problem with most examples on TDD is that, for the sake of brevity, they seldom let us in on the conversation that goes on while refactoring and TDD is happening. We only get to see the results. That's understandable but unfortunate.

I think that TDD remains a mystery to many because they miss out on certain nuances about TDD that can only be discerned when we have a better understanding of the thought processes involved in arriving at the refactored code. Conversations and experimentation are what weaves the threads of knowledge into the cloth of understanding.

So here's another chance at being a fly on the wall during a reconstruction of some of the conversations I have with participants in my TDD workshop when we go over the Compose Method refactoring.

Caveat: As you read through the dialog below, remember that these are still fragments of a much longer conversation. For the sake of relative brevity and coherence, I left out parts where we go from Red to Green to Refactor or where we discuss the merits of others choices we needed to make. This illustrates my point about the challenges in giving good, complete stories about TDD. You really have to be there to appreciate the whole thing.

Me: First up, let's address the Arrow Code problem. We know how to recognize it now, right? And to remediate the problem, we introduce a guard clause, right?

Participants: Right.

Me: Hmmm, since we're doing TDD, do we really want to do that right away or should we do something else first?

Participants: We should probably write a test first to make sure we don't break anything.

Me: Great idea! What test do we write first?

Invariably, somebody will want to address the null case first. I steer them away from that and any other test idea not related to introducing the guard clause. I'll skip that part of the conversation for now; there are other topics where that's relevant but we'll get into those later.

Me: So, say we write the test to check that the guard clause works. What would this test look like?

Participants: We should check the readOnly property.

Somebody will offer something like this:

public class MyListTest {
 
   @Test
   public void testReadOnly() {
       MyList list = new MyList();
       list.setReadOnly(true);

       assertTrue(list.isReadOnly());
   }
}


Me
: Ok, so we add a setter, that's fine. But how does this test help us make sure that the "introduce guard clause" refactoring won't mess anything up?

Participants: Well, if the list is read only, then the guard clause should work, right? The guard clause is checking the readOnly attribute and it will work if the readOnly attribute is true, right?

Me: But is that really what this test is telling us?  Here's what I'm getting from this: When you make a list readonly, then its readOnly property is true.

Participants: We don't get it. We're confused. What's your point?

Me: That's like saying that if you're a U.S. citizen, then your passport is blue.

Participants: Ok, that's true, a U.S. passport is blue. But we still don't get your point.

Me: If I'm a U.S. citizen, the fact that my passport is blue doesn't mean much. Being a U.S. citizen means I can vote. Otherwise, I can't. What does the readOnly attribute allow or prevent you from doing?

Participants: Ok... well, if readOnly is true, we can't add any new elements to the list.

Me: Right! So instead of checking the readOnly attribute, we need to check the elements, right?  If we try to add to a list that is read-only, then the code should ignore that request and the elements array shouldn't change, right? Would anybody care to modify the test?

Somebody will offer something like this:

public class MyListTest {
 
   @Test
   public void testReadOnly() {
       MyList list = new MyList("First");
       list.setReadOnly(true);
 
       list.add("new element");

       assertEquals(1, list.getElements().length);
   }
}


Me: Ok, so we add a constructor that takes an initial element. We can easily make that take a variable size argument list so it can be any number of elements. That makes sense. Then we get the elements and check its length. Is everybody ok with this?

There is usually a nod of agreement or an awkward pause here, sometimes both. Sometimes somebody will object but let's go with the usual case where they still feel a little lost. I won't even go down the thread of conversation that addresses the fact that the above code has a big bug in it. That can also be the motivation for getting away from this option.

Me: Ok, is there an object-oriented design concept that we're violating here? Are we breaking anything, like say by examining the length of the elements array that we get from the list?

Participants: Yeah, we're breaking encapsulation. But we do this all the time in our code...

Me: Doesn't mean it's right, does it? This is what talking about these things does: it puts your bad habits in a spotlight. It makes you rethink your approach.  What can we do so we don't break the encapsulation of the list?

After more prodding and discussion, someone will come up with this:

public class MyListTest {
 
   @Test
   public void testReadOnly() {
       MyList list = new MyList("First");
       list.setReadOnly(true);
 
       list.add("new element");

       assertEquals(1, list.size());
   }
}


Me: Ok, that looks better.  By adding the size() method, we pushed down the knowledge about the implementation of the elements array and its length, and replaced it with a higher level concept of size().  At this level of abstraction, it doesn't matter whether there's an array or whatever involved.

Me: That's a step forward. But there's some implied dependency in this test. Where does that 1 in the assertion come from?

Participants: There's only one initial element in the list.

Me: But what if somebody comes in later and decides they want more than one initial element? Are they going to remember to modify that assertion accordingly? What are the chances of them forgetting to do that? There's an implied connection between that first line where we initialize the list and the assertion.  Can we do something to make this more explicit and less brittle? Either bring it all out in the open or hide it behind a name that reveals our intent better?

After more prodding and discussion, someone might come up with this:

public class MyListTest {
 
   @Test
   public void testReadOnly() {
       MyList list = new MyList("First");
       int initialSize = 1;
       list.setReadOnly(true);
 
       list.add("new element");

       assertEquals(initialSize, list.size());
   }
}

Me: That's good. We definitely made the intent of the test a little clearer. But there's still a need to change that line that sets the value of initialSize if we add more initial list elements in the line above. We can make that better, right? How about if we did this?

public class MyListTest {
 
   @Test
   public void testReadOnly() {
       MyList list = new MyList("First");
       int initialSize = list.size();
       list.setReadOnly(true);
 
       list.add("new element");

       assertEquals(initialSize, list.size());
   }
}

Me: That's kind of obvious now, right? Why didn't we think of that before? Now we won't have to worry about having to keep these two lines synchronized. But there's still something not quite right with that name. There seems to be a disconnect between its intent and the test intent.

I want the test to read pretty much like this: When I try to add new elements to a read-only list, the list's size should not be changed after I call the add() method. Or something along those lines. How can we refactor this test so that it says that? Is there a better name we can use instead of initialSize?

After more discussion, we have these versions to choose from:

// Choice #1

@Test
public void testReadOnly() {
   MyList list = new MyList("First");
   int initialSize = list.size();
   list.setReadOnly(true);
 
   list.add("new element");

   int finalSize = list.size();

   assertEquals(initialSize, finalSize);
}

// Choice #2

@Test
public void testReadOnly() {
   MyList list = new MyList("First");
   int sizeBefore = list.size();
   list.setReadOnly(true);
 
   list.add("new element");

   assertEquals(sizeBefore, list.size());
}

// Choice #3
 
@Test
public void testReadOnly() {
   MyList list = new MyList("First");
   int sizeBefore = list.size();
   list.setReadOnly(true);
 
   list.add("new element");

   int sizeAfter = list.size();

   assertEquals(sizeBefore, sizeAfter);
}

// Choice #4

@Test
public void add_does_not_change_size_of_readOnly_list() {
   MyList list = new MyList("First");
   list.setReadOnly(true);
   int notChanged = list.size();
 
   list.add("new element");

   assertThat(list.size(), is(notChanged));
}

Style Note: The test name convention I use here is something I picked up from Neal Ford at a No Fluff Just Stuff Symposium many years ago. Neal's assertion was that test names are special and don't really need to follow the normal convention for regular method names. I find this convention more readable and it also helps me highlight regular program element names when I use them in the test name.

That last option above actually involves a lot more discussion than I let on here. There's discussion about the test name itself, the convention (see the Style note above), whether we really need that second explaining variable, whether we want to use a more fluent API for assertions, and even a discussion about whether to move the first explaining variable closer to the Act part of the test (see Arrange, Act, Assert).

After more discussion, we settle on this:

@Test
public void size_of_readOnly_list_does_not_change_after_add() {
   MyList list = new MyList("initial element");
   list.setReadOnly(true);
   int notChanged = list.size();
 
   list.add("new element");

   assertThat(list.size(), is(notChanged));
}

There are a lot of details in this particular arch of a story that I have left out. I will leave it to you as an exercise with your programming partners to explore and discover your own version of this story.

Again with the reflections


You might be getting tired of this by now but it's not going to end. These posts are going to be full of reflections.

Reflection #1: Refactoring is a lot harder in the field than it is on paper.

On paper, the Compose Method refactoring example seems simple and straightforward. You see the end result and the way to get there seems obvious. Reality is very different, however, especially when there are people who are not familiar with the thought processes involved.  The end result is not known and getting there requires experimentation, discussion, and sometimes, negotiation.

In my experience from giving interviews/auditions and from leading a TDD workshop, this exercise can easily take 30 minutes to go through when I do it with as few as three or four people. And if you're thinking otherwise, this seems to have very little to do with the intelligence or apparent lack thereof of the people involved.

Rather, I have found that the level of familiarity and understanding of the principles of design, the mechanics of the refactoring techniques, and the kind of questions we ask each other and the quality of the ensuing discussions determines how quickly we get to the desired end state of the code or at least a good enough approximate of it.

Reflection #2: Finding good names for program elements is challenging.

Programmers have a tendency to be very technical when choosing names in their programs. Names tend to leak implementation details. Names can mislead the reader. They can obfuscate the intent of the code. I'm not even talking about the names that are chosen out of lack of effort to find anything better than numElmnts.  Names like this are a pet peeve of mine. Read the relevant chapter in Uncle Bob's "Clean Code" book. Please.

There are a few things you can do to find good names. Keep the language of the conversation focused on high-level concepts. Constantly remind each other that you want to keep as much of the implementation details hidden and make the intent abundantly clear. Pay attention to grammar, spelling, and context. These will usually help you stumble upon a good name.

In the five TDD workshops that I have led in the last two years, there has been only one person who came up with the name atCapacity() right off the bat.  And no, this guy had never seen Joshua Kerievsky's example before. I was very excited when that happened and my reaction got a few strange looks. All the other times, we needed a few iterations to get something as succinct. Some usual suspects include: hasEnoughElements(), isMaximumSize(), isMaxSizeReached(), and other similar alternatives, each with their own problems and reasons for disqualification.

I will talk about names in more detail in another post.

Reflection #3: It is important to have these kinds of conversations and thought processes to spur action down a path that leads to better code.

The conversations usually revolve around revealing intent and improving clarity. Revealing intent includes the intent of tests as well as the intent of the program code. These two intents must be in agreement. The tests need to say what it expects of the program and the program must say what it does to meet those expectations.

Questions must be asked in a way that elicits ideas for saying what you want to say both with the tests and the code in as clear and succinct way as possible. Don't settle on the first idea that pops up either. There are usually more questions and ideas that can be brought to the surface after the first few rounds of refactoring.

Reflection #4: Prefer to verify behaviors and capabilities rather than attributes

Tests should focus on verifying behavior, not so much the value of an attribute. This meshes very nicely with the idea that object-oriented programming is mostly about properly assigning responsibilities. A good set of guidelines to use is GRASP, the general responsibility assignment software principles. There's also the Law of Demeter and code smells like feature envy and inappropriate intimacy.

In the example conversation, I used the analogy of being concerned with a person's ability to vote rather than the color of a person's passport.

Since tests also should tell what is expected of the program, you should make sure the test says what these expectations are in a clear and unambiguous way. Nothing implied, nothing assumed, and nothing that the reader must deduce from clues that are found anywhere other than the immediate test. Parts of the example conversation revolved around this.

I find that reading the code out loud really helps in discovering subtle problems related to unclear intent and mis-assigned responsibilities. Try to explain what the code is doing in non-technical terms to each other. I'll write more about making code more expressive and readable and how to "listen" to the code in a subsequent post.

Reflection #5: Diligently and consistently apply the Code Hygiene Starter Kit techniques in the refactoring step.

These three simple techniques will help you quickly fix problems you introduced in the Red and Green steps of TDD and avoids the buildup of plaque in your code that can cause bigger problems and pain in the long run.

Just as you should brush, floss, and gargle daily, you should also Rename, Extract, and Compose Method constantly throughout the TDD cycle. Refactor ruthlessly.

Reflection #6: The "strategy" that I alluded to earlier in this post is this: have an idea of the end in mind and use that to guide you along the way.

It's kind of like getting a bearing on the sun and having a general sense of north, south, east, and west. Or finding the North star if you're traveling at night in the northern hemisphere. This is your "big picture" view.

As you slowly make your way through the fog of uncertainty and the unknown, the brush and undergrowth of quickly written code, and the thick of trees that are your program requirements, don't lose sight of where that guiding light is and keep a bead on it so you know you're still heading in the right direction.

Don't sweat the small stuff or let yourself get distracted by them but don't forget to pay attention to the little details, too. Little details can often make a big difference in how your code ends up saying what it's doing.

You may not see any connection between what I discussed above and the last reflection but it's there. It's just very subtle and for me, it's one of those things that is elevated to the level of a general principle for TDD.

Again with the Martial Arts connections


You may also be getting tired of these martial arts analogies but these concepts have really helped me see things in my TDD practice in a different light.

Miyamoto Musashi wrote in the "Use of the Gaze in Strategy" section of his "Book of Five Rings":

Use the eyes in a broad manner. There are two aspects of sight—perception and seeing. Perception is strong and seeing is weak. It is vital to see things which are at a distance as if they were close, and things which are close as if they were far away. It is vital in strategy to know the opponent's sword, but not to look at it.

Aikido has a similar concept called Zanshin which also pertains to a broad focus and general awareness of your surroundings.  In western terms, the closest thing to zanshin is the concept of situational awareness.  It's an awareness of your own relative position and connection to everything else in any given setting.

In an often-told story about events that led to a spiritual awakening that inspired the development of Aikido, O'Sensei was said to have seen and perceived his connection to the entire universe, thus allowing him to easily and deftly evade an expert swordsman's strikes until the swordsman was so exhausted from exertion that he had to concede defeat to O'Sensei.

I like to explain it like this: In Aikido we are taught to not let our minds be captured by the swinging katana (sword). The blade moves faster than you can ever hope to follow. We are taught to not allow our mind be captured by the other person's eyes but rather to look at nothing in particular and perceive everything as a whole. Connect with uke's center this way and you can control him before he even moves. In Aikido, the conflict is over before it even begins.

If this makes no sense to you, that's fine. Hopefully you'll start getting a sense for what it means as you practice TDD more.  More on this later.

That's it for this "little" detour. Hopefully, it has helped set the tone for a better understanding and quicker recognition of ways around the challenges that Phil and Carol are going to encounter when we rejoin them in their TDD exercise next time.

Next: TDD and Getting Lost in the Trees (Part 3)

Sunday, October 11, 2015

TDD and Getting Lost in the Trees (Part 1)

"Experience enables you to recognize a mistake when you make it again." – Franklin P. Jones

This is the fifth of a series of posts about Test-Driven Development (TDD).  Here's a recap of what I've posted so far:


As promised in my last post, I'm going to show some code examples this time. But before we get to that, bear with me as I wax philosophical just a little bit more.

How do we get ourselves in trouble? Let me count the ways…


I've seen many ways that programmers, including myself, can paint themselves into a corner. You have to admit, we can be pretty good at getting ourselves wedged between a rock and a hard place and I wouldn't be surprised if you found a mess of other ways besides the ones that I'll put under a microscope here. (Quick, how many idiomatic expressions did I manage to squeeze into this paragraph?)

I started out writing this post thinking that I could cover several scenarios but it quickly became clear that this first one that I'm going to bring up has so many facets to it that it deserves to be picked apart in isolation. Buckle up because this is going to be a doozy!

Recognizing that we don't know what we don't know


If you're one of those who I predicted would struggle with TDD and the simple calculator problem, then you probably didn't do so well on your first few attempts, even if you think you did.

You might think it a bit presumptuous of me to say that or maybe even arrogant but if you think about it for a minute, it's really a natural consequence of just starting out with TDD. Without a frame of reference, how can you know if what you just did was really what TDD is supposed to be like? If you're relatively new to TDD, then you won't really know that you've done it poorly until you do it better. As Donald Rumsfeld famously put it, "There are things we don't know we don't know."

This is kind of what the phrase "Aikido works, your Aikido doesn't" is getting at. It's what we sometimes tell people who think Aikido is ineffective. Similarly, people who say the same about Agile are sometimes told that "Agile works, your Agile doesn't." Or with TDD, "TDD works, your TDD doesn't." I honestly don't say this often because it tends to rub people the wrong way and often the people you want to say it to the most aren't really worth the aggravation. Nevertheless, there is a ring of truth to it that can't be denied and maybe that's why they get mad, because now they know it's them.

When I was just starting to learn Aikido, I got better by pairing up with black belts because they would usually point out things that I could improve, things that I would have never recognized otherwise. Now that I'm a black belt myself, I try to pay it forward by helping new students recognize ways they can improve their practice.

Remember, practice only makes habit. Only perfect practice makes perfect. Unless we start recognizing aspects of our practice that are flawed or could be strengthened, we're not going to be able to make it better, much less perfect. So, here are some things that I first focused on recognizing and making better in my TDD practice. Hopefully, these will help you see how you can get better, too.

Getting Lost in the Trees


You've probably heard the expression "seeing the forest for the trees". This first scenario that we will examine is about losing sight of the forest and getting lost in the trees. There are many dimensions to this but it happens quite often and it mainly comes from not understanding the principles that Michael Feathers and Steve Freeman alluded to in their presentation "Test-Driven Development: Ten Years Later." It is rooted in at least two thought processes that I think most developers find difficult to avoid. These are boxes that most of us find hard to think out of.

(Grammar police wannabes be like, "Really, ending a sentence with a preposition?" and I'd be like, "Chill out, dude, don't get all Carol with me. It's fine." Carol is hilarious though. If you're not quite following me, I'm talking about the TV comedy show, "The Last Man on Earth")

First, let's take a look at some test code that's typical of when developers are getting themselves "lost in the trees."

// CalculatorTest.java

public class CalculatorTest {

   private static final double TOLERANCE = 0.0001;
 
   private Calculator calc;
 
   @Before
   public void setUp() {
      calc = new Calculator();
   }

   @Test
   public void testAddition() {
      assertEquals(3.0, calc.evaluate("1 + 2"), TOLERANCE);
   }

   @Ignore
   @Test
   public void testMultiplication() {
      assertEquals(6.0, calc.evaluate("2 * 3"), TOLERANCE);
   }

   @Ignore
   @Test
   public void testAdditionAndMultiplication() {
      assertEquals(7.0, calc.evaluate("1 + 2 * 3"), TOLERANCE);
   }
 
   @Ignore
   @Test
   public void testWithParentheses() {
      assertEquals(9.0, calc.evaluate("(1 + 2) * 3"), TOLERANCE);
   }
}

// Calculator.java

public class Calculator {
   
   public double evaluate(String expression) {
       return 0.0;
   }
}

If you're thinking that this code doesn't look all that bad, you should probably pay close attention to the rest of this post. You might learn a few things you don't know you don't know. And no, that's not a typo; see the Donald Rumsfeld quote above.

If you're thinking that this is way too much code to start with when you're doing TDD, it's great that you recognize that. You're right, if this is the code that you have after just one or two TDD cycles, then you're probably writing too much up front. But just humor me for a bit. Let's just say for now that this is code you might have after a few rounds of following the Red-Green-Refactor flow. Then we'll step through how we might get here.

They say that imitation is the greatest form of flattery so I'm going to fashion this after Uncle Bob's archetypical XP Episode. Since I already mentioned Carol, let's get Phil in on this and pretend that they are pairing up on the calculator TDD exercise. Suspend your disbelief as necessary.

(Phil, of course, is the title character in "The Last Man on Earth")

Phil: Hey, Carol, I started that calculator TDD exercise like you asked me to. Here's the code so far. I even have my first failing test. See?

Carol: Really, Phil? That looks like a lot of code. Did you really get that through proper TDD or did you just write it all at once because it's what you thought you'd need anyway?

Phil: (sheepishly) I wrote it all at once because it's what I thought I'd need anyway...

Carol: That's Ok, Phil, I understand. You just wanted to save some time and cut to the chase. But it's important to understand that the way we get there is just as important as what we get in the end when we do TDD. The means is just as important as the end, Phil. Remember that.

Phil: I don't know what we're getting out of doing all those little steps though, Carol. It seems like a waste of time and it feels silly to do the wrong things when the right things are so obvious.

Carol: Oh, like how you knew that NOT leaving Todd in the desert to die so you could have more booty for yourself was obviously the right thing to do?

Phil: Aw jeez, Carol, that was a low blow, even for you. I already said I was sorry! And I went back for him, didn't I?! Am I ever going to live that mistake down? That was one small lapse of judgement in a moment of weakness!

Carol: Relax, Tandy. That's all behind us now but you should remember the lessons from the past. Doing TDD reminds us that you often have to see the wrong thing before you realize what the right thing to do is. Tell you what, why don't we write that calculator again and this time, let's go through the TDD cycle and thought process, step by step.

Phil: (grudgingly) Ok. Lucky for us, I'm using Git and I just happened to make that default JUnit test my initial commit. Let me revert to that really quick.


// CalculatorTest.java

public class CalculatorTest {

   @Test
   public void test() {
      fail("Not yet implemented");
   }

}

(PAUSE)

Let's hit the pause button for just a minute and reflect on what just transpired. You might be thinking that this conversation is a bit contrived (Well, duh. Unlike Uncle Bob and Bob Koss, Phil and Carol are not real people) but it's based on observations and questions from my own experience with doing TDD and teaching it to other developers.

Reflection #1: Phil went ahead and just wrote all the test code that he thought he was going to need. His intention was to save time. He also felt that the small intermediate steps were a waste of time and that it was silly because he was writing code that was obviously wrong.

The thing to recognize here is that writing correct code and avoiding or eliminating incorrect code is part of the complex algorithm for writing a program that you've followed for years. Your brain has most likely also been trained to be averse to rework, based on the belief that rework is wasteful and costly and that you can avoid rework by doing things right the first time.

Does that strike a chord with you? Do you not hold these things in high regard, if not dearly, as a software developer?

These ideas form part of your understanding of what it takes to write good software. It's the same kind of understanding that you have about how to ride a bicycle. It is the kind of understanding that is so entrenched in your brain that it has become almost a second nature to you. It's that rigid idea that Destin said is stuck in your head and you're finding it hard to change it, even if you want to.

Destin said that "Knowledge is not understanding." In other words, understanding runs deeper than knowledge. Much deeper. That's why it's so hard to change. Even if you have the knowledge about how to write programs through TDD, it's going to take a lot of effort to supplant your brain's current understanding of how to write programs with a new understanding that comes from all that TDD knowledge.

Does that make sense?

Reflection #2: TDD is as much about the way you get there as it is about what you get in the end.

Very skilled programmers can probably produce software with the same level of quality and testing that you could get by doing TDD properly. The problem is, I don't think I've ever met one these mythical super rock stars of programming in real life, even after decades in this profession. I've worked with many companies and met many developers and from what I can tell, the average developer just doesn't write very good programs and they don't write very good tests, sometimes even when they claim to be doing TDD. Color me cynical but I'm just stating my opinion based on experience.

If you recall what Michael Feathers and Steve Freeman said in their presentation, TDD is much more than just testing. Each of the steps you take in the TDD cycle has a very distinct purpose and focus. And that's why the path to get to the end product is very important. More about this later.

Reflection #3: TDD reminds us that we often need to see what's wrong before we can recognize what's right. You've probably heard or said this before: "I'll know it when I see it." We say this because it's probably the case that

  1. You've never really seen "It" before
  2. "It" is just a vague notion or an immature or incomplete idea
  3. You can't completely describe or wrap your head around what "It" is
  4. There are a few things "It" could be; there are multiple possibilities for "It"
  5. You have to actually try some of the possibilities for "It" first before you can say what "It" is

Take a minute to reflect on the above again before we hit the play button to see what happens next with Phil and Carol.

(PLAY)

Phil: Ok, first step is to write some test code and see it fail. I'm going to need a Calculator class. Agreed?

Carol: Sure, Phil, I agree.

Phil: (sigh) This is going to be a long day. (makes changes to the code)

// CalculatorTest.java

public class CalculatorTest {

   @Test
   public void test() {
      Calculator calc = new Calculator();
   }
}

Phil: Ok, I'm instantiating Calculator but this doesn't compile because we don't have a Calculator class yet. That still qualifies as Red because according to Uncle Bob's Three Rules of TDD, code that doesn't compile is still a failure.

Carol: Yup, you got it. Go ahead and fix it then.

Phil: (uses keyboard shortcuts) There. Now the test compiles.

Carol: Oooh, now you're just showing off, Phil. But that's cool. There's nothing more annoying than mousing around for every little thing you need to do. The keyboard shortcuts are much faster. Ok, run the test then.

Phil: What do you mean? There's nothing in the test to run.

Carol: Oh, Ok. Well, I suppose we should write another line of test code then.

Phil: See, this is getting pretty silly now. Why do we have to do this when we can easily just write everything out and then fix the problems all at once? This is a really tedious way to write a program, isn't it?

Carol: Believe me, Phil, I've been there. It really does feel tedious and downright stupid at first but once we get a hang of the flow and a better idea of where we're going, we can start trying to write bigger chunks of code. Right now, we're writing tiny chunks of code to force ourselves to take things slow. Really slow.

Phil: Why do we need to go so slow?

Carol: Because we want to fail fast and fail small.

Phil: Wait, what? I thought we were going slow. Now we want to go fast? WTH, Carol! Make up your mind! Are we going slow or fast?

Carol: Silly Care Bear. I know that sounds contradictory but by "going slow," I mean that we write smaller increments of new code. Smaller increments means that it'll take us longer to get all the code that we eventually want to get. That's because we're failing fast and failing small, which means that we run our tests every time we add a small increment of code. This way, we'll know right away if we messed anything up. And if we do mess anything up, we know it's probably the last small increment of code that messed it up. Bottom line is that we go slow by adding only small bits of code, fail fast by always running the tests and getting feedback right away, and fail small because we can only mess up a little bit with each increment of code.

Carol: It's kind of like driving in the dark, right? If you go fifty miles before checking if you're still on the right road, then you can drive faster but there's a bigger chance that you can go miles out of your way and get really lost. But if you stop every five miles to check with someone at a store or gas station if you're still on track, then it might take you longer to get wherever you're going, but at least you won't get too far off track at any point. Does that make sense?

Phil: That driving thing makes sense because that's kind of what I did when I took scenic routes from Billings, Montana back home to Tucson. I think I follow your twisted logic though, Carol. You still might have to explain it to me again a few times. I'm not sure I really understanding it all.

Carol: That's Ok, Phil. At least now you know. Understanding will come with practice. You just gotta keep doing it for a while for it to really sink in and make sense. And when it does, it does. It's kind of like those 3D pictures that you have to stare at cross-eyed, you know? It's hard to see the 3D at first but once you get it, you get it and it's a lot easier to see the 3D in other pictures afterwards.

Phil: Huh, I guess I never really looked at it like that before. Yeah, it took me a while to get those 3D pictures. I always saw them as just random patterns until one day, I just suddenly saw the 3D in one of them. Ok, I'll try to see this through some more. Now I'm curious to see if this picture of TDD that you have is as great as you say it is.

(PAUSE)

I'm going to hit the pause button again here for some more reflection.

Reflection #4: Adding small increments of code may seem silly and tedious but it allows you to go slow and keep checking your progress more often. I think Carol explained it quite well.

Reflection #5: Fail Fast and Fail Small. Run your tests every time you add an increment of code so you get feedback quickly and often. The smaller your code increments, the smaller the problems are that they might introduce. The smaller code also makes it easier to revert to a previous state where all the tests were passing.

Reflection #6: Knowledge is NOT Understanding. Understanding will come later, when the knowledge you have is exercised in a context. The more contexts there are in which you apply that knowledge, the more it gets ingrained in your brain as Understanding.

Reflection #7: The tests provide different contexts in which you can exercise your knowledge of what your program is doing. The more you test your program, the more you understand what it's doing. See Reflection #6.

Reflection #8: The algorithm for writing a program with TDD is very complex and complicated and belies the simplicity of the Red-Green-Refactor mantra. As you can see from the number of things we can reflect upon from just this short "conversation," there are many things to think about when you're doing TDD.

You may be asking where those "trees" are that we supposedly can get lost in. If you don't see them, then remember what I said before about not having enough experience to recognize what you don't know? The trees are there but maybe you just didn't recognize them for what they are.

I know you're probably disappointed to not see more code than what I've shown but if you have a little bit more patience, it will be rewarded. (Remember what I said in the first post about needing some virtues?)

We'll continue our eavesdropping on Phil and Carol's TDD session next time. They are going to be writing a lot more now that they have a bit of a shared understanding of TDD.

In the meantime, I encourage you to reflect on all this some more. As you do that TDD exercise again, keep in mind whatever new knowledge you've managed to pick up here. This is how you develop recognition, which in turn opens up possibilities for improvement and getting closer to perfection in your practice of TDD.

Next: TDD and Getting Lost in the Trees (Part 2)

Saturday, October 10, 2015

TDD: First Time Crash and Burn

This is the fourth in a series of posts about Test-Driven Development (TDD). Previously, I posted these:



Eliminating the Self-Centered Ego 


One of the things that makes Aikido different from many other popular martial arts is that there are no competitions in Aikido, unlike say, in Tae Kwon Do or Karate or Wu Shu.

The following are excerpts from the book "Advanced Aikido" by Phong Thong Dang and Lynn Seiser:

O'Sensei Morihei Ueshiba... discouraged competition. He felt that competition... diminished Aikido's application as a legitimately applied martial art. He also believed that competition and the striving for championship status would only further encourage, develop, and inflate the self-centered ego. The learned identity ego, in Eastern thought, is one of the major barriers to sprituality... He felt... that competition would create ego blocks to spirituality.
"Aikido is for the development of the entire human being," states Doshu Moriteru Ueshiba... The mental discipline of Aikido is to overcome our own ego for spiritual development, as opposed to employing Aikido in the service of the learned ego identity, to overcome someone else, thus strengthening what Aikido is trying to minimize.

No Contest


You may be wondering what any of this has to do with TDD. Well, in my previous post, I posited that people who have never written a program with TDD would not be able to write a simple calculator program with TDD. I put out the challenge and gave some requirements for how the calculator should work.

And then came the beat down: "You can't do it! You might think you can, but you can't!"

This is what Destin, the engineer who learned how to ride the backwards bicycle, told viewers of his short and hugely popular video on YouTube. He even offers to pay $200 to anyone who can ride the bicycle for ten feet. I don't think anyone has cashed in on that offer yet. Quite frankly, I think it's a sucker bet and Destin knows it.

I'm not going to put any money on the table for this but I'm betting that at the very least, half the people who take up the challenge of writing a program through TDD for the first time will falter after a few minutes and quickly fall back to their old ways of writing programs, much like those who take up Destin's challenge on the backwards bicycle.

The Agony of Defeat 


As Destin said: You wouldn't think it, but the algorithm for riding a bicycle is just that complicated and any change to one thing throws the whole control system off balance. The algorithm for writing a simple program is pretty complicated, too, maybe even more complicated than that for riding a bike.

"But what does that stuff about Aikido have to do with any of this?" you might ask again.

It all goes back to what I said before about being afraid to screw up. It's the ego, man. It's about not wanting to look stupid. It's about wanting to be a winner.  It's about proving Destin and me wrong about you not being able to do it.

"Ok, whatever," you might say with a roll of your eyes.

But am I wrong? Were you really able to write a little bit of test code, then some production code, then refactor, then some more test code, yada, yada, yada?  Were you able to get through the whole exercise without, at some point, stopping and racking your brain as to what test you should write next? Did you not, at some point, stop and look at the mess you'd made and say, "What the (bleep)?!!! This is horrible! This is frustrating! How can I possibly not get this right?!"

I know this is starting to sound like a lawyer badgering the witness but admit it, it's not as easy as it seems, is it? Admit that you started all over at least one or two times, maybe even more. C'mon, if I can be man enough to say that it also happened to me, which it did, you can at least admit that it didn't go swimmingly well for you either.

I hate to say "I told you so," but I told you so. The truth hurts, doesn't it?

Keep Calm and TDD On


Ok, so if you can admit to crashing and burning on this first outing with TDD, then there's still a glimmer of hope for you. Here's why: I couldn't do it at first but after a while, I finally could. After about two years actually.

As self-deprecating, down-to-earth, and approachable as Kent Beck and most of his contemporaries in the field can be, we all know that the apparent ease and dexterity with which these tests and code can be wrangled is belied by the fact that they are the gurus, gods, and demigods of TDD. It's easy for them to talk about and demonstrate how powerful and effective TDD is as a technique because they practically and quite literally invented all this stuff.

It's kind of like how my fellow students of Aikido around the world and I will watch videos on the Internet (isn't technology wonderful?) of all the Aikido masters and even of the founder himself from way back in the 1960s, and wonder in awe at how easy and effortless they make all these techniques appear to be. And then we try to do them and find out that it's not easy at all.

I, on the other hand, hold about as much candle to any of those guys as a baby to her octogenarian grandmother. I'm one of you, just another Joe Schmoe programmer trying to learn how to get a little better at this job. If I can pick myself up, dust myself off, and power through the pain and frustration of failing at TDD, I think most of you can, too.

So relax, take a deep breath, and try it again. It gets better, I promise. You just gotta keep keepin' on.

Eliminating Competition


So now we know that most of us go into TDD with a good measure of knowledge about it but with little to no understanding that we have what Destin calls "a very rigid way of thinking" which we cannot easily change even if we wanted to.

Were you not at some level fighting with this new paradigm? Were you not struggling against the urge to just write production code first, even for just a little bit? Once you entered that ring with TDD, you quickly realize that despite having all the weapons you thought you needed, you were still ill-prepared for this battle with your mind.

Did you really think it would be that easy to find the magical switch that controls the flow of your thought processes, creating big up front designs and writing production code before the tests? The bias from your past experiences guards that switch very jealously, trying to keep you from flipping it. And when, on occasion, you are able to flip it and alter the flow of your thought processes, your bias continues to fight to flip it back to its "normal" position.

Even if you don't buy all that or want to dismiss it as just more hokey martial arts, Eastern philosophy baloney, that's fine. All I'm saying is that this is how I retrospected about the start of my journey in learning TDD. Retrospecting about it helped me re-align my outlook and attitude towards TDD so that I could be more open minded and accepting of what was happening and what TDD wanted me to do.

Call it mind games, or neurolinguistic programming, or whatever. The biggest obstacle to you being able to go with the flow of TDD is your own bias from past experience and you need to eliminate at least some of that bias to keep going, to get unstuck. That's the competition you have to deal with.

Where's the Code, Dude?


All right, I promised I'd give some demonstrations on how I do TDD and all I've given so far are a bunch of philosophical essays. I get it. You're a developer and you want to see some code.

In the next post, I promise I'll finally get to it. Sorry, but it's the weekend now and the Buckeyes are playing Maryland, with kickoff in less than twelve hours. My daughter is finally going to go for her road test and my wife is bugging me to finish putting up curtains on the new windows and a shower head in our newly-renovated master bathroom.

Next week, I promise, you'll see a lot more code.

Friday, October 9, 2015

TDD and that First Awkward Moment

This is the third in a series of posts about Test-Driven Development (TDD). These are what I posted before:

These posts were partly inspired by a question I answered at coderanch.com about how to develop a simple calculator program with TDD.  It turned out to be quite a long discussion and you can head over to the Ranch and read through all 200+ responses there, if you have that kind of time to kill. Or you can stick around here as I review and summarize some of the things I mentioned in that thread.

Ok, I created a new test... now what?


So you've decided to do TDD and you're all excited and eager to see and experience just what everyone has been talking about. You've created a new JUnit test class, which typically looks something like this:

public class MyNewTest {

    @Test
    public void test() {
        fail("Not yet implemented");
    }
}

Now comes that first of many awkward moments people new to TDD are most likely to have. It's just like being on a first date and not knowing exactly how to break the ice, so you sit there uncomfortably, hoping and praying for a flash of inspiration for a witty remark. 

You say to yourself or your programming partner(s), "Now what?"

Why do we hesitate? What are we afraid of?

Think about that for a minute.

Here's my best armchair psychologist's analysis: We get stuck because we're afraid of being wrong. We're afraid that we'll mess up, that we'll look stupid, or that we'll feel stupid. 

My wife would probably say it's a typical "guy thing," knowing that we're hopelessly lost but not wanting to ask directions for fear of being called "a dumb schmuck" or looking like one.

Ok, I'll admit that I think the invention of portable GPS navigation systems and mobile apps has probably saved more marriages than AAA TripTiks, Dr. Phil, and Couples Counseling combined ever will. 

I see the same thing with beginners in the dojo. They'll start the technique, then stop midstride and ask for validation. Then they'll want to start all over. We always tell them to just keep going and do what they think they saw sensei demonstrate just a few minutes ago.

The Challenge


As pointed out in my previous post, knowing is not understanding. You may be nodding your head in agreement to all of the above but to really understand, you need to get on that backwards bicycle and feel it for yourself! You need to step on the mat and practice!

Let's try to create a simple calculator through TDD. Now, if you've never written a program using TDD (or maybe even if you think you have), then I'm sorry to tell you this but "You can't do it! You may think you can, but you can't!

Destin's words, not mine. But yeah, he's probably right.

At this point, hopefully you're saying "Ok, homes, challenge accepted!" as you're firing up your favorite IDE and deftly clicking around various menus with your mouse, or better yet, quickly typing in the keyboard shortcuts, to create a new project.

What, are you still just reading? Go ahead and fire up your IDE, dude!

Enter the TDD dojo


I expect that by now, you have your brand spanking new project laid out in front of you. Hopefully, you've navigated to the /src/test/java folder or wherever you plan to keep your unit test classes. If you're a real go-git-'er-done type, then you probably have already created a new JUnit test called CalculatorTest.  I am, of course, assuming that you're programming in Java and using JUnit. If you're not, feel free to go ahead and make the necessary mental adjustments.

Now what?

What's that? You need requirements? Ok, here they are:

Write a simple calculator program that can handle the four basic arithmetic operations of addition, subtraction, multiplication, and division. The calculator should adhere to the order of precedence rules associated with the mnemonic "Please Excuse My Dear Aunt Sally," except this calculator need not support exponents for now. Your calculator should honor the order of precedence dictated by parentheses.
The calculator should be able to handle decimal calculations and a result that is a rational number supported by your platform, presumably a typical laptop or desktop computer. Any underflow or overflow should be reported as an error. (Note: if you're trying to do this on a mobile device or a mainframe, I salute you, but maybe ease up a little and find a PC or Mac for this, you weirdo)
For example, given the expression 1 + 2 * 3, the calculator should return 7 as the result rather than 9.  However, given the expression (1 + 2) * 3, the calculator should return 9 as the result.
Also, the calculator should be able to display intermediate results as they become available throughout the process of evaluating an expression. 
For example, while evaluating the expression 3 * 4 + 5, the calculator should be able to display the intermediate result of 12 when the "+" operator is reached.  And while evaluating the expression (1 + 2) * 5, the intermediate result of 3 (the sum of 1 and 2) should be available for display when the closed parenthesis is reached.
That's all I have for now.

I'm going to end this post here and give you a chance to go and prove Destin and me wrong. (And no, you grammar police wanna-be, the correct form is "Destin and me," not "Destin and I."  Look it up for yourself if you doubt it.)

In the next article, I will go over what typically happens with TDD neophytes and cargo culters when they try to do this.

Hajime!

Next: TDD: First Time Crash and Burn

Thursday, October 8, 2015

TDD and the Backwards Brain

This is a followup to the first article I posted about TDD and why it's so hard to learn and sustain.

In that first article, I proposed that TDD is hard because it requires Mastery, which takes a lot of time, and requires virtue and lots of practice.  Some might read that and think it's just a lot of philosophical mumbo-jumbo that doesn't really help anyone get closer to being able to understand and learn TDD, or get better at it, or keep doing it.

Some might think that the "Mastery" thing was just a lead up to a mention of Dan Pink's talk about "Drive," and the whole Autonomy, Mastery, and Purpose thing. Well, there I said it. Yeah, there's going to be a bit of that, too. But that's for later on.

Right now, I want to extend an olive branch to the practical-minded folks out there who didn't really know what to make of the whole Mastery thing from the last article. I want to give you something more scientific, more fact-based, or at least empirical. I present, for your enjoyment and edification, the "Backwards Brain Bicycle!"

Go ahead and watch that video first. It's only about seven minutes long.

Interesting, isn't it? Do you think you could ride it? Do you think you could handle that kind of frustration and perhaps even, humiliation?

Let's go over some of the points that were made by Destin (that's the guy's name, in case you didn't catch it. Yeah, just like that one time, in Florida, where you went for Spring break...)

Point #1: Knowledge IS NOT Understanding 


Just because you have knowledge about something doesn't mean that you understand it. To fully understand the Backwards Brain Bicycle, you have to actually get on it and learn to ride it. A lot of people tried it but were unsuccessful.

It's the exact same thing with TDD!

Just because you know about Red-Green-Refactor, that you're supposed to write tests first, follow Uncle Bob's Three Rules and Kent Beck's Four Rules of Simple Design, know about SOLID, DRY, SLAP, GRASP, KISS, ... (Wait, what?! Are we still talking about software development here? Because it was starting to sound like we might need a "safe word"... ¯\_(ツ)_/¯ )

Ok, so maybe you didn't realize that there was so much to know besides that catchy TDD mantra. Well, that's kind of what I was getting at with the Mastery thing, too. But I digress.

To fully understand TDD, you have to experience it yourself. You have to fire up your IDE, create a new test, and somehow get past that awkward moment when you're just sitting there staring at the monitor and realize that you have not the slightest clue of what test you should write first. And you sit there valiantly fighting the urge to navigate out of the /src/test folder and go back to the /src/main folder where you're much more comfortable whipping out code.

You have to spend the time to rewire your brain somehow and break it of its long-time habit of writing production code first, and thinking that tests should be written afterwards. You now have to make it follow this weird, backwards way of writing programs. And that process of retraining your brain takes time. And it takes a lot of patience, discipline, humility, perseverance, practice, practice, practice, ... hmmm, does that sound familiar?

Point #2: The algorithm is just that complicated!


Destin says, "Think about it (the algorithm for riding a bicycle): downwards force on the pedals, leaning your whole body, pulling and pushing the handle bars, gyroscopic precession in the wheels... every single force is part of this algorithm and if you change any one part, it affects the entire control system."

Again, this is the same situation we have with TDD. As I mentioned above, there are a lot of things you have to think about when you're writing software. For years, your brain has been trained to think in a certain way, and then suddenly, you tell it to do the exact opposite. And that's not even the start of it!  We're not just changing one thing, we're throwing a whole toolbox of monkey wrenches at your brain!

Need I go over what those monkey wrenches are again? Uncle Bob's Three Rules, Kent Beck's Four Rules, SOLID, DRY, ... "Omaha!!!" (that's our safe word).

Which brings me to Destin's most obnoxious point and most ego-crushing realization that you must resign yourself to accept:

Point #3: You can't ride this bicycle! You might think you can, but you can't!


Likewise, very few people can really grok TDD without a struggle, with little effort. You can't just read a book or watch a video about TDD, or go to a five-day coding boot camp or whatever, and think that you're going to go back to work and merrily start whipping out tests and production code, in that order.

Destin claims it took him eight months of trying, spending five minutes every day practicing (that adds up to a lot of practice), wrecking, and putting up with his neighbors' teasing, before he could ride the backwards brain bicycle.

It took me two years of dedicated solo TDD practice, probably spending four extra hours a week on top of the time that I put in at work, before I finally got a hang of TDD. I can't pinpoint the exact moment in time and I don't even know if there was one particular moment when the "switch flipped" and I was off to the TDD races.

Point #4: One day I couldn't, the next day I could!


Unlike Destin's experience with learning to ride the backwards bicycle, I don't think I had that kind of magical moment that I can pinpoint in my TDD practice. I can't say it was a gradual thing either though. I don't know, maybe I was just too caught up in a moment of ruthless refactoring to realize that I was actually doing it well. It was probably more of a "you don't know how much fun you're having until you look back and recall how much fun you had before" kind of thing for me.

At any rate, it doesn't happen overnight but it does seem like all of a sudden the things you used to struggle with, the things you thought were so backwards and against the grain of the way you are used to doing things doesn't seem that way any more. At least not as much.

And so, we come to the final point I'd like to highlight:

Point #5: You're looking at the world with a bias, whether you like it or not.


Remember how it took Destin's son only two weeks to learn to ride the backwards bicycle versus his eight months? Destin speculated that this was because kids have better neuroplasticity and could adapt more quickly to the new control algorithm of the backwards bicycle.

Maybe that's also why younger and less experienced developers seem to be able to pick up on TDD much faster than the ones with more experience. It's not always the case but I've seen it often enough to think that it might be a common thing.

I think this is one of the biggest challenges in being able to grok TDD.  We all carry some kind of bias because it naturally comes with experience. That experience came at a personal cost. We've all worked hard and even made sacrifices to gain the knowledge and skills we have, knowledge and skills that are based on principles we have long stood for and defended as a mark and measure of our professionalism. That's not something that everyone will just willingly toss out the window.

But then again, that's a kind of bias too, isn't it? What if we change our mode of thinking? What if we, in the same spirit of TDD, flip our thinking around instead?

What if, instead of seeing it as a loss of the investments we've made in becoming the professionals we are today, we see it as an opportunity to diversify? Let's take the gains of our past labors and apply them to a new venture, one that can potentially give back huge returns if we're just willing to tough it out for a while, go back to square one, and see this thing through all its growing pains, and power through the suck zone until we finally land in that sweet kick ass zone.

What do you think?

In the next few installments, I'll start ramping up on the TDD code examples but I'll be using them mostly as the basis for further discussion about principles, philosophy, and mindset.  

In the meantime, I'd love to hear from you about your own experience with TDD and how you did, good or bad, successes and failures. All relevant comments are welcome and appreciated.

Next: TDD and that First Awkward Moment

Wednesday, October 7, 2015

Why is TDD so hard?

A little late is better than never, as they say.

I attended the Agile 2015 conference in Washington, D.C. back in August and was happy to see that there was at least one session about Test-Driven Development (TDD). I was even happier to see that someone was going to talk about sustaining TDD. I have also found that TDD can be very difficult to sustain and I have witnessed this with many of my colleagues at work. I suspect that it's not an uncommon problem across the board and the number of articles that Google turns up on this subject pretty much confirms this.

I had high hopes going into Scott Bain's session on "Sustainable Test-Driven Development" at Agile 2015 and he made some great points that I'll touch upon in subsequent posts. However, I think that Scott only scratched the surface of the problem and I believe his failure to dig deeper into it means that, ironically, his answers are not going to help much in sustaining the effort to promote TDD either.

TDD in Principle, not just Form


I just recently got to watch "Test-Driven Development: Ten Years Later," a presentation by Michael Feathers and Steve Freeman, both stalwarts of TDD, given way back in 2009.  Like I said, better late than never. Anyway, this is a great presentation and if you are an even bigger laggard than I am, you should go and watch it, too.

Despite having said them over six years ago, the things Michael and Steve talk about in the presentation are still very relevant today. And I see that Michael's skin condition still hasn't cleared up. Just kidding, Michael, I realize it's just a nervous habit you have. On the off chance that you really have been suffering from a recurring rash, I suggest a bit of moisturizing lotion or steroid-based itch cream <big grin, wink>. I kiiid, I kiiid. I'm a big fan of Michael's work and I think his WELC book should be on every programmer's desk, within easy reach for quick reference, right beside Steve's GOOS book.

One of the things in Michael's and Steve's presentation really resonated with me and got me thinking about possibly giving a presentation on it. Towards the end, at about the 45 minute mark, Steve says this about TDD:

"If you want to make this work, you have to understand how it works and what the practices are that go with it.  You have to understand the principles behind the practices."

Let me say that again with emphasis, because this is what I find myself saying over and over to the developers who attend my TDD workshop: "You have to understand the principles behind the practices."

I think that Steve is hinting at Cargo Cult TDD here. I always like to bring Aikido into the conversation so I compare it to what beginners experience in the Shu stage of ShuHaRi.  They are simply following the form, without necessarily fully understanding, or even partially understanding, the principles behind the practices.

The other thing that Michael and Steve mention at the end of their presentation that resonated with me was the need for craftsmanship and professionalism. Again, my Aikido training kicks in and tells me that this is not unlike the attitudes needed for shugyo, which is explained very briefly here and in great detail here. The documentary, "Jiro Dreams of Sushi," is really an exploration of the shokunin spirit, which is along the same lines.

The Truth about TDD


The hard truth is that TDD is difficult and I think the main reason people have a hard time learning and sustaining it is quite simple: It requires Mastery.

Just as with anything else that is worth doing, it's worth doing well. And you can't do something well without a healthy measure of Mastery.

Mastery takes Time


In "Jiro Dreams of Sushi," you learn that Jiro's sons spend years under their father's tutelage. Even young apprentices in their shop must spend many months paying their dues and doing other things in the kitchen before they are allowed to just cook the sushi rice. Getting moved up from cooking the sushi rice takes even longer and the standard of quality is higher and more difficult to meet.

In traditional Aikido dojos, uchi-deshi or live-in students have to help with the upkeep and care of the dojo, its surrounding compound, and the dojocho or headmaster, who often lives in the same compound. The uchi-deshi stay with the dojo for extended periods of time, anywhere from months to years, so that they can study intensively with the master. They put in a lot more time than the other students but it pays off for them in the long run.

It took me more than four years to work my way up through the kyu ranks before I was deemed ready to test for my shodan (1st degree black belt) in Aikido.  The black belt doesn't even mean that I'm really good at Aikido. It just means that I'm good enough and I understand enough to start actually learning Aikido. All that time before was spent just getting ready to learn. Again, I consider it time well invested.

It took me four years of dabbling and two years of more intensive solo TDD study to get comfortable with TDD and really grok it. That's a long time to be in the Shu phase of learning but it was well worth it.

Programmers are an impatient lot though. Ain't none of them got time for that! Well, most of them at least. Programmers need to get results in an internet minute and Mastery just doesn't go by that kind of timetable.

Mastery requires Virtue


In particular, Mastery requires the virtues of patience, humility, discipline, dedication, and perseverance. Each of these is challenging by itself, let alone all of them together. But that's what Mastery requires.

The master sushi maker, Jiro, had two sons. They learned about these virtues from their father. Only the eldest son would eventually take over the sushi shop from their father. The younger son would have to go and set up his own shop. He knew this all along and accepted his lot. He patiently studied and learned the trade over many, many years. He persevered for many, many years before he was given the blessing to set out on his own.  The same is true for Aikido instructors who want to branch off from the Hombu dojo, or the main school, and set up their own dojos.

Jiro's sons had no illusions about their ability to equal, much less surpass, their father's ability at making sushi. They knew that in time, they might be able to approach the kind of quality with which their father made sushi but they were humble enough to realize that they probably would never be as accomplished as he. This kind of attitude requires a tremendous amount of humility and self-awareness. It also requires dedication and perseverance to keep learning regardless.

I don't think being able to do TDD well demands quite those levels of virtue but mastery of it certainly requires levels that only a minority of programmers I have met seem to have or are willing to attain.

Mastery requires Perfect Practice


My son's viola teacher liked to say, "Practice makes habit. Only perfect practice makes perfect."

That's also true with sushi making, Aikido, and TDD. In fact, it's universally accepted that the only way to get to Carnegie Hall is through "Practice, Practice, Practice!" And even more practice.

Perfect practice.

Most developers who fail at TDD fail because of imperfect practice. Michael and Steve hint at that in their presentation, at around the 43 minute mark, where they show a table that involves some kind of magic number that has something to do with the spread of complexity within a code base and its relation to automated unit testing in various open source projects. You can search for related work by Keith Braithwaite if you're interested in the details. 

Anyway, Steve says that he feels that the kind of people who wrote the code in the projects that scored well (2.0 or higher) were the early adopters of TDD. He continues by saying that he's seen code bases with tests that are definitely not on the good side of the scale.

Steve's feeling and observation pretty much aligns with my suspicion that over time, the bulk of the later generations of programmers who have tried to do TDD have lost sight of what it means to practice it perfectly. Over time, TDD practice in the wild has become more and more cargo cultish. This happens in Aikido, too, and probably any other kind of practice of an art or craft as it proliferates among those who are further and further removed from the teachings of the original school of thought. I think it's very much like the phenomenon of Semantic Diffusion that Martin Fowler wrote about, only on a much larger scale.

What chance then do we mere mortal programmers have?


The above doesn't even begin to address the various challenges in adopting and sustaining TDD but it gives you an idea of how deep the problem goes, not just technically but philosophically as well.

I don't for one second believe that TDD is for everyone just as I do not think that Aikido is a martial art that everyone can get into or even just appreciate. Most long-time Aikido practitioners I have met have a certain kind of general mindset and attitude and I feel this is the same kind that you need to get into and appreciate TDD. You don't have to be a lifelong student of Aikido to have that mindset and attitude. You could just as well be a football player, a musician, or even a sushi maker. It's not an Aikido thing, it's a shokunin thing.

In subsequent articles, I plan to demonstrate some of the things I do when I practice TDD and delve deeper into the philosophy, principles, and mindset behind them. Hopefully, this will help other developers find a better understanding and some measure of sustainability in their TDD practice. I know it has helped me a lot.

And hopefully, after a few articles I'll have enough of my thoughts and material gathered and organized to be able to share in a presentation at Agile 2016 in Atlanta next year.

Next: TDD and the Backwards Brain

Friday, May 1, 2015

On Belief, Honesty, Integrity, and Predictability

"It ain't what you don't know that gets you into trouble; it's what you know for sure that just ain't so." —Mark Twain

I wish I could have attended the Craft Conference 2015 in Budapest, Hungary last week. Not only does Budapest look gorgeous at this time of the year, judging from all the pictures they posted on the conference website and Twitter feed, but it would have been a great opportunity to make a pilgrimage to the birthplace of one of my favorite things to play with, the Rubik's Cube, which celebrates its 35th anniversary of being introduced to the world on July 26 this year.

As it was, I was busy trying to fix some loose tiles in my bathroom instead. My wife had been asking me to take care of these since the winter but until last week I had procrastinated doing anything about it, reassuring her whenever she reminded me that the material on which the tile was laid was some kind of special water-resistant board designed specifically for use in bathrooms and toilets. It can wait until warmer weather, I told her each time, which was about every two weeks or so.

Well, Mark Twain proved to be right again.

Ever since I read Kent Beck's white book about Extreme Programming, I have believed in the promise of Agile methods. And ever since I read Martin Fowler's book on refactoring at about the same time that I read "XP Explained: Embrace Change," I believed that TDD, or test-first programming as it was first known way back then, with its short feedback loops, ruthless refactoring, and collaborative development and design, was the best way to develop software. I still do, as a matter of fact.

And yet, what's most perplexing to me is that to this day and perhaps into the foreseeable near future, there are those who still don't and won't buy into Agile and/or TDD. I just don't understand this. Even notable figures in the industry like DHH and Jim Coplien, who are undoubtably very influential in shaping the opinions of many, have come out against TDD. I won't rehash all that here. You can follow the "Is TDD Dead?" conversations that Martin Fowler moderated and read the InfoQ article that looks at the controversy about TDD in detail. Listening to and reading the reasons behind the dissenting opinions, I was once again reminded of Mark Twain's epigram and left wondering whether or not something I knew (or did I?) to be true really wasn't.

I mention all this because I have been preparing to give a two-day workshop on TDD for some engineers in my group at work. I have a Prezi where I go though some of the ideas on which I base my practice of TDD and kaizen, citing articles and quotes from "Uncle Bob" Martin and others. I'm really more or less trying to channel Uncle Bob in this workshop, promoting the ideas of craftsmanship and professionalism and bringing back pride in the work that we developers produce.

This brings me back to #craftconf. Marty Cagan's keynote was really thought-provoking and I just want to thank the conference organizers and speakers for making the videos of their talks available to those who couldn't attend first-hand. Marty talks about how most of the companies that he works with have really no clue what agility is about, despite claiming to be Agile. He talks about a number of fatal flaws in the development process and rails against how companies are using roadmaps and backlogs to trick developers into believing that they were doing Agile, even though when you really think about it, it really ain't so.

Dan North added even more doubt to my mind by saying that backlog grooming should be outlawed. This was said almost in passing during his talk, "Beyond Features". Dan North, the guy behind BDD and doing TDD the right way. Luckily, his article, "The Perils of Estimation", sheds some light on his statement and dispels some of the doubt that it cast in my head.

Coming back to my bathroom tile issue, when I took out the loose tiles, I was horrified to see some black scum covering the back of the tiles and the board underneath. To make matters even worse, the board, which appeared to be plain old gypsum drywall, was soaking wet and crumbling to the slightest touch. I started to work off the surrounding tiles and soon I had two gaping holes in my shower, exposing the nastiness that had been festering underneath the shiny tile all this while. What's more, one of the holes opens up to an external wall and the insulation was soaked and had more of that black scum, which is likely some kind of mold or fungi, on it. Now I'm looking at some major repairs and a big chunk of change taken out of my rainy day savings. When it rains, it pours.

Just as my belief that the material under the tile would be protection enough against water damage got me in trouble when it turned out not to be so, it seems that some of the beliefs about Agile and TDD that I hold dearly and steadfastly defend may not be so after all either. At least not in every way nor all of the time.

I'm reminded of a recent conversation I had on JavaRanch and how people new to Agile might feel about the enthusiasm of proponents and, admittedly, often obnoxious way it sometimes comes across. Sometimes the nice shiny exteriors can hide the nastiness that lurks inside the walls. I may be guilty of contributing to this kind of disingenuousness in the past but I know I have recently made a conscious effort to empathize more with skeptics and dissenters.

Reflecting on my own beliefs and attitudes, I realize that I have gone off on my own little "rantifestos"—I will take the liberty of co-opting that word, too, Dan— on the Ranch as well, particularly when it comes to software craftsmanship.

Here are a few more things that I had to reflect upon after watching Marty's and Dan's talks:

I thought that backlogs were a good way to get the users and product owners involved and engaged in defining the work to be done and planning how we would proceed in delivering the most value quickly. I thought that grooming backlogs was a good way to get a common understanding between everyone on the delivery team and the product owner as to exactly what needed to be done and approximately how long it would take to complete the work.

I thought that TDD would help developers write better code. I thought that with TDD, the testers could get along better with developers and that they both could work towards the common goal of ensuring that the software had the level of quality needed to go to production without testers expressly focusing on finding where the developers had messed up.

Turns out that Dan North, Marty Cagan, and even DHH before them, made some pretty strong arguments to make me reconsider what I knew so surely to be true.

Yes, TDD can be taken the wrong way and turned into something that developers use to satisfy the strong desire of some managers to see metrics on productivity, quality, and code coverage. This as opposed to just measuring the lead time to delivering a viable product that solved our customer's problem.

Yes, it does seem that the business types have tricked us into accepting their Gantt charts disguised as product roadmaps and backlogs. And yes, we're back to pulling estimates out of our rear ends again and feeling guilty when we don't "meet" those estimates, or rather, "forecasts" which apparently are, in fact, the "new commitments." These behaviors are detrimental to our honesty and integrity as developers and responsible members of a project team.

And yes, after watching the video of Marty Cagan's keynote, I believe that this insidious push for predictability in our projects is jeopardizing developers' integrity, making them less inclined to be honest, and creating a vicious cycle that works against our belief that Agile, or rather true Agility is the best way that we can quiet the business types' incessant badgering about "when can we get this done?"

So as I get ready to pay my $500 deductible and perhaps whatever is above the limit on my home insurance policy endorsement for mold and fungi damage, I must also prepare to warn the developers who attend my workshop next week about the perils of taking TDD the wrong way. I must also prepare to talk to our senior leadership about the perils of taking roadmaps, backlogs, estimates, and metrics the wrong way.

Lastly and most importantly, I must now gird my loins against those inevitable looks from my ever-knowing-what's-going-on wife —you know which looks I'm talking about— and man up and say "You were right, dear, there really was something going on behind those tiles and I should have taken care of it when you told me to."

Mark Twain may have been right but he obviously wasn't talking about wives.