TDD and Getting Lost in the Trees (Part 1)

"Experience enables you to recognize a mistake when you make it again." – Franklin P. Jones

This is the fifth of a series of posts about Test-Driven Development (TDD).  Here's a recap of what I've posted so far:

• Why is TDD so Hard?
• TDD and the Backwards Brain
• TDD and that First Awkward Moment
• TDD: First Time Crash and Burn 

As promised in my last post, I'm going to show some code examples this time. But before we get to that, bear with me as I wax philosophical just a little bit more.

How do we get ourselves in trouble? Let me count the ways…

I've seen many ways that programmers, including myself, can paint themselves into a corner. You have to admit, we can be pretty good at getting ourselves wedged between a rock and a hard place and I wouldn't be surprised if you found a mess of other ways besides the ones that I'll put under a microscope here. (Quick, how many idiomatic expressions did I manage to squeeze into this paragraph?)

I started out writing this post thinking that I could cover several scenarios but it quickly became clear that this first one that I'm going to bring up has so many facets to it that it deserves to be picked apart in isolation. Buckle up because this is going to be a doozy!

Recognizing that we don't know what we don't know

If you're one of those who I predicted would struggle with TDD and the simple calculator problem, then you probably didn't do so well on your first few attempts, even if you think you did.

You might think it a bit presumptuous of me to say that or maybe even arrogant but if you think about it for a minute, it's really a natural consequence of just starting out with TDD. Without a frame of reference, how can you know if what you just did was really what TDD is supposed to be like? If you're relatively new to TDD, then you won't really know that you've done it poorly until you do it better. As Donald Rumsfeld famously put it, "There are things we don't know we don't know."

This is kind of what the phrase "Aikido works, your Aikido doesn't" is getting at. It's what we sometimes tell people who think Aikido is ineffective. Similarly, people who say the same about Agile are sometimes told that "Agile works, your Agile doesn't." Or with TDD, "TDD works, your TDD doesn't." I honestly don't say this often because it tends to rub people the wrong way and often the people you want to say it to the most aren't really worth the aggravation. Nevertheless, there is a ring of truth to it that can't be denied and maybe that's why they get mad, because now they know it's them.

When I was just starting to learn Aikido, I got better by pairing up with black belts because they would usually point out things that I could improve, things that I would have never recognized otherwise. Now that I'm a black belt myself, I try to pay it forward by helping new students recognize ways they can improve their practice.

Remember, practice only makes habit. Only perfect practice makes perfect. Unless we start recognizing aspects of our practice that are flawed or could be strengthened, we're not going to be able to make it better, much less perfect. So, here are some things that I first focused on recognizing and making better in my TDD practice. Hopefully, these will help you see how you can get better, too.

Getting Lost in the Trees

You've probably heard the expression "seeing the forest for the trees". This first scenario that we will examine is about losing sight of the forest and getting lost in the trees. There are many dimensions to this but it happens quite often and it mainly comes from not understanding the principles that Michael Feathers and Steve Freeman alluded to in their presentation "Test-Driven Development: Ten Years Later." It is rooted in at least two thought processes that I think most developers find difficult to avoid. These are boxes that most of us find hard to think out of.

(Grammar police wannabes be like, "Really, ending a sentence with a preposition?" and I'd be like, "Chill out, dude, don't get all Carol with me. It's fine." Carol is hilarious though. If you're not quite following me, I'm talking about the TV comedy show, "The Last Man on Earth")

First, let's take a look at some test code that's typical of when developers are getting themselves "lost in the trees."

// CalculatorTest.java

public class CalculatorTest {

   private static final double TOLERANCE = 0.0001;
 
   private Calculator calc;
 
   @Before
   public void setUp() {
      calc = new Calculator();
   }

   @Test
   public void testAddition() {
      assertEquals(3.0, calc.evaluate("1 + 2"), TOLERANCE);
   }

   @Ignore
   @Test
   public void testMultiplication() {
      assertEquals(6.0, calc.evaluate("2 * 3"), TOLERANCE);
   }

   @Ignore
   @Test
   public void testAdditionAndMultiplication() {
      assertEquals(7.0, calc.evaluate("1 + 2 * 3"), TOLERANCE);
   }
 
   @Ignore
   @Test
   public void testWithParentheses() {
      assertEquals(9.0, calc.evaluate("(1 + 2) * 3"), TOLERANCE);
   }
}

// Calculator.java

public class Calculator {
   
   public double evaluate(String expression) {
       return 0.0;
   }
}

If you're thinking that this code doesn't look all that bad, you should probably pay close attention to the rest of this post. You might learn a few things you don't know you don't know. And no, that's not a typo; see the Donald Rumsfeld quote above.

If you're thinking that this is way too much code to start with when you're doing TDD, it's great that you recognize that. You're right, if this is the code that you have after just one or two TDD cycles, then you're probably writing too much up front. But just humor me for a bit. Let's just say for now that this is code you might have after a few rounds of following the Red-Green-Refactor flow. Then we'll step through how we might get here.

They say that imitation is the greatest form of flattery so I'm going to fashion this after Uncle Bob's archetypical XP Episode. Since I already mentioned Carol, let's get Phil in on this and pretend that they are pairing up on the calculator TDD exercise. Suspend your disbelief as necessary.

(Phil, of course, is the title character in "The Last Man on Earth")

Phil: Hey, Carol, I started that calculator TDD exercise like you asked me to. Here's the code so far. I even have my first failing test. See?

Carol: Really, Phil? That looks like a lot of code. Did you really get that through proper TDD or did you just write it all at once because it's what you thought you'd need anyway?

Phil: (sheepishly) I wrote it all at once because it's what I thought I'd need anyway...

Carol: That's Ok, Phil, I understand. You just wanted to save some time and cut to the chase. But it's important to understand that the way we get there is just as important as what we get in the end when we do TDD. The means is just as important as the end, Phil. Remember that.

Phil: I don't know what we're getting out of doing all those little steps though, Carol. It seems like a waste of time and it feels silly to do the wrong things when the right things are so obvious.

Carol: Oh, like how you knew that NOT leaving Todd in the desert to die so you could have more booty for yourself was obviously the right thing to do?

Phil: Aw jeez, Carol, that was a low blow, even for you. I already said I was sorry! And I went back for him, didn't I?! Am I ever going to live that mistake down? That was one small lapse of judgement in a moment of weakness!

Carol: Relax, Tandy. That's all behind us now but you should remember the lessons from the past. Doing TDD reminds us that you often have to see the wrong thing before you realize what the right thing to do is. Tell you what, why don't we write that calculator again and this time, let's go through the TDD cycle and thought process, step by step.

Phil: (grudgingly) Ok. Lucky for us, I'm using Git and I just happened to make that default JUnit test my initial commit. Let me revert to that really quick.

// CalculatorTest.java

public class CalculatorTest {

   @Test
   public void test() {
      fail("Not yet implemented");
   }

}

(PAUSE)

Let's hit the pause button for just a minute and reflect on what just transpired. You might be thinking that this conversation is a bit contrived (Well, duh. Unlike Uncle Bob and Bob Koss, Phil and Carol are not real people) but it's based on observations and questions from my own experience with doing TDD and teaching it to other developers.

Reflection #1: Phil went ahead and just wrote all the test code that he thought he was going to need. His intention was to save time. He also felt that the small intermediate steps were a waste of time and that it was silly because he was writing code that was obviously wrong.

The thing to recognize here is that writing correct code and avoiding or eliminating incorrect code is part of the complex algorithm for writing a program that you've followed for years. Your brain has most likely also been trained to be averse to rework, based on the belief that rework is wasteful and costly and that you can avoid rework by doing things right the first time.

Does that strike a chord with you? Do you not hold these things in high regard, if not dearly, as a software developer?

These ideas form part of your understanding of what it takes to write good software. It's the same kind of understanding that you have about how to ride a bicycle. It is the kind of understanding that is so entrenched in your brain that it has become almost a second nature to you. It's that rigid idea that Destin said is stuck in your head and you're finding it hard to change it, even if you want to.

Destin said that "Knowledge is not understanding." In other words, understanding runs deeper than knowledge. Much deeper. That's why it's so hard to change. Even if you have the knowledge about how to write programs through TDD, it's going to take a lot of effort to supplant your brain's current understanding of how to write programs with a new understanding that comes from all that TDD knowledge.

Does that make sense?

Reflection #2: TDD is as much about the way you get there as it is about what you get in the end.

Very skilled programmers can probably produce software with the same level of quality and testing that you could get by doing TDD properly. The problem is, I don't think I've ever met one these mythical super rock stars of programming in real life, even after decades in this profession. I've worked with many companies and met many developers and from what I can tell, the average developer just doesn't write very good programs and they don't write very good tests, sometimes even when they claim to be doing TDD. Color me cynical but I'm just stating my opinion based on experience.

If you recall what Michael Feathers and Steve Freeman said in their presentation, TDD is much more than just testing. Each of the steps you take in the TDD cycle has a very distinct purpose and focus. And that's why the path to get to the end product is very important. More about this later.

Reflection #3: TDD reminds us that we often need to see what's wrong before we can recognize what's right. You've probably heard or said this before: "I'll know it when I see it." We say this because it's probably the case that

1. You've never really seen "It" before
2. "It" is just a vague notion or an immature or incomplete idea
3. You can't completely describe or wrap your head around what "It" is
4. There are a few things "It" could be; there are multiple possibilities for "It"
5. You have to actually try some of the possibilities for "It" first before you can say what "It" is

Take a minute to reflect on the above again before we hit the play button to see what happens next with Phil and Carol.

(PLAY)

Phil: Ok, first step is to write some test code and see it fail. I'm going to need a Calculator class. Agreed?

Carol: Sure, Phil, I agree.

Phil: (sigh) This is going to be a long day. (makes changes to the code)

// CalculatorTest.java

public class CalculatorTest {

   @Test
   public void test() {
      Calculator calc = new Calculator();
   }
}

Phil: Ok, I'm instantiating Calculator but this doesn't compile because we don't have a Calculator class yet. That still qualifies as Red because according to Uncle Bob's Three Rules of TDD, code that doesn't compile is still a failure.

Carol: Yup, you got it. Go ahead and fix it then.

Phil: (uses keyboard shortcuts) There. Now the test compiles.

Carol: Oooh, now you're just showing off, Phil. But that's cool. There's nothing more annoying than mousing around for every little thing you need to do. The keyboard shortcuts are much faster. Ok, run the test then.

Phil: What do you mean? There's nothing in the test to run.

Carol: Oh, Ok. Well, I suppose we should write another line of test code then.

Phil: See, this is getting pretty silly now. Why do we have to do this when we can easily just write everything out and then fix the problems all at once? This is a really tedious way to write a program, isn't it?

Carol: Believe me, Phil, I've been there. It really does feel tedious and downright stupid at first but once we get a hang of the flow and a better idea of where we're going, we can start trying to write bigger chunks of code. Right now, we're writing tiny chunks of code to force ourselves to take things slow. Really slow.

Phil: Why do we need to go so slow?

Carol: Because we want to fail fast and fail small.

Phil: Wait, what? I thought we were going slow. Now we want to go fast? WTH, Carol! Make up your mind! Are we going slow or fast?

Carol: Silly Care Bear. I know that sounds contradictory but by "going slow," I mean that we write smaller increments of new code. Smaller increments means that it'll take us longer to get all the code that we eventually want to get. That's because we're failing fast and failing small, which means that we run our tests every time we add a small increment of code. This way, we'll know right away if we messed anything up. And if we do mess anything up, we know it's probably the last small increment of code that messed it up. Bottom line is that we go slow by adding only small bits of code, fail fast by always running the tests and getting feedback right away, and fail small because we can only mess up a little bit with each increment of code.

Carol: It's kind of like driving in the dark, right? If you go fifty miles before checking if you're still on the right road, then you can drive faster but there's a bigger chance that you can go miles out of your way and get really lost. But if you stop every five miles to check with someone at a store or gas station if you're still on track, then it might take you longer to get wherever you're going, but at least you won't get too far off track at any point. Does that make sense?

Phil: That driving thing makes sense because that's kind of what I did when I took scenic routes from Billings, Montana back home to Tucson. I think I follow your twisted logic though, Carol. You still might have to explain it to me again a few times. I'm not sure I really understanding it all.

Carol: That's Ok, Phil. At least now you know. Understanding will come with practice. You just gotta keep doing it for a while for it to really sink in and make sense. And when it does, it does. It's kind of like those 3D pictures that you have to stare at cross-eyed, you know? It's hard to see the 3D at first but once you get it, you get it and it's a lot easier to see the 3D in other pictures afterwards.

Phil: Huh, I guess I never really looked at it like that before. Yeah, it took me a while to get those 3D pictures. I always saw them as just random patterns until one day, I just suddenly saw the 3D in one of them. Ok, I'll try to see this through some more. Now I'm curious to see if this picture of TDD that you have is as great as you say it is.

(PAUSE)

I'm going to hit the pause button again here for some more reflection.

Reflection #4: Adding small increments of code may seem silly and tedious but it allows you to go slow and keep checking your progress more often. I think Carol explained it quite well.

Reflection #5: Fail Fast and Fail Small. Run your tests every time you add an increment of code so you get feedback quickly and often. The smaller your code increments, the smaller the problems are that they might introduce. The smaller code also makes it easier to revert to a previous state where all the tests were passing.

Reflection #6: Knowledge is NOT Understanding. Understanding will come later, when the knowledge you have is exercised in a context. The more contexts there are in which you apply that knowledge, the more it gets ingrained in your brain as Understanding.

Reflection #7: The tests provide different contexts in which you can exercise your knowledge of what your program is doing. The more you test your program, the more you understand what it's doing. See Reflection #6.

Reflection #8: The algorithm for writing a program with TDD is very complex and complicated and belies the simplicity of the Red-Green-Refactor mantra. As you can see from the number of things we can reflect upon from just this short "conversation," there are many things to think about when you're doing TDD.

You may be asking where those "trees" are that we supposedly can get lost in. If you don't see them, then remember what I said before about not having enough experience to recognize what you don't know? The trees are there but maybe you just didn't recognize them for what they are.

I know you're probably disappointed to not see more code than what I've shown but if you have a little bit more patience, it will be rewarded. (Remember what I said in the first post about needing some virtues?)

We'll continue our eavesdropping on Phil and Carol's TDD session next time. They are going to be writing a lot more now that they have a bit of a shared understanding of TDD.

In the meantime, I encourage you to reflect on all this some more. As you do that TDD exercise again, keep in mind whatever new knowledge you've managed to pick up here. This is how you develop recognition, which in turn opens up possibilities for improvement and getting closer to perfection in your practice of TDD.

Next: TDD and Getting Lost in the Trees (Part 2)

TDD: First Time Crash and Burn

This is the fourth in a series of posts about Test-Driven Development (TDD). Previously, I posted these:

• Why is TDD so Hard?
• TDD and the Backwards Brain
• TDD and that First Awkward Moment

Eliminating the Self-Centered Ego 

One of the things that makes Aikido different from many other popular martial arts is that there are no competitions in Aikido, unlike say, in Tae Kwon Do or Karate or Wu Shu.

The following are excerpts from the book "Advanced Aikido" by Phong Thong Dang and Lynn Seiser:

O'Sensei Morihei Ueshiba... discouraged competition. He felt that competition... diminished Aikido's application as a legitimately applied martial art. He also believed that competition and the striving for championship status would only further encourage, develop, and inflate the self-centered ego. The learned identity ego, in Eastern thought, is one of the major barriers to sprituality... He felt... that competition would create ego blocks to spirituality.

"Aikido is for the development of the entire human being," states Doshu Moriteru Ueshiba... The mental discipline of Aikido is to overcome our own ego for spiritual development, as opposed to employing Aikido in the service of the learned ego identity, to overcome someone else, thus strengthening what Aikido is trying to minimize.

No Contest

You may be wondering what any of this has to do with TDD. Well, in my previous post, I posited that people who have never written a program with TDD would not be able to write a simple calculator program with TDD. I put out the challenge and gave some requirements for how the calculator should work.

And then came the beat down: "You can't do it! You might think you can, but you can't!"

This is what Destin, the engineer who learned how to ride the backwards bicycle, told viewers of his short and hugely popular video on YouTube. He even offers to pay $200 to anyone who can ride the bicycle for ten feet. I don't think anyone has cashed in on that offer yet. Quite frankly, I think it's a sucker bet and Destin knows it.

I'm not going to put any money on the table for this but I'm betting that at the very least, half the people who take up the challenge of writing a program through TDD for the first time will falter after a few minutes and quickly fall back to their old ways of writing programs, much like those who take up Destin's challenge on the backwards bicycle.

The Agony of Defeat 

As Destin said: You wouldn't think it, but the algorithm for riding a bicycle is just that complicated and any change to one thing throws the whole control system off balance. The algorithm for writing a simple program is pretty complicated, too, maybe even more complicated than that for riding a bike.

"But what does that stuff about Aikido have to do with any of this?" you might ask again.

It all goes back to what I said before about being afraid to screw up. It's the ego, man. It's about not wanting to look stupid. It's about wanting to be a winner.  It's about proving Destin and me wrong about you not being able to do it.

"Ok, whatever," you might say with a roll of your eyes.

But am I wrong? Were you really able to write a little bit of test code, then some production code, then refactor, then some more test code, yada, yada, yada?  Were you able to get through the whole exercise without, at some point, stopping and racking your brain as to what test you should write next? Did you not, at some point, stop and look at the mess you'd made and say, "What the (bleep)?!!! This is horrible! This is frustrating! How can I possibly not get this right?!"

I know this is starting to sound like a lawyer badgering the witness but admit it, it's not as easy as it seems, is it? Admit that you started all over at least one or two times, maybe even more. C'mon, if I can be man enough to say that it also happened to me, which it did, you can at least admit that it didn't go swimmingly well for you either.

I hate to say "I told you so," but I told you so. The truth hurts, doesn't it?

Keep Calm and TDD On

Ok, so if you can admit to crashing and burning on this first outing with TDD, then there's still a glimmer of hope for you. Here's why: I couldn't do it at first but after a while, I finally could. After about two years actually.

As self-deprecating, down-to-earth, and approachable as Kent Beck and most of his contemporaries in the field can be, we all know that the apparent ease and dexterity with which these tests and code can be wrangled is belied by the fact that they are the gurus, gods, and demigods of TDD. It's easy for them to talk about and demonstrate how powerful and effective TDD is as a technique because they practically and quite literally invented all this stuff.

It's kind of like how my fellow students of Aikido around the world and I will watch videos on the Internet (isn't technology wonderful?) of all the Aikido masters and even of the founder himself from way back in the 1960s, and wonder in awe at how easy and effortless they make all these techniques appear to be. And then we try to do them and find out that it's not easy at all.

I, on the other hand, hold about as much candle to any of those guys as a baby to her octogenarian grandmother. I'm one of you, just another Joe Schmoe programmer trying to learn how to get a little better at this job. If I can pick myself up, dust myself off, and power through the pain and frustration of failing at TDD, I think most of you can, too.

So relax, take a deep breath, and try it again. It gets better, I promise. You just gotta keep keepin' on.

Eliminating Competition

So now we know that most of us go into TDD with a good measure of knowledge about it but with little to no understanding that we have what Destin calls "a very rigid way of thinking" which we cannot easily change even if we wanted to.

Were you not at some level fighting with this new paradigm? Were you not struggling against the urge to just write production code first, even for just a little bit? Once you entered that ring with TDD, you quickly realize that despite having all the weapons you thought you needed, you were still ill-prepared for this battle with your mind.

Did you really think it would be that easy to find the magical switch that controls the flow of your thought processes, creating big up front designs and writing production code before the tests? The bias from your past experiences guards that switch very jealously, trying to keep you from flipping it. And when, on occasion, you are able to flip it and alter the flow of your thought processes, your bias continues to fight to flip it back to its "normal" position.

Even if you don't buy all that or want to dismiss it as just more hokey martial arts, Eastern philosophy baloney, that's fine. All I'm saying is that this is how I retrospected about the start of my journey in learning TDD. Retrospecting about it helped me re-align my outlook and attitude towards TDD so that I could be more open minded and accepting of what was happening and what TDD wanted me to do.

Call it mind games, or neurolinguistic programming, or whatever. The biggest obstacle to you being able to go with the flow of TDD is your own bias from past experience and you need to eliminate at least some of that bias to keep going, to get unstuck. That's the competition you have to deal with.

Where's the Code, Dude?

All right, I promised I'd give some demonstrations on how I do TDD and all I've given so far are a bunch of philosophical essays. I get it. You're a developer and you want to see some code.

In the next post, I promise I'll finally get to it. Sorry, but it's the weekend now and the Buckeyes are playing Maryland, with kickoff in less than twelve hours. My daughter is finally going to go for her road test and my wife is bugging me to finish putting up curtains on the new windows and a shower head in our newly-renovated master bathroom.

Next week, I promise, you'll see a lot more code.

Next: TDD and Getting Lost in the Trees (Part 1)

TDD and that First Awkward Moment

This is the third in a series of posts about Test-Driven Development (TDD). These are what I posted before:

• Why is TDD so Hard?
• TDD and the Backwards Brain

These posts were partly inspired by a question I answered at coderanch.com about how to develop a simple calculator program with TDD.  It turned out to be quite a long discussion and you can head over to the Ranch and read through all 200+ responses there, if you have that kind of time to kill. Or you can stick around here as I review and summarize some of the things I mentioned in that thread.

Ok, I created a new test... now what?

So you've decided to do TDD and you're all excited and eager to see and experience just what everyone has been talking about. You've created a new JUnit test class, which typically looks something like this:

public class MyNewTest {

    @Test
    public void test() {
        fail("Not yet implemented");
    }
}

Now comes that first of many awkward moments people new to TDD are most likely to have. It's just like being on a first date and not knowing exactly how to break the ice, so you sit there uncomfortably, hoping and praying for a flash of inspiration for a witty remark. 

You say to yourself or your programming partner(s), "Now what?"

Why do we hesitate? What are we afraid of?

Think about that for a minute.

Here's my best armchair psychologist's analysis: We get stuck because we're afraid of being wrong. We're afraid that we'll mess up, that we'll look stupid, or that we'll feel stupid. 

My wife would probably say it's a typical "guy thing," knowing that we're hopelessly lost but not wanting to ask directions for fear of being called "a dumb schmuck" or looking like one.

Ok, I'll admit that I think the invention of portable GPS navigation systems and mobile apps has probably saved more marriages than AAA TripTiks, Dr. Phil, and Couples Counseling combined ever will. 

I see the same thing with beginners in the dojo. They'll start the technique, then stop midstride and ask for validation. Then they'll want to start all over. We always tell them to just keep going and do what they think they saw sensei demonstrate just a few minutes ago.

The Challenge

As pointed out in my previous post, knowing is not understanding. You may be nodding your head in agreement to all of the above but to really understand, you need to get on that backwards bicycle and feel it for yourself! You need to step on the mat and practice!

Let's try to create a simple calculator through TDD. Now, if you've never written a program using TDD (or maybe even if you think you have), then I'm sorry to tell you this but "You can't do it! You may think you can, but you can't!" 

Destin's words, not mine. But yeah, he's probably right.

At this point, hopefully you're saying "Ok, homes, challenge accepted!" as you're firing up your favorite IDE and deftly clicking around various menus with your mouse, or better yet, quickly typing in the keyboard shortcuts, to create a new project.

What, are you still just reading? Go ahead and fire up your IDE, dude!

Enter the TDD dojo

I expect that by now, you have your brand spanking new project laid out in front of you. Hopefully, you've navigated to the /src/test/java folder or wherever you plan to keep your unit test classes. If you're a real go-git-'er-done type, then you probably have already created a new JUnit test called CalculatorTest.  I am, of course, assuming that you're programming in Java and using JUnit. If you're not, feel free to go ahead and make the necessary mental adjustments.

Now what?

What's that? You need requirements? Ok, here they are:

Write a simple calculator program that can handle the four basic arithmetic operations of addition, subtraction, multiplication, and division. The calculator should adhere to the order of precedence rules associated with the mnemonic "Please Excuse My Dear Aunt Sally," except this calculator need not support exponents for now. Your calculator should honor the order of precedence dictated by parentheses.

The calculator should be able to handle decimal calculations and a result that is a rational number supported by your platform, presumably a typical laptop or desktop computer. Any underflow or overflow should be reported as an error. (Note: if you're trying to do this on a mobile device or a mainframe, I salute you, but maybe ease up a little and find a PC or Mac for this, you weirdo)

For example, given the expression 1 + 2 * 3, the calculator should return 7 as the result rather than 9.  However, given the expression (1 + 2) * 3, the calculator should return 9 as the result.

Also, the calculator should be able to display intermediate results as they become available throughout the process of evaluating an expression. 

For example, while evaluating the expression 3 * 4 + 5, the calculator should be able to display the intermediate result of 12 when the "+" operator is reached.  And while evaluating the expression (1 + 2) * 5, the intermediate result of 3 (the sum of 1 and 2) should be available for display when the closed parenthesis is reached.

That's all I have for now.

I'm going to end this post here and give you a chance to go and prove Destin and me wrong. (And no, you grammar police wanna-be, the correct form is "Destin and me," not "Destin and I."  Look it up for yourself if you doubt it.)

In the next article, I will go over what typically happens with TDD neophytes and cargo culters when they try to do this.

Hajime!

Next: TDD: First Time Crash and Burn

TDD and the Backwards Brain

This is a followup to the first article I posted about TDD and why it's so hard to learn and sustain.

In that first article, I proposed that TDD is hard because it requires Mastery, which takes a lot of time, and requires virtue and lots of practice.  Some might read that and think it's just a lot of philosophical mumbo-jumbo that doesn't really help anyone get closer to being able to understand and learn TDD, or get better at it, or keep doing it.

Some might think that the "Mastery" thing was just a lead up to a mention of Dan Pink's talk about "Drive," and the whole Autonomy, Mastery, and Purpose thing. Well, there I said it. Yeah, there's going to be a bit of that, too. But that's for later on.

Right now, I want to extend an olive branch to the practical-minded folks out there who didn't really know what to make of the whole Mastery thing from the last article. I want to give you something more scientific, more fact-based, or at least empirical. I present, for your enjoyment and edification, the "Backwards Brain Bicycle!"

Go ahead and watch that video first. It's only about seven minutes long.

Interesting, isn't it? Do you think you could ride it? Do you think you could handle that kind of frustration and perhaps even, humiliation?

Let's go over some of the points that were made by Destin (that's the guy's name, in case you didn't catch it. Yeah, just like that one time, in Florida, where you went for Spring break...)

Point #1: Knowledge IS NOT Understanding 

Just because you have knowledge about something doesn't mean that you understand it. To fully understand the Backwards Brain Bicycle, you have to actually get on it and learn to ride it. A lot of people tried it but were unsuccessful.

It's the exact same thing with TDD!

Just because you know about Red-Green-Refactor, that you're supposed to write tests first, follow Uncle Bob's Three Rules and Kent Beck's Four Rules of Simple Design, know about SOLID, DRY, SLAP, GRASP, KISS, ... (Wait, what?! Are we still talking about software development here? Because it was starting to sound like we might need a "safe word"... ¯\_(ツ)_/¯ )

Ok, so maybe you didn't realize that there was so much to know besides that catchy TDD mantra. Well, that's kind of what I was getting at with the Mastery thing, too. But I digress.

To fully understand TDD, you have to experience it yourself. You have to fire up your IDE, create a new test, and somehow get past that awkward moment when you're just sitting there staring at the monitor and realize that you have not the slightest clue of what test you should write first. And you sit there valiantly fighting the urge to navigate out of the /src/test folder and go back to the /src/main folder where you're much more comfortable whipping out code.

You have to spend the time to rewire your brain somehow and break it of its long-time habit of writing production code first, and thinking that tests should be written afterwards. You now have to make it follow this weird, backwards way of writing programs. And that process of retraining your brain takes time. And it takes a lot of patience, discipline, humility, perseverance, practice, practice, practice, ... hmmm, does that sound familiar?

Point #2: The algorithm is just that complicated!

Destin says, "Think about it (the algorithm for riding a bicycle): downwards force on the pedals, leaning your whole body, pulling and pushing the handle bars, gyroscopic precession in the wheels... every single force is part of this algorithm and if you change any one part, it affects the entire control system."

Again, this is the same situation we have with TDD. As I mentioned above, there are a lot of things you have to think about when you're writing software. For years, your brain has been trained to think in a certain way, and then suddenly, you tell it to do the exact opposite. And that's not even the start of it!  We're not just changing one thing, we're throwing a whole toolbox of monkey wrenches at your brain!

Need I go over what those monkey wrenches are again? Uncle Bob's Three Rules, Kent Beck's Four Rules, SOLID, DRY, ... "Omaha!!!" (that's our safe word).

Which brings me to Destin's most obnoxious point and most ego-crushing realization that you must resign yourself to accept:

Point #3: You can't ride this bicycle! You might think you can, but you can't!

Likewise, very few people can really grok TDD without a struggle, with little effort. You can't just read a book or watch a video about TDD, or go to a five-day coding boot camp or whatever, and think that you're going to go back to work and merrily start whipping out tests and production code, in that order.

Destin claims it took him eight months of trying, spending five minutes every day practicing (that adds up to a lot of practice), wrecking, and putting up with his neighbors' teasing, before he could ride the backwards brain bicycle.

It took me two years of dedicated solo TDD practice, probably spending four extra hours a week on top of the time that I put in at work, before I finally got a hang of TDD. I can't pinpoint the exact moment in time and I don't even know if there was one particular moment when the "switch flipped" and I was off to the TDD races.

Point #4: One day I couldn't, the next day I could!

Unlike Destin's experience with learning to ride the backwards bicycle, I don't think I had that kind of magical moment that I can pinpoint in my TDD practice. I can't say it was a gradual thing either though. I don't know, maybe I was just too caught up in a moment of ruthless refactoring to realize that I was actually doing it well. It was probably more of a "you don't know how much fun you're having until you look back and recall how much fun you had before" kind of thing for me.

At any rate, it doesn't happen overnight but it does seem like all of a sudden the things you used to struggle with, the things you thought were so backwards and against the grain of the way you are used to doing things doesn't seem that way any more. At least not as much.

And so, we come to the final point I'd like to highlight:

Point #5: You're looking at the world with a bias, whether you like it or not.

Remember how it took Destin's son only two weeks to learn to ride the backwards bicycle versus his eight months? Destin speculated that this was because kids have better neuroplasticity and could adapt more quickly to the new control algorithm of the backwards bicycle.

Maybe that's also why younger and less experienced developers seem to be able to pick up on TDD much faster than the ones with more experience. It's not always the case but I've seen it often enough to think that it might be a common thing.

I think this is one of the biggest challenges in being able to grok TDD.  We all carry some kind of bias because it naturally comes with experience. That experience came at a personal cost. We've all worked hard and even made sacrifices to gain the knowledge and skills we have, knowledge and skills that are based on principles we have long stood for and defended as a mark and measure of our professionalism. That's not something that everyone will just willingly toss out the window.

But then again, that's a kind of bias too, isn't it? What if we change our mode of thinking? What if we, in the same spirit of TDD, flip our thinking around instead?

What if, instead of seeing it as a loss of the investments we've made in becoming the professionals we are today, we see it as an opportunity to diversify? Let's take the gains of our past labors and apply them to a new venture, one that can potentially give back huge returns if we're just willing to tough it out for a while, go back to square one, and see this thing through all its growing pains, and power through the suck zone until we finally land in that sweet kick ass zone.

What do you think?

In the next few installments, I'll start ramping up on the TDD code examples but I'll be using them mostly as the basis for further discussion about principles, philosophy, and mindset.  

In the meantime, I'd love to hear from you about your own experience with TDD and how you did, good or bad, successes and failures. All relevant comments are welcome and appreciated.

Next: TDD and that First Awkward Moment

Why is TDD so hard?

A little late is better than never, as they say.

I attended the Agile 2015 conference in Washington, D.C. back in August and was happy to see that there was at least one session about Test-Driven Development (TDD). I was even happier to see that someone was going to talk about sustaining TDD. I have also found that TDD can be very difficult to sustain and I have witnessed this with many of my colleagues at work. I suspect that it's not an uncommon problem across the board and the number of articles that Google turns up on this subject pretty much confirms this.

I had high hopes going into Scott Bain's session on "Sustainable Test-Driven Development" at Agile 2015 and he made some great points that I'll touch upon in subsequent posts. However, I think that Scott only scratched the surface of the problem and I believe his failure to dig deeper into it means that, ironically, his answers are not going to help much in sustaining the effort to promote TDD either.

TDD in Principle, not just Form

I just recently got to watch "Test-Driven Development: Ten Years Later," a presentation by Michael Feathers and Steve Freeman, both stalwarts of TDD, given way back in 2009.  Like I said, better late than never. Anyway, this is a great presentation and if you are an even bigger laggard than I am, you should go and watch it, too.

Despite having said them over six years ago, the things Michael and Steve talk about in the presentation are still very relevant today. And I see that Michael's skin condition still hasn't cleared up. Just kidding, Michael, I realize it's just a nervous habit you have. On the off chance that you really have been suffering from a recurring rash, I suggest a bit of moisturizing lotion or steroid-based itch cream <big grin, wink>. I kiiid, I kiiid. I'm a big fan of Michael's work and I think his WELC book should be on every programmer's desk, within easy reach for quick reference, right beside Steve's GOOS book.

One of the things in Michael's and Steve's presentation really resonated with me and got me thinking about possibly giving a presentation on it. Towards the end, at about the 45 minute mark, Steve says this about TDD:

"If you want to make this work, you have to understand how it works and what the practices are that go with it.  You have to understand the principles behind the practices."

Let me say that again with emphasis, because this is what I find myself saying over and over to the developers who attend my TDD workshop: "You have to understand the principles behind the practices."

I think that Steve is hinting at Cargo Cult TDD here. I always like to bring Aikido into the conversation so I compare it to what beginners experience in the Shu stage of ShuHaRi.  They are simply following the form, without necessarily fully understanding, or even partially understanding, the principles behind the practices.

The other thing that Michael and Steve mention at the end of their presentation that resonated with me was the need for craftsmanship and professionalism. Again, my Aikido training kicks in and tells me that this is not unlike the attitudes needed for shugyo, which is explained very briefly here and in great detail here. The documentary, "Jiro Dreams of Sushi," is really an exploration of the shokunin spirit, which is along the same lines.

The Truth about TDD

The hard truth is that TDD is difficult and I think the main reason people have a hard time learning and sustaining it is quite simple: It requires Mastery.

Just as with anything else that is worth doing, it's worth doing well. And you can't do something well without a healthy measure of Mastery.

Mastery takes Time

In "Jiro Dreams of Sushi," you learn that Jiro's sons spend years under their father's tutelage. Even young apprentices in their shop must spend many months paying their dues and doing other things in the kitchen before they are allowed to just cook the sushi rice. Getting moved up from cooking the sushi rice takes even longer and the standard of quality is higher and more difficult to meet.

In traditional Aikido dojos, uchi-deshi or live-in students have to help with the upkeep and care of the dojo, its surrounding compound, and the dojocho or headmaster, who often lives in the same compound. The uchi-deshi stay with the dojo for extended periods of time, anywhere from months to years, so that they can study intensively with the master. They put in a lot more time than the other students but it pays off for them in the long run.

It took me more than four years to work my way up through the kyu ranks before I was deemed ready to test for my shodan (1st degree black belt) in Aikido.  The black belt doesn't even mean that I'm really good at Aikido. It just means that I'm good enough and I understand enough to start actually learning Aikido. All that time before was spent just getting ready to learn. Again, I consider it time well invested.

It took me four years of dabbling and two years of more intensive solo TDD study to get comfortable with TDD and really grok it. That's a long time to be in the Shu phase of learning but it was well worth it.

Programmers are an impatient lot though. Ain't none of them got time for that! Well, most of them at least. Programmers need to get results in an internet minute and Mastery just doesn't go by that kind of timetable.

Mastery requires Virtue

In particular, Mastery requires the virtues of patience, humility, discipline, dedication, and perseverance. Each of these is challenging by itself, let alone all of them together. But that's what Mastery requires.

The master sushi maker, Jiro, had two sons. They learned about these virtues from their father. Only the eldest son would eventually take over the sushi shop from their father. The younger son would have to go and set up his own shop. He knew this all along and accepted his lot. He patiently studied and learned the trade over many, many years. He persevered for many, many years before he was given the blessing to set out on his own.  The same is true for Aikido instructors who want to branch off from the Hombu dojo, or the main school, and set up their own dojos.

Jiro's sons had no illusions about their ability to equal, much less surpass, their father's ability at making sushi. They knew that in time, they might be able to approach the kind of quality with which their father made sushi but they were humble enough to realize that they probably would never be as accomplished as he. This kind of attitude requires a tremendous amount of humility and self-awareness. It also requires dedication and perseverance to keep learning regardless.

I don't think being able to do TDD well demands quite those levels of virtue but mastery of it certainly requires levels that only a minority of programmers I have met seem to have or are willing to attain.

Mastery requires Perfect Practice

My son's viola teacher liked to say, "Practice makes habit. Only perfect practice makes perfect."

That's also true with sushi making, Aikido, and TDD. In fact, it's universally accepted that the only way to get to Carnegie Hall is through "Practice, Practice, Practice!" And even more practice.

Perfect practice.

Most developers who fail at TDD fail because of imperfect practice. Michael and Steve hint at that in their presentation, at around the 43 minute mark, where they show a table that involves some kind of magic number that has something to do with the spread of complexity within a code base and its relation to automated unit testing in various open source projects. You can search for related work by Keith Braithwaite if you're interested in the details. 

Anyway, Steve says that he feels that the kind of people who wrote the code in the projects that scored well (2.0 or higher) were the early adopters of TDD. He continues by saying that he's seen code bases with tests that are definitely not on the good side of the scale.

Steve's feeling and observation pretty much aligns with my suspicion that over time, the bulk of the later generations of programmers who have tried to do TDD have lost sight of what it means to practice it perfectly. Over time, TDD practice in the wild has become more and more cargo cultish. This happens in Aikido, too, and probably any other kind of practice of an art or craft as it proliferates among those who are further and further removed from the teachings of the original school of thought. I think it's very much like the phenomenon of Semantic Diffusion that Martin Fowler wrote about, only on a much larger scale.

What chance then do we mere mortal programmers have?

The above doesn't even begin to address the various challenges in adopting and sustaining TDD but it gives you an idea of how deep the problem goes, not just technically but philosophically as well.

I don't for one second believe that TDD is for everyone just as I do not think that Aikido is a martial art that everyone can get into or even just appreciate. Most long-time Aikido practitioners I have met have a certain kind of general mindset and attitude and I feel this is the same kind that you need to get into and appreciate TDD. You don't have to be a lifelong student of Aikido to have that mindset and attitude. You could just as well be a football player, a musician, or even a sushi maker. It's not an Aikido thing, it's a shokunin thing.

In subsequent articles, I plan to demonstrate some of the things I do when I practice TDD and delve deeper into the philosophy, principles, and mindset behind them. Hopefully, this will help other developers find a better understanding and some measure of sustainability in their TDD practice. I know it has helped me a lot.

And hopefully, after a few articles I'll have enough of my thoughts and material gathered and organized to be able to share in a presentation at Agile 2016 in Atlanta next year.

Next: TDD and the Backwards Brain

On Belief, Honesty, Integrity, and Predictability

"It ain't what you don't know that gets you into trouble; it's what you know for sure that just ain't so." —Mark Twain

I wish I could have attended the Craft Conference 2015 in Budapest, Hungary last week. Not only does Budapest look gorgeous at this time of the year, judging from all the pictures they posted on the conference website and Twitter feed, but it would have been a great opportunity to make a pilgrimage to the birthplace of one of my favorite things to play with, the Rubik's Cube, which celebrates its 35th anniversary of being introduced to the world on July 26 this year.

As it was, I was busy trying to fix some loose tiles in my bathroom instead. My wife had been asking me to take care of these since the winter but until last week I had procrastinated doing anything about it, reassuring her whenever she reminded me that the material on which the tile was laid was some kind of special water-resistant board designed specifically for use in bathrooms and toilets. It can wait until warmer weather, I told her each time, which was about every two weeks or so.

Well, Mark Twain proved to be right again.

Ever since I read Kent Beck's white book about Extreme Programming, I have believed in the promise of Agile methods. And ever since I read Martin Fowler's book on refactoring at about the same time that I read "XP Explained: Embrace Change," I believed that TDD, or test-first programming as it was first known way back then, with its short feedback loops, ruthless refactoring, and collaborative development and design, was the best way to develop software. I still do, as a matter of fact.

And yet, what's most perplexing to me is that to this day and perhaps into the foreseeable near future, there are those who still don't and won't buy into Agile and/or TDD. I just don't understand this. Even notable figures in the industry like DHH and Jim Coplien, who are undoubtably very influential in shaping the opinions of many, have come out against TDD. I won't rehash all that here. You can follow the "Is TDD Dead?" conversations that Martin Fowler moderated and read the InfoQ article that looks at the controversy about TDD in detail. Listening to and reading the reasons behind the dissenting opinions, I was once again reminded of Mark Twain's epigram and left wondering whether or not something I knew (or did I?) to be true really wasn't.

I mention all this because I have been preparing to give a two-day workshop on TDD for some engineers in my group at work. I have a Prezi where I go though some of the ideas on which I base my practice of TDD and kaizen, citing articles and quotes from "Uncle Bob" Martin and others. I'm really more or less trying to channel Uncle Bob in this workshop, promoting the ideas of craftsmanship and professionalism and bringing back pride in the work that we developers produce.

This brings me back to #craftconf. Marty Cagan's keynote was really thought-provoking and I just want to thank the conference organizers and speakers for making the videos of their talks available to those who couldn't attend first-hand. Marty talks about how most of the companies that he works with have really no clue what agility is about, despite claiming to be Agile. He talks about a number of fatal flaws in the development process and rails against how companies are using roadmaps and backlogs to trick developers into believing that they were doing Agile, even though when you really think about it, it really ain't so.

Dan North added even more doubt to my mind by saying that backlog grooming should be outlawed. This was said almost in passing during his talk, "Beyond Features". Dan North, the guy behind BDD and doing TDD the right way. Luckily, his article, "The Perils of Estimation", sheds some light on his statement and dispels some of the doubt that it cast in my head.

Coming back to my bathroom tile issue, when I took out the loose tiles, I was horrified to see some black scum covering the back of the tiles and the board underneath. To make matters even worse, the board, which appeared to be plain old gypsum drywall, was soaking wet and crumbling to the slightest touch. I started to work off the surrounding tiles and soon I had two gaping holes in my shower, exposing the nastiness that had been festering underneath the shiny tile all this while. What's more, one of the holes opens up to an external wall and the insulation was soaked and had more of that black scum, which is likely some kind of mold or fungi, on it. Now I'm looking at some major repairs and a big chunk of change taken out of my rainy day savings. When it rains, it pours.

Just as my belief that the material under the tile would be protection enough against water damage got me in trouble when it turned out not to be so, it seems that some of the beliefs about Agile and TDD that I hold dearly and steadfastly defend may not be so after all either. At least not in every way nor all of the time.

I'm reminded of a recent conversation I had on JavaRanch and how people new to Agile might feel about the enthusiasm of proponents and, admittedly, often obnoxious way it sometimes comes across. Sometimes the nice shiny exteriors can hide the nastiness that lurks inside the walls. I may be guilty of contributing to this kind of disingenuousness in the past but I know I have recently made a conscious effort to empathize more with skeptics and dissenters.

Reflecting on my own beliefs and attitudes, I realize that I have gone off on my own little "rantifestos"—I will take the liberty of co-opting that word, too, Dan— on the Ranch as well, particularly when it comes to software craftsmanship.

Here are a few more things that I had to reflect upon after watching Marty's and Dan's talks:

I thought that backlogs were a good way to get the users and product owners involved and engaged in defining the work to be done and planning how we would proceed in delivering the most value quickly. I thought that grooming backlogs was a good way to get a common understanding between everyone on the delivery team and the product owner as to exactly what needed to be done and approximately how long it would take to complete the work.

I thought that TDD would help developers write better code. I thought that with TDD, the testers could get along better with developers and that they both could work towards the common goal of ensuring that the software had the level of quality needed to go to production without testers expressly focusing on finding where the developers had messed up.

Turns out that Dan North, Marty Cagan, and even DHH before them, made some pretty strong arguments to make me reconsider what I knew so surely to be true.

Yes, TDD can be taken the wrong way and turned into something that developers use to satisfy the strong desire of some managers to see metrics on productivity, quality, and code coverage. This as opposed to just measuring the lead time to delivering a viable product that solved our customer's problem.

Yes, it does seem that the business types have tricked us into accepting their Gantt charts disguised as product roadmaps and backlogs. And yes, we're back to pulling estimates out of our rear ends again and feeling guilty when we don't "meet" those estimates, or rather, "forecasts" which apparently are, in fact, the "new commitments." These behaviors are detrimental to our honesty and integrity as developers and responsible members of a project team.

And yes, after watching the video of Marty Cagan's keynote, I believe that this insidious push for predictability in our projects is jeopardizing developers' integrity, making them less inclined to be honest, and creating a vicious cycle that works against our belief that Agile, or rather true Agility is the best way that we can quiet the business types' incessant badgering about "when can we get this done?"

So as I get ready to pay my $500 deductible and perhaps whatever is above the limit on my home insurance policy endorsement for mold and fungi damage, I must also prepare to warn the developers who attend my workshop next week about the perils of taking TDD the wrong way. I must also prepare to talk to our senior leadership about the perils of taking roadmaps, backlogs, estimates, and metrics the wrong way.

Lastly and most importantly, I must now gird my loins against those inevitable looks from my ever-knowing-what's-going-on wife —you know which looks I'm talking about— and man up and say "You were right, dear, there really was something going on behind those tiles and I should have taken care of it when you told me to."

Mark Twain may have been right but he obviously wasn't talking about wives.

MongoDB Java Driver Breaks the Principle of Least Astonishment

I have been going through the MongoDB University introductory course for Java Developers  these past few weeks and happy to report that I have learned quite a few things about MongoDB.

The course I'm taking is pretty decent as an intro and it's free so be sure to check it out if you've been wanting to learn a little more about MongoDB. The video lectures are short and sweet, averaging about two to three minutes each, with only a few that go a little longer but no more than ten minutes or so. Each week, there will be a dozen or so such lectures with each lecture usually ending with a quick quiz that doesn't factor in to your final course grade.

Despite its title, the course, designated as M101J, doesn't focus on using MongoDB from Java as much as it does on introducing MongoDB concepts in general. In fact, most of the examples shown in the lectures are done in the MongoDB shell, which takes JavaScript commands, and there are even a few mentions of PyMongo, the MongoDB driver for Python, in the Week 6 lectures.

The session I'm in right now ends on May 5 and all I have left to do before I complete this course and, hopefully, earn a certificate of completion (yay!) is to score at least 65% on the final exam. Actually, I could get less and still qualify for a certificate since I pretty much aced all the homework but I'm not the kind to set the bar low like that.

Anyway, to the point of this post: I discovered that the MongoDB Java Driver has violated the Principle of Least Astonishment in the MongoCollection.insertOne() method.

I came across the surprising behavior in the MongoDB Java Driver as I was answering one of the final exam questions in the M101J course. I am using the most current version of the driver, 3.0.0, which was released a few weeks ago as of this writing (April 2015).

There doesn't appear to be any equivalent of MongoCollection.insertOne() in the Mongo shell, which is fine. That's not a biggie. To add a new document to a collection in the Mongo shell, you would just use the collection.insert()method. Figure 1 below shows an example of how that's done with the fubar collection in the test database.


Figure 1. Inserting a document in Mongo Shell

$ mongo

> use test
switched to db test

> doc = {a:5, b:5, c:5555}
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.find({a:5})
{ "_id" : ObjectId("5542234eb8aa3c1b4dd7b89b"), "a" : 5, "b" : 5, "c" : 5555 }

The Mongo Shell commands are pretty straightforward and the results are very reasonably what you would expect them to be. Notice that after I successfully insert the document and display it again, nothing in the document changes. "Duh," you might say, "Why would it?" Exactly. When I run find() on the fubar collection to retrieve the document I just inserted, the result is a document that has an extra key which I didn't specify in the document that I passed to the insert method. Again, this is not surprising since MongoDB automatically appends an _id key to any document you insert into a collection and assigns it a unique value.

Now here's where the Mongo Shell and the MongoDB Java Driver diverge in behavior. Figure 2 below shows what happens when I use the same doc variable to insert yet another document. The command succeeds, doc is unchanged as before, and when I do a find, I find that I now have two documents which are identical except for their _id. Again, nothing really surprising about this.


Figure 2. Inserting a Document multiple times

// continuing from before...

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.find({a:5})
{ "_id" : ObjectId("5542234eb8aa3c1b4dd7b89b"), "a" : 5, "b" : 5, "c" : 5555 }
{ "_id" : ObjectId("55422366b8aa3c1b4dd7b89c"), "a" : 5, "b" : 5, "c" : 5555 }

The MongoDB Java Driver behavior for inserting documents is quite different though and this is very surprising. Figure 3 shows how you would insert a document into the same collection.


Figure 3. Inserting a Document in Java

public static void main(String[] args) {
   MongoClient c =  new MongoClient();
   MongoDatabase db = c.getDatabase("test");
   MongoCollection<Document> fubar = db.getCollection("fubar");

   Document foo = new Document("a", 5)
           .append("b", 5)
           .append("c", 555);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);
}

The console output from running this Java code looks something like this:

Before: Document{{a=5, b=5, c=555}}
After: Document{{a=5, b=5, c=555, _id=5542351b5f6987eb755b795a}}

This is quite surprising since the driver mutates the Document that you passed in as a parameter to the insertOne method. What's more, the insertOne method does not return anything: it's declared as a void method. This spells trouble for the unwary Java developer who has come to expect otherwise from their experience in the Mongo Shell. There are a couple of consequences of this design flaw, both of which are not good.

First, you can't readily use the same Document instance as a template to insert multiple documents into a collection. That is, you can't just initialize a bunch of keys with common values and set up a loop to just change the keys that vary. Doing this will result in a MongoWriteException being thrown with a message telling you that a duplicate key was found in the _id index. To work around this problem, you need to remove the _id key before each subsequent call to insertOne using the same document instance. This is the second bad consequence of the design flaw.


Figure 4. Avoiding duplicate key error with MongoDB for Java

public static void main(String[] args) {
   MongoClient c =  new MongoClient();
   MongoDatabase db = c.getDatabase("test");
   MongoCollection<Document> fubar = db.getCollection("fubar");

   Document foo = new Document("a", 5)
           .append("b", 5)
           .append("c", 555);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);

   // fubar.insertOne(foo);  <== duplicate _id error!

   foo.remove("_id")
   foo.remove("a");
   foo.append("a", 55);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);
}

The output of this program looks something like this:

Before: Document{{a=5, b=5, c=555}}
After: Document{{a=5, b=5, c=555, _id=5542392f5f6987019de3a75b}}
Before: Document{{b=5, c=555, a=55}}
After: Document{{b=5, c=555, a=55, _id=5542392f5f6987019de3a75c}}

Not nice, MongoDB Java Driver!

In my opinion, it would have been more reasonable for the insertOne method to instead return the Document that was inserted, the one with the _id key in it. The original Document passed in as a parameter should not experience any change at all. This would have been the least astonishing behavior for this method and the behavior most symmetrical with the MongoDB shell behavior for inserting documents.

(Update #1) Upon further investigation, I found that this is the behavior in the Mongo Shell:

Figure 5. Value returned by collection.insert() in Mongo Shell

> newdoc = db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> newdoc
WriteResult({ "nInserted" : 1 })

> typeof newdoc
object

Which means that the insert method in the Mongo Shell doesn't return the inserted document either. No biggie, I suppose that's a reasonable design choice, too. So, I guess if you were to really make the MongoDB Java Driver be aligned with this behavior and adhere to the Principle of Least Astonishment, you wouldn't return the inserted Document as I suggested above but instead return some kind of object, like a WriteResult maybe.

(Update #2) Looking at the PyMongo Driver, the behavior of the insert_one method is to return a pymongo.results.InsertOneResult object so something similar in the Java driver would make more sense.