TDD and the Backwards Brain

This is a followup to the first article I posted about TDD and why it's so hard to learn and sustain.

In that first article, I proposed that TDD is hard because it requires Mastery, which takes a lot of time, and requires virtue and lots of practice.  Some might read that and think it's just a lot of philosophical mumbo-jumbo that doesn't really help anyone get closer to being able to understand and learn TDD, or get better at it, or keep doing it.

Some might think that the "Mastery" thing was just a lead up to a mention of Dan Pink's talk about "Drive," and the whole Autonomy, Mastery, and Purpose thing. Well, there I said it. Yeah, there's going to be a bit of that, too. But that's for later on.

Right now, I want to extend an olive branch to the practical-minded folks out there who didn't really know what to make of the whole Mastery thing from the last article. I want to give you something more scientific, more fact-based, or at least empirical. I present, for your enjoyment and edification, the "Backwards Brain Bicycle!"

Go ahead and watch that video first. It's only about seven minutes long.

Interesting, isn't it? Do you think you could ride it? Do you think you could handle that kind of frustration and perhaps even, humiliation?

Let's go over some of the points that were made by Destin (that's the guy's name, in case you didn't catch it. Yeah, just like that one time, in Florida, where you went for Spring break...)

Point #1: Knowledge IS NOT Understanding 

Just because you have knowledge about something doesn't mean that you understand it. To fully understand the Backwards Brain Bicycle, you have to actually get on it and learn to ride it. A lot of people tried it but were unsuccessful.

It's the exact same thing with TDD!

Just because you know about Red-Green-Refactor, that you're supposed to write tests first, follow Uncle Bob's Three Rules and Kent Beck's Four Rules of Simple Design, know about SOLID, DRY, SLAP, GRASP, KISS, ... (Wait, what?! Are we still talking about software development here? Because it was starting to sound like we might need a "safe word"... ¯\_(ツ)_/¯ )

Ok, so maybe you didn't realize that there was so much to know besides that catchy TDD mantra. Well, that's kind of what I was getting at with the Mastery thing, too. But I digress.

To fully understand TDD, you have to experience it yourself. You have to fire up your IDE, create a new test, and somehow get past that awkward moment when you're just sitting there staring at the monitor and realize that you have not the slightest clue of what test you should write first. And you sit there valiantly fighting the urge to navigate out of the /src/test folder and go back to the /src/main folder where you're much more comfortable whipping out code.

You have to spend the time to rewire your brain somehow and break it of its long-time habit of writing production code first, and thinking that tests should be written afterwards. You now have to make it follow this weird, backwards way of writing programs. And that process of retraining your brain takes time. And it takes a lot of patience, discipline, humility, perseverance, practice, practice, practice, ... hmmm, does that sound familiar?

Point #2: The algorithm is just that complicated!

Destin says, "Think about it (the algorithm for riding a bicycle): downwards force on the pedals, leaning your whole body, pulling and pushing the handle bars, gyroscopic precession in the wheels... every single force is part of this algorithm and if you change any one part, it affects the entire control system."

Again, this is the same situation we have with TDD. As I mentioned above, there are a lot of things you have to think about when you're writing software. For years, your brain has been trained to think in a certain way, and then suddenly, you tell it to do the exact opposite. And that's not even the start of it!  We're not just changing one thing, we're throwing a whole toolbox of monkey wrenches at your brain!

Need I go over what those monkey wrenches are again? Uncle Bob's Three Rules, Kent Beck's Four Rules, SOLID, DRY, ... "Omaha!!!" (that's our safe word).

Which brings me to Destin's most obnoxious point and most ego-crushing realization that you must resign yourself to accept:

Point #3: You can't ride this bicycle! You might think you can, but you can't!

Likewise, very few people can really grok TDD without a struggle, with little effort. You can't just read a book or watch a video about TDD, or go to a five-day coding boot camp or whatever, and think that you're going to go back to work and merrily start whipping out tests and production code, in that order.

Destin claims it took him eight months of trying, spending five minutes every day practicing (that adds up to a lot of practice), wrecking, and putting up with his neighbors' teasing, before he could ride the backwards brain bicycle.

It took me two years of dedicated solo TDD practice, probably spending four extra hours a week on top of the time that I put in at work, before I finally got a hang of TDD. I can't pinpoint the exact moment in time and I don't even know if there was one particular moment when the "switch flipped" and I was off to the TDD races.

Point #4: One day I couldn't, the next day I could!

Unlike Destin's experience with learning to ride the backwards bicycle, I don't think I had that kind of magical moment that I can pinpoint in my TDD practice. I can't say it was a gradual thing either though. I don't know, maybe I was just too caught up in a moment of ruthless refactoring to realize that I was actually doing it well. It was probably more of a "you don't know how much fun you're having until you look back and recall how much fun you had before" kind of thing for me.

At any rate, it doesn't happen overnight but it does seem like all of a sudden the things you used to struggle with, the things you thought were so backwards and against the grain of the way you are used to doing things doesn't seem that way any more. At least not as much.

And so, we come to the final point I'd like to highlight:

Point #5: You're looking at the world with a bias, whether you like it or not.

Remember how it took Destin's son only two weeks to learn to ride the backwards bicycle versus his eight months? Destin speculated that this was because kids have better neuroplasticity and could adapt more quickly to the new control algorithm of the backwards bicycle.

Maybe that's also why younger and less experienced developers seem to be able to pick up on TDD much faster than the ones with more experience. It's not always the case but I've seen it often enough to think that it might be a common thing.

I think this is one of the biggest challenges in being able to grok TDD.  We all carry some kind of bias because it naturally comes with experience. That experience came at a personal cost. We've all worked hard and even made sacrifices to gain the knowledge and skills we have, knowledge and skills that are based on principles we have long stood for and defended as a mark and measure of our professionalism. That's not something that everyone will just willingly toss out the window.

But then again, that's a kind of bias too, isn't it? What if we change our mode of thinking? What if we, in the same spirit of TDD, flip our thinking around instead?

What if, instead of seeing it as a loss of the investments we've made in becoming the professionals we are today, we see it as an opportunity to diversify? Let's take the gains of our past labors and apply them to a new venture, one that can potentially give back huge returns if we're just willing to tough it out for a while, go back to square one, and see this thing through all its growing pains, and power through the suck zone until we finally land in that sweet kick ass zone.

What do you think?

In the next few installments, I'll start ramping up on the TDD code examples but I'll be using them mostly as the basis for further discussion about principles, philosophy, and mindset.  

In the meantime, I'd love to hear from you about your own experience with TDD and how you did, good or bad, successes and failures. All relevant comments are welcome and appreciated.

Next: TDD and that First Awkward Moment

Why is TDD so hard?

A little late is better than never, as they say.

I attended the Agile 2015 conference in Washington, D.C. back in August and was happy to see that there was at least one session about Test-Driven Development (TDD). I was even happier to see that someone was going to talk about sustaining TDD. I have also found that TDD can be very difficult to sustain and I have witnessed this with many of my colleagues at work. I suspect that it's not an uncommon problem across the board and the number of articles that Google turns up on this subject pretty much confirms this.

I had high hopes going into Scott Bain's session on "Sustainable Test-Driven Development" at Agile 2015 and he made some great points that I'll touch upon in subsequent posts. However, I think that Scott only scratched the surface of the problem and I believe his failure to dig deeper into it means that, ironically, his answers are not going to help much in sustaining the effort to promote TDD either.

TDD in Principle, not just Form

I just recently got to watch "Test-Driven Development: Ten Years Later," a presentation by Michael Feathers and Steve Freeman, both stalwarts of TDD, given way back in 2009.  Like I said, better late than never. Anyway, this is a great presentation and if you are an even bigger laggard than I am, you should go and watch it, too.

Despite having said them over six years ago, the things Michael and Steve talk about in the presentation are still very relevant today. And I see that Michael's skin condition still hasn't cleared up. Just kidding, Michael, I realize it's just a nervous habit you have. On the off chance that you really have been suffering from a recurring rash, I suggest a bit of moisturizing lotion or steroid-based itch cream <big grin, wink>. I kiiid, I kiiid. I'm a big fan of Michael's work and I think his WELC book should be on every programmer's desk, within easy reach for quick reference, right beside Steve's GOOS book.

One of the things in Michael's and Steve's presentation really resonated with me and got me thinking about possibly giving a presentation on it. Towards the end, at about the 45 minute mark, Steve says this about TDD:

"If you want to make this work, you have to understand how it works and what the practices are that go with it.  You have to understand the principles behind the practices."

Let me say that again with emphasis, because this is what I find myself saying over and over to the developers who attend my TDD workshop: "You have to understand the principles behind the practices."

I think that Steve is hinting at Cargo Cult TDD here. I always like to bring Aikido into the conversation so I compare it to what beginners experience in the Shu stage of ShuHaRi.  They are simply following the form, without necessarily fully understanding, or even partially understanding, the principles behind the practices.

The other thing that Michael and Steve mention at the end of their presentation that resonated with me was the need for craftsmanship and professionalism. Again, my Aikido training kicks in and tells me that this is not unlike the attitudes needed for shugyo, which is explained very briefly here and in great detail here. The documentary, "Jiro Dreams of Sushi," is really an exploration of the shokunin spirit, which is along the same lines.

The Truth about TDD

The hard truth is that TDD is difficult and I think the main reason people have a hard time learning and sustaining it is quite simple: It requires Mastery.

Just as with anything else that is worth doing, it's worth doing well. And you can't do something well without a healthy measure of Mastery.

Mastery takes Time

In "Jiro Dreams of Sushi," you learn that Jiro's sons spend years under their father's tutelage. Even young apprentices in their shop must spend many months paying their dues and doing other things in the kitchen before they are allowed to just cook the sushi rice. Getting moved up from cooking the sushi rice takes even longer and the standard of quality is higher and more difficult to meet.

In traditional Aikido dojos, uchi-deshi or live-in students have to help with the upkeep and care of the dojo, its surrounding compound, and the dojocho or headmaster, who often lives in the same compound. The uchi-deshi stay with the dojo for extended periods of time, anywhere from months to years, so that they can study intensively with the master. They put in a lot more time than the other students but it pays off for them in the long run.

It took me more than four years to work my way up through the kyu ranks before I was deemed ready to test for my shodan (1st degree black belt) in Aikido.  The black belt doesn't even mean that I'm really good at Aikido. It just means that I'm good enough and I understand enough to start actually learning Aikido. All that time before was spent just getting ready to learn. Again, I consider it time well invested.

It took me four years of dabbling and two years of more intensive solo TDD study to get comfortable with TDD and really grok it. That's a long time to be in the Shu phase of learning but it was well worth it.

Programmers are an impatient lot though. Ain't none of them got time for that! Well, most of them at least. Programmers need to get results in an internet minute and Mastery just doesn't go by that kind of timetable.

Mastery requires Virtue

In particular, Mastery requires the virtues of patience, humility, discipline, dedication, and perseverance. Each of these is challenging by itself, let alone all of them together. But that's what Mastery requires.

The master sushi maker, Jiro, had two sons. They learned about these virtues from their father. Only the eldest son would eventually take over the sushi shop from their father. The younger son would have to go and set up his own shop. He knew this all along and accepted his lot. He patiently studied and learned the trade over many, many years. He persevered for many, many years before he was given the blessing to set out on his own.  The same is true for Aikido instructors who want to branch off from the Hombu dojo, or the main school, and set up their own dojos.

Jiro's sons had no illusions about their ability to equal, much less surpass, their father's ability at making sushi. They knew that in time, they might be able to approach the kind of quality with which their father made sushi but they were humble enough to realize that they probably would never be as accomplished as he. This kind of attitude requires a tremendous amount of humility and self-awareness. It also requires dedication and perseverance to keep learning regardless.

I don't think being able to do TDD well demands quite those levels of virtue but mastery of it certainly requires levels that only a minority of programmers I have met seem to have or are willing to attain.

Mastery requires Perfect Practice

My son's viola teacher liked to say, "Practice makes habit. Only perfect practice makes perfect."

That's also true with sushi making, Aikido, and TDD. In fact, it's universally accepted that the only way to get to Carnegie Hall is through "Practice, Practice, Practice!" And even more practice.

Perfect practice.

Most developers who fail at TDD fail because of imperfect practice. Michael and Steve hint at that in their presentation, at around the 43 minute mark, where they show a table that involves some kind of magic number that has something to do with the spread of complexity within a code base and its relation to automated unit testing in various open source projects. You can search for related work by Keith Braithwaite if you're interested in the details. 

Anyway, Steve says that he feels that the kind of people who wrote the code in the projects that scored well (2.0 or higher) were the early adopters of TDD. He continues by saying that he's seen code bases with tests that are definitely not on the good side of the scale.

Steve's feeling and observation pretty much aligns with my suspicion that over time, the bulk of the later generations of programmers who have tried to do TDD have lost sight of what it means to practice it perfectly. Over time, TDD practice in the wild has become more and more cargo cultish. This happens in Aikido, too, and probably any other kind of practice of an art or craft as it proliferates among those who are further and further removed from the teachings of the original school of thought. I think it's very much like the phenomenon of Semantic Diffusion that Martin Fowler wrote about, only on a much larger scale.

What chance then do we mere mortal programmers have?

The above doesn't even begin to address the various challenges in adopting and sustaining TDD but it gives you an idea of how deep the problem goes, not just technically but philosophically as well.

I don't for one second believe that TDD is for everyone just as I do not think that Aikido is a martial art that everyone can get into or even just appreciate. Most long-time Aikido practitioners I have met have a certain kind of general mindset and attitude and I feel this is the same kind that you need to get into and appreciate TDD. You don't have to be a lifelong student of Aikido to have that mindset and attitude. You could just as well be a football player, a musician, or even a sushi maker. It's not an Aikido thing, it's a shokunin thing.

In subsequent articles, I plan to demonstrate some of the things I do when I practice TDD and delve deeper into the philosophy, principles, and mindset behind them. Hopefully, this will help other developers find a better understanding and some measure of sustainability in their TDD practice. I know it has helped me a lot.

And hopefully, after a few articles I'll have enough of my thoughts and material gathered and organized to be able to share in a presentation at Agile 2016 in Atlanta next year.

Next: TDD and the Backwards Brain

On Belief, Honesty, Integrity, and Predictability

"It ain't what you don't know that gets you into trouble; it's what you know for sure that just ain't so." —Mark Twain

I wish I could have attended the Craft Conference 2015 in Budapest, Hungary last week. Not only does Budapest look gorgeous at this time of the year, judging from all the pictures they posted on the conference website and Twitter feed, but it would have been a great opportunity to make a pilgrimage to the birthplace of one of my favorite things to play with, the Rubik's Cube, which celebrates its 35th anniversary of being introduced to the world on July 26 this year.

As it was, I was busy trying to fix some loose tiles in my bathroom instead. My wife had been asking me to take care of these since the winter but until last week I had procrastinated doing anything about it, reassuring her whenever she reminded me that the material on which the tile was laid was some kind of special water-resistant board designed specifically for use in bathrooms and toilets. It can wait until warmer weather, I told her each time, which was about every two weeks or so.

Well, Mark Twain proved to be right again.

Ever since I read Kent Beck's white book about Extreme Programming, I have believed in the promise of Agile methods. And ever since I read Martin Fowler's book on refactoring at about the same time that I read "XP Explained: Embrace Change," I believed that TDD, or test-first programming as it was first known way back then, with its short feedback loops, ruthless refactoring, and collaborative development and design, was the best way to develop software. I still do, as a matter of fact.

And yet, what's most perplexing to me is that to this day and perhaps into the foreseeable near future, there are those who still don't and won't buy into Agile and/or TDD. I just don't understand this. Even notable figures in the industry like DHH and Jim Coplien, who are undoubtably very influential in shaping the opinions of many, have come out against TDD. I won't rehash all that here. You can follow the "Is TDD Dead?" conversations that Martin Fowler moderated and read the InfoQ article that looks at the controversy about TDD in detail. Listening to and reading the reasons behind the dissenting opinions, I was once again reminded of Mark Twain's epigram and left wondering whether or not something I knew (or did I?) to be true really wasn't.

I mention all this because I have been preparing to give a two-day workshop on TDD for some engineers in my group at work. I have a Prezi where I go though some of the ideas on which I base my practice of TDD and kaizen, citing articles and quotes from "Uncle Bob" Martin and others. I'm really more or less trying to channel Uncle Bob in this workshop, promoting the ideas of craftsmanship and professionalism and bringing back pride in the work that we developers produce.

This brings me back to #craftconf. Marty Cagan's keynote was really thought-provoking and I just want to thank the conference organizers and speakers for making the videos of their talks available to those who couldn't attend first-hand. Marty talks about how most of the companies that he works with have really no clue what agility is about, despite claiming to be Agile. He talks about a number of fatal flaws in the development process and rails against how companies are using roadmaps and backlogs to trick developers into believing that they were doing Agile, even though when you really think about it, it really ain't so.

Dan North added even more doubt to my mind by saying that backlog grooming should be outlawed. This was said almost in passing during his talk, "Beyond Features". Dan North, the guy behind BDD and doing TDD the right way. Luckily, his article, "The Perils of Estimation", sheds some light on his statement and dispels some of the doubt that it cast in my head.

Coming back to my bathroom tile issue, when I took out the loose tiles, I was horrified to see some black scum covering the back of the tiles and the board underneath. To make matters even worse, the board, which appeared to be plain old gypsum drywall, was soaking wet and crumbling to the slightest touch. I started to work off the surrounding tiles and soon I had two gaping holes in my shower, exposing the nastiness that had been festering underneath the shiny tile all this while. What's more, one of the holes opens up to an external wall and the insulation was soaked and had more of that black scum, which is likely some kind of mold or fungi, on it. Now I'm looking at some major repairs and a big chunk of change taken out of my rainy day savings. When it rains, it pours.

Just as my belief that the material under the tile would be protection enough against water damage got me in trouble when it turned out not to be so, it seems that some of the beliefs about Agile and TDD that I hold dearly and steadfastly defend may not be so after all either. At least not in every way nor all of the time.

I'm reminded of a recent conversation I had on JavaRanch and how people new to Agile might feel about the enthusiasm of proponents and, admittedly, often obnoxious way it sometimes comes across. Sometimes the nice shiny exteriors can hide the nastiness that lurks inside the walls. I may be guilty of contributing to this kind of disingenuousness in the past but I know I have recently made a conscious effort to empathize more with skeptics and dissenters.

Reflecting on my own beliefs and attitudes, I realize that I have gone off on my own little "rantifestos"—I will take the liberty of co-opting that word, too, Dan— on the Ranch as well, particularly when it comes to software craftsmanship.

Here are a few more things that I had to reflect upon after watching Marty's and Dan's talks:

I thought that backlogs were a good way to get the users and product owners involved and engaged in defining the work to be done and planning how we would proceed in delivering the most value quickly. I thought that grooming backlogs was a good way to get a common understanding between everyone on the delivery team and the product owner as to exactly what needed to be done and approximately how long it would take to complete the work.

I thought that TDD would help developers write better code. I thought that with TDD, the testers could get along better with developers and that they both could work towards the common goal of ensuring that the software had the level of quality needed to go to production without testers expressly focusing on finding where the developers had messed up.

Turns out that Dan North, Marty Cagan, and even DHH before them, made some pretty strong arguments to make me reconsider what I knew so surely to be true.

Yes, TDD can be taken the wrong way and turned into something that developers use to satisfy the strong desire of some managers to see metrics on productivity, quality, and code coverage. This as opposed to just measuring the lead time to delivering a viable product that solved our customer's problem.

Yes, it does seem that the business types have tricked us into accepting their Gantt charts disguised as product roadmaps and backlogs. And yes, we're back to pulling estimates out of our rear ends again and feeling guilty when we don't "meet" those estimates, or rather, "forecasts" which apparently are, in fact, the "new commitments." These behaviors are detrimental to our honesty and integrity as developers and responsible members of a project team.

And yes, after watching the video of Marty Cagan's keynote, I believe that this insidious push for predictability in our projects is jeopardizing developers' integrity, making them less inclined to be honest, and creating a vicious cycle that works against our belief that Agile, or rather true Agility is the best way that we can quiet the business types' incessant badgering about "when can we get this done?"

So as I get ready to pay my $500 deductible and perhaps whatever is above the limit on my home insurance policy endorsement for mold and fungi damage, I must also prepare to warn the developers who attend my workshop next week about the perils of taking TDD the wrong way. I must also prepare to talk to our senior leadership about the perils of taking roadmaps, backlogs, estimates, and metrics the wrong way.

Lastly and most importantly, I must now gird my loins against those inevitable looks from my ever-knowing-what's-going-on wife —you know which looks I'm talking about— and man up and say "You were right, dear, there really was something going on behind those tiles and I should have taken care of it when you told me to."

Mark Twain may have been right but he obviously wasn't talking about wives.

MongoDB Java Driver Breaks the Principle of Least Astonishment

I have been going through the MongoDB University introductory course for Java Developers  these past few weeks and happy to report that I have learned quite a few things about MongoDB.

The course I'm taking is pretty decent as an intro and it's free so be sure to check it out if you've been wanting to learn a little more about MongoDB. The video lectures are short and sweet, averaging about two to three minutes each, with only a few that go a little longer but no more than ten minutes or so. Each week, there will be a dozen or so such lectures with each lecture usually ending with a quick quiz that doesn't factor in to your final course grade.

Despite its title, the course, designated as M101J, doesn't focus on using MongoDB from Java as much as it does on introducing MongoDB concepts in general. In fact, most of the examples shown in the lectures are done in the MongoDB shell, which takes JavaScript commands, and there are even a few mentions of PyMongo, the MongoDB driver for Python, in the Week 6 lectures.

The session I'm in right now ends on May 5 and all I have left to do before I complete this course and, hopefully, earn a certificate of completion (yay!) is to score at least 65% on the final exam. Actually, I could get less and still qualify for a certificate since I pretty much aced all the homework but I'm not the kind to set the bar low like that.

Anyway, to the point of this post: I discovered that the MongoDB Java Driver has violated the Principle of Least Astonishment in the MongoCollection.insertOne() method.

I came across the surprising behavior in the MongoDB Java Driver as I was answering one of the final exam questions in the M101J course. I am using the most current version of the driver, 3.0.0, which was released a few weeks ago as of this writing (April 2015).

There doesn't appear to be any equivalent of MongoCollection.insertOne() in the Mongo shell, which is fine. That's not a biggie. To add a new document to a collection in the Mongo shell, you would just use the collection.insert()method. Figure 1 below shows an example of how that's done with the fubar collection in the test database.


Figure 1. Inserting a document in Mongo Shell

$ mongo

> use test
switched to db test

> doc = {a:5, b:5, c:5555}
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.find({a:5})
{ "_id" : ObjectId("5542234eb8aa3c1b4dd7b89b"), "a" : 5, "b" : 5, "c" : 5555 }

The Mongo Shell commands are pretty straightforward and the results are very reasonably what you would expect them to be. Notice that after I successfully insert the document and display it again, nothing in the document changes. "Duh," you might say, "Why would it?" Exactly. When I run find() on the fubar collection to retrieve the document I just inserted, the result is a document that has an extra key which I didn't specify in the document that I passed to the insert method. Again, this is not surprising since MongoDB automatically appends an _id key to any document you insert into a collection and assigns it a unique value.

Now here's where the Mongo Shell and the MongoDB Java Driver diverge in behavior. Figure 2 below shows what happens when I use the same doc variable to insert yet another document. The command succeeds, doc is unchanged as before, and when I do a find, I find that I now have two documents which are identical except for their _id. Again, nothing really surprising about this.


Figure 2. Inserting a Document multiple times

// continuing from before...

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> doc
{ "a" : 5, "b" : 5, "c" : 5555 }

> db.fubar.find({a:5})
{ "_id" : ObjectId("5542234eb8aa3c1b4dd7b89b"), "a" : 5, "b" : 5, "c" : 5555 }
{ "_id" : ObjectId("55422366b8aa3c1b4dd7b89c"), "a" : 5, "b" : 5, "c" : 5555 }

The MongoDB Java Driver behavior for inserting documents is quite different though and this is very surprising. Figure 3 shows how you would insert a document into the same collection.


Figure 3. Inserting a Document in Java

public static void main(String[] args) {
   MongoClient c =  new MongoClient();
   MongoDatabase db = c.getDatabase("test");
   MongoCollection<Document> fubar = db.getCollection("fubar");

   Document foo = new Document("a", 5)
           .append("b", 5)
           .append("c", 555);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);
}

The console output from running this Java code looks something like this:

Before: Document{{a=5, b=5, c=555}}
After: Document{{a=5, b=5, c=555, _id=5542351b5f6987eb755b795a}}

This is quite surprising since the driver mutates the Document that you passed in as a parameter to the insertOne method. What's more, the insertOne method does not return anything: it's declared as a void method. This spells trouble for the unwary Java developer who has come to expect otherwise from their experience in the Mongo Shell. There are a couple of consequences of this design flaw, both of which are not good.

First, you can't readily use the same Document instance as a template to insert multiple documents into a collection. That is, you can't just initialize a bunch of keys with common values and set up a loop to just change the keys that vary. Doing this will result in a MongoWriteException being thrown with a message telling you that a duplicate key was found in the _id index. To work around this problem, you need to remove the _id key before each subsequent call to insertOne using the same document instance. This is the second bad consequence of the design flaw.


Figure 4. Avoiding duplicate key error with MongoDB for Java

public static void main(String[] args) {
   MongoClient c =  new MongoClient();
   MongoDatabase db = c.getDatabase("test");
   MongoCollection<Document> fubar = db.getCollection("fubar");

   Document foo = new Document("a", 5)
           .append("b", 5)
           .append("c", 555);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);

   // fubar.insertOne(foo);  <== duplicate _id error!

   foo.remove("_id")
   foo.remove("a");
   foo.append("a", 55);

   System.out.println("Before: " + foo);
   fubar.insertOne(foo);
   System.out.println("After: " + foo);
}

The output of this program looks something like this:

Before: Document{{a=5, b=5, c=555}}
After: Document{{a=5, b=5, c=555, _id=5542392f5f6987019de3a75b}}
Before: Document{{b=5, c=555, a=55}}
After: Document{{b=5, c=555, a=55, _id=5542392f5f6987019de3a75c}}

Not nice, MongoDB Java Driver!

In my opinion, it would have been more reasonable for the insertOne method to instead return the Document that was inserted, the one with the _id key in it. The original Document passed in as a parameter should not experience any change at all. This would have been the least astonishing behavior for this method and the behavior most symmetrical with the MongoDB shell behavior for inserting documents.

(Update #1) Upon further investigation, I found that this is the behavior in the Mongo Shell:

Figure 5. Value returned by collection.insert() in Mongo Shell

> newdoc = db.fubar.insert(doc)
WriteResult({ "nInserted" : 1 })

> newdoc
WriteResult({ "nInserted" : 1 })

> typeof newdoc
object

Which means that the insert method in the Mongo Shell doesn't return the inserted document either. No biggie, I suppose that's a reasonable design choice, too. So, I guess if you were to really make the MongoDB Java Driver be aligned with this behavior and adhere to the Principle of Least Astonishment, you wouldn't return the inserted Document as I suggested above but instead return some kind of object, like a WriteResult maybe.

(Update #2) Looking at the PyMongo Driver, the behavior of the insert_one method is to return a pymongo.results.InsertOneResult object so something similar in the Java driver would make more sense.

Exploring Java 8: Lambda expressions and default interface methods

The introduction of lambda expressions in Java 8 is arguably the biggest change in Java since generics and annotations were added to the language in Java 5. This week, I got a chance to experiment with lambda expressions a little bit when I helped answer a question posted on JavaRanch.

The topic was about filtering a portion of a list. I have changed it up a little here but the idea is essentially the same.

The challenge is to write a method which, given a list of numbers, would remove all odd numbers that occurred between a starting index (inclusive) and an ending index (exclusive). Elements that are outside the given index range should not be affected.

For example, given a list of eleven numbers, {3, 18, 7, 1, 16, 11, 9, 4, 33, 5, 10}, a starting index of 3, and an ending index of 9, the method would remove the odd numbers in the highlighted range: {3, 18, 7, 1, 16, 11, 9, 4, 33, 5, 10}, producing a new list, {3, 18, 7, 16, 4, 5, 10}.

To appreciate how lambda expressions can dramatically improve Java code, let's start with a solution that doesn't use them.

Revision #1 - The "standard" implementation

    List<Integer> removeOdd(List<Integer> aList,
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();

        for (int i = 0; i < fromIndex; i++) {
            newList.add(aList.get(i));
        }

        for (int i = fromIndex; i < toIndex; i++) {
            Integer e = aList.get(i);
            if (e % 2 == 0) {
                newList.add(e);
            }
        }

        for (int i = toIndex; i < aList.size(); i++) {
            newList.add(aList.get(i));
        }

        return newList;
    }

This solution uses three for-loops to process the head, body, and tail of the list, respectively. The head includes elements with an index less than the starting index, while the tail includes elements with an index greater than or equal to the ending index. It works but it's not exactly the most succinct piece of code you've ever seen.

The trouble with this solution is that it doesn't clearly reveal its intent. Anyone coming into this code without prior knowledge of it will have to spend a bit of time reading through it to grok what's going on. We can improve this with a little bit of refactoring.

The bookend for-loops can be eliminated by using the List.subList() method which returns a view of a portion of a list. This helps make the code that copies elements from the head and tail portions a little bit clearer. Now we are left to contend with only one for-loop.


Revision #2 - Refactored to use List.subList()

    List<Integer> removeOdd(List<Integer> aList, 
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();

        newList.addAll(aList.subList(0, fromIndex));

        for (int i = fromIndex; i < toIndex; i++) {
            Integer e = aList.get(i);
            if (e % 2 == 0) {
                newList.add(e);
            }
        }

        newList.addAll(aList.subList(toIndex; aList.size())

        return newList;
    }

The remaining for-loop still is not quite up to snuff with the Single Level of Abstraction Principle or SLAP for short. To make this a well-composed method, we can extract the detailed code into another method. This results in the following, which is relatively clean as far as Java code goes.


Revision #3 - Refactored to SLAP

    List<Integer> removeOdd(List<Integer> aList, 
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();

        newList.addAll(aList.subList(0, fromIndex));
        newList.addAll(evenNumbers(aList, fromIndex, toIndex));
        newList.addAll(aList.subList(toIndex, aList.size())

        return newList;
    }

    List<Integer> evenNumbers(List<Integer> aList, 
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();
        
        for (int i = fromIndex; i < toIndex; i++) {
            Integer e = aList.get(i);
            if (e % 2 == 0) {
                newList.add(e);
            }
        }

        return newList;
    }
   

The removeOdd method is about as close to having a single level of abstraction as we can get but the evenNumbers helper method is still a bit of an eyesore. The business of having to create a new list and return it is a bit repetitious if you ask me. Refactoring evenNumbers a little bit more makes it somewhat better but you may have trouble even noticing the difference from the previous revision.


Revision #4 - Refactored helper method

    List<Integer> removeOdd(List<Integer> aList, 
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();

        newList.addAll(aList.subList(0, fromIndex));
        newList.addAll(evenNumbers(aList.subList(fromIndex, toIndex)));
        newList.addAll(aList.subList(toIndex, aList.size());
    }

    List<Integer> evenNumbers(List<Integer> aList) {

        List<Integer> newList = new ArrayList<>();
        
        for (Integer e : aList) {
            if (e % 2 == 0) {
                newList.add(e);
            }
        }

        return newList;
    }

Now, let's see what lambda expressions can do.


Revision #5 - Using a lambda expression

    List<Integer> removeOdd(List<Integer> aList, 
            int fromIndex, int toIndex) {

        List<Integer> newList = new ArrayList<>();
        newList.addAll(aList);

        newList.subList(fromIndex, toIndex).removeIf(e -> e % 2 != 0);

        return newList;
    }

If you're wondering why I didn't choose to modify the given list in place, it's because I program defensively. I prefer to not do anything that would change the List passed to the method and avoid potentially giving callers an unexpected and unwanted surprise. This approach will also help me transition to a more functional style later.

The lambda expression, with the help of the new List.removeIf() method, replaces the arguably ugly helper method and dramatically simplifies the code. The removeIf method was introduced in Java 8 as a default method of the Collection interface. Default methods are also new in Java 8 and are a boon for developers of libraries who want to unobtrusively extend their existing APIs.

One subtle difference here is that the expression used to select elements changed from (e % 2 == 0) to (e % 2 != 0). This difference is key to getting the correct behavior but it seems like every time I look at it, I have to do a double take and pause to make sure I didn't make a mistake. That's a little annoying.

This annoyance made me feel like I could do a little more to make the code really reveal its intent. To ease my angst, I performed one final refactoring to introduce an explaining variable.


Revision #6 - Introduce a local explaining variable for the lambda

    List<Integer> removeOdd(List<Integer> aList, 
            int fromIndex, int toIndex) {

        final Predicate<Integer> oddNumber = (e -> e % 2 != 0);

        List<Integer> newList = new ArrayList<>();
        newList.addAll(aList);

        newList.subList(fromIndex, toIndex).removeIf(oddNumber);

        return newList;
    }

Looking back at the previous versions again, I noticed that there was a little bit of a shift from the way the problem is stated, "remove odd numbers from the list" versus the name of helper method, evenNumbers. I used that name because it's the only one that makes sense as the argument to the call to addAll.

This little shift goes away with the name oddNumber. It matches the problem statement and it works in both the context of the lambda expression and as an argument to the removeIf method. I'm hard-pressed to think of a way to make that line of code any more expressive. That's pretty cool.

Meanwhile, back in the JavaRanch thread that started this, I got into a bit of a discussion about the merits of introducing a local explaining variable. Some may think that adding that line to create a local variable defeats the purpose of using lambdas but I think that the small sacrifice made to brevity is worth the gain in clarity. I suppose it's just a matter of taste at this point. I like this one more.

In conclusion, this exercise helped me appreciate just how much lambda expressions can help to make for better, cleaner code.  And as an added bonus, I also saw the utility of default interface methods in making it easier to extend existing APIs.

The Sound of Code Smells

Caveat: A friend told me that this is a bit of a rambling post so proceed if you must but be prepared to bear with my blabbering if you have nothing better to do. I will work on refactoring this for clarity and post an update when I can get around to it. Thanks.

If a piece of code smells and there are no developers around, does it make a sound?

That's an interesting twist to the tree in the forest thing, isn't it? We may forever ponder the question about that poor tree in the forest--one company even advertises its opinion about it--but my answer to the code smell question is a definite and resounding "Yes!"

What sound does a code smell make?

At worst, I would imagine that it sounds something like whatever inspired Edvard Munch to produce his arguably most famous piece of work, The Scream. The eternal juveniles and jaded cynics out there will probably say "Plop!" or "Schploosh!" or "Really bad code smells are silent but deadly." If you've read Uncle Bob's "Clean Code" book, you might suggest that the sound of a code smell is "Dude, WTF?!" As crude as some of these retorts may be, you have to admit there's a distinct ring of truth to them.

Seriously though, you'd think that a very apt metaphor like "code smell" should be enough to motivate developers to take another look at their code and clean it up, right? So why would you want to add to a perfectly good metaphor? In this day and age when everyone is supposed to be familiar with—and presumably doing!—refactoring and unit testing, shouldn't this "smell" thing make sense to developers already?

Well, Virginia, I hate to break it to you like this but apparently, the "code smell" metaphor is still not good enough to help a large number (I shudder to think that "a majority" applies here) of developers in the real world start writing better code.

While I'm sure it would be easy for your average agile developer to think as far back as a day ago to remember when he last got a whiff of bad code, tracked it down, and cleaned it up, the average Joe Schmoe developer just doesn't have that kind of olfactory ability. At best, their code smell genes are still lying dormant somewhere inside them, just waiting for that moment of XP epiphany to finally break out in full glory so these developers can reach their true potential as lean, mean, code smell busting machines, the way nature intended all good developers to be.

Ok, it isn't that glorious either. Not usually. For many, "breaking through" those walls of ignorance that imprisoned us in rotting, smelly code doesn't conjure up images of emerging from a pit of darkness into glorious sunshine, birds singing, and gentle breezes blowing over endless seas of undulating green grass.

Getting code smells is really more like XP puberty, right? It's like those episodes of adolescent "break outs" that meant painful hours of intense scrutiny, self-loathing, and daily scrubbing in front of a mirror. For the unfortunate few who don't get it quite right and are left permanently scarred, code smells are just a fact of life that you learn to live with and things like Agile, TDD, and refactoring are nothing more than bottles of snake oil that unscrupulous traveling salesmen hawk to the many eager but desperate, unsuspecting, and gullible developers out there.

For those who do get code smells, TDD, and refactoring, "breaking out" is often followed by a few of those post-escape Andy Dufresne moments when our initial jubilation of making it out of the sewer is replaced by cold fear, confusion, and self-doubt. And certainly there are those poor old souls like Brooks Hatlen whose senses become overwhelmed--everybody is such a big damn hurry!--that they just simply give up in despair, longing for those simpler times before they were given so much freedom and a new lease on life. Some people just can't handle the truth. There has to be something that can be done to help out guys like that.

Who among us, apart from the very fortunate and brilliant few, have never said to themselves after they first started truly grokking and experiencing code smells, things like "I can't believe I wrote that sh*t!" or "Was I even sober when I wrote that cr*p?!" or just simply, "Dude, WTF?!" and then follow it up with the inevitable "Now what?" and "Maybe I was better off back in Shawshank and not having to deal with this."

Can any of us claim that we can always point a finger at the source of a code smell and decisively and incisively take care of it lickety-split? Heck, even gurus like Kent Beck admit to getting stuck sometimes. What hope is there then for us mere mortal developers who aspire to keep our code as clean as humanly possible? Ok, that's a bit too dramatic, I admit, but if you've been there, you know what I'm talking about.

Anyway, lately I've been asking people on JavaRanch to read their code out loud. Here's one reply where I do that (look for my reply that has a bunch of sample unit test code). This is actually what motivated me to write all this but there are quite a few other threads where I've asked the OP to do the same thing: read the code out loud.

If you try it and read some smelly code out loud, you might see what I mean by "the sound of code smells". Or rather, you might hear what I mean.

For example, the sound of redundancy is often a stammer, as with the Booking thing in that JavaRanch thread (Listing 1).

Listing 1. The sound that Redundancy makes

    public void addBooking(Booking booking) {
        bookings.add(booking);
    }

Comments that smell of redundancy sound very maddenesque (Listing 2).

Listing 2. The sound that Redundant Comments make

    public void addBooking(Booking booking) {
        // adds a booking to bookings
        bookings.add(booking);
    }

The sound of poorly chosen names can be heard in this thread and this thread and there are quite a few others where you may hear different kinds of sounds.

(Sorry if the links to JavaRanch are off-putting to you. I'm not trying to click-bait or spam-link you with these -- it's just that I have yet to ask the bosses at JavaRanch if I can take code that was posted there and put them up here as examples. I'll update this later if I get a nod from them, which I'm hoping I will).

I'm sure there are many other kinds of code smell "sounds" out there that I've heard but, just like with "smell", it's difficult to describe "sound", especially to someone else who has never heard the same sound before. To someone who doesn't know any better, "The sound of a poorly chosen name" is very Zen-like. I tweeted the other day about how difficult it would be to train a dog named "Stay". The sound of a poorly named method inspired that tweet. Here's my reply on JavaRanch about it.

If you think about it, adding the sense of hearing to the already good metaphor of "code smell" makes a lot of, well, sense. We're already saying things like "the code tells a story" and "the code is telling you what it wants to do" and "that code just screams out to be refactored" and "that variable is asking to be renamed," right? These all suggest that we need to listen more carefully to our code. What better way to get developers to listen than to get them to read their code OUT LOUD?

The point of all this, and thank you for listening to me blabber away for this long, is that I think it would help those developers, the ones who don't quite know how to recognize a code smell yet, if we older and more experienced developers who do would tell them more often to read their code out loud and listen to the sounds that their code is making.

Even for us old paroled jail birds, who sometimes get stuck and are experiencing that "maybe I was better off back in Shawshank" feeling, it might help us to read our code out loud once in a while. Something seems to happen when we actually hear what the code is doing and sometimes that's all it takes to get unstuck.

Maybe for us developers, the best way to develop our metaphorical sense of code smell is by exercising our literal sense of hearing.

So, to all the young developers and all the old fans of Casablanca, "Here's to hearing from you, kid."

(Update, Tuesday 11-Nov-2014)

I was looking around for other references to code sounds and found one on Ward's Wiki on, of all places, the page on Code Smells. Russell Gold mentions the auditory metaphor in a comment about how saying code "smells" has a different connotation than saying that it is "off-key" or "doesn't sound right." While his point is not quite the same as the one I'm trying to make here, I still think it's interesting that others have felt that it's at least worth considering other sensory metaphors besides smell.

Mob Programming for Distributed Agile Teams

Looks like Woody Zuill (@woodyzuill) is getting pretty busy promoting the practice of Mob Programming these days. I applaud his efforts and I'm happy to see his apparent success with it. I think it's a great practice and I think more teams should at least try it on for size.

Responding to Tampere Goes Agile (@tmpgoesagile) question on why you should use mob programming, I said that mob programming has helped our distributed agile team get better by keeping everyone on the same page. @tmpgoesagile still wanted to hear more about it so here are some details on how that came to be and how it works for my team.

It's a Pattern

I think it was either Esther Derby (@estherderby) or Jeff Patton (@jeffpatton) whom I heard say something like "I didn't invent this. Other people have told me that they've done the same kind of thing before. So I guess the only thing I can claim here is to have discovered and documented a pattern." They were, of course, talking about something other than mob programming, but the same thing seems to apply here.

I first heard the practice being called "Mob Programming" at the Agile 2013 conference in Nashville, TN. A team from a company in California was talking about it in an Open Jam Huddle in the expansive lobby of the Gaylord Opryland Conference Center and I thought I'd stopped by for a quick listen while I was between sessions. A group of young guys were talking about how they did all their development anymore and how much better and enjoyable mob programming was than just plain old pair programming.

We had been doing the same thing on our team for at least a year already, except we called it "swarming," a name we'd picked up somewhere for the practice of having the whole scrum team work on a single user story at a time. I liked the ring that "Mob Programming" had to it so I started using the name with my team, too, although I still tend to say "mob or group programming" when I describe what we do to other people in our organization. Now that Woody had made the name more popular, I guess I'll go with "mob programming" from here on.

First Rule of Distributed Teams

We are a distributed team with members in Ohio, Texas, and California. Our concession to not being co-located was to require that team members be no more than three timezones apart. This gave us at least five hours in a day when nobody had to make crazy adjustments to their schedules. If we did have to schedule something that was a "little bit" early or late for one or two people on the team, we could at least avoid swinging out more than an hour or two on either side of our core work hours. Any crazy adjustments to schedules, which a few of us did anyway, were totally your personal choice and discretion.

I think if you're going to do mob programming with your distributed agile team, you have to at least have this rule in place. If you have a few guys on your team like me and a couple of my colleagues, that's fine. But make sure people agree that crazy hours are the exception rather than the rule.

Share Images to Build a Shared Understanding 

In his book "User Story Mapping", Jeff Patton notes that pictures play an important part in building shared understanding in a team. This is especially true with distributed teams and it has been borne out by our experience.

The right collaboration tools are very important for distributed teams like us. Before I go on, I want to make it clear that this is by no means a thinly veiled attempt at promoting our company's products, no matter what it looks like. I assure you that I mention the tool we use purely as a matter-of-fact. I have to admit though that it is nice to be a rare case of the cobbler's kids getting nice shoes for a change.

Anyway, we use Cisco WebEx for meetings and collaborative development all the time. Most of us work from home and we love the flexibility and convenience that comes with it. At the start of our journey to agility, people would just email each other or use IM to have conversations. I nipped that in the bud. I reminded everyone that if we couldn't have face-to-face conversations, the next best thing was to have a live conversation on WebEx. We could share our screens, which we often do, and we could even share our cameras, which we still don't do very often for some reason.

I guess that's one curious observation to note: Being able to share your screen, priceless. Being able to share your face... meh.

Nowadays, our conversations usually start with the simple IM: "WebEx?"

Pair-Programming to the Extreme

As I alluded to earlier, we kind of just fell naturally into the practice of mob programming from "swarming" on a user story.  As with most distributed teams, we'd been having trouble completing user stories when we divided them up among ourselves and tried to work on them concurrently. We often ended up moving stories to the next iteration to complete them. Swarming was our attempt to get at least one story fully done by the end of the iteration that it was pulled into the first time.

I'm also a big proponent of Test-Driven Development. I had first instituted the practice of code reviews which naturally led to my goal of having frequent pair programming sessions so that after-the-fact code reviews had to be done less frequently.

When we started swarming on user stories, this led to us doing group programming on WebEx for at least three or four hours a day. Yeah, we called it "group programming" and we liked it a lot. The mechanics were kind of the same for pair programming except that there was a lot more discussion and different perspectives brought to the table. Also, there were more people to do little sidebar tasks like Google for something or experiment with an API call or write documentation or whatever.

As a tech lead, I loved that we didn't have to do as many after-the-fact code reviews. I could keep my finger on the pulse of the design and head off any design choices that could potentially make trouble for us later. I could also do my mentoring on refactoring practices and design principles all at once and not have to repeat myself with other team members at a later time.

The Uncomfortable and Unspoken Rule of Distributed Teams

If you've never seen the movie "Office Space" then any claim you make to geekdom rings a bit hollow. Anyway, I think mob programming helps you avoid having any Milton Waddamses on your distributed team. It's all too easy for someone to say, "Let me work on this for a little bit then I'll get with you for a code review on WebEx tomorrow." Then next thing you know, that guy is in the basement plotting to set the building on fire. Ok, maybe that's a bit of a leap but you know what I mean, don't you? If you don't, Google and NetFlix are your friends.

Since everyone is working together for a good portion of the day, mob programming makes it almost unnecessary to even think about the Uncomfortable and Unspoken Rule of Distributed Teams, which is that "Team members will have the integrity and discipline to work even when nobody is watching."

I know we all want to assume the best of our team members but not giving anyone a chance to give in to the temptation of working alone can sometimes be the best way of establishing trust on the team.  I think mob programming helps with that.

Conclusion

I'm really glad to see that Woody Zuill's efforts to spread the word about Mob Programming are succeeding. Mob programming is great for any agile team and it can be especially beneficial for distributed agile teams.