Don't trust your instincts

Ours is a young discipline with no rules or laws except the ones we choose for ourselves.  We are still living in the wild wild west of software development.  There is no exam to pass to become a software developer.  There are no standard evaluation procedures, or checks and balances.  If you can make your software execute, you can install it, host it, and sell it.  This is one of the things I find very attractive about our industry, but also occasionally frustrating.

There have been many attempts to standardize engineering in the form of processes and methodologies.  But process is only a small -- and very uninteresting -- portion of building software.  Code itself is far more challenging, interesting, and diverse.  But it is very lacking in recognized rules or techniques or principles or even just ideas.  Certainly there are some code principles and practices, but how successful or accepted are they?

It's hard to get a true sense of the opinions and practices of our industry, but there is clearly a very vocal minority that eschews "software engineering" practices in favor of a loosely defined aesthetic.  I'll use "software engineering" as a label for structured principles, patterns, and practices.  For example, consider the Gang of Four's design patterns, or Bob Martin's SOLID principles.  But the vocal minority, which seems to me at least to be getting increasingly vocal these days, would argue these concepts (patterns and principles) are more harmful than helpful.  That a better approach is to simply take the time to feel the pain in your code, and adjust, rewrite, and refactor as needed.

A really solid example of this argument being made can be heard in this Ruby Rogues podcast interview of DHH.  If you stick with it, the conversation covers a lot of really interesting topics including how DHH applies this thinking to rails and basecamp, YAGNI, thoughts on education and the necessity of stubbing your toe to learn, and more (Thanks to Lee Muro for referring me to that podcast).

I agree that stubbing your toe is a good teacher, but I don't think it's the only way to learn.  I agree that abstract concepts are easy to over use and misapply, especially after first learning about them, but I don't think that's inevitable.  While I find the refactoring and continuous learning part of this attitude very pragmatic, there is one element I do disagree with: the idea that we don't need abstract rules and principles and guidance and science.  That all we need is our sense of aesthetic.  The idea that by simply looking at some code, maybe comparing it to a different version, you can derive an intuitive understanding of which code is better.

I don't buy this, because I don't think that's how humans work, as outlined by Malcolm Gladwell's book Blink and this article by Jonah Lehrer.  I recommend them both, but if you're short on time, just read the Jonah Lehrer article as it's short and the most directly relevant.

Blink is all about the influence our subconscious mind has on us.  We like to think that we are rational and in full conscious control of what we do and what we think.  But Blink has plenty of research to prove that this simply is not the case.  We depend on our subconscious to make snap decisions and influence our general mood and thoughts much more than we realize.  And Blink goes to great lengths to present the fact that this can be both very powerful and harmful.  Your mind is capable of "thin slicing" a situation, pulling out many relevant factors from all the thousands of details, and coming to a conclusion based on those details.  But, not surprisingly, you need both extensive practice AND exposure to all the needed factors for this to work.  And it's worth mentioning that even when it does work, your conscious mind may never understand what it was your unconscious did to come to it's conclusion!

You might read that and think, "Experts can use their unconscious to recognize good and bad code, the vocal minority is right!"  I believe that is true, but only on a local level.  When you look at code, you are always drilled in to the lowest level.  I think you could intuit a fair amount at this level, but it's the higher concepts that have the larger influence, and I'm not sure you can effectively thin slice that stuff.  Many of the concepts of good architecture are about higher level structure: low coupling, high cohesion, SRP, ISP, DRY.  But if I showed you one code file and asked you to tell me if the application suffered from coupling issues, you wouldn't be able to say.  And that's because you haven't been provided with enough information.  And without that information, how can you possible thin slice your way to an intuitive understanding of good code?  I worry that a focus on "aesthetic" and "elegance" leans too heavily on this intuitive feel for code, and carries a serious risk of leading you down a path that feels smooth and easy, but ultimately leads straight into the woods.

But I would take this argument even further.  Jonah Lehrer's article tells a story of a psychology experiment that went something like this.  Study participants were shown two videos, both showed two different sized balls, one larger than the other, falling toward the ground.  In one video the balls hit the ground at the same time, and in the other the larger ball hit the ground first.  The participants were asked which video was a more accurate representation of gravity.



And the answer is: the video where they hit the ground at the same time is the correct one.  This is not intuitive, most of us would expect the larger ball to hit first.  So the way the world actually works comes as quite a surprise.  But where this gets interesting is in the second part of the study.  This time, the participants were all physics majors, who had studied this and learned the correct answer.  The participants brains were being monitored with an fMRI machine and what the researchers discovered is that in the non-physics majors a certain part of the brain was lighting up which is associated with detecting errors, the "Oh-shit! circuit" as Jonah calls it.  When they saw the video of the balls hitting the ground at the same time, their brains raised the bull shit flag.  So what was different with the physics majors that allowed them to get the right answer?

But it turned out that something interesting was happening inside their brains that allowed them to hold this belief. When they saw the scientifically correct video, blood flow increased to a part of the brain called the dorsolateral prefrontal cortex, or D.L.P.F.C. The D.L.P.F.C. is located just behind the forehead and is one of the last brain areas to develop in young adults. It plays a crucial role in suppressing so-called unwanted representations, getting rid of those thoughts that aren’t helpful or useful.

This other section of the brain allows us to override our intuitive primal expectations, the Oh-shit! circuit, and replace them with learned ones.  But in order for this circuit to work, you must have studied and learned the material!  Which requires that there be something to learn!

The connection to the aesthetic instinctive approach to software should be pretty clear.  If you shun what "science" our industry has to offer, however admittedly weak and young it may be, you're not training your brain to suppress the intuitive but worse-for-you-in-the-end code!

So I think it's important to be cautious when relying on your intuition and sense of aesthetic, especially in an industry as young as ours with so little widely accepted guidance.  We need to follow that pragmatic approach of continuing to learn, but at the same time we have to continue to question our intuition.  And just as important, we should take the science/engineering of our industry seriously, even while recognizing it's limitations.  


Software is hard, be careful how much you trust your instincts!