Rails has no place at the office

This is a milestone post for me! My first ever purposefully incendiary title!

I should probably run with it and try to get everybody super offended, to the point where you have no idea what my point is because all you can see is red. I guess I'll have to leave that for a future milestone...

Because, yeah, I'm not really serious. Rails has a place at the office. And no, this isn't going to be one of those "Is Rails ready for the Enterprise?" posts. Rails is perfectly ready for use in the Enterprise, but that's the wrong question.  As usual, the right question is much more complicated.

To start with, let me point out that this conversation has nothing to do with Ruby vs. C#. It doesn't really have anything to do with Rails vs ASP.NET MVC either. Instead I'm going to be talking about Active Record vs. Data Mapper, and View-Models vs. no View-Models, and this general concept of "the straight and narrow" vs. explicit abstraction and control. These are design patterns which apply to any language and appear in many different frameworks.

Rob Conery recently wrote a blog post in which he said,

For a lot of .NET/Java devs this will look "messy" - you shouldn't elevate "data concerns" into your model. This argument makes good sense for a large, complex site - that you're building in C# or Java. Typically Ruby focuses on the straight, narrow path and with that comes a dramatic turn towards "doing what you need to do... and no more". This resonates with me...

The part about Ruby/Rails focusing on "the straight and narrow path" really struck a chord for me.  Ruby, being a dynamic language, is very much on the "straight and narrow."  It dispenses with all kinds of things found in strongly typed languages like private, internal, protected, interfaces, etc.  These are things that are usually considered very important in a strongly typed language, and practices like DDD, but Ruby doesn't really bother with them.  Ruby favors documentation and convention over strict control.

Rails has a similar story.  It uses the Active Record pattern for its data access, which requires a 1-1 correspondence with your database.  Further, the models don't even really exist!  They're built dynamically from the schema of the database tables.

If you compare ASP.NET MVC to Rails one of the differences you'll quickly discover is this concept of a "View-Model". ASP.NET peeps seem to like these, whereas I haven't found a Rails sample anywhere that uses these.  Both Active Record and this lack of a View-Model are accomplishing the same thing: removing abstraction in favor of directness and simplicity.

Now lets step back from this for a second and ask a question.  Who in there right mind would want to have to deal with things like class and method visibility and extra layers of abstraction, which more often than not appear to be just duplication?  No one!  No one would want to deal with these things!  It's extra work!  I _hate_ extra work!

So why do we do it?  Why does DDD make a big deal out of private constructors and Factories?  Why does Fowler recommend the Data Mapper pattern over Active Record?  Why do we create View-Models to separate our Views from our Models?  Why do we do all these things that seem to just make life more complicated?  Why don't we all take the straight and narrow path on all of our projects all of the time?

Certainly it's not as simple as the language we're using.  Just because you're writing in C# and Java doesn't mean you can't use Active Record.  And it doesn't mean you can't pass your Model straight to your View.  There is also nothing about C# or Java that forces you to use interfaces, or follow the Dependency Inversion Principle.  That said, there's also no reason why you couldn't use the Data Mapper pattern in Ruby, or create View-Models.  The language certainly HELPS with some of these issues, but it's not the real difference.  These are just patterns, and they apply equally well to any language.

The reason why we introduce this complexity and divert from the straight and narrow path in our technical approach is actually due primarily to non-technical reasons.  Here are some of the reasons I think lead us to adopt these "enterprise" patterns:

• There are more than two or three developers on the project
• You have more than 6 entities in the domain
• The project has a timeline longer than 3 months
• The developers aren't intimately familiar with the domain
• The project is likely to grow in fits and starts
• The team members are more likely to come and go

These are not technical issues but they have technical IMPLICATIONS!

The practices prescribed by DDD are a big deal if you're working with a large complicated domain with lots of potential for change.  If you're not, then you don't need DDD.  Fowler's enterprise patterns are a big deal for the same reasons.  If you know things are complicated, likely to change, and not possible for everyone on the team to grok completely, then you need to build abstraction into your code.  And you need to try to be as explicit as you possibly can about what the code does and how it works.  And you need to look for opportunities to prevent error and misunderstanding before it happens.  These things will allow you to keep things clean, organized, and ultimately make your project successful when you're faced with "enterprise" challenges.

This is obvious.  I'm sure you're sitting there (or standing) thinking, "duh!" or "when is this dude going to get to the point?" or "does this moron really think this is revolutionary?!"

My point is as simple as this.  Rails is awesome.  Simplicity is awesome.  But as I sit here in my ivory tower looking out over the landscape I see lots of quiet subtle backlash from people against the "enterprise-y" patterns in favor of the simplicity of Rails.  This makes a lot of sense to me because, as we pointed out, who would WANT to deal with the complexity of enterprise problems and patterns?  But it is easy to be tempted by the appeal of simple solutions to simple projects.  And certainly we should always strive to find the simplest solution that could possibly work.  But we can not close our eyes to the complexities of the problem or the environment in which we are solving the problem.  And we cannot allow ourselves to be boiled alive either.

So by all means, choose the right tool for the right job, but make sure you understand the job as well as you understand the tool.

Vim Commenting

Recently there has been some renewed interest in my series of posts on using Vim for C# development, so I thought I should add a few more posts to the series to bring it up to date.

You can find the introduction to this series here.

Visual Studio has a feature which allows you to select a bunch of lines of code and have them all commented out, or uncommented.  In my setup it is bound to Ctrl+k+c to comment and Ctrl+k+u to uncomment.

I use this particular feature pretty regularly, so I definitely wanted it in Vim, and wouldn't you know it, there's a plugin for that!  NERD Commenter.

With NERD Commenter installed you can select a bunch of lines of code (I typically do something like Vjjjj) and then type either

,cc

or

,c<space>

The first is the comment command, the second is the toggle command.  If you use the comment command you'll need to use

,cu

to uncomment.  If you use the toggle command you don't need to remember two commands!

That's all there is to it!

Now, the first step is to visually select the lines you want to comment or uncomment.  You can just hit j,k a bunch of times but there are usually better ways.  For example, if you want to comment out an entire method:

public void IAmAMethod()
{
  ... lots of lines here...
}

If there are lots of lines in  the method, you don't want to be hitting j all day.  Instead, with your cursor on the method declaration line, do:

Vj%

The % is the "match" motion.  So when your cursor is on the { it will find the matching }.

Alternatively, if you just want to comment out a group of text that is arranged in a paragraph:

public void IAmAMethod()
{
  ...paragraph of code...

  // I want to comment out this block of code
  ...more code...
  ...goes here...
}

For this you can use the "paragraph" motion.  With your cursor on the first line of the block, do:

V}

That will select the whole block.

Once again, we have approximated a feature found in Visual Studio, but made it even better with the power Vim!

Vim File Navigation

Recently there has been some renewed interest in my series of posts on using Vim for C# development, so I thought I should add a few more posts to the series to bring it up to date.

You can find the introduction to this series here.

In a previous post I talked about how to open and edit files in Vim.  That post discusses just the basics of opening files.  Since then I've started using the wonderful NERDTree plugin.  This plugin opens a small buffer on the left of your Vim window which contains the file system tree.  You can then navigate through directories and open files.  The nifty part is the NERDTree is just a Vim buffer, so you can navigate with h,j,k,l and you can search with / etc.  To see what I'm talking about, you can watch a demo of the NERDTree in action here.

Before I go on I should mention that Vim actually has a built in file system navigation plugin called Netrw.  NERDTree adds a few features and is in someways a bit easier to use, but Netrw is capable of doing all of this stuff and it's built right into Vim.

Using NERDTree has completely changed the way I work in Vim.  When I'm ready to start working, I navigate to my .sln directory in the terminal and I type

gvim .

This opens Vim with NERDTree showing the contents of the current directory.  From here you can navigate around and find the file you want to start editing.

When you open the first file NERDTree will go away.  If you want to pull up NERDTree so it's always visible docked to the left of your Vim window you can type

:NERDTree

To toggle it open and closed you use

:NERDToggle

That's kind of a lot to type. So to shorten it up I've mapped it to F2 by adding this to my vimrc:

" toggles NERDTree on and off
map <f2> :NERDTreeToggle<cr>
imap <f2> <esc>:NERDTreeToggle<cr>i

Now hitting F2 will open and close the NERDTree. Fast and easy.

So NERDTree is great for finding and opening files (you can even open files in splits with i for horizontal and s for vertical), but NERDTree can also manipulate the file system.

For example, to add a new file, put your cursor over (or in) the directory you want to add the file to and hit the "m" key.  This will open up a menu with some options.  Type a to "add a childnode" and then just type in the name of the file. This works for creating directories too.  You can also move (and rename) files as well as delete.

Coding Style Preferences

I recently did a little poll on twitter and in my office to get a feel for what people's coding style preferences were.  The fun thing about coding style preferences is that they are completely irrelevant, and yet a topic that people can easily get pretty passionate about.

It was only a little poll, with 35 people responding to these few questions:

Curly braces?

• On new line
• On same line

Spaces in control statements (if, foreach, etc)?

• Spaces outside and in ex: if ( this.HadSomeCandy )
• Spaces inside only ex: if( this.HadSomeCandy )
• Spaces outside only ex: if (this.HadSomeCandy)
• No spaces ex: if(this.HadSomeCandy)

Spaces in method calls?

• Spaces outside and in ex: someone.ShouldJustDecide ( "what", "is", "right );
• Spaces inside only ex: someone.ShouldJustDecide( "what", "is", "right" );
• Spaces outside only ex: someone.ShouldJustDecide ("what", "is", "right");
• No spaces ex: someone.ShouldJustDecide("what", "is", "right");

Spaces in method declaration?

• Spaces outside and in
• Spaces inside only
• Spaces outside only
• No spaces

Spaces in method calls with no args?

• No space ex: someone.ShouldJustDecide();
• Space ex: someone.ShouldJustDecide( );

How many spaces in indentation?

• 2
• 4
• 8

And here are the results:

So, clearly, the winner is spaces outside/no spaces as in:

if (who.Cares("about coding style?!");

An interesting observation here is that the people who "don't like spaces" are very consistent in their preferences whereas the people who "do like spaces" are much more varied.  This is evident in that of the 22 people who voted for spaces outside only in control statements, 19 also voted for no spaces in method calls.

Note that only 25 people answered the question about spaces in indentation because I added it to the poll later.  I expect the results would have been much different because it was the people from my office who didn't get to answer and our internal standard is 2 spaces.

For curly braces it was 22 to 13 in favor of braces on a new line.

There were answers for just about every combination, no matter how weird.  For example, some people put spaces outside and in for control statements but no spaces in method calls.

The sample size of this poll is too small to actually mean anything, but it is still interesting that the preferences line up pretty closely with Microsoft's coding style standards.  I didn't verify this, but I wonder if this could be influenced by Visual Studio's default code style settings.

Personally I was very much in the minority here.  For the last five years I've been a spaces outside and in/spaces inside guy as in:

if ( who.Cares( "about coding styles?!" );

I've also been a 2 spaces guy and if you go back to college I was a curly braces on the same line proponent.  I started doing the curly braces on a new line when I started full time at my job.  I recently tried curly braces on the same line when I started learning jQuery and I have to admit, I didn't like it anymore.  Could be just because my javascript is still pretty ugly though.

I'm also starting to second guess the whole 2 spaces thing.  I always preferred it because it made it so you could see more code.  But now that I've embraced the SOLID principles, if the lines of code in my methods were so indented as to cause a problem reading them, I'd suspect a "design" problem with that method.  And I'm starting to think that 4 spaces would make a pretty big readability difference, since it would be much easier to spot where indentations start and end.  I think its especially important if you do curly braces on the end of the line, or if you're writing Python or Haml.

Finally, I always liked the spaces in control flow because I believed it made it easier to read.  But when I was preparing this poll I wrote the different styles out side by side and I started to wonder if the spaces actually bring out the "noise" of the different characters...  I'm still not sure about this one.

This whole exercise also made we question WHY there is so much possible variation in the languages.  Wouldn't it be nice of the details of the language were done in such a way that there was 1 right way to do it and we didn't have to concern ourselves with silly details like where to put spaces?

View-Model design question

Here's a design question for you!

Lets say you are working in ASP.NET MVC 2 (or your favorite MVC web framework).  Lets also say you have a nice rich model.  Your controllers have to fetch the model objects and get that data to your view.  How do you do that?

There are a bunch of ways:

1. Pass the model directly to the view
2. Create a "View-Model" class and put a property on it that exposes the model
3. Create a "View-Model" that completely hides the model behind properties of its own

And there are a bunch of variations on those too.  But those are the main options.

The best thing about #1 is it's as simple as can be.

#2 is almost as simple as #1 but adds the ability for you to create other custom properties on the "View-Model" object that can perform various operations for the view.  For example, you might format values, or retrieve the latest object from a list, etc.

The downside to these two options is your View is directly coupled to your Model.  This might become a problem if you end up with lots of Views that depend on the same Model, or if the Model keeps evolving and being refactored over time.

That's where #3 comes in.  By creating all new properties on the View-Model, you're basically applying the Dependency Inversion Principle and saying, "This view needs this data, I don't care where it comes from as long as someone provides it."  You will now need some form of mapping layer to get the data from the Model to the properties of the View-Model.  This is more work, but it's also nice.  When the Model changes, you only need to update the mapping, which is dramatically easier than digging into lots of HTML and finding what could be many references to your properties.

Now, that said, there are still lots of changes that will require you to make changes to the Model, View-Model, and View.  Any change that is a change in the *meaning* of the Model will cascade this way.  But there is a whole set of changes that won't cause this update cascade.  Like any refactoring of the Model for example.

The obvious downside with #3 is more code and more work (Though tools like AutoMapper certainly help).

So, how do you know when to apply which pattern?  Is one pattern always better than the others, or does it depend.  And if it depends, on what?  And how do you know when it's time to switch from one to the other?

Thoughts?  Experiences?

SRP and complexity

The Single Responsibility Principle (SRP) is probably the most important concept of good design.  But even once you know about it, and have read up on it, and seen countless blog articles describe and reference it, you may find yourself hesitant to actually follow it in real life.

The usual argument against it is that it seems like it might increase the complexity of your code.  Lets look at an example of applying SRP to a method.

public void UpdatePrimaryThingStatus( string status )
{
  Thing primaryThing = null;
  foreach( Thing t in something.AllThings )
  {
    if ( t.IsPrimary )
    {
      primaryThing = t;
      break;
    }
  }

  if ( primaryThing != null )
    primaryThing.Status = status;
}

There's nothing _wrong_ with this code, but it doesn't really follow SRP because the method is updating the primary thing's status, as advertised, but it's also finding the primary thing.  Lets factor out the finding of the primary thing into its own method:

public void UpdatePrimaryThingStatus( string status )
{
  var t = GetPrimaryThing();
  if ( t != null )
    t.Status = status;
}

public Thing GetPrimaryThing()
{
  foreach( Thing t in something.AllThings )
  {
    if ( t.IsPrimary )
      return t;
  }
  return null;
}

Notice how much code actually disappeared here.  And notice how simple each method is.  But, we did add a new method to the class.  Do we intended to reuse this method?  That depends, it IS a useful method that could easily be reused, but since we didn't have it already, lets assume we don't need to reuse it right now.  So yes, we simplified the code in the individual methods, but by adding a new method we've increased the complexity of the class.

We should probably ask why adding a new method is a problem.  It's only one new method!  It's well defined with a single responsibility, with an intention revealing interface, and simple code to boot.  Why would we think this is going to increase the complexity of the class?  Probably because we're used to working with classes that are thousands of lines long with lots and lots and lots of methods!  So yes, if you're applying SRP to your methods, but not to your classes, things might get a little complex.  But make sure your classes have a single responsibility, and you'll find that this wont be the case anymore.

OK, so if our classes are following SRP, then we'll be breaking large classes into more smaller classes.  But now we have lots of classes!  Doesn't that make our code more complex?

This same pattern will follow right up the chain through namespaces and assemblies...  is this getting out of control?  What's the solution?

The solution is cohesion!  You can add lots of small classes as long as they are all part of a cohesive whole.  This is actually a really beautiful thing.  If your classes are well organized, and obviously form a cohesive unit, you get an amazing benefit.  Lets say you need to go into the code and find a bug.  You know what area the bug is in, even if you don't know exactly what it is.  There may be 20 files that make up your code, but you'll probably only need to crack open 3 or 4 of them to find and fix the bug.  And each file you do open will be understandable, dare I say, easily understandable.

You may think there is still a problem in understanding the WHOLE.  To understand how it all works, don't you now need to open all these little classes and figure out how they all work together?  Yes and No.  Again, I think the fact that each class has a clear single responsibility (and therefore an intention revealing interface) means you can actually understand the WHOLE and read less code than if you had it all squished into a single class.

So, whatever you do, don't let fear of complexity drive you away from SRP.

Framework Disease

A lot of software engineers have a Computer Science background.  My college education, for example, included  the standard things like Data Structures and "Operating Systems."  It also included some cool things like Artificial Intelligence, Peer to Peer, Automata Theory, and Evolutionary Computation.  I was also fortunate enough to have the opportunity to participate in some research projects as well which included Mobile Agents, Evolutionary Computation, and Swarm computing.

These things are cool.

But now I spend my time figuring out how to get data from a UI into a database and back again.  That's basically the bottom line of "Enterprise Application Development".  Surprisingly, there are enough challenges in this space to keep you busy for a very long time.  And while the topic itself doesn't have a lot of sex appeal, the work is actually amazingly broad, even just from a technical standpoint.  And once you add the "business" concerns in, it has the potential to become very interesting indeed.

But still I'm a computer scientist, and I'm inexorably drawn by computer science-y problems.  In the Enterprise space, the most computer science-y problems tend to be those of building "frameworks."  And what I mean by framework, is re-usable code bases that developers use to avoid having to write the same (or similar) code over and over again.

Frameworks are to some people as cat nip is to cats, or street lights are to moths, drugs to drug addicts, or cigarettes to smokers, or...  Some people loooooooove building frameworks.  They are always on the look out for an opportunity to build a framework.  At the first sign of duplication, or recurring pattern, you can see the light in their eye... framework!

I call this Framework Disease.  Frameworks are tricky.  They can be huge time savers.  And they certainly are fun to work on, since they're so computer science-y.  But at the same time, they can be real time wasters.

Sometimes the problem you are trying to solve with a framework simply isn't worth the time it takes to build the framework.  This could be because all the framework does is replace some standard boilerplate code that could easily be copied and pasted or generated.  In these cases, centralizing the boilerplate can actually be a bad thing because you're forcing every use to be identical forever.  Just because they are the same now doesn't mean they always will be, or always should be.

Other times the framework ends up being written in such a way that it actually becomes a problem.  This can happen when the framework starts limiting what you can do, or when it continues to grow and grow and grow, or when the complexity of the framework obscures the simplicity of the problem being solved.  When this happens you're spending more time working on and fighting with your framework than you are on actually getting things done.

Another problem with frameworks is the tendency to build them too soon.  If you set out to write a framework without having seen plenty of examples of what your framework will be replacing, your framework is probably doomed.  To be really successful, you have to write the code your framework will replace in a number of different places.  If you don't, you'r just guessing about what should be abstracted into the framework.  This means you really don't know what should go in the framework, nor where the framework should be flexible or where it should be rigid.

I have personally fallen into all these traps many times, and just about everyone I know has suffered from a bit of Framework Disease at one time or another.  It is very contagious.

I think Framework Disease is a symptom of not being connected enough with the goals of the development effort.  At Codemash, Mary Poppendieck told a little parable that went something like this:

A philosopher walked into a quarry and saw three people working with pickaxes.  He walked up to the first man and asked him, "What are you doing?"  The man irritably looked up and said, "I'm cutting stone, what the hell does it look like?!"  The philosopher moved on to the second man, asking the same question.  "I'm making a living for my family."  Finally the philosopher asked the third man, who responded, "I'm building a Cathedral!"

The third guy clearly understood the context of his work.  I think a lack of understanding of the context of work is frequently what leads to Framework Disease.  Passionate people in particular are susceptible to this.  Without a broad understanding of why you are doing what you are doing every day, how can you possibly stay focused on the important things?  How can you possibly stay energized?

So if you find yourself exhibiting the symptoms of Framework Disease, step back and ask yourself, "If I'm not building a Cathedral, what am I building?  And does this framework really further that goal?"

Passion

I think there are two qualities that set really great developers apart:

1. Technical competence
2. Passion

Pretty much in that order.  If Bob has strong tech skills, it means he can solve complicated problems to a reasonable level of quality independently.  But if he is lacking passion, it means he wont be looking for opportunities to improve, or to push the envelope on issues like code quality, cleanliness, productivity, etc.

If Bill has a lot of passion, it means he'll be on the lookout for ways to improve.  Both in his own work, and his team's work.  But if Bill doesn't have the technical competence to back it up it means he's simply unreliable.  The impression will be that he pays lots of lip service to quality and improvement but never manages to actually deliver any.

Bennie, on the other hand, might have both of these qualities.  This makes him reliable and constantly improving. And not just improving himself, but improving those around him.  Bennie is the guy who's likely to not only complain about some "policy" his office has that he feels is hurting more than helping, but to actually work on getting that policy changed.  Effectively, Bennie is a leader.

I think people with passion naturally end up leading, regardless of whether they are in "leadership" roles.  You don't have to be the boss to influence how your company works.  And you don't have to be the team lead to influence what technologies get used and how.

However it is this fact that makes passion a double edged sword.

Passionate people are more likely to challenge the status quo.  Which is good.  Unless they are challenging it in ways that actually hurt their team or hurt their company.  This runs along the same lines as some issues I discussed in an earlier post called Engaged Employees.  In a nut shell, if the priorities of the passionate people don't line up with the priorities of the business, you've got trouble ("With a capital T.  And that rhymes with P and that stands for..." Passion?).

When the passionate people become dis-engaged, there are two likely outcomes.  They might start "farting around" with "improvements" that don't actually help the team or the business accomplish any of its goals.  It is probably still true that these "improvements" are "good" in their own context.  But in the context of the business, they might actually be "bad," or possibly just unimportant (and therefore a waste of time).  On the other hand, the passionate people might decide to simply checkout.  They might decide to put in the minimal amount of effort possible, with an attitude of "screw those guys."  If just one person was acting in either of these ways, it wouldn't be that big of a deal.  But when it's your passionate people, it can lead to much more trouble.  These are your leaders after all, and their attitude and behavior rubs off on everyone else.

A company plagued with dis-engaged passionate individuals (like Bennie) would probably like to trade them all for simply technically compete people (like Bob).  The Bobs would stop causing so many problems and stop challenging everything and stop being in such a bad mood all the time and just get their work done.

But we have to ask, what has led the passionate people to be so removed from the goals of the business?  I think there is always a simple one word answer to that question: Management.  The book First Break All The Rules makes the case that a managers job is simply to bring out the best in his people.  And not the best out of context.  If you're an amazing yo-yo player but you program for a living it's not your manager's job to bring our your best yo-yoing.  Obviously.  It's his job to bring out your best in the context of the goals of the business (What a Programmer Wants in a Manager).  If your passionate people don't know what the priorities of the business are, or if they can't figure out what they could do today to have the biggest impact on the company, then management has messed up somehow.

But just because management has messed up doesn't mean we get to blame them and call it a day.  It doesn't mean we can just give up, stop trying, decide not to care, or adopt a bad attitude.  That is the path to the dark side.  As Bobbies, we have to do what we've always done: Fix it!  This is going to be harder for us than, say, designing a better data layer.  This is now a people problem.  And as technical nerds, we are probably not the best suited individuals to address people problems.  But sometimes we have to embrace the circumstances life throws at us, however uncomfortable they may be, and we have to grow up, step up, and fix it!

Quality Code

A month ago or so Jeremy D. Miller wrote a blog post where he briefly, but effectively, tackles the issues of why writing good code is important. I've written about this in the past as well. Now, I think this is one of those issues that no one would REALLY argue against, but that we all know lots of people don't FULLY agree with or understand.

I don't think anyone would argue that bad code is better than good code. But I think people (developers and business people) misunderstand how important good code actually is. This issue is very near and dear for me because I have quite a history with bad code. Really bad code... So figuring out what good code looks like is more than just an academic exercise for me.

What Jeremy says is,

Let’s be realistic here, you never have the perfect requirements. Your business partners with the vision will need to iterate and refine their vision, and the entire product team is better off if the development team is technically able to efficiently deliver features that weren’t even imagined at project inception. You can succeed with bad code, but all things being equal, I think you maximize your business’s chances of succeeding by taking software quality very seriously.

This is one of those things that is easy to overlook: change.  Especially unexpected change.  The "business people" don't know the difference between expected change and unexpected change.  Even "business people" who used to be really really smart technical people.  Once you're outside the code, you have no idea what magic is needed to make changes.  That's why these business people always come to you with this odd look on their face and ask, "Can we do this??"  Sometimes you look at them like they're crazy and say, "Duh."  But other times you blow up at them, "What?!  That wasn't in the original spec!"  That's why those business people have that funny look on their face, they never know what to expect.  Of course, that's also why some of these business people adopt the attitude of, "I don't want to hear about it, just get it done by tomorrow."

Let me be very clear here, I’m defining software quality as the structural qualities of code structure that enable a team to be productive within that codebase for an extended amount of time.

So that's why code quality is important.  The business people should want you to be writing quality code because it means you can respond to their changes with a good attitude and quick turn around time.  And you should want to be writing quality code because it means you can deal with those changes without going out of your mind, and without adding more and more hacks into your code.

Jeremy's blog post goes on to list some "qualities" of good code.  One of these I think is very important:

Feedback. I think the best way to be successful building software is to assume that everything you do is wrong. We need rapid feedback cycles to find and correct our inevitable mistakes.

He also has a list of links to his MSDN articles on various valuable patterns and principles which you can apply to help keep your code maintainable.

My TDD Struggle

I'm a huge fan of the concept of TDD (Test Driven Development).  I've done it a few times with varying success but I intend to make it a constant practice on anything new I write.  If you want to see people doing it right, go watch some videos at http://katacasts.com/.  Now on to the words!

TDD is the red/green/refactor process.  Write the test, watch it fail.  Go write the bare minimum of code possible to make it pass.  Refactor the code, and refactor the tests.  Repeat.

This process lends itself to what people call "emergent design."  This is the concept that you don't stress out trying to devise some all encompassing design before you begin coding.  You sit down, you write tests, and you let the design emerge from the process.  The reasoning here is that you'll end up with the simplest possible design that does exactly what you need and nothing more.

That point hits home very strongly for me because of my experience with both applications and code that have been over designed and end up causing all sorts of long term problems.  So the call for simplicity is one I am very eager to answer.

BUT.  Clearly you can't just close your eyes and code away and assume it will all work out.  There is an interesting tight rope walk happening here.  As you are coding you have to be constantly evaluating the design and refactoring to represent the solution in the simplest possible way.  But what TDD is really trying to get you to do is not think too much about what is coming next and instead pass the current test as though there wasn't going to be a next test.

It's that "ignoring" the future part that I really struggle with. The knee jerk negative reaction is that this will cost you time because you're constantly re-doing work.  There are times when this is probably true, but in general the tests lead you through the solution incrementally, affirming you're on the right track each step of the way.  And when you suddenly discover something that causes you to back track, you've got all the tests ready to back you up.

But there are a few things that I don't think this technique is good for.  One is "compact" algorithms, the other is large systems.  We'll take them one at a time.  I was recently practicing the Karate Chop Kata which is a binary search.  My testing process went like this:

1. When array is null or empty it should return negative one
   
2. When array has one item 
1. it should return zero if item matches
2. it should return negative one if item does not match
3. When array has two items
1. it should return zero if first item matches
2. it should return one if second item matches
3. it should return negative one if nothing matches
4. When array has three items
1. ...

Numbers 1-3 were all implemented in the straight forward way you would expect.  But when I get to #4, now I have to actually write the binary search algorithm.  So now I have to decide if I'm going to write it with a loop, with recursion, with some form of "slices", etc.  I also have to figure out what the terminating conditions are and verify that my indexes and increments are all correct.  In other words I have to do all the work after writing that one test.  

And worse, these tests are stupid.  What am I going to do, write a test for every array length and every matching index?  I re-factored the tests later to be a bit more generic and more specific to the edge cases of the algorithm in question.  If you'd like to see what I ended up with you can checkout the code on bitbucket.

In general, writing your tests with knowledge of the implementation you're writing is bad, bad, bad.  Like @mletterle reminded me of on twitter, tests should test the behavior of the code, not the implementation of the code.  Bob Martin just recently wrote a post that made the same kind of argument in regards to Mocks.  

Now don't get me wrong.  The tests are still valuable in this example, they're just not as useful in an "emergent design" kind of way.

Moving on, the second thing that the emergent design mindset isn't very good for is complex system design.  Systems that are complicated enough to warrant DDD (Domain Driven Design).  In this case you really want to step back and take a big picture view and do a real Domain Model.  The emergent design approach may lead to a design with fewer objects or something, but this may not be a good thing if you're interested in a design that excels at communication.

With these systems you'd do your Domain Driven Design, then drop into your TDD and allow it to "emergently" design the actual implementation code of your model.  You're kind of getting the best of both worlds this way.  But its important to recognize that TDD is still an important part of this, even though you didn't let it guide the ENTIRE design.

So TDD and emergent design might not be the answer in all circumstances.  But I still think that you'll find a strong place for it in even these circumstances.

Anybody in the blogosphere strongly disagree with this?  Or perhaps, dare I ask, agree?

SCM Trade-Offs

Source Control Management (SCM) is, on the surface, a very simple topic.  You put all your source code in one place, then you check in and check out from there.  Advanced systems support merging changes if two people edit the same file.  And there you go, SCM in a nut shell.

And if you are just one person, or a small team, that's probably where the story ends for you.  But with larger teams, or more complicated application environments SCM inevitably morphs into Application Lifecycle Management (ALM).  ALM covers topics from requirements, to architecture, to testing, and release management.

I have found that typically it is ALM requirements that end up introducing branching to your SCM.  Branching is another simple concept.  You make a copy of the source code which can later be "easily" merged back with the original.  Unfortunately branching is another area that quickly gets much more complicated than all that (which I've written about before, in a surprisingly humorous post if I may say so myself...).  As the number of branches increase the amount of work required to track them and merge between them increases.

So you will always find that there is a delicate trade-off that must be made when you introduce a branch.  Branches give you isolation, letting people work independently from each other.  There are tons of reasons why this can be helpful, some of which include:

1. You can work without fear of breaking other people's work
2. You can try "experimental" stuff which may never actually be finished
3. You can work on parallel features and release one without releasing the others
4. You can do new work while still enhancing an old release (ex: service pack releases)

But branches also introduce the need to integrate.  If everyone works on the same branch, every check in is an integration.  And these happen so frequently it's hard to even think of them as "integration."  But if you introduce long lived branches, integration can now be delayed indefinitely.  There are plenty of horror stories about companies which tried to delay integration to the very end of application development and then spent almost as long "integrating" as they spent "developing."

This is where Continuous Integration comes from.  The idea is that you want to integrate very often (at least once a day according to Fowler), and that you want integration to not just mean merging code, but also running tests to verify that the integration is successful.  The point is that you want to catch integration issues early.  The reasoning is the earlier you catch them the easier they will be to fix.  The longer you wait to fix them, the more things will have diverged, or be built on code that needs to change.  So integrate often.

And this is where we arrive at a problem I've been struggling with for quite some time.  On the one hand, I want isolation for all the reasons I've already mentioned.  And on the other hand I want to integrate continuously to avoid all the pitfalls mentioned.  But you can't have both.

Naturally I've tried to look at how other people tend to deal with this issue.  There seem to be two big picture approaches:

1. "Centralized" in which everything starts integrated, and isolation must be purposefully added
2. "Distributed" in which everything starts isolated, and integration is tightly controlled

All open source projects I've looked at are run very "distributed."  Anyone can download and update the code in their own sandbox.  But to actually get that code into the project you must submit patches, which the project leaders review and apply if they pass their standards.  This works well for Open Source projects because you can't let anybody waltz in and start changing your code.  You need control over what code is being added and modified, and you want it to all be reviewed and tested.

In contrast most company projects seem to be run "centralized."  Mostly I've found accounts of a main branch with release branches for past releases and maybe a few feature branches.  You can work code reviews into a process like this.  Some systems support preventing check ins without code reviews.  Other people just make it part of the "process."  But in general, people are trusted to be making the right changes in the right way, so the control required in OSS projects isn't as strictly needed here.  You trust your own employees.

I suppose I should mention quickly that my words "centralized" and "distributed" DO line up with the differences between Distributed and Centralized Source Control Systems (like Mercurial vs. Subversion), but you can still use a Centralized Source Control System and work in a "distributed" fashion by using patches.

There are lots of reasons why you might work one way or the other.  But I still have a hard time figuring out what's right for me.  This could be because my requirements are unusual.  I'm not sure why they would be, but I haven't found other people worrying about them.  This makes me think either my requirements are very special, or they're crazy and I'm missing something.

Briefly, here's what I'm dealing with.  I have a number of different people working on the same application, but they are all working on un-related or only loosely related features.  We have frequent and regular releases of the application being done.  We try to target features for the appropriate release, but its incredibly hard to know if a feature will really be "done" on time.  Features get pushed back when:

1. They under go dramatic changes when it reaches completion and users try to put it through its paces
2. The development takes longer than expected
3. The developer gets pulled off and placed on another feature
4. The feature gets put on hold for "business reasons"

If everyone is working on all this stuff in the "main" branch and integrating continuously, what do we do when one feature is not ready to release but the rest are?  The changes are all tangled up in the same branch now and there is no easy way to un-tangle them.

The only answer seems to be doing any "non-trivial" work in Feature Branches.

But feature branches pose whole challenges of their own!  If you're writing Mercurial, then feature branches are no problem.  If you're developing a website with a database then feature branches are a bit harder, because you have to create a new database and fill it with test data.  But if you're developing an enterprise application with a complicated database (which can't just be loaded with generated test data) that talks to other databases (with linked servers and service broker) and has a large number of supporting services many of which depend on OTHER 3rd party licensed services; then what?  In that case, creating the full application environment for each feature branch is actually impossible.  And creating a partial environment is certainly not easy.

But it is precisely because of all of this complexity that we want the isolation!  But the complexity makes the isolation difficult (if it's even possible, which I'm not sure of yet) and time consuming.  I love paradox!

Perhaps you're saying to yourself, "Maybe you wont be able to actually run the app and do everything it can do in your feature branch, but surely the code is written so all these things that depend on other things are nicely abstracted so that you WILL be able to run the unit tests!"  Um, no.  Sorry.  It pains me to admit it, but I'm afraid that isn't the case just yet.

So where does that leave me?  Well, I want to implement continuous integration to reduce the integration issues and make releasing easier, but I also want the flexibility to develop features independently so they don't prevent other features from being released, but my environment is so ridiculous that I can't figure out how to set it up so feature branches are fast and easy to create (and more importantly useful).

So, can you help me?  Do you have any of these problems?  Have you managed to mitigate any of these problems?  Am I missing something?

Publish ClickOnce with Albacore and Rake

Rake is a great build system written on Ruby.  It uses the Ruby language to build a wonderfully simple Domain Specific Language (DSL) for writing build scripts.

If you look around a bit you'll find that people are using Rake for all kinds of things.  Even .NET projects use Rake, including Fluent NH and Machine.Specifications.

Albacore is a Ruby library for Rake which gives you a whole set of useful predefined tasks for doing common .NET tasks, like executing msbuild.

With Albacore, building a .NET solution is as simple as writing this rake script:

desc "Run a sample build using the MSBuildTask"
msbuildtask do |msb|
  msb.properties = {:configuration => :Debug}
  msb.targets [:Clean, :Build]
  msb.solution = "spec/support/TestSolution/TestSolution.sln"
end

You execute that by simply typing "rake msbuild" at the command prompt.

You can even use Rake and Albacore to automatically perform a ClickOnce Publish!  For my office, I'm hoping that this will be a VERY useful thing.  To set it up, get the ClickOnce publish working in Visual Studio first.  With that done, write a rake script like this:

desc "Publish ClickOnce"
msbuildtask("publish") do |msb|
  msb.properties = {
    "configuration" => "Release",
    "PublishDir" => "C:/temp/",
    "PublishUrl" => "C:/temp/",
    "InstallUrl" => "C:/temp/"
  }
  msb.targets [:Publish]
  msb.solution = "slnFile.sln"
end

There are a few important things you have to get right for this to work.  First, you have to specify the PublishDir parameter or it wont copy your files to the PublishUrl.  Visual Studio makes this work automatically, but you have to do it manually if you want msbuild to do it.

The second thing to note is the use of forward slashes in the paths.  If you use backslashes you have to double escape them, once for Ruby, and once for cmd.  I have no idea why cmd needs the backslashes escaped, but that's the behavior I ran into and it's simpler to just use forward slashes.

With this all setup, you can run "rake publish" at the command line and your app will be ClickOnce deployed automatically.  You can also easily setup different ClickOnce deployments if you need to (ex: Test vs Production).

If you have any build steps at all that you think could be automated I highly recommend you check out Rake and Albacore.

Snow Blower Efficiency

Hi there, I'm a big nerd, and I'd like to prove it to you.

We've been getting a lot of snow in Cleveland, especially in the city where I live.  I use a snow blower to clear my drive way, and naturally I think about what the most efficient way to clear the drive way is while I'm out there in the cold doing it.

The solution I've come up with is somewhat interesting, so I figured I'd share.

There are a few factors to consider:

1. Length of driveway
2. Width of driveway
3. Number of turns you make
4. Number of turns of the thrower you make (the direction the snow is being thrown)

#1 and #2 are fixed, and there is nothing I can do (aside from buying a larger snow blower) to change those.  So the number of passes I have to make to clear the driveway is fixed.

Clearly, I'm going to go up and down the drive way, not side to side, as that will reduce the number of turns.  Given that, the number of turns I'm going to make is pretty much fixed.  Thus, the only other factor left for me to consider is the number of turns of the thrower I have to make.  As it "turns out" (har har), I actually can do something about this.  In fact, if I do it right, I shouldn't have to re-direct the thrower at all the entire time I'm clearing the drive way.

Here's two diagrams to help explain.  In the first diagram, I'm starting at the left top of the driveway and going up and down to the right.  In the second diagram, I'm starting in the middle and going down on the left and up on the right.

Notice that in the first image every time you turn, you have to also turn the thrower.  But in the second image the thrower is ALWAYS pointing to the right.  The downside to the pattern on the right is that your turns are wider, but I've actually found this is a good thing.  It's easier for me to make a wide turns than a sharp turn.

The last question is why start in the middle?  The reason is that when you're in the middle, the snow has to be thrown the farthest, and sometimes because of the wind or whatever it doesn't quite make it fully off the driveway.  By starting in the middle you make sure you never throw snow over ground you already cleared!

And with that, I have to go snow blow the driveway...  again.

People Problems

I'm sorry to have to tell you this, but your job is to solve problems for people.

I don't even have to know what your job is, and I can still say that with a pretty high level of confidence.  Are you an engineer building bridges?  You're solving a problem for people who need to get across that divide.  Are you a cashier at the grocery store?  You're solving a problem for people who need to pay for their groceries (and for people who need to take money from people for groceries, double whammy).  And of course, are you a computer programmer?  You're solving SOME, maybe not very well specified, problem for your users.

It really doesn't matter how far removed from people you are, you're still solving problems for people.  You could be completely devoted to the inner most workings of the Windows kernel.  You're actually worse off!  You're solving problems for users who want to check their email, and programmers who want to write MS Word, and designers who want to make Windows look pretty, and marketers who want to sell Windows as the most stable ever, and finance people who want to show market growth, and the list just goes on and on and on...

Sometimes you are aware that you are solving problems for people, but maybe you don't really know what problem you're solving.  Like the grocery store clerk who is really nice and friendly and makes all the customers happy going through the line.  Maybe they think their job is to make you happy, but it's not.  The problem you want them to solve is to figure out how much money you owe, and take it from you.  And you want them to do it quickly.  If they can do that AND make you happy, then they are a very good grocery clerk.  If they only make you happy, then they just aren't very good.

A similar thing happens with programmers.  I frequently get confused and think that my job is to make high quality software and write high quality code and do good high quality design work.  But its not.  My job is to solve some problem my client has.  As long as I fix their problem, I've done my job.  Maybe I could have fixed their problem better with beautiful code I'd be proud to hang on my wall at home.  That would be a bonus.  But all I'm supposed to do is solve their problem.

That is, unless I'm actually supposed to be developing a long life product, which will require all kinds of future enhancement and maintenance and re-configuring.  In that case my job is to both solve the client's problem AND develop a strong software product.  These are two competing goals.

And that's the thing about People Problems:

• They are never clear cut
• They tend to overlap
• They always involve trade-offs

They're not clear cut because frequently you don't know what problem your supposed to be solving.  Or there are a bunch of problems you're supposed to be solving and you don't know which is most important.  Or different people have different problems and you need to simultaneously solve them all.  Create an app for a user!  Create it in the time your managers wants!  Make the app a platform to build a product on for your boss!

And this is where the trade-offs start.  You simply can't HAVE your cake AND eat it.  See?  Once you EAT it, you don't HAVE it anymore.  Or if you HAVE it, then you haven't EATEN it.  See?  I just thought that up. Pretty good huh?

The "real world" is all about People Problems.  And People Problems inevitably lead to trade offs. And trade offs inevitably lead to disappointment.  For example, if you choose to make the code super high quality, you wont be disappointed, but your boss will be because of how much it cost.

The trick is to understand that above all you are solving people's problems.  Then understand that necessarily involves trade offs.  Then try to take a big picture view when deciding on how to make those trade offs.  Hopefully that will at least help you deal with the inevitable disappointment.  At least you'll know you made the best decision you could for the specific People Problem you were faced with.

Of course, you'll probably be disappointed when you find out later that you didn't have all the facts straight and your decision was based on faulty information or just information that has since changed.  But that's a whole different dimension of dealing with People Problems.

WPF UserControl IsEnabled in WinForms host

While debugging today, I ran into something rather odd.  Two things actually.  And since I'm falling behind in my posting I thought I'd share.

Here's what I was dealing with.  I have a WinForms UserControl which contains a WPF UserControl within an ElementHost.  The WPF control is being informed of what is selected on the form and is Enabling or Disabling itself accordingly.  However, there is a special case in which the form knows it wants the control to be disabled all the time.

Before I go any further.  If you're working with WPF controls hosted in WinForms and you're dealing with enabling and disabling things, you should be aware of this bug and this workaround on Microsoft Support.  It will cause all kinds of havoc if you set things to Enabled=false before showing them.

The WPF user control is using MVVM, so it's IsEnabled property was bound to an IsEnabled property on the View-Model.  This is where we run into the first interesting thing: this doesn't work.  The vm.IsEnabled property was changed, and the PropertyChanged event was fired with "IsEnabled" as the property name, but the WPF user control's IsEnabled property did not change.  I found someone else who had this same issue and posted about it on this blog.  His work around cracks me up, I can't believe it works...  WPF is crazy.

My work around was to just hook the PropertyChanged event myself like this:

void _vm_PropertyChanged( object sender, PropertyChangedEventArgs e )
{
  if ( e.PropertyName == "IsEnabled" )
    this.IsEnabled = _vm.IsEnabled;
}

I was stunned when this code didn't work. It fired, it set the property, but after setting it the property value didn't change. And it didn't change after WPF got a chance to run through a layout pass either.

This is the second interesting thing.  Turns out the WinForm's control that the WPF control was in was disabled. Why?  Because the parent form had disabled it due to that "special case" I mentioned.  So when I set the WPF user control's IsEnabled to true it didn't matter because it's parent's Enabled property was false. So it just shrugged it's shoulders and ignored me.

So before I can enable the WPF control, I need to enable it's parent, the WinForms control.

In order to get this whole mess working what I ended up doing looks something like this:

1. View-Model fires change event for IsEnabled
2. WPF User Control fires custom change event for IsEnabled (it does NOT try to set IsEnabled)
3. WinForms User Control sets Enabled = CustomEnabled from WPF User Control
4. WinForms User Control's EnabledChanged fires
5. WinForms User Control sets WPF UserControl's IsEnabled property = this.Enabled
6. WPF User Control's IsEnabledChanged fires
7. WPF User Control sets View-Model's IsEnabled = this.IsEnabled

This ensures that all Enabled properties on all the objects involved here will always be in sync, no matter which one you change.

Part of the problem here I'm going to blame on bad design.  I don't think the control should be being enabled and disabled in two completely different ways (one from the top (WinForms), and one from the bottom (View-Model)).  The rest of the problem I'm going to blame on Enabled properties being really confusing.  They have mystical relationships with their parents and their values can change at different times.  Most properties when you give them a value either have that value after the set, or throw an exception.  But not Enabled!  It's mystical.  I find that I know this but that I still get bit by it when I'm not expecting it.

Usability: Locking Doors

Reading The Design of Everyday Things has caused me to start paying attention to usability issues I run into in day to day life. Some of them are interesting and I'm going to try to remember to share those here.

This one is about doors. DOET talks a lot about doors. The specific part I want to talk about is how they lock. A normal door works something like this:

• Use the lock mechanism to lock the door
• When locked the door cannot be opened
• Use the lock mechanism to unlock the door
• When unlocked the door can be opened

A while back I encountered a door that worked differently than this though. Unlike the normal door, which when locked can not be opened, this door could still be opened from the inside even when it was locked without unlocking it! But it could not be opened from the outside when it was locked.

You can see where the designers were coming from here. The point of locking the door isn't to keep people inside locked in. The point is to keep people outside from getting in. Letting you open the door from inside even when it's locked aligns more closely with the purpose of locking the door. And you can imagine all kinds of situations where this would be nice: answering the door when someone knocks, opening the door to leave the house, etc.

So this change seems to be a great idea: it fits the purpose more closely, and it eliminates some small annoyances. Unfortunately it introduces a really big annoyance of its own: its super easy to lock yourself out.

All you have to do to lock yourself out of the house is walk out and close the door behind you.

Preventing you from locking yourself out of the house isn't one of the stated purposes of a normal door, but because of how it works it manages it all the same.

But lets look at how a person uses the door the way a software engineer looks at a person using software, in terms of "clicks". Lets assume the door is already locked and a person wants to leave the house and leave the door locked behind them.

With a normal door:

1. Unlock the door
2. Open the door
3. Close the door behind you
4. Lock the door

With the special open-while-locked door:

1. Open the door
2. Close the door behind you

Which of these doors is better designed?

Don't Write Another Line Of Code Unless...

Don't write another line of code unless you know and have been influenced by the following things:

SOLID (from Uncle Bob)

SRP* - Single Responsibility Principle

Every class should have one clear and well defined responsibility

OCP* - Open Closed Principle (from MSDN)

Extend the behavior of a class without modifying that class

LSP - Liskov Substitution Principle

Derived classes must be substitutable for each other (this one is kind of obvious, but still important)

ISP - Interface Segregation Principle

Make fine grained interfaces, not fat interfaces (this goes along well with DIP)

DIP* - Dependency Inversion Principle

Classes define their dependencies, which other classes implement

* The three I've starred I believe are the most important as they make the biggest difference in fighting spaghetti code.

YAGNI (from Wikipedia) - You Ain't Gonna Need It

This is an Extreme Programming concept which basically says you shouldn't add functionality until you actually need it

TDD (from Wikipedia)- Test Driven Development

Loose Coupling/High Cohesion (from MSDN)

Code Smells (from Fowler)
A code smell is a "surface indication" that there may be a deeper problem with your code.  It is very useful to know these, especially when practicing TDD.

Basic Design Patterns - Singleton, Observer, Mediator, Strategy, Decorator, etc

There are lots of books on the major design patterns. I personally haven't read this one, but it has been highly recommended to me: Head First Design Patterns

MVC/MVP/MVVM (from Fowler) - Model View {Controller|Presenter|View-Model}

The biggest piece to take away from this, in my opinion, is the responsibility of the Model and the separation of the UI (the view) from the Application layer (the Controller, Presenter, View-Model).

DI/IoC/Service Locator (from Fowler) - Dependency Injection/Inversion of Control/Service Locator

Law of Demeter (from Wikipedia)

This is a useful concept to be aware of, but one that really shouldn't be thought of as a law. See what Phil Haack has to say.

Database Isolation Levels (from MSDN)

Anyone doing anything with databases needs to know the Isolation Levels, what they do, and when to use them.

Concurrency Models - Optimistic, Pessimistic (from Fowler's Patterns of Enterprise Application Architecture)

You can't write any multi-user application that updates data without understanding concurrency and the various patterns for dealing with it.

Version Control (from Wikipedia)

You should be familiar with centralized version control (like Subversion). And I think you should also understand distributed version control (like Mercurial).

--

You don't have to be an expert in all these things. But any decent developer should have at least a basic understanding of these concepts and be able to understand what they are when someone else mentions one.

Our industry is REALLY weak on education. We go to school and learn about data structures, semaphores, virtual memory, file systems, etc. Then we graduate, get jobs in Software Engineering and promptly never use any of that. I'm not saying it's not useful stuff to know; it is useful to know. But its not what you deal with day to day as a software engineer.

You can't be a software engineer without knowing the stuff on this list. I'm serious. If you don't know it, you're just a dude who's hacking out code.

That said, I'm pretty sure I'm just a dude who's hacking out code... What things do you think I should have on this list before I write another line of code?

PS. There are also lots of good books you should probably read if you're thinking about being, or already are, a software engineer.

UPDATES:
1/28/2010: added Code Smells and re-ordered list

DDD is Not About Perfection

The main practice of DDD (Domain Driven Design) is refactoring to deeper insight. The idea is very similar to much of what the Agile practices preach. Namely, when you find you have

1. misunderstood something or made a mistake
2. been given bad information
3. learned something new that changes previous assumptions

You go back and you update your code to reflect your new understanding. In Agile, this is called embracing change, in DDD its refactoring to deeper insight.

This is one of those principles that seems like it shouldn't need to be said. If someone knows that what they've done isn't right anymore who WOULDN'T go back and fix it?! But it turns out this is one place where real life and theory don't line up. The thinking usually goes something like this:

We've spent a lot of time working on this feature, we're out of budget, it works fine in 90% of the cases, and we can just add this little hack that will take care of the other 10%. Therefore it makes more sense for us to just do the little hack.

I'm sure you've seen this line of thinking before, so you're already anticipating that I'm going to say this is stupid. But hold on. Its not stupid. This is actually totally sensible thinking, but there are two problems:

1. You made up that thing about "90% of the cases." You actually have no idea how often the "edge cases" that are a problem for you will crop up. And for all you know, those may be the more important cases, which your system can now only accommodate with a weird hack.
2. You haven't considered changes that might arise in the future, or insights you may have in the future. If any do crop up that are related to the "10%" edge cases, you're now going to be forced to build hacks on top of hacks (on top of hacks, on top of hacks, on top of hacks...).

So obviously you should never write the hack, you should always embrace the change and refactor to deeper insight.

Nope, sorry, wrong again. Unfortunately we are here living in the real world. We have real world constraints: Time and Money. If we always took the time to fix everything we discovered we got slightly wrong we would never deliver a product, ever. And shipping is a feature.

So... sometimes we're going to have to hack, which might get us into a world of trouble? And sometimes we're going to have to blow our budget refactoring? How do we know when to do which?

In Strategic Design - Responsibility Traps Eric Evans (the founder of DDD) says "the whole system will not be well designed." This is an inescapable fact if you're working on a large complex system. If you're working on a small simple system, maybe you can pull it off, but even then I doubt it.

How can the guy who's whole development technique revolves around refactoring to deeper insight say the whole system will not be well designed? Well, he has an answer of sorts. Evans says that what you need to do is identify your application's Core Domain. What is it that your application does that sets it apart, makes it important, or provides the most value for the users? That's your Core Domain. Now if a change crops up in the Core Domain, you refactor to deeper insight. This is the most important part of your app! It's the part you'll be building on for everything else in your app. This is the part of your app has to be perfectly designed.

But what about parts that are NOT the Core Domain? Or what if it's not so easy to define your app's Core Domain? Then what do you do? Well, you weigh the options, and take your best guess.

• Is there time in the budget?
• How hard will it be to refactor to account for this change?
• How much better will the app be if you do change it (time saved? # people affected? usability? performance? correctness?)?
• Can you convince yourself it is unlikely other changes will have to be made in the future that will be related to this change?

Once you have answers to these questions that are as accurate as you can manage, then you have to just guess. Because your answers to these questions are NOT scientific. And you have no way to predict the future. But you have to try anyway, so you guess.

If you think it's going to be very hard to accommodate the change and you think the change wont make your app all that much better, then hack it. If that's reversed, refactor it.

I'm a programmer, and I'm hungry for perfection, so I always lean toward wanting to refactor it. I think refactoring to make it correct anytime you have the time and ability is the right move. You'll end up with an app you're proud of, that works better for your users, and is easier to maintain and update. But we have to face facts! Sometimes the cost is simply too high. Sometimes we're forced to hack it now, and pay the consequences later.

The good news is that DDD helps tremendously with this. If you can define a Core Domain, you're that much better off because you have now decided what your application is all about. This will help you in every decision you need to make.

Further more, DDD is drastically easier to refactor and maintain than spaghetti code. So the cost of refactoring to deeper insight is lowered when you're writing DDD.

But in the end, we have to realize that DDD is not about writing perfect code. Its about writing good code, that makes the complexity of your application manageable. But we know that despite all the benefits DDD can bring, it doesn't promise perfection. The simple truth is that software is very very hard to write, and designing enterprise applications is even harder. So the best we can do is write Good Enough software, like they say in The Pragmatic Programmer. But hopefully DDD will help us to raise the bar on how good that software is.

Knowledge in the Head and in the World

A looooooooong time ago I wrote a post called Theory of Software Usability.  This post was primarily about the tradeoff between “Ease of Learning” and “Ease of Use.”

I used my favorite example of Vim vs. Notepad.  Vim is an advanced modal editor that a n00b wont even be able to get text into if they don’t know what they’re doing.  Whereas Notepad is just about the simplest application you can imagine that anyone can figure out how to use.

My argument was that Vim is extremely usable but difficult to learn while Notepad is extremely learnable, but not really all that usable.  So there is an implicit tradeoff between Learnability and Usability. 

Recently I have been reading The Design of Everyday Things and I can across a concept that is an interesting corollary to the Usability vs. Learnability issue.  The book presents “The Tradeoff between Knowledge in the World and in the Head”.  Knowledge in the world is simply information that is readily available in the world, so you don’t have to learn it, or at least you don’t have to learn too much.  In The Design of Everyday Things an example of a typist is used. 

“Many typists have not memorized the keyboard.  Usually each letter is labeled, so nontypists can hunt and peck letter by letter, relying on knowledge in the world and minimizing the time required for learning.  The problem is that such typing is slow and difficult… But as long as the typist needs to watch the keyboard, the speed is limited.

If a person needs to type large amounts of material regularly, further investment is worthwhile: a course, a book, or an interactive computer program… It takes several hours to learn the system and several months to become expert.  But the payoff of all this effort is increased typing speed, increased accuracy, and decreased mental load and effort at the time of typing.”

At the end the book presents some tradeoffs between knowledge in the head and in the world in terms of 5 properties, retrievability, learning, efficiency of use, ease of use at first encounter, and aesthetics.  It breaks down like this, knowledge in the world is retrievable, requires little to no learning, is not efficient, is easy to use at first encounter, and can be unaesthetic and inelegant.  On the other hand, knowledge in the head is not retrievable, requires lots of learning, is efficient, is not easy at first encounter, and can lead to better aesthetics.  So basically, they are at odds with each other.

Bringing back my Vim vs Notepad example, we can see how this fits right in.  Notepad puts all the knowledge you need “in the world.” All the labeled keys on your keyboard do exactly what you’d expect and the other functions are clearly labeled in the menus.  In Vim on the other hand, you can’t even enter text until you learn the “i” command.  The knowledge must be in your head.  All the tradeoffs listed above apply perfectly to this example.

I think this is a very important concept to keep in mind when doing software design.  Who is your user?  What job are they doing?  Often often will they be doing that job?  Will new people need to figure it out on the fly, or will the same people always do it over and over again?  The answers to these questions will help you decide if you should emphasize knowledge in the world or knowledge in the head.  If you are building a public facing website that many people will visit, you want to emphasize knowledge in the world.  If you are building an application for something like data entry you may want to emphasize knowledge in the head.

The important thing to take away from this is that there is a tradeoff and you have to make a decision one way or the other.  Knowledge in the world is not always better than knowledge in the head, and vice versa.  Pay attention to what you are building and who you are building it for and design accordingly.

Intangible Value

Rory Sutherland is a marketer who recently gave a TED talk. The talk is about 16 minutes long and is hilarious, you should totally watch it.

He opens the talk with this gem, "if you want to live in a world in the future where there are fewer material goods, you can either live in a world which is poorer (which people generally don't like), or you can live in a world where actually intangible value constitutes a greater part of overall value."

Seth Godin is another marketer who has a short and sweet blog post called Creating sustainable competitive advantage in which he argues that competitive advantage rarely comes from proprietary technology or technological barrier to entry. In other words, technology alone will not allow a business to succeed because its competitors will quickly be able to copy the technology.

He has a list of things you can do to gain competitive advantage, 3 of which apply here:

• You can build a network (which can take many forms--natural monopolies are organizations where the market is better off when there's only one of you).
• You can build a brand (shorthand for relationships, beliefs, trust, permission and word of mouth).
• You can create a constantly innovating organization where extraordinary employees thrive.

Tying in with Sutherland, these are about adding intangible value. You can gain intangible value by building a network around your product, or by building trust and a name for yourself ("brand"), or by being a constantly innovating organization.

The last is half intangible, half tangible. The actual innovations produced are tangible, but being innovative adds its own intangible value, both to your customers as well as your own employees and even to job applicants! Emphasizing extraordinary employees is a relatively intangible thing which can produce tangible benefits across the board (better products, faster delivery, lower employee turn over rate, and better employees). Thus the competitive advantage.

I think this concept of Intangible Value can be extended into software itself. Possibly the best example is usability. Some software usability concerns are tangible: how long does it take someone to accomplish a task, how many clicks are required, etc. But other usability concerns are intangible: does the user enjoy using the software or does it make them want to shoot themselves in the face.

Seth Godin's post talks about things you can do today to gain competitive advantage, but Rory Sutherland's talk is about how we as a people need to learn to value intangible things more. This is much harder than it sounds.

When you're evaluating two products, you look at the feature lists. If one product has more features, you're likely to decide that it is the better product. But while it may have more features, it may also make people want to shoot themselves in the face. Can we include that on the list of features?

As an example, compare Microsoft Visio to Balsamiq Mockups. Visio is a very full featured product which is ridiculously flexible and powerful compared to Balsamiq. But everyone I know likes Balsamiq better. Why? It's the shoot myself in the face factor. Balsamiq is faster and easier to use. In fact, it's a joy to work with. That is a relatively intangible benefit, but it's real.

As another example, take 37 signals. I have not personally used their products, but I know from what I've heard and from what they've said that their focus as a company is on building slim lined and usable software. There are big box alternatives to their products that have been around for much longer and are far more "configurable," but people love 37 signals. Again, for mostly intangible reasons I think.

So intangible value is a real thing which is often overlooked by the "deciders" but always appreciated by the users. The challenge for those of us who design software is to figure out how to add that intangible value into our products, and how to make potential users aware of it. The challenge for people in general is to recognize intangible value when they come across it and not dismiss it as unimportant.